A cloud game picture processing method and device, electronic equipment and storage medium

Abstract

The application provides a cloud game picture processing method and device, electronic equipment and storage medium, relates to the technical field of image processing, and includes that a client sends a display refresh rate to a server; the server determines a rendering frame rate of the server based on the display refresh rate; the server performs rendering and encoding processing on cloud game picture data based on the rendering frame rate of the server, generates encoding data, and transmits the encoding data to the client; the client receives the encoding data, decodes and renders the encoding data, obtains decoded data, and the rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data. The application adjusts the rendering frame rate of the server according to the display refresh rate of the client, provides an effective upward redundancy space for the frame rate of data transmission, and effectively eliminates the picture tearing problem in cloud games by adjusting the display refresh rate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a cloud gaming screen processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] The digital wave is sweeping the globe, and cloud computing has become the core of the IT industry's development. Cloud gaming, as an emerging technological force, shifts game rendering from player devices to centralized servers, revolutionizing the gaming experience. However, high frame rate cloud gaming faces the challenge of screen tearing. Vertical sync and G-Sync / FreeSync technologies can respectively solve the problem of screen tearing caused by the graphics card's frame rate being higher or lower than the monitor's refresh rate; the former causes latency and stuttering, while the latter is only suitable for local gaming scenarios.

[0003] Cloud gaming is more complex due to the processes involved, including server rendering, encoding, network transmission, client decoding, and display. G-Sync / FreeSync technologies require communication between the graphics card and the monitor to synchronize frame rates, but in cloud gaming, the graphics card is on the server and the monitor is on the client, making synchronization impossible. Furthermore, due to network transmission, even if the server-side frame rate and the client's monitor refresh rate are kept constant, the frame generation time on the client side becomes unstable after encoding / decoding and network transmission delays, resulting in screen tearing. Therefore, eliminating screen tearing in cloud gaming is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, aiming to solve the problem of screen tearing in cloud gaming.

[0005] The first aspect of this application provides a cloud gaming screen processing method, applied to a system including a server and a client, the method comprising:

[0006] The client sends the display refresh rate to the server.

[0007] The server determines its rendering frame rate based on the display refresh rate, and the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate.

[0008] The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and generates encoded data which is then transmitted to the client.

[0009] The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0010] The client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0011] In one optional implementation, the server determines its rendering frame rate based on the display refresh rate, including:

[0012] The client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and sends it to the server.

[0013] The server receives the frame rate adjustment ratio for the current second from the client, and adjusts the display refresh rate downward based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

[0014] In one optional implementation, the client generates a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including:

[0015] The client obtains the latency difference ratio of all frames in the previous second based on the decoded data obtained in the previous second.

[0016] Based on the latency difference ratio of all frames in the previous second, the first perturbation frame in the previous second is determined. The first perturbation frame is a frame whose latency difference ratio is greater than or equal to a first threshold. The first threshold is equal to the frame rate adjustment ratio corresponding to the previous second.

[0017] If the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment proportion for the current second.

[0018] In one alternative implementation, after obtaining the latency difference ratio of all frames in the previous second, the method further includes:

[0019] Based on the latency difference ratio of all frames in the previous second, a second perturbation frame in the previous second is determined. The second perturbation frame is a frame whose latency difference ratio is less than or equal to a second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value.

[0020] If the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment proportion of the current second.

[0021] In one optional implementation, the method further includes:

[0022] If the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset proportion, and / or if the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset proportion, the frame rate adjustment proportion corresponding to the previous second shall be used as the frame rate adjustment proportion for the current second.

[0023] In one optional implementation, the delay difference ratio of each frame in the decoded data is obtained as follows:

[0024] The client obtains the delay of each frame in the decoded data;

[0025] The client obtains the delays corresponding to a preset number of frames before the target frame, and determines the average delay based on the delays corresponding to the preset number of frames. The target frame is any frame in the decoded data.

[0026] The client determines the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

[0027] In one optional implementation, the client obtains the latency of each frame in the decoded data, including:

[0028] When the server encodes the cloud gaming screen data, the server generates a first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client.

[0029] When the client decodes the encoded data, the client generates a second timestamp information corresponding to each frame in the decoded data;

[0030] The client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0031] A second aspect of this application provides a cloud gaming screen processing apparatus, the apparatus comprising:

[0032] The first sending module is used by the client to send the display refresh rate to the server.

[0033] A frame rate determination module is used for the server to determine the rendering frame rate of the server based on the display refresh rate, wherein the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate.

[0034] The encoding module is used by the server to render and encode the cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client.

[0035] A decoding module is used for the client to receive the encoded data and perform decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate;

[0036] An adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data.

[0037] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the cloud gaming screen processing method described in any one of the first aspects of this application.

[0038] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the cloud gaming screen processing method described in any one of the first aspects of this application.

[0039] Beneficial effects:

[0040] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, applied to a system including a server and a client. The method includes: the client sending its display refresh rate to the server; the server determining its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate; the server rendering and encoding cloud gaming screen data based on the server's rendering frame rate to generate encoded data which is then transmitted to the client; the client receiving the encoded data and decoding and rendering it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusting its display refresh rate to match the rendering frame rate of the decoded data. This application, by adjusting the server's rendering frame rate according to the client's display refresh rate, provides effective upward redundancy for the data transmission frame rate, ensuring that the client's decoded frame rate remains less than or equal to the display refresh rate, thereby effectively eliminating screen tearing issues in cloud gaming. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the steps of a cloud gaming screen processing method proposed in one embodiment of this application;

[0043] Figure 2 This is an example flowchart of a cloud gaming screen processing method proposed in one embodiment of this application;

[0044] Figure 3 This is a schematic diagram illustrating the logical interactions between various modules in an existing cloud gaming screen processing method.

[0045] Figure 4 This is a schematic diagram of the logical interaction of each module in a cloud gaming screen processing method proposed in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the structure of a cloud gaming screen processing device according to an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the accompanying drawings, the size of constituent elements, the thickness of layers, or areas may sometimes be exaggerated for clarity. Therefore, any implementation of this disclosure is not necessarily limited to the dimensions shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect true proportions. Furthermore, the drawings schematically illustrate ideal examples, and any implementation of this disclosure is not limited to the shapes or values ​​shown in the drawings.

[0050] The digital wave is sweeping the globe, and cloud computing has become the core of the IT industry's development. Cloud gaming, as an emerging technological force, shifts game rendering from player devices to centralized servers, revolutionizing the gaming experience. However, high frame rate cloud gaming faces the challenge of screen tearing.

[0051] Screen tearing is a display issue that can occur in video games. It typically happens in fast-moving scenes or when frame rates are out of sync (the graphics card's output frame rate is higher or lower than the monitor's refresh rate). This problem causes a mismatch between two adjacent frames on the screen, making the image appear "torn" into two parts, with the upper and lower halves not fully synchronized, severely impacting the user and player experience. Screen tearing has plagued players since the birth of gaming. In traditional native games, screen tearing occurs because the monitor's refresh rate and the graphics card's frame rate are out of sync. When the monitor refreshes the screen, if the graphics card simultaneously generates the next frame, the registers in the video memory that store the rendered image are switching between the two frames. When the monitor reads the registers, the upper and lower halves of the image come from different frames, causing screen tearing. Therefore, screen tearing occurs when the graphics card's output frame rate is higher or lower than the monitor's refresh rate.

[0052] To address screen tearing, the industry commonly uses Vertical Sync (V-Sync) technology. This technology effectively reduces screen tearing by lowering the graphics card's frame rate to match the monitor's refresh rate. However, V-Sync is not a perfect solution. It causes the graphics card to wait for the monitor to refresh before rendering the image, resulting in input lag. Furthermore, this function uses a fixed monitor refresh rate. When the graphics card's frame rate is lower than the monitor's refresh rate, V-Sync will limit the graphics card's frame rate to half or a quarter of the monitor's refresh rate (multiples will not cause screen tearing), resulting in a significant drop in frame rate and affecting game smoothness. Therefore, this technology is mainly used in scenarios where the graphics card's frame rate is higher than the monitor's refresh rate. Weighing the pros and cons, some gamers choose to disable V-Sync to pursue higher gaming performance.

[0053] To address the shortcomings of vertical synchronization, graphics card manufacturers introduced G-Sync and FreeSync technologies. These technologies work by dynamically adjusting the monitor's refresh rate to match the graphics card's frame rate. When the graphics card's frame rate is between its lower limit (G-Sync supports ultra-lower limits of 4-10Hz, while G-Sync Compatible and FreeSync are mostly 42Hz) and the monitor's maximum refresh rate, the graphics card does not need to reduce its frame rate. This effectively solves the problem of screen tearing while ensuring smooth gameplay and low latency. Therefore, this technology is mainly used in scenarios where the graphics card's frame rate is lower than the monitor's maximum refresh rate, but it requires monitor compatibility.

[0054] Vertical sync and G-Sync / FreeSync technologies can solve the screen tearing problem caused by the graphics card's frame rate being higher or lower than the monitor's refresh rate, respectively. Vertical sync causes latency and stuttering, while G-Sync / FreeSync is only suitable for local gaming scenarios. Cloud gaming is more complex due to the server-side rendering, encoding, network transmission, client-side decoding, and display processes. G-Sync / FreeSync requires communication between the graphics card and the monitor to achieve frame rate synchronization, but in cloud gaming, the graphics card is on the server and the monitor is on the client, making communication synchronization impossible. Furthermore, due to the network transmission involved, even if the server-side frame rate and the client's monitor refresh rate are fixed, the frame generation time on the client side becomes unstable after encoding / decoding and network transmission delays. For example, even if the server stably generates a 60FPS frame, the frame rate received by the client often fluctuates between 50FPS and 70FPS after encoding / decoding and network transmission. Therefore, even if the monitor is set to a 60FPS refresh rate, screen tearing will still occur on the client side.

[0055] In view of this, this application proposes a cloud gaming screen processing method. This method is applied to a system including a server and a client. The server is the core of the cloud gaming service and typically resides on a virtual server (vMonitor) provided by the cloud gaming provider. The virtual server uses software to simulate the function of a physical monitor. The screen rendered by the graphics card is transmitted to the virtual monitor for cloud gaming screen capture. Compared to a physical monitor, the virtual monitor offers more flexible settings to support a large number of players simultaneously online and provide a smooth gaming experience. A complete game program runs on the cloud gaming server. When a player connects to the server through a client, the server renders the game screen in real time, processes the game logic, and sends the results to the client as a video stream. Simultaneously, the server receives player input (such as key presses and mouse movements) from the client and updates the game status accordingly. The client typically refers to the device or interface used by the player to access the cloud gaming service; these devices can be smartphones, tablets, smart TVs, personal computers (PCs), etc.

[0056] In the embodiments of this application, Figure 1 This application illustrates a flowchart of the steps involved in a cloud gaming screen processing method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0057] S101, The client sends the display refresh rate to the server.

[0058] In specific implementation step S101, the client first obtains the refresh rate of its display. Since screen tearing is caused by the asynchrony between the client's display refresh rate and the graphics card frame rate, when the display refresh rate is higher than the graphics card frame rate, the client's display can adjust its refresh rate through adaptive synchronization to keep it consistent with the graphics card frame rate, avoiding screen tearing caused by some frames having a lower frame rate than the display refresh rate. However, in cloud gaming scenarios, when the server transmits data to the client, the frame rate received by the client will fluctuate further, causing more frames to have a higher frame rate than the display refresh rate. This reduces the number of frames that the display refresh rate can cover, increasing the screen tearing problem. Therefore, in this embodiment, the client's display refresh rate is sent to the server so that the server can adjust its rendering frame rate based on the display refresh rate. This ensures that the frame rate of more or even all frames in the data transmitted to the client can be covered by the display refresh rate, thereby using adaptive synchronization to keep the display refresh rate consistent with the frame rate of all frames, effectively eliminating the screen tearing problem.

[0059] In some optional implementations, the server in this application embodiment includes a virtual display, which has the function of supporting dynamic modification of the server's rendering frame rate, so that the server can dynamically adjust the rendering frame rate of the virtual display based on the display refresh rate.

[0060] S102. The server determines its rendering frame rate based on the display refresh rate, and the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate.

[0061] In specific implementation step S102, in the cloud gaming scenario, after the cloud gaming screen data from the server is transmitted to the client, the client's rendering frame rate will fluctuate further compared to the server's rendering frame rate due to factors such as encoding / decoding latency and network latency. This results in a greater difference from the display refresh rate. For example, if the display refresh rate is 60 FPS, and the server's rendering frame rate is set to the same 60 FPS as the display refresh rate, the client's rendering frame rate will often fluctuate between 50 FPS and 70 FPS after the data is transmitted to the client. This will cause half of the frames to have a rendering frame rate greater than the display refresh rate, resulting in screen tearing in these frames.

[0062] Therefore, to reduce or even eliminate the problem of the client's rendering frame rate exceeding the monitor's refresh rate, the server's rendering frame rate needs to be adjusted based on the monitor's refresh rate. This ensures that the upper limit of the server's rendering frame rate is less than or equal to the monitor's refresh rate. The portion of the server's rendering frame rate that is less than the monitor's refresh rate serves as upward redundancy space for unstable frame generation time (frame rate fluctuation) caused by latency after data frames arrive at the client from the server. This minimizes screen tearing caused by the client's decoded frame rate exceeding the monitor's refresh rate. Optionally, you can enable the server's graphics card's adaptive vertical synchronization function via the graphics card API interface to control the server's graphics card's rendering frame rate between 0 FPS and the upper limit of the server's rendering frame rate.

[0063] For example, the display refresh rate is 60 FPS. The client obtains the display refresh rate and sends it to the server. The server uses the 60 FPS display refresh rate as a benchmark and lowers its rendering frame rate to a maximum of 50 FPS. When the cloud gaming screen data is transmitted to the client after encoding, decoding, and network transmission based on a 50 FPS rendering frame rate, the client's maximum rendering frame rate is within the range of 40 FPS to 60 FPS due to latency disturbances, which is still less than or equal to the display refresh rate. The client can use adaptive synchronization to adjust the display refresh rate and the client's rendering frame rate to be consistent, effectively eliminating screen tearing.

[0064] In some optional implementations, since the frame rate fluctuations caused by the transmission delay between the server and the client may change at different time points, in order to better ensure that the client's rendering frame rate is effectively covered by the display refresh rate, the server's rendering frame rate is dynamically adjusted during transmission based on the delay of the previous second. That is, if the frame rate fluctuation caused by the transmission delay of the previous second is large, in order to ensure that the client's rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the server's rendering frame rate is increased, thereby increasing the upward redundancy range of frame rate fluctuations. Conversely, if the frame rate fluctuation caused by the transmission delay of the previous second is small, in order to ensure that the client's rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the server's rendering frame rate is decreased, thereby decreasing the upward redundancy range of frame rate fluctuations.

[0065] Specifically, the server's rendering frame rate is determined as follows: First, the client generates a frame rate adjustment ratio (n%) for the current second based on the decoded data obtained in the previous second, and sends it to the server. The frame rate adjustment ratio is configured to determine the fluctuation of all frames in the decoded data obtained in the previous second. The frame rate adjustment ratio is determined by the client based on the timestamp information sent by the server to the client in the previous second. Subsequently, the server receives the frame rate adjustment ratio for the current second sent by the client, and adjusts the display refresh rate downward based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

[0066] Optionally, the server's rendering frame rate for the current second is obtained by reducing the frame rate adjustment ratio for the current second based on the display refresh rate. That is, the frame rate adjustment ratio for the current second forms an upward redundancy space range for the server's rendering frame rate for the current second. After the server encodes and transmits the cloud game screen data to the client based on the server's rendering frame rate for the current second, the frame rate of the decoded data formed by the client after decoding fluctuates upward within the upward redundancy space, but is still less than or equal to the display refresh rate. This ensures that the display refresh rate can be adaptively synchronized with the client's rendering frame rate.

[0067] For example, the display refresh rate is 240Hz. Based on the decoded data obtained in the previous second, the client generates a frame rate adjustment ratio (n%) of 10% for the current second. The client sends the frame rate adjustment ratio for the current second to the server. Subsequently, the server receives the frame rate adjustment ratio for the current second sent by the client and reduces the display refresh rate by 10% based on the frame rate adjustment ratio for the current second, resulting in an upper limit of the server's rendering frame rate for the current second of 216Hz. The server's rendering frame rate for the current second is within the range of 0 to 216Hz. At this time, the rendering frame rate of the data frame after decoding by the client remains within the range of 240Hz.

[0068] S103. The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and generates encoded data which is then transmitted to the client.

[0069] In specific implementation step S103, after the server determines the server's rendering frame rate, the server renders the cloud gaming screen data according to the server's rendering frame rate, then performs encoding processing to generate encoded data, and sends the encoded data to the client.

[0070] In some optional implementations, since the client needs to obtain the frame rate adjustment ratio for the current second based on the decoded data of the previous second, and the frame rate adjustment ratio for the current second needs to be determined based on the latency between the server and the client, the server needs to provide the client with encoding time nodes when generating the encoded data. Specifically, after the streaming starts, after the server's graphics card has rendered a frame, when the server is encoding the cloud gaming screen data, the server generates first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client. The first timestamp information is used to characterize the encoding time node of each frame in the cloud gaming screen data.

[0071] S104. The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0072] In specific implementation step S104, the client receives the encoded data sent by the server, and sequentially decodes and renders the encoded data to obtain the decoded data. The decoded data is then stored in the client's frame buffer, and the cloud gaming screen is displayed on the client's display interface using the decoded data in the frame buffer. Simultaneously, the client also generates the frame rate adjustment ratio for the current second, dynamically adjusting the frame rate adjustment ratio every second through negative feedback, ensuring that the client's rendering frame rate is overridden by the display refresh rate every second.

[0073] Figure 3 This diagram illustrates the logical interaction of various modules in an existing cloud gaming screen processing method, such as... Figure 3 As shown, in existing cloud gaming screen processing methods, the server-side encoding module encodes the rendered cloud gaming screen data and sends it to the client. Since the server-side rendering frame rate is directly synchronized with the display refresh rate, when the client-side decoding module decodes the data and performs rendering on the client, the transmission delay between the server and the client causes the rendering frame rate of the decoded data to fluctuate compared to the server-side rendering frame rate. This results in some frames having a rendering frame rate higher than the display refresh rate, making it impossible for the display refresh rate of these frames to be adjusted to match the rendering frame rate through adaptive vertical synchronization, causing screen tearing in cloud gaming.

[0074] In the embodiments of this application, Figure 4 This paper illustrates a schematic diagram of the logical interactions between various modules in a cloud gaming screen processing method according to an embodiment of this application. Figure 4As shown, the server includes a rendering module, an encoding module, and a virtual display. The rendering module renders the cloud gaming screen data based on a vertical synchronization adaptive mode, and simultaneously adjusts the server's rendering frame rate by a frame rate adjustment ratio n%. The virtual display's refresh rate is (100-n)%A, ensuring that the virtual display's refresh rate (the server's rendering frame rate) is less than or equal to the display's refresh rate A. This provides sufficient upward redundancy for the rendering frame rate, ensuring that even with the transmission latency between the server and client, the rendered frame rate of the decoded data generated by the client's decoding module remains less than or equal to the display's refresh rate A after being disturbed compared to the server's rendering frame rate. Furthermore, to further improve the effective coverage of the display's refresh rate on the decoded data's rendering frame rate each second, the client also generates the current second's frame rate adjustment ratio based on the previous second's decoded data. This is achieved through negative feedback, dynamically adjusting the frame rate adjustment ratio each second, thereby dynamically adjusting the upper limit of the server's rendering frame rate each second.

[0075] In some optional implementations, firstly, the client obtains the latency difference ratio of all frames within the previous second based on the decoded data obtained in the previous second. This latency difference ratio characterizes the frame rate disturbance caused by transmission and latency after each frame of data is transmitted from the server to the client. In this embodiment, the adjustment direction of the frame rate adjustment ratio for the current second is determined by using the latency difference ratio of all frames within the previous second. Specifically, the latency difference ratio of each frame in the decoded data is obtained as follows: the client obtains the latency of each frame in the decoded data, where the latency characterizes the time delay of each frame's transmission from the server to the client; the client obtains the latency corresponding to a preset number of frames before the target frame, and determines the average latency based on the latency corresponding to the preset number of frames, where the target frame is any frame in the decoded data. For example, the preset number is 10 frames; the client obtains the latency corresponding to the 10 frames before the target frame, and uses the average of the latency corresponding to the 10 frames before the target frame as the average latency corresponding to the target frame.

[0076] Subsequently, the client obtains the difference between the delay of the target frame and the average delay based on the delay of the target frame and the average delay, as the delay difference; and obtains the proportion of the delay difference to the average delay of the target frame, which is determined as the delay difference proportion of the target frame.

[0077] In some optional implementations, a first perturbation frame is determined based on the latency difference ratio of all frames in the previous second. The first perturbation frame is a frame whose latency difference ratio is greater than or equal to a first threshold, where the first threshold is equal to the frame rate adjustment ratio corresponding to the previous second. The first perturbation frame characterizes frames with significant perturbation due to transmission latency in the previous second. If the first perturbation frame accounts for a large proportion of all frames in the previous second, it indicates that the adjustment range of the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is small. This may cause the client's rendering frame rate to exceed the display refresh rate during data transmission in the current second, resulting in screen tearing in some frames. Therefore, compared to the frame rate adjustment ratio of the previous second, the server data in the current second requires a larger frame rate adjustment ratio during rendering.

[0078] Specifically, if the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased by the first preset value to obtain the frame rate adjustment proportion for the current second. For example, the frame rate adjustment proportion corresponding to the previous second is n%, the first preset proportion is 10%, the first preset value is 10, and the first threshold is n%; if the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to the first preset proportion of 10%, the frame rate adjustment proportion n% corresponding to the previous second is increased by 10 to obtain the frame rate adjustment proportion for the current second (n+10)%.

[0079] In some optional implementations, a second perturbation frame is determined based on the latency difference ratio of all frames in the previous second. The second perturbation frame is a frame whose latency difference ratio is less than or equal to a second threshold, which is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value. The second perturbation frame is used to characterize frames with a smaller degree of perturbation due to transmission latency in the previous second. If the proportion of the second perturbation frame among all frames in the previous second is large, it indicates that the adjustment range of the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is large. Under the premise of avoiding screen tearing, the rendering frame rate can be increased by reducing the frame rate adjustment ratio to ensure the display effect on the client. Therefore, compared with the frame rate adjustment ratio of the previous second, the server data in the current second requires a smaller frame rate adjustment ratio during rendering.

[0080] Specifically, when the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to a second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced by a first preset value to obtain the frame rate adjustment proportion for the current second. For example, if the frame rate adjustment proportion corresponding to the previous second is n%, the second preset proportion is 90%, and the first preset value is 10, then the second threshold is (n-10)%; when the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion of 90%, the frame rate adjustment proportion n% corresponding to the previous second is reduced by 10 to obtain the frame rate adjustment proportion for the current second, which is (n-10)%. It should be noted that since the frame rate adjustment proportion for each second is dynamically adjusted based on the frame rate adjustment proportion for the previous second, the frame rate adjustment proportion for each second is greater than or equal to 0.

[0081] In some optional implementations, if the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset proportion, and / or the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset proportion, it indicates that adjusting the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is appropriate. Adjusting the server's rendering frame rate according to the frame rate adjustment ratio of the previous second can ensure the display effect on the client while avoiding screen tearing. Therefore, in this case, the frame rate adjustment ratio corresponding to the previous second is directly used as the frame rate adjustment ratio of the current second.

[0082] In some optional implementations, the delay of each frame in the decoded data is the duration between the time node when the cloud gaming screen data is encoded on the server and the time node when the cloud gaming screen data is decoded on the client. Specifically, the delay of each frame in the decoded data is obtained according to the following steps: In this embodiment of the application, referring to the content in step S103 above, when the server encodes the cloud gaming screen data, the server generates first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client; subsequently, the client receives the first timestamp information corresponding to each frame in the cloud gaming screen data; when the client decodes the encoded data, the client generates second timestamp information corresponding to each frame in the decoded data, the second timestamp information being used to characterize the decoding time node of each frame in the cloud gaming screen data; the client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0083] S105, The client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0084] In specific implementation step S105, since the rendering frame rate of the server is adjusted based on the display refresh rate, the rendering frame rate of the decoded data obtained after decoding and rendering by the client is less than or equal to the display refresh rate. The rendering frame rate of each frame in the decoded data is overridden by the display refresh rate. In this embodiment, the server uses adaptive vertical synchronization technology to adjust the display refresh rate to be consistent with the rendering frame rate of the decoded data, thereby achieving real-time synchronization between the rendering frame rate of each frame on the client and the display refresh rate, effectively eliminating screen tearing issues in cloud gaming on the client side.

[0085] Optionally, the client enables the G-Sync / FreeSync function, wherein the G-Sync / FreeSync function is used to adjust the display refresh rate in real time to adapt to the change in the rendered frame rate after decoding, based on the client rendering frame rate which is lower than the display refresh rate, when generating the decoding data for each frame, so that the display refresh rate of each frame drops to the same as the rendered frame rate of each frame, thereby effectively eliminating the problem of screen tearing in cloud gaming.

[0086] To enable those skilled in the art to more clearly understand this application, the cloud gaming screen processing method described in this application will now be described in detail through the following embodiments.

[0087] Figure 2 This application illustrates an example flowchart of a cloud gaming screen processing method according to an embodiment of the present application, as follows: Figure 2 As shown, the client first obtains the refresh rate of its own display and sends it to the server, so that the server adjusts its rendering frame rate based on the refresh rate. The server determines its rendering frame rate based on the refresh rate, then renders screen data within the server's virtual display refresh rate (the server's rendering frame rate range), and performs encoding processing to generate encoded data. Simultaneously, during encoding processing, the server generates a first timestamp for each frame in the cloud gaming screen data. The server then sends the encoded data and the corresponding first timestamp information to the client.

[0088] The client is equipped with G-Sync / FreeSync enabled. After the streaming starts, it receives the encoded data and first timestamp information of each frame. It decodes and renders the encoded data to obtain the decoded data, which is then stored in the frame buffer within the client. Simultaneously, during the decoding process, it generates second timestamp information corresponding to each frame in the cloud gaming screen data. Based on the first and second timestamp information, it determines the delay of each frame and dynamically adjusts the frame rate adjustment ratio for the next second based on the delay of each frame. This is then sent to the server so that the server modifies its rendering frame rate limit based on the dynamically adjusted frame rate adjustment ratio in the next second. This ensures that after all frames are transmitted within each second, the client's rendering frame rate is less than the display refresh rate.

[0089] Finally, the client enables G-Sync / FreeSync. This G-Sync / FreeSync function adjusts the display refresh rate in real-time, using a client-side rendering frame rate lower than the display's refresh rate as a baseline, to adapt to changes in the decoded rendering frame rate. This ensures that the display refresh rate for each frame decreases to match the rendering frame rate. Furthermore, the solution provided in this embodiment reduces latency by eliminating the need for the graphics card to wait for the display to refresh, and also avoids a significant drop in frame rate.

[0090] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, applied to a system including a server and a client. The method includes: the client sending its display refresh rate to the server; the server determining its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate; the server rendering and encoding cloud gaming screen data based on the server's rendering frame rate to generate encoded data which is then transmitted to the client; the client receiving the encoded data and decoding and rendering it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusting its display refresh rate to match the rendering frame rate of the decoded data. This application, by adjusting the server's rendering frame rate according to the client's display refresh rate, provides effective upward redundancy for the data transmission frame rate, ensuring that the client's decoded frame rate remains less than or equal to the display refresh rate, thereby effectively eliminating screen tearing issues in cloud gaming.

[0091] Based on the same inventive concept, this application discloses a cloud gaming screen processing system. The cloud gaming screen processing system is used to execute the cloud gaming screen processing method described in this application embodiment. The cloud gaming screen processing system includes a server and a client. The client is used to send the display refresh rate to the server. The server is used to determine its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate. The server then renders and encodes cloud gaming screen data based on the server's rendering frame rate to generate encoded data which is transmitted to the client. The client is also used to receive the encoded data and decode and render it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate. Finally, the client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0092] Based on the same inventive concept, this application discloses a cloud gaming screen processing device. Figure 5 A schematic diagram of the structure of a cloud gaming screen processing device according to an embodiment of this application is shown, as follows: Figure 5 As shown, the device includes:

[0093] The first sending module is used by the client to send the display refresh rate to the server.

[0094] A frame rate determination module is used for the server to determine the rendering frame rate of the server based on the display refresh rate, wherein the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate.

[0095] The encoding module is used by the server to render and encode the cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client.

[0096] A decoding module is used for the client to receive the encoded data and perform decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate;

[0097] An adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data.

[0098] In one optional implementation, the frame rate determination module includes:

[0099] The frame rate adjustment ratio submodule is used by the client to generate the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and send it to the server.

[0100] The frame rate determination submodule is used to receive the frame rate adjustment ratio for the current second sent by the client, and adjust the display refresh rate based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

[0101] In one optional implementation, the frame rate determination submodule includes:

[0102] The latency difference ratio determination submodule is used by the client to obtain the latency difference ratio of all frames in the previous second based on the decoded data obtained in the previous second.

[0103] The first perturbation frame submodule is used to determine the first perturbation frame in the previous second based on the delay difference ratio of all frames in the previous second. The first perturbation frame is a frame whose delay difference ratio is greater than or equal to a first threshold, and the first threshold is equal to the frame rate adjustment ratio corresponding to the previous second.

[0104] The first frame rate adjustment submodule is used to increase the frame rate adjustment ratio corresponding to the previous second by a first preset value when the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, so as to obtain the frame rate adjustment ratio of the current second.

[0105] In one optional implementation, the frame rate determination submodule further includes:

[0106] The second perturbation frame submodule is used to determine the second perturbation frame in the previous second based on the delay difference ratio of all frames in the previous second. The second perturbation frame is a frame whose delay difference ratio is less than or equal to a second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value.

[0107] The second frame rate adjustment submodule is used to reduce the frame rate adjustment ratio corresponding to the previous second by a first preset value when the proportion of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to a second preset proportion, so as to obtain the frame rate adjustment ratio of the current second.

[0108] In one optional implementation, the frame rate determination submodule further includes:

[0109] The third frame rate adjustment submodule is used to take the frame rate adjustment ratio corresponding to the previous second as the frame rate adjustment ratio of the current second when the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset ratio, and / or the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset ratio.

[0110] In one optional implementation, the delay difference ratio determination submodule includes:

[0111] A delay unit is used by the client to obtain the delay of each frame in the decoded data;

[0112] An average latency unit is used to determine the average latency based on the latency corresponding to a preset number of frames before the client obtains the target frame, wherein the target frame is any frame in the decoded data.

[0113] The latency difference ratio unit is used by the client to determine the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

[0114] In one alternative implementation, the delay unit includes:

[0115] The first timestamp information subunit is used to generate first timestamp information corresponding to each frame in the cloud game screen data and send it to the client when the server encodes the cloud game screen data.

[0116] The second timestamp information subunit is used to generate second timestamp information corresponding to each frame in the decoded data when the client decodes the encoded data.

[0117] The delay subunit is used by the client to use the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0118] Based on the same inventive concept, this application discloses an electronic device. Figure 6 A schematic diagram of an electronic device according to an embodiment of this application is shown, such as... Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set can be loaded and run on the processor 120 to implement the steps in the cloud gaming screen processing method disclosed in the embodiments of this application.

[0119] Based on the same inventive concept, embodiments of this application disclose a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set thereon. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps in the cloud gaming screen processing method disclosed in embodiments of this application.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0121] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0125] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0126] The above provides a detailed description of a cloud gaming screen processing method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A cloud gaming screen processing method, characterized in that, Applied to systems including both server and client components, the method includes: The client sends the display refresh rate to the server. The server determines its rendering frame rate based on the display refresh rate, and the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate. The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and generates encoded data which is then transmitted to the client. The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate. The client adjusts the display refresh rate to match the rendering frame rate of the decoded data. The server determines its rendering frame rate based on the display refresh rate, including: The client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and sends it to the server. The server receives the frame rate adjustment ratio for the current second from the client, and adjusts the display refresh rate downward based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

2. The cloud gaming screen processing method according to claim 1, characterized in that, The client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including: The client obtains the latency difference ratio of all frames in the previous second based on the decoded data obtained in the previous second. Based on the latency difference ratio of all frames in the previous second, the first perturbation frame in the previous second is determined. The first perturbation frame is a frame whose latency difference ratio is greater than or equal to a first threshold. The first threshold is equal to the frame rate adjustment ratio corresponding to the previous second. If the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment proportion for the current second.

3. The cloud gaming screen processing method according to claim 2, characterized in that, After obtaining the latency difference ratio of all frames in the previous second, the method further includes: Based on the latency difference ratio of all frames in the previous second, a second perturbation frame in the previous second is determined. The second perturbation frame is a frame whose latency difference ratio is less than or equal to a second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value. If the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment proportion of the current second.

4. The cloud gaming screen processing method according to claim 3, characterized in that, The method further includes: If the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset proportion, and / or if the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset proportion, the frame rate adjustment proportion corresponding to the previous second shall be used as the frame rate adjustment proportion for the current second.

5. The cloud gaming screen processing method according to claim 2, characterized in that, The latency difference ratio of each frame in the decoded data is obtained as follows: The client obtains the delay of each frame in the decoded data; The client obtains the delays corresponding to a preset number of frames before the target frame, and determines the average delay based on the delays corresponding to the preset number of frames. The target frame is any frame in the decoded data. The client determines the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

6. The cloud gaming screen processing method according to claim 5, characterized in that, The client obtains the delay of each frame in the decoded data, including: When the server encodes the cloud gaming screen data, the server generates a first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client. When the client decodes the encoded data, the client generates a second timestamp information corresponding to each frame in the decoded data; The client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

7. A cloud gaming screen processing device, characterized in that, The device includes: The first sending module is used by the client to send the display refresh rate to the server. A frame rate determination module is used for the server to determine the rendering frame rate of the server based on the display refresh rate, wherein the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate. The encoding module is used by the server to render and encode the cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client. A decoding module is used for the client to receive the encoded data and perform decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; An adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data; The frame rate determination module includes: The frame rate adjustment ratio submodule is used by the client to generate the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and send it to the server. The frame rate determination submodule is used to receive the frame rate adjustment ratio for the current second sent by the client, and adjust the display refresh rate based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the cloud gaming screen processing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the cloud gaming screen processing method according to any one of claims 1-6.

Images