Cloud game picture processing method and device, electronic equipment and storage medium

Abstract

The invention provides a cloud game picture processing method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the cloud game picture processing method comprises the steps that a client sends a display refresh rate to a server; the server determines a rendering frame rate of the server based on the display refresh rate; the server renders and encodes the cloud game picture data based on the rendering frame rate of the server to generate encoded data and transmits the encoded data to the client; the client receives the coded data and decodes and renders the coded data to obtain decoded data, and the rendering frame rate of the decoded data is smaller than or equal to the refresh rate of the display; and the client adjusts the refresh rate of the display to be consistent with the rendering frame rate of the decoded data. According to the method, the rendering frame rate of the server side is adjusted according to the display refresh rate of the client side, and an effective upward redundant space is provided for the frame rate of data transmission, so that the picture tearing problem in the cloud game is effectively eliminated by adjusting the display refresh rate.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and specifically, to a cloud game screen processing method, device, electronic device and storage medium. Background Art

[0002] The wave of digitalization has swept the world, and cloud computing has become the core of the development of the IT industry. As an emerging technology force, cloud gaming transfers game rendering from players' devices to centralized servers, revolutionizing the gaming experience. However, high-frame-rate cloud gaming faces the problem of screen tearing. Vertical synchronization and G-Sync / FreeSync technology can respectively solve the problem of screen tearing caused by graphics card frame rates higher and lower than the monitor refresh rate. The former will cause delays and freezes, while the latter can only be applied to local gaming scenarios.

[0003] Cloud gaming is more complicated because it needs to go through server rendering, encoding, network transmission, client decoding, display and other processes. Since G-Sync / FreeSync technology requires communication between the graphics card and the monitor to achieve frame rate synchronization, the graphics card of cloud gaming is on the server side and the monitor is on the client side, so communication synchronization cannot be achieved; and because of the network transmission in the middle, even if the image frame rate on the server side and the refresh rate of the client monitor are fixed and unified, the frame generation time of the client will be unstable after the encoding and decoding delay and network transmission delay, and the screen will also be torn on the client side. Therefore, how to eliminate the problem of screen tearing in cloud gaming has become a problem that needs to be solved urgently in this field. Summary of the invention

[0004] The embodiments of the present application provide a cloud game screen processing method, device, electronic device and storage medium, aiming to solve the problem of how to eliminate screen tearing in cloud games.

[0005] A first aspect of an embodiment of the present application provides a cloud game screen processing method, which is applied to a system including a server and a client. The method includes:

[0006] The client sends the display refresh rate to the server;

[0007] The server determines a rendering frame rate of the server based on the display refresh rate, and an upper limit value of the rendering frame rate of the server is less than or equal to the display refresh rate;

[0008] The server renders and encodes the cloud game screen data based on the rendering frame rate of the server, generates encoded data and transmits it to the client;

[0009] The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the refresh rate of the display;

[0010] The client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data.

[0011] In an optional implementation manner, the server determines the rendering frame rate of the server based on the display refresh rate, including:

[0012] The client generates a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and sends the frame rate adjustment ratio to the server;

[0013] The server receives the frame rate adjustment ratio of the current second sent by the client, and lowers the refresh rate of the display based on the frame rate adjustment ratio of the current second to obtain the rendering frame rate of the server in the current second.

[0014] In an optional implementation manner, the client generates a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including:

[0015] The client obtains the delay difference ratio of all frames in the previous second based on the decoded data obtained in the previous second;

[0016] Determine a first disturbance frame in the previous second based on the delay difference ratio of all frames in the previous second, wherein the first disturbance frame is a frame whose delay difference ratio is greater than or equal to a first threshold, and the first threshold is equal to the frame rate adjustment ratio corresponding to the previous second;

[0017] When the ratio of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset ratio, the frame rate adjustment ratio corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment ratio of the current second.

[0018] In an optional implementation manner, after obtaining the delay difference ratio of all frames in the previous second, the method further includes:

[0019] Based on the delay difference ratio of all frames in the previous second, determining a second disturbance frame in the previous second, the second disturbance frame being a frame whose delay difference ratio is less than or equal to a second threshold, and the second threshold being a difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value;

[0020] When the ratio of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to the second preset ratio, the frame rate adjustment ratio corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment ratio of the current second.

[0021] In an optional implementation, the method further includes:

[0022] When the ratio of the first disturbance frame in the previous second to all frames in the previous second is less than the first preset ratio, and / or the ratio of the second disturbance frame in the previous second to all frames in the previous second is less than the second preset ratio, the frame rate adjustment ratio corresponding to the previous second is used as the frame rate adjustment ratio of the current second.

[0023] In an optional implementation manner, the delay difference ratio of each frame in the decoded data is obtained as follows:

[0024] The client obtains a delay of each frame in the decoded data;

[0025] The client obtains delays corresponding to a preset number of frames before a target frame, and determines an average delay based on the delays corresponding to the preset number of frames, wherein the target frame is any frame in the decoded data;

[0026] The client determines the delay difference ratio of the target frame based on the delay of the target frame and the average delay.

[0027] In an optional implementation manner, the client obtains the delay of each frame in the decoded data, including:

[0028] When the server encodes the cloud game screen data, the server generates first timestamp information corresponding to each frame in the cloud game screen data and sends it to the client;

[0029] When the client decodes the encoded data, the client generates second timestamp information corresponding to each frame in the decoded data;

[0030] The client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0031] A second aspect of an embodiment of the present application provides a cloud game screen processing device, the device comprising:

[0032] A first sending module, used for the client to send the display refresh rate to the server;

[0033] A frame rate determination module, used for the server to determine the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate;

[0034] The encoding module is used for the server to render and encode the cloud game screen data based on the rendering frame rate of the server, generate encoded data and transmit it to the client;

[0035] A decoding module, used for the client to receive the encoded data and decode and render the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the refresh rate of the display;

[0036] An adaptive synchronization module is used for the client to adjust the refresh rate of the display to be consistent with the rendering frame rate of the decoded data.

[0037] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the cloud game screen processing method described in any one of the first aspect of the embodiment of the present application.

[0038] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the cloud game screen processing method described in any one of the first aspect of the embodiment of the present application are implemented.

[0039] Beneficial effects:

[0040] The present application provides a cloud game screen processing method, device, electronic device and storage medium, which are applied to a system including a server and a client, and the method includes: the client sends the display refresh rate to the server; the server determines the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate; the server renders and encodes the cloud game screen data based on the rendering frame rate of the server, generates encoded data and transmits it to the client; the client receives the encoded data and decodes and renders the encoded data to obtain decoded data, and the rendering frame rate of the decoded data is less than or equal to the display refresh rate; the client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data. The present application provides effective upward redundant space for the frame rate of data transmission by adjusting the rendering frame rate of the server according to the display refresh rate of the client, so that the frame rate after decoding by the client is still less than or equal to the display refresh rate, thereby effectively eliminating the screen tearing problem in cloud games by adjusting the display refresh rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 This is a flowchart of a method for processing a cloud game screen proposed in one embodiment of the present application;

[0043] Figure 2 This is a flowchart of an example step of a cloud game screen processing method proposed in an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of the logical interaction of various modules in an existing cloud game screen processing method;

[0045] Figure 4 This is a schematic diagram of the logical interaction of various modules in a cloud game screen processing method proposed in one embodiment of the present application;

[0046] Figure 5 It is a structural schematic diagram of a cloud game screen processing device proposed in one embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In the drawings, the size of the constituent elements, the thickness of the layer or the area may be exaggerated for the sake of clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the size shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect the true proportions. In addition, the drawings schematically show ideal examples, and any implementation of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0050] The wave of digitalization has swept the world, and cloud computing has become the core of the development of the IT industry. As an emerging technology force, cloud gaming transfers game rendering from player devices to centralized servers, revolutionizing the gaming experience. However, high-frame-rate cloud gaming faces the problem of screen tearing.

[0051] Screen tearing is an image display problem that may occur in electronic games. It usually occurs in fast-moving scenes or when the frame rate is not synchronized (the graphics card output frame rate is greater than or less than the monitor refresh rate). This problem causes a mismatch between two adjacent frames on the screen, making the image seem to be "torn" into two parts, with the upper and lower halves not completely synchronized, seriously affecting the user and player experience. Screen tearing has troubled players since the birth of games. In traditional local games, screen tearing occurs because the monitor refresh rate is not synchronized with the graphics card frame rate. When the monitor refreshes the screen, if the graphics card generates the next frame at the same time, the registers in the video memory that store the rendered screen are switching between the two frames of screen data. After the monitor reads the registers, the upper and lower parts of the screen will come from different frames, resulting in screen tearing. Therefore, screen tearing will occur when the frame rate output by the graphics card is greater than or less than the monitor refresh rate.

[0052] In response to the screen tearing problem, the industry generally uses vertical synchronization (V-Sync) technology to solve it. This technology effectively reduces screen tearing by reducing the graphics card frame rate to the monitor refresh rate. However, vertical synchronization is not a perfect solution. It will cause the graphics card to wait for the monitor to refresh before rendering the screen, which will cause input delay. In addition, the monitor refresh rate of this function is fixed. When the graphics card frame rate is lower than the monitor refresh rate, vertical synchronization will also limit the graphics card frame rate to half or one-quarter of the monitor refresh rate (the multiple relationship will not cause screen tearing), causing a significant drop in frame rate and affecting the smoothness of the game. Therefore, this technology is mainly used in scenes where the graphics card frame rate is higher than the monitor refresh rate. Weighing the pros and cons, some players will choose to turn off vertical synchronization in order to pursue game performance.

[0053] To address the shortcomings of vertical synchronization, graphics card manufacturers have introduced G-Sync and FreeSync technologies, which reversely match the graphics card frame rate by dynamically adjusting the refresh rate of the monitor. This technology does not require the graphics card to reduce the frame rate when the graphics card frame rate is between the lower limit frame rate (G-Sync supports an ultra-low limit of 4-10Hz, and G-Sync Compatible (G-Sync compatible version) and FreeSync are mostly 42Hz) and the maximum refresh rate of the monitor. It can effectively solve the problem of screen tearing while ensuring game smoothness and low latency. Therefore, this technology is mainly used in scenarios where the graphics card frame rate is lower than the maximum refresh rate of the monitor, but this technology requires monitor compatibility.

[0054] Vertical synchronization and G-Sync / FreeSync technology can solve the problem of screen tearing caused by graphics card frame rate being higher or lower than the monitor refresh rate, respectively. The former will cause delays and freezes, while the latter can only be used in local game scenarios. Cloud games are more complicated because they need to go through server rendering, encoding, network transmission, client decoding, and display. Since G-Sync / FreeSync technology requires communication between the graphics card and the monitor to achieve frame rate synchronization, the graphics card of cloud games is on the server side and the monitor is on the client side, so communication synchronization cannot be achieved. In addition, due to the network transmission in the middle, even if the image frame rate of the server side and the monitor refresh rate of the client side are fixed and unified, the frame generation time of the client side will be unstable after the encoding and decoding delay and network transmission delay. For example, even if the server stably generates a 60FPS image, after encoding and decoding and network transmission, the frame rate will often fluctuate between 50FPS and 70FPS when the client receives it. Therefore, even if the monitor is set to a 60FPS refresh rate, screen tearing will occur on the client side.

[0055] In view of this, the embodiment of the present application proposes a cloud game screen processing method, which is applied to a system including a server and a client, wherein the server is the core part of the cloud game service, which usually resides on the virtual server (vMonitor) of the cloud game provider, and the virtual server simulates the function of the physical display by software methods. The picture rendered by the graphics card will be transmitted to the virtual display to facilitate the picture capture of the cloud game, and compared with the physical display, the setting of the virtual display is more flexible to support a large number of players playing online games at the same time and provide a smooth gaming experience. On the server side of the cloud game, a complete game program is running. When the player connects to the server side through the client, the server side will render the game screen in real time, process the game logic, and send the result to the client in the form of a video stream; at the same time, the server side will also receive the player input (such as keystrokes, mouse movements, etc.) sent by the client, and update the game status accordingly. The client generally refers to the device or interface used by the player to access the cloud game service, which can be a smartphone, tablet computer, smart TV, personal computer (PC), etc.

[0056] In the embodiment of the present application, Figure 1 A flowchart of a method for processing a cloud game screen according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0057] S101. The client sends a display refresh rate to the server.

[0058] When the step S101 is specifically implemented, the client first obtains the display refresh rate of the client display. Since the screen tearing problem is caused by the asynchronous display refresh rate of the client and the frame rate of the graphics card, when the display refresh rate is higher than the frame rate of the graphics card, the display of the client can adjust the display refresh rate through the adaptive synchronization function to keep it consistent with the frame rate of the graphics card, so as to avoid the screen tearing problem caused by the frame rate of some frames being less than the display refresh rate. However, in the cloud game scenario, when the server transmits data to the client, the frame rate of the client when receiving will further fluctuate, so that the frame rate of more frames is greater than the display refresh rate, thereby reducing the number of frames that can be covered by the display refresh rate, and increasing the generation of screen tearing problems. Therefore, in an embodiment of the present application, the display refresh rate of the client display is sent to the server, so that the server adjusts the rendering frame rate of the server based on the display refresh rate, ensuring that the frame rates of more or even all frames in the data transmitted to the client can be covered by the display refresh rate, thereby keeping the display refresh rate consistent with the frame rate of all frames through the adaptive synchronization function, effectively eliminating the screen tearing problem.

[0059] In some optional implementations, the server in the embodiments of the present application includes a virtual display, and the virtual display has the function of supporting dynamic modification of the server rendering frame rate, so that the server can dynamically adjust the rendering frame rate of the virtual display based on the display refresh rate.

[0060] S102: The server determines a rendering frame rate of the server based on the display refresh rate, and an upper limit value of the rendering frame rate of the server is less than or equal to the display refresh rate.

[0061] When step S102 is specifically implemented, in the scenario of the cloud game, after the cloud game screen data of the server is transmitted to the client, under the influence of factors such as encoding and decoding delay and network delay, the rendering frame rate of the client will further fluctuate compared with the rendering frame rate of the server, thereby being more different from the display refresh rate. Exemplarily, the display refresh rate is 60FPS. If the rendering frame rate of the server is set to 60FPS, which is the same as the display refresh rate, the rendering frame rate of the client data will often fluctuate between 50FPS and 70FPS after the data is transmitted to the client, thereby making the rendering frame rate of half of the frames greater than the display refresh rate, causing screen tearing problems in these frames.

[0062] Therefore, in order to reduce or even eliminate the problem that the rendering frame rate of the client data is greater than the refresh rate of the display, it is necessary to adjust the rendering frame rate of the server based on the refresh rate of the display to ensure that the upper limit of the rendering frame rate of the server is less than or equal to the refresh rate of the display. The portion of the rendering frame rate of the server that is less than the refresh rate of the display serves as an upward redundant space for the unstable frame generation time (frame rate fluctuation) caused by the delay after the data frame arrives at the client from the server, thereby eliminating the screen tearing problem caused by the frame rate after the client decodes being higher than the refresh rate of the display as much as possible. Optionally, you enable the adaptive vertical synchronization function of the server graphics card through the graphics card API interface, and control the rendering frame rate of the server graphics card between 0FPS and the upper limit of the rendering frame rate of the server.

[0063] Exemplarily, the display refresh rate is 60FPS, the client obtains the display refresh rate and sends the display refresh rate to the server, and the server lowers the rendering frame rate of the server based on the display refresh rate of 60FPS, so that the rendering frame rate upper limit of the server reaches 50FPS. When the cloud game screen data is encoded and decoded and transmitted over the network based on the rendering frame rate of 50FPS, the rendering frame rate upper limit of the client reaches 40FPS to 60FPS under the influence of delay disturbance, that is, it is still less than or equal to the display refresh rate. The client can adjust the display refresh rate to be consistent with the rendering frame rate of the client through the adaptive synchronization, effectively eliminating the screen tearing problem.

[0064] In some optional embodiments, since the frame rate disturbance caused by the transmission delay between the server and the client at different time nodes will change, in order to better ensure that the client rendering frame rate is effectively covered by the display refresh rate, the rendering frame rate of the server is dynamically adjusted during the transmission process based on the delay of the previous second, that is, when the frame rate fluctuation caused by the transmission delay of the previous second is large, in order to ensure that the client rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the rendering frame rate of the server is increased, thereby increasing the upward redundant spatial range of the frame rate fluctuation; conversely, when the frame rate fluctuation caused by the transmission delay of the previous second is small, in order to ensure that the client rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the rendering frame rate of the server is reduced, thereby reducing the upward redundant spatial range of the frame rate fluctuation.

[0065] Specifically, the rendering frame rate of the server is determined as follows: first, the client generates a frame rate adjustment ratio (n%) for the current second based on the decoded data obtained in the previous second, and sends it to the server, and the frame rate adjustment ratio is configured to determine the fluctuation of all frames in the decoded data obtained in the previous second, wherein the frame rate adjustment ratio is determined by the client based on the timestamp information sent by the server to the client in the previous second; subsequently, the server receives the frame rate adjustment ratio for the current second sent by the client, and lowers the refresh rate of the display based on the frame rate adjustment ratio for the current second to obtain the rendering frame rate of the server in the current second.

[0066] Optionally, the rendering frame rate of the server in the current second is obtained by reducing the frame rate adjustment ratio of the current second on the basis of the display refresh rate, that is, the frame rate adjustment ratio of the current second forms an upward redundant space range for the rendering frame rate of the server in the current second. After the server encodes the cloud game screen data and transmits it to the client based on the rendering frame rate of the server in the current second, the frame rate of the decoded data formed by the decoding process on the client fluctuates upward within the upward redundant space and is still less than or equal to the display refresh rate, thereby ensuring that the display refresh rate can be kept consistent with the client rendering frame rate through adaptive synchronization.

[0067] Exemplarily, the refresh rate of the display is 240 Hz, and the client generates a frame rate adjustment ratio (n%) of 10% for the current second based on the decoded data obtained in the previous second, and the client sends the frame rate adjustment ratio of the current second to the server; subsequently, the server receives the frame rate adjustment ratio of the current second sent by the client, and lowers the refresh rate of the display by 10% based on the frame rate adjustment ratio of the current second, so that the upper limit of the rendering frame rate of the server in the current second is 216 Hz, and the rendering frame rate of the server in the current second is in the range of 0 to 216 Hz. At this time, the rendering frame rate of the data frame after decoding and processing by the client remains in the range of 240 Hz.

[0068] S103: The server renders and encodes the cloud game screen data based on the rendering frame rate of the server, generates encoded data and transmits it to the client.

[0069] When step S103 is specifically implemented, after the server determines the rendering frame rate of the server, the server renders the cloud game screen data according to the rendering frame rate of the server, then performs encoding processing to generate encoded data, and sends the encoded data to the client.

[0070] In some optional implementations, since the client needs to obtain the frame rate adjustment ratio of the current second based on the decoded data of the previous second, and the frame rate adjustment ratio of the current second needs to be determined based on the delay between the server and the client, the server needs to provide the client with an encoding time node when generating the encoded data. Specifically, after the streaming starts, after the server graphics card renders a frame, when the server encodes the cloud game screen data, the server generates the first timestamp information corresponding to each frame in the cloud game screen data and sends it to the client. The first timestamp information is used to represent the encoding time node of each frame in the cloud game screen data.

[0071] S104: The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data, wherein a rendering frame rate of the decoded data is less than or equal to a refresh rate of the display.

[0072] When step S104 is specifically implemented, the client receives the encoded data sent by the server, and decodes and renders the encoded data in sequence to obtain the decoded data, and then stores the decoded data in the frame buffer in the client, and displays the cloud game screen on the display interface of the client through the decoded data in the frame buffer. At the same time, the client is also used to generate the frame rate adjustment ratio of the current second, so as to dynamically adjust the frame rate adjustment ratio of each second by negative feedback, so that the rendering frame rate of the client in each second is covered by the refresh rate of the display.

[0073] Figure 3 A schematic diagram of the logical interaction of each module in an existing cloud game screen processing method is shown, such as Figure 3 As shown, in the existing cloud game screen processing method, the encoding module of the server encodes the rendered cloud game screen data and sends it to the client. Since the rendering frame rate of the server is directly synchronized with the display refresh rate, the decoding processing generated by the decoding processing of the decoding module of the client is rendered by the client. Under the influence of the transmission delay between the server and the client, the rendering frame rate of the decoded data will be disturbed compared with the rendering frame rate of the server, resulting in the rendering frame rate of some frames being greater than the display refresh rate, making it impossible to adjust the display refresh rate of these frames through adaptive vertical synchronization to be consistent with the rendering frame rate, resulting in the problem of cloud game screen tearing.

[0074] In the embodiment of the present application, Figure 4 FIG. 1 shows a schematic diagram of the logical interaction of each module in a cloud game screen processing method proposed in an embodiment of the present application, such as Figure 4As shown, the server includes a rendering module, an encoding module and the virtual display. The rendering module renders the cloud game screen data based on the vertical synchronization adaptive mode, and adjusts the rendering frame rate of the server by the frame rate adjustment ratio n%. The refresh rate of the virtual display is (100-n)% A, so that the refresh rate of the virtual display (the rendering frame rate of the server) is less than or equal to the display refresh rate A, giving the rendering frame rate enough upward redundant space, so that when the client performs rendering processing, the decoding processing generated by the decoding processing of the decoding module of the client, under the influence of the transmission delay between the server and the client, the rendering frame rate of the decoded data is still less than or equal to the display refresh rate A after the rendering frame rate of the server is disturbed. At the same time, in order to further improve the effective coverage of the display refresh rate of each second to the rendering frame rate of the decoded data, the client also generates the frame rate adjustment ratio of the current second through the decoding data of the previous second, so as to dynamically adjust the frame rate adjustment ratio of each second by negative feedback, thereby dynamically adjusting the upper limit of the rendering frame rate of the server in each second.

[0075] In some optional implementations, first, the client obtains the delay difference ratio of all frames in the last second based on the decoded data obtained in the last second, and the delay difference ratio is used to characterize the situation that each frame of data is disturbed due to transmission and delay after being transmitted to the client through the server. The embodiment of the present application determines the adjustment direction of the frame rate adjustment ratio of the current second through the delay difference ratio of all frames in the last second. Specifically, the delay difference ratio of each frame in the decoded data is obtained in the following manner: the client obtains the delay of each frame in the decoded data, and the delay is used to characterize the time delay of each frame transmitted from the server to the client; the client obtains the delay corresponding to a preset number of multiple frames before the target frame, and determines the average delay based on the delay corresponding to the preset number of multiple frames, and the target frame is any frame in the decoded data. Exemplarily, the preset number is 10 frames, and the client obtains the delay corresponding to the 10 frames before the target frame, and takes the average value of the delay corresponding to the 10 frames before the target frame as the average delay corresponding to the target frame.

[0076] Subsequently, the client obtains the difference between the delay of the target frame and the average delay based on the delay of the target frame and the average delay as the delay difference; obtains the ratio of the delay difference to the average delay of the target frame, and determines it as the delay difference ratio of the target frame.

[0077] In some optional embodiments, based on the delay difference ratio of all frames in the previous second, the first disturbance frame in the previous second is determined, and the first disturbance frame is a frame whose delay difference ratio is greater than or equal to a first threshold, and the first threshold is equal to the frame rate adjustment ratio corresponding to the previous second. The first disturbance frame is used to characterize the frame with a large degree of disturbance under the influence of transmission delay in the previous second. If the first disturbance frame accounts for a large proportion of all frames in the previous second, it means that the adjustment amplitude of the rendering frame rate of the server according to the frame rate adjustment ratio of the previous frame is small, which may cause the rendering frame rate of the client to be greater than the display refresh rate when the data is transmitted in the current second under the condition of large disturbance, resulting in screen tearing problems in some frames. Therefore, compared with the frame rate adjustment ratio of the previous second, the server data of the current second requires a larger frame rate adjustment ratio during the rendering process.

[0078] Specifically, when the ratio of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset ratio, the frame rate adjustment ratio corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment ratio of the current second. Exemplarily, the frame rate adjustment ratio corresponding to the previous second is n%, the first preset ratio is 10%, the first preset value is 10, and the first threshold is n%; when the ratio of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset ratio of 10%, the frame rate adjustment ratio n% corresponding to the previous second is increased by 10 to obtain the frame rate adjustment ratio of the current second (n+10)%.

[0079] In some optional embodiments, based on the delay difference ratio of all frames in the previous second, a second disturbance frame in the previous second is determined, the second disturbance frame being a frame whose delay difference ratio is less than or equal to a second threshold, the second threshold being the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value. The second disturbance frame is used to characterize the frame with a smaller disturbance degree under the influence of the transmission delay in the previous second. If the second disturbance frame accounts for a larger proportion of all frames in the previous second, it means that the adjustment amplitude of the rendering frame rate of the server according to the frame rate adjustment ratio of the previous frame is larger. Under the premise of avoiding the problem of screen tearing, the rendering frame rate can be increased by reducing the frame rate adjustment ratio to ensure the display effect on the client. Therefore, compared with the frame rate adjustment ratio of the previous second, the server data of the current second requires a smaller frame rate adjustment ratio during the rendering process.

[0080] Specifically, when the proportion of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, the frame rate adjustment ratio corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment ratio of the current second. Exemplarily, the frame rate adjustment ratio corresponding to the previous second is n%, the second preset ratio is 90%, and the first preset value is 10, then the second threshold value is (n-10)%; when the proportion of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to 90% of the second preset ratio, the frame rate adjustment ratio n% corresponding to the previous second is reduced by 10 to obtain the frame rate adjustment ratio of the current second (n-10)%. It should be noted that since the frame rate adjustment ratio of each second is dynamically adjusted based on the frame rate adjustment ratio of the previous second, the frame rate adjustment ratio of each second is greater than or equal to 0.

[0081] In some optional embodiments, when the ratio of the first disturbance frame in the previous second to all frames in the previous second is less than the first preset ratio, and / or the ratio of the second disturbance frame in the previous second to all frames in the previous second is less than the second preset ratio, it indicates that the adjustment amplitude of adjusting the rendering frame rate of the server according to the frame rate adjustment ratio of the previous frame is more appropriate, and adjusting the rendering frame rate of the server according to the frame rate adjustment ratio of the previous second can ensure the display effect of the client while avoiding the problem of screen tearing. Therefore, in this case, the frame rate adjustment ratio corresponding to the previous second is directly used as the frame rate adjustment ratio of the current second.

[0082] In some optional embodiments, the delay of each frame in the decoded data is the duration between the time node when the cloud game screen data is encoded on the server and the time node when the cloud game screen data is decoded on the client. Specifically, the delay of each frame in the decoded data is obtained according to the following steps: In the embodiment of the present application, referring to the content in the above step S103, when the server encodes the cloud game screen data, the server generates a first timestamp information corresponding to each frame in the cloud game screen data and sends it to the client; then, the client receives the first timestamp information corresponding to each frame in the cloud game screen data; when the client decodes the encoded data, the client generates a second timestamp information corresponding to each frame in the decoded data, and the second timestamp information is used to represent the decoding time node of each frame in the cloud game screen data; the client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0083] S105: The client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data.

[0084] When step S105 is specifically implemented, since the rendering frame rate of the server is adjusted based on the display refresh rate, the rendering frame rate of the decoded data obtained after the client performs decoding and rendering processing is less than or equal to the display refresh rate, and the rendering frame rate of each frame in the decoded data is covered by the display refresh rate. In an embodiment of the present application, the server adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data based on adaptive vertical synchronization technology, thereby achieving real-time synchronization between the rendering frame rate of each frame on the client and the display refresh rate, which can effectively eliminate the tearing problem of the client cloud game screen.

[0085] Optionally, the client turns on the G-Sync\FreeSync function, wherein the G-Sync\FreeSync function is used to adjust the display refresh rate in real time to adapt to the changes in the rendering frame rate after decoding, based on the client rendering frame rate that is less than the display refresh rate when generating decoded data for each frame, so as to make the display refresh rate of each frame drop to the same as the rendering frame rate of each frame, thereby effectively eliminating the problem of screen tearing in cloud games.

[0086] In order to enable those skilled in the art to understand the present application more clearly, the cloud game screen processing method described in the present application is now described in detail through the following embodiments.

[0087] Figure 2 An example step flow chart of a cloud game screen processing method proposed in an embodiment of the present application is shown. Figure 2 As shown, the client first obtains the display refresh rate of the client display, and sends the display refresh rate of the client display to the server, so that the server adjusts the rendering frame rate of the server based on the display refresh rate. The server determines the rendering frame rate of the server based on the display refresh rate, and then renders the screen data within the virtual display refresh rate of the server (the rendering frame rate range of the server), and then performs encoding processing to generate encoded data. At the same time, the server generates the first timestamp information corresponding to each frame in the cloud game screen data when performing the encoding processing, and the server sends the encoded data and the corresponding first timestamp information to the client.

[0088] The client is equipped with G-Sync\FreeSync function, receives the encoded data and first timestamp information of each frame after the streaming starts, decodes and renders the encoded data to obtain the decoded data, and then stores the decoded data in the frame buffer in the client; at the same time, generates second timestamp information corresponding to each frame in the cloud game screen data during decoding; determines the delay of each frame based on the first timestamp information and the second timestamp information, dynamically adjusts the frame rate adjustment ratio of the next second based on the delay of each frame, and sends it to the server, so that the server modifies the rendering frame rate upper limit of the server based on the dynamically adjusted frame rate adjustment ratio in the next second, thereby ensuring that after all frames in each second are transmitted, the rendering frame rate of the client is less than the refresh rate of the display.

[0089] Finally, the client turns on the G-Sync\FreeSync function, wherein the G-Sync\FreeSync function is used to adjust the display refresh rate in real time to adapt to the changes in the rendering frame rate after decoding, based on the client rendering frame rate that is less than the display refresh rate, when generating decoded data for each frame, so that the display refresh rate of each frame drops to the same as the rendering frame rate of each frame. In addition, the solution provided in the embodiment of the present application reduces latency because it does not require the graphics card to render and wait for the display to refresh, and does not cause a significant drop in the frame rate.

[0090] The present application provides a cloud game screen processing method, device, electronic device and storage medium, which are applied to a system including a server and a client, and the method includes: the client sends the display refresh rate to the server; the server determines the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate; the server renders and encodes the cloud game screen data based on the rendering frame rate of the server, generates encoded data and transmits it to the client; the client receives the encoded data and decodes and renders the encoded data to obtain decoded data, and the rendering frame rate of the decoded data is less than or equal to the display refresh rate; the client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data. The present application provides effective upward redundant space for the frame rate of data transmission by adjusting the rendering frame rate of the server according to the display refresh rate of the client, so that the frame rate after decoding by the client is still less than or equal to the display refresh rate, thereby effectively eliminating the screen tearing problem in cloud games by adjusting the display refresh rate.

[0091] Based on the same inventive concept, an embodiment of the present application discloses a cloud game screen processing system, which is used to execute the cloud game screen processing method described in the embodiment of the present application, wherein the cloud game screen processing system includes a server and a client; the client is used to send the display refresh rate to the server; the server is used to determine the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate; the cloud game screen data is rendered and encoded based on the rendering frame rate of the server, and the encoded data is generated and transmitted to the client; the client is also used to receive the encoded data and decode and render the encoded data to obtain decoded data, and the rendering frame rate of the decoded data is less than or equal to the display refresh rate; the display refresh rate is adjusted to be consistent with the rendering frame rate of the decoded data.

[0092] Based on the same inventive concept, the embodiment of the present application discloses a cloud game screen processing device. Figure 5 A schematic diagram of the structure of a cloud game screen processing device proposed in an embodiment of the present application is shown. Figure 5 As shown, the device comprises:

[0093] A first sending module, used for the client to send the display refresh rate to the server;

[0094] A frame rate determination module, used for the server to determine the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate;

[0095] The encoding module is used for the server to render and encode the cloud game screen data based on the rendering frame rate of the server, generate encoded data and transmit it to the client;

[0096] A decoding module, used for the client to receive the encoded data and decode and render the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the refresh rate of the display;

[0097] An adaptive synchronization module is used for the client to adjust the refresh rate of the display to be consistent with the rendering frame rate of the decoded data.

[0098] In an optional implementation manner, the frame rate determination module includes:

[0099] A frame rate adjustment ratio submodule, used for the client to generate a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and send it to the server;

[0100] The frame rate determination submodule is used for the server to receive the frame rate adjustment ratio of the current second sent by the client, and to lower the display refresh rate based on the frame rate adjustment ratio of the current second to obtain the rendering frame rate of the server in the current second.

[0101] In an optional implementation manner, the frame rate determination submodule includes:

[0102] A delay difference ratio determination submodule is used for the client to obtain the delay difference ratio of all frames in the last second based on the decoded data obtained in the last second;

[0103] A first disturbance frame submodule, configured to determine a first disturbance frame in the previous second based on a delay difference ratio of all frames in the previous second, wherein the first disturbance frame is a frame whose delay difference ratio is greater than or equal to a first threshold, and the first threshold is equal to a frame rate adjustment ratio corresponding to the previous second;

[0104] The first frame rate adjustment submodule is used to increase the frame rate adjustment ratio corresponding to the previous second according to the first preset value to obtain the frame rate adjustment ratio of the current second when the ratio of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset ratio.

[0105] In an optional implementation manner, the frame rate determination submodule further includes:

[0106] A second disturbance frame submodule, configured to determine a second disturbance frame in the previous second based on the delay difference ratio of all frames in the previous second, wherein the second disturbance frame is a frame whose delay difference ratio is less than or equal to a second threshold, and the second threshold is a difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value;

[0107] The second frame rate adjustment submodule is used to reduce the frame rate adjustment ratio corresponding to the previous second according to the first preset value to obtain the frame rate adjustment ratio of the current second when the ratio of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to the second preset ratio.

[0108] In an optional implementation manner, the frame rate determination submodule further includes:

[0109] The third frame rate adjustment submodule is used to use the frame rate adjustment ratio corresponding to the previous second as the frame rate adjustment ratio of the current second when the ratio of the first disturbance frame in the previous second to all frames in the previous second is less than the first preset ratio and / or the ratio of the second disturbance frame in the previous second to all frames in the previous second is less than the second preset ratio.

[0110] In an optional implementation manner, the delay difference ratio determination submodule includes:

[0111] A delay unit, used for the client to obtain a delay of each frame in the decoded data;

[0112] An average delay unit, configured for the client to obtain delays corresponding to a preset number of frames before a target frame, and determine an average delay based on the delays corresponding to the preset number of frames, wherein the target frame is any frame in the decoded data;

[0113] A delay difference ratio unit is used for the client to determine the delay difference ratio of the target frame based on the delay of the target frame and the average delay.

[0114] In an optional implementation manner, the delay unit includes:

[0115] A first timestamp information subunit, configured to generate first timestamp information corresponding to each frame in the cloud game screen data and send the first timestamp information to the client when the server encodes the cloud game screen data;

[0116] A second timestamp information subunit, configured to enable the client to generate second timestamp information corresponding to each frame in the decoded data when the client decodes the encoded data;

[0117] The delay subunit is used for the client to use the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0118] Based on the same inventive concept, the embodiment of the present application discloses an electronic device, Figure 6 A schematic diagram of an electronic device proposed in an embodiment of the present application is shown, such as Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected to each other through a bus communication. The memory 110 stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set can be loaded and run on the processor 120 to implement the steps in the cloud game screen processing method disclosed in the embodiment of the present application.

[0119] Based on the same inventive concept, an embodiment of the present application discloses a computer-readable storage medium, on which is stored at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the steps in the cloud game screen processing method disclosed in the embodiment of the present application.

[0120] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0121] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0125] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0126] The above is a detailed introduction to a cloud game screen processing method, device, electronic device and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cloud game screen processing method, characterized in that: Applied to a system including a server and a client, the method includes: The client sends the display refresh rate to the server; The server determines a rendering frame rate of the server based on the display refresh rate, and an upper limit value of the rendering frame rate of the server is less than or equal to the display refresh rate; The server renders and encodes the cloud game screen data based on the rendering frame rate of the server, generates encoded data and transmits it to the client; The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the refresh rate of the display; The client adjusts the display refresh rate to be consistent with the rendering frame rate of the decoded data.

2. The cloud game screen processing method according to claim 1, characterized in that: The server determines the rendering frame rate of the server based on the display refresh rate, including: The client generates a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and sends the frame rate adjustment ratio to the server; The server receives the frame rate adjustment ratio of the current second sent by the client, and lowers the refresh rate of the display based on the frame rate adjustment ratio of the current second to obtain the rendering frame rate of the server in the current second.

3. The cloud game screen processing method according to claim 2, characterized in that: The client generates a frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including: The client obtains the delay difference ratio of all frames in the previous second based on the decoded data obtained in the previous second; Determine a first disturbance frame in the previous second based on the delay difference ratio of all frames in the previous second, wherein the first disturbance frame is a frame whose delay difference ratio is greater than or equal to a first threshold, and the first threshold is equal to the frame rate adjustment ratio corresponding to the previous second; When the ratio of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset ratio, the frame rate adjustment ratio corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment ratio of the current second.

4. The cloud game screen processing method according to claim 3, characterized in that: After obtaining the delay difference ratio of all frames in the last second, the method further includes: Based on the delay difference ratio of all frames in the previous second, determining a second disturbance frame in the previous second, the second disturbance frame being a frame whose delay difference ratio is less than or equal to a second threshold, and the second threshold being a difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value; When the ratio of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to the second preset ratio, the frame rate adjustment ratio corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment ratio of the current second.

5. The cloud game screen processing method according to claim 4, characterized in that: The method further comprises: When the ratio of the first disturbance frame in the previous second to all frames in the previous second is less than the first preset ratio, and / or the ratio of the second disturbance frame in the previous second to all frames in the previous second is less than the second preset ratio, the frame rate adjustment ratio corresponding to the previous second is used as the frame rate adjustment ratio of the current second.

6. The cloud game screen processing method according to claim 3, characterized in that: The delay difference ratio of each frame in the decoded data is obtained as follows: The client obtains a delay of each frame in the decoded data; The client obtains delays corresponding to a preset number of frames before a target frame, and determines an average delay based on the delays corresponding to the preset number of frames, wherein the target frame is any frame in the decoded data; The client determines the delay difference ratio of the target frame based on the delay of the target frame and the average delay.

7. The cloud game screen processing method according to claim 6, characterized in that: The client obtains the delay of each frame in the decoded data, including: When the server encodes the cloud game screen data, the server generates first timestamp information corresponding to each frame in the cloud game screen data and sends it to the client; When the client decodes the encoded data, the client generates second timestamp information corresponding to each frame in the decoded data; The client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

8. A cloud game screen processing device, characterized in that: The device comprises: A first sending module, used for the client to send the display refresh rate to the server; A frame rate determination module, used for the server to determine the rendering frame rate of the server based on the display refresh rate, and the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate; The encoding module is used for the server to render and encode the cloud game screen data based on the rendering frame rate of the server, generate encoded data and transmit it to the client; A decoding module, used for the client to receive the encoded data and decode and render the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the refresh rate of the display; An adaptive synchronization module is used for the client to adjust the refresh rate of the display to be consistent with the rendering frame rate of the decoded data.

9. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps in the cloud game screen processing method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by the processor, the steps in the cloud game screen processing method described in any one of claims 1-7 are implemented.

Images