Video stream optimization method and system for desktop cloud

Abstract

The invention relates to the technical field of cloud information, in particular to a video stream optimization method and system for desktop cloud. According to the video stream optimization method for the desktop cloud, a WebRTC retransmission feedback and congestion control mechanism is utilized, the performance and congestion condition of network connection are obtained by counting the packet loss rate and the packet transmission delay, the coding resolution and the frame rate are dynamically adjusted, a super-resolution algorithm and a frame generation algorithm are started at a client, and the video stream optimization efficiency is improved. Therefore, the resolution and the refresh rate preset by the user are kept unchanged. According to the video stream optimization method and system for the desktop cloud, the problems of jamming, frame dropping and black screen can be avoided, and the user experience of the desktop cloud in a weak network scene is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud information technology, and particularly relates to a method and system for optimizing video streams for a desktop cloud. Background Art

[0002] Desktop as a Service, i.e., cloud computer, is a service model based on cloud computing technology. Through virtualization technology, a user's operating system, data, application programs, etc. are hosted in the cloud and processed by a cloud server. The user accesses on demand through a network connection to achieve sharing of computing resources. In different scenarios, the desktop cloud can meet the needs of enterprise employees for remote work and mobile work, improving work efficiency; it can meet scenarios with high computing performance requirements such as visual design and film production, saving the cost of purchasing hardware; it can also provide a flexible teaching and learning environment for students and teachers.

[0003] Currently, limited by the jitter and congestion in the network connection performance between the desktop cloud client and the desktop cloud server, the transmission quality of the desktop cloud video stream is often difficult to guarantee, resulting in problems such as stuttering, frame dropping, and even black screens, seriously affecting the user experience in scenarios such as cloud games.

[0004] In view of the above problems, the present invention proposes a method and system for optimizing video streams for a desktop cloud. Summary of the Invention

[0005] In order to make up for the defects of the prior art, the present invention provides a simple and efficient method and system for optimizing video streams for a desktop cloud.

[0006] The present invention is implemented by the following technical solutions:

[0007] A method for optimizing a video stream for a desktop cloud uses the retransmission feedback and congestion control mechanisms of WebRTC, obtains the performance and congestion conditions of the network connection by statistically analyzing the packet loss rate and packet transmission delay, dynamically adjusts the encoding resolution and frame rate, and enables a super-resolution algorithm and a frame generation algorithm on the client side to ensure that the user's preset resolution and refresh rate remain unchanged;

[0008] It includes the following steps:

[0009] Step S1: When establishing a session, the desktop cloud server and the client negotiate to determine the client's preset target resolution and frame rate, and the support conditions of the video stream super-resolution algorithm and frame generation algorithm;

[0010] Step S2: Based on the WebRTC protocol, the desktop cloud server periodically statistically analyzes the packet loss rate and packet transmission delay of the connection with the client at a custom period.

[0011] For clients that do not support video stream super-resolution algorithms and frame generation algorithms, the video stream bitrate is only dynamically adjusted based on the packet loss rate and packet transmission delay.

[0012] For clients that only support video stream super-resolution algorithms, a video stream bitrate threshold X is pre-configured on the desktop cloud server, and dynamic resolution adjustment is enabled.

[0013] For clients that only support frame generation algorithms, a video stream bitrate threshold Y is pre-configured on the desktop cloud server, and dynamic frame rate adjustment is enabled.

[0014] For clients that support both video stream super-resolution algorithms and frame generation algorithms, a video stream bitrate threshold Z is pre-configured on the desktop cloud server, and both dynamic resolution and frame rate adjustments are enabled.

[0015] In the step S2, the video stream bitrate threshold X, the video stream bitrate threshold Y, and the video stream bitrate threshold Z are all calculated by the video quality assessment algorithm VQA.

[0016] In the step S2, for clients that only support video stream super-resolution algorithms, the desktop cloud server obtains the pre-configured video stream bitrate threshold X on the desktop cloud server based on the client's preset target resolution and frame rate obtained in the negotiation phase.

[0017] When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than the threshold X, the desktop cloud server never adjusts the video stream resolution.

[0018] When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted video stream target bitrate being lower than the threshold X, query the bitrate-resolution mapping table pre-configured on the desktop cloud server to obtain the resolution configuration applicable to the target bitrate, and dynamically adjust the resolution of the video stream based on this configuration.

[0019] The bitrate-resolution mapping table is calculated by the video quality assessment algorithm VQA.

[0020] In the step S2, for clients that only support video stream frame generation algorithms, the desktop cloud server obtains the pre-configured video stream bitrate threshold Y on the desktop cloud server based on the client's preset target resolution and frame rate obtained in the negotiation phase.

[0021] When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than the threshold Y, the desktop cloud server never adjusts the video stream frame rate.

[0022] When the packet loss rate and packet transmission delay deteriorate, resulting in the target bitrate of the dynamically adjusted video stream being lower than threshold Y, query the bitrate-frame rate mapping table pre-configured on the desktop cloud server to obtain the frame rate configuration applicable to the target bitrate, and dynamically adjust the frame rate of the video stream based on this configuration.

[0023] The bitrate-frame rate mapping table is calculated by the video quality assessment algorithm VQA.

[0024] In step S2, for a client that supports both the video stream super-resolution algorithm and the frame generation algorithm, the desktop cloud server obtains the video stream bitrate threshold Z pre-configured on the desktop cloud server based on the client's preset target resolution and frame rate obtained during the negotiation phase.

[0025] When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than threshold Z, the desktop cloud server never adjusts the video stream resolution and frame rate.

[0026] When the packet loss rate and packet transmission delay deteriorate, resulting in the target bitrate of the dynamically adjusted video stream being lower than threshold Z, query the bitrate-resolution mapping table and the bitrate-frame rate mapping table pre-configured on the desktop cloud server to obtain the resolution and frame rate configurations applicable to the target bitrate, and dynamically adjust the resolution and frame rate of the video stream based on this configuration.

[0027] Step S3: When the desktop cloud server adjusts the resolution and / or frame rate of the video stream, carry the original client preset target resolution and frame rate information in the packet of the adjusted video frame to instruct the client to enable the super-resolution and / or frame generation algorithm.

[0028] Step S4: When the client receives the adjusted video frame, use the super-resolution and / or frame generation algorithm to restore the video stream to the original preset target resolution and frame rate according to the preset target resolution and bitrate information.

[0029] A video stream optimization system for a desktop cloud, including a desktop cloud server and a client, is used to implement the above method.

[0030] A video stream optimization computing device for a desktop cloud, including:

[0031] One or more processors, one or more memories, and one or more programs, where one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the above methods.

[0032] A computer-readable storage medium storing one or more programs, the one or more programs including instructions which, when executed by a video stream optimization computing device for a desktop cloud, cause the video stream optimization computing device for the desktop cloud to execute any of the above methods.

[0033] The beneficial effects of the present invention are as follows: The video stream optimization method and system for a desktop cloud can dynamically adjust the encoding resolution and frame rate, and enable a super-resolution algorithm and a frame generation algorithm on the client side, ensuring that the preset resolution and refresh rate of the user remain unchanged, thereby enhancing the user experience of the desktop cloud in a weak network scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0035] Attached Figure 1 is a schematic diagram of the video stream optimization method for the desktop cloud of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] In order to enable those skilled in the art of the present technology to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0037] Desktop cloud relies on a stable network connection to transmit data and display images. When the network quality is poor, data transmission delay or loss may cause the user client to fail to receive the image data sent by the desktop cloud server in a timely manner, resulting in a lag phenomenon. A network connection interruption may cause the connection between the user client and the desktop cloud server to be lost, resulting in the interface of the client losing response and going black.

[0038] Currently, mainly through prediction algorithms to insert intermediate frames and super-resolution algorithms, reducing the transmission resolution to reduce the impact of such problems. However, the enabling and parameter adjustment of these algorithms are mainly through static configuration or triggered by detection of the desktop cloud client, lacking flexibility.

[0039] The video stream optimization method for desktop cloud utilizes the retransmission feedback and congestion control mechanisms of WebRTC to obtain the performance and congestion situation of the network connection by statistically calculating the packet loss rate and packet transmission delay, dynamically adjusts the encoding resolution and frame rate, and enables super-resolution algorithms and frame generation algorithms on the client side to ensure that the user's preset resolution and refresh rate remain unchanged;

[0040] It includes the following steps:

[0041] Step S1: When establishing a session, the desktop cloud server and the client negotiate to determine the client's preset target resolution and frame rate, and the support situation of the video stream super-resolution algorithm and frame generation algorithm;

[0042] Step S2: Based on the WebRTC protocol, the desktop cloud server periodically calculates the packet loss rate and packet transmission delay of the connection with the client at a custom period;

[0043] For clients that do not support the video stream super-resolution algorithm and frame generation algorithm, only the video stream bitrate is dynamically adjusted based on the packet loss rate and packet transmission delay situation;

[0044] For clients that only support the video stream super-resolution algorithm, a video stream bitrate threshold X is pre-configured on the desktop cloud server, and dynamic resolution adjustment is enabled;

[0045] For clients that only support the frame generation algorithm, a video stream bitrate threshold Y is pre-configured on the desktop cloud server, and dynamic frame rate adjustment is enabled;

[0046] For clients that support both the video stream super-resolution algorithm and the frame generation algorithm, a video stream bitrate threshold Z is pre-configured on the desktop cloud server, and both dynamic resolution and frame rate adjustments are enabled.

[0047] In step S2, the video stream bitrate threshold X, the video stream bitrate threshold Y, and the video stream bitrate threshold Z are all calculated by the video quality assessment algorithm VQA.

[0048] In step S2, for clients that only support the video stream super-resolution algorithm, the desktop cloud server obtains the pre-configured video stream bitrate threshold X on the desktop cloud server based on the client's preset target resolution and frame rate obtained in the negotiation phase;

[0049] When the target video stream bitrate dynamically adjusted based on the packet loss rate and packet transmission delay situation is higher than the threshold X, the desktop cloud server never adjusts the video stream resolution;

[0050] When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted target bitrate of the video stream being lower than threshold X, query the bitrate-resolution mapping table pre-configured on the desktop cloud server, obtain the resolution configuration applicable to the target bitrate, and dynamically adjust the resolution of the video stream based on this configuration.

[0051] The bitrate-resolution mapping table is calculated by the video quality assessment algorithm VQA.

[0052] In step S2, for a client that only supports the video stream frame generation algorithm, the desktop cloud server obtains the video stream bitrate threshold Y pre-configured on the desktop cloud server based on the client's preset target resolution and frame rate obtained during the negotiation phase;

[0053] When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than threshold Y, the desktop cloud server never adjusts the video stream frame rate;

[0054] When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted target bitrate of the video stream being lower than threshold Y, query the bitrate-frame rate mapping table pre-configured on the desktop cloud server, obtain the frame rate configuration applicable to the target bitrate, and dynamically adjust the frame rate of the video stream based on this configuration.

[0055] The bitrate-frame rate mapping table is calculated by the video quality assessment algorithm VQA.

[0056] In step S2, for a client that supports both the video stream super-resolution algorithm and the frame generation algorithm, the desktop cloud server obtains the video stream bitrate threshold Z pre-configured on the desktop cloud server based on the client's preset target resolution and frame rate obtained during the negotiation phase;

[0057] When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than threshold Z, the desktop cloud server never adjusts the video stream resolution and frame rate;

[0058] When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted target bitrate of the video stream being lower than threshold Z, query the bitrate-resolution mapping table and the bitrate-frame rate mapping table pre-configured on the desktop cloud server, obtain the resolution and frame rate configuration applicable to the target bitrate, and dynamically adjust the resolution and frame rate of the video stream based on this configuration.

[0059] Step S3, when the desktop cloud server adjusts the resolution and / or frame rate of the video stream, carry the original client preset target resolution and frame rate information in the packet of the adjusted video frame to instruct the client to enable the super-resolution and / or frame generation algorithm;

[0060] Step S4. When the client receives the adjusted video frame, according to the preset target resolution and bitrate information, use the super-resolution and / or frame generation algorithm to restore the video stream to the original preset target resolution and frame rate.

[0061] The video stream optimization system for the desktop cloud includes a desktop cloud server and a client, and is used to implement the above method.

[0062] The video stream optimization computing device for the desktop cloud includes:

[0063] One or more processors, one or more memories, and one or more programs, where the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors. The one or more programs include instructions for executing any of the above methods.

[0064] The computer-readable storage medium storing the one or more programs, the one or more programs including instructions that, when executed by the video stream optimization computing device for the desktop cloud, cause the video stream optimization computing device for the desktop cloud to execute any of the above methods.

[0065] The above embodiments are only one of the specific implementation manners of the present invention. The ordinary variations and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A video stream optimization method for desktop cloud, characterized in that: Utilize the retransmission feedback and congestion control mechanisms of WebRTC to obtain the performance and congestion status of the network connection by statistically calculating the packet loss rate and packet transmission delay, dynamically adjust the encoding resolution and frame rate, and enable super-resolution algorithms and frame generation algorithms on the client side to ensure that the user's preset resolution and refresh rate remain unchanged; It includes the following steps: Step S1: The desktop cloud server and the client negotiate during session establishment to determine the client's preset target resolution and frame rate, as well as the support for video stream super-resolution algorithms and frame generation algorithms; Step S2: The desktop cloud server, based on the WebRTC protocol, periodically calculates the packet loss rate and packet transmission delay of the connection with the client at a custom period; For clients that do not support video stream super-resolution algorithms and frame generation algorithms, only dynamically adjust the video stream bitrate based on the packet loss rate and packet transmission delay; For clients that only support video stream super-resolution algorithms, pre-configure a video stream bitrate threshold X on the desktop cloud server and enable dynamic resolution adjustment; For clients that only support frame generation algorithms, pre-configure a video stream bitrate threshold Y on the desktop cloud server and enable dynamic frame rate adjustment; For clients that support both video stream super-resolution algorithms and frame generation algorithms, pre-configure a video stream bitrate threshold Z on the desktop cloud server and enable both dynamic resolution and frame rate adjustment; Step S3: When the desktop cloud server adjusts the resolution and / or frame rate of the video stream, carry the original client preset target resolution and frame rate information in the packet of the adjusted video frame to instruct the client to enable the super-resolution and / or frame generation algorithm; Step S4: When the client receives the adjusted video frame, restore the video stream to the original preset target resolution and frame rate using the super-resolution and / or frame generation algorithm based on the preset target resolution and bitrate information.

2. The video stream optimization method for desktop cloud according to claim 1, characterized in that: In step S2, the video stream bitrate threshold X, the video stream bitrate threshold Y, and the video stream bitrate threshold Z are all calculated by the video quality assessment algorithm VQA.

3. The video stream optimization method for desktop cloud according to claim 2, wherein: In step S2, for clients that only support video stream super-resolution algorithms, the desktop cloud server obtains the pre-configured video stream bitrate threshold X on the desktop cloud server based on the client preset target resolution and frame rate obtained during the negotiation phase; When the target video stream bitrate dynamically adjusted based on the packet loss rate and packet transmission delay is higher than the threshold X, the desktop cloud server never adjusts the video stream resolution; When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted video stream target bitrate being lower than the threshold X, query the pre-configured bitrate-resolution mapping table on the desktop cloud server to obtain the resolution configuration applicable to the target bitrate, and dynamically adjust the video stream resolution based on this configuration.

4. The video stream optimization method for desktop cloud according to claim 3, characterized in that: The bitrate-resolution mapping table is calculated by the video quality assessment algorithm VQA.

5. The video stream optimization method for desktop cloud according to claim 2, wherein: In step S2, for clients that only support video stream frame generation algorithms, the desktop cloud server obtains the pre-configured video stream bitrate threshold Y on the desktop cloud server based on the client preset target resolution and frame rate obtained during the negotiation phase; When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than threshold Y, the desktop cloud server never adjusts the video stream frame rate. When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted target video stream bitrate being lower than threshold Y, query the bitrate-frame rate mapping table pre-configured on the desktop cloud server to obtain the frame rate configuration applicable to the target bitrate, and dynamically adjust the frame rate of the video stream based on this configuration.

6. The video stream optimization method for desktop cloud according to claim 5, characterized in that: The bitrate-frame rate mapping table is calculated by the video quality assessment algorithm VQA.

7. The video stream optimization method for a desktop cloud according to claim 2, characterized in that: In step S2, for a client that supports both the video stream super-resolution algorithm and the frame generation algorithm, the desktop cloud server obtains the video stream bitrate threshold Z pre-configured on the desktop cloud server based on the client's preset target resolution and frame rate obtained during the negotiation phase. When the target video stream bitrate dynamically adjusted according to the packet loss rate and packet transmission delay is higher than threshold Z, the desktop cloud server never adjusts the video stream resolution and frame rate. When the packet loss rate and packet transmission delay deteriorate, resulting in the dynamically adjusted target video stream bitrate being lower than threshold Z, query the bitrate-resolution mapping table and the bitrate-frame rate mapping table pre-configured on the desktop cloud server to obtain the resolution and frame rate configurations applicable to the target bitrate, and dynamically adjust the resolution and frame rate of the video stream based on this configuration.

8. A video stream optimization system for desktop cloud, characterized in that: It includes a desktop cloud server and a client, and is used to implement the method according to any one of claims 1 to 7.

9. A video stream optimization calculation device for a desktop cloud, characterized in that: It includes: One or more processors, one or more memories, and one or more programs, where one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors. The one or more programs include instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a video stream optimization computing device for a desktop cloud, cause the video stream optimization computing device for the desktop cloud to execute the method according to any one of claims 1 to 7.

Images