Cloud desktop streaming collaboration method and device, equipment, storage medium and program product

Abstract

The invention provides a cloud desktop streaming collaboration method and device, electronic equipment, a storage medium and a program product, the cloud desktop streaming collaboration method can be applied to a server of a cloud desktop, and the method comprises the following steps: determining a streaming frame rate for streaming transmission of a cloud desktop image to a client, and sending the streaming frame rate to the client, the client adjusts the sampling frequency of the desktop input event based on the streaming frame rate; the streaming frame rate represents the frame rate of streaming transmission of the cloud desktop image to the client by the server; and receiving a desktop input event uploaded by the client, and processing the desktop input event based on the sampling frequency of the desktop input event. According to the application, smoother and more efficient cloud desktop service can be provided, and the phenomenon of jamming of the cloud desktop in weak network scenes such as network jitter is solved.

Description

Technical Field

[0001] This application belongs to the field of cloud desktop technology, specifically relating to a cloud desktop streaming collaboration method, device, electronic device, storage medium, and program product. Background Technology

[0002] With the development and maturation of virtualization technology, cloud computing has been rapidly popularized and applied. As one of the cloud computing service models, cloud desktops enable users to quickly access their own desktop environment via the network using portable terminal devices, improving the flexibility and convenience of office work.

[0003] In existing technologies, users can access cloud desktops through client devices such as computers and mobile phones. The cloud desktop server can receive input commands from input devices such as keyboards and mice uploaded by the client, then parse and execute the input commands, and use the cloud desktop transmission protocol to transmit the image changes of the cloud desktop to the client in real time, thereby achieving the effect of operating a local desktop.

[0004] Since cloud desktop transmission protocols transmit data over a network, their transmission efficiency is highly dependent on the network environment. In weak network scenarios, such as those with network jitter, the jitter in the server's reception of user input events can easily lead to unsmooth input processing and stuttering. Therefore, ensuring the smooth and efficient operation of cloud desktops under weak network conditions is a technical problem that needs to be solved by those skilled in the art.

[0005] It should be noted that the above statements are only used to provide background information related to this application and do not necessarily constitute prior art. Summary of the Invention

[0006] This application proposes a cloud desktop streaming collaboration method, device, electronic device, storage medium, and program product, which can provide smoother and more efficient cloud desktop services and solve the lag problem of cloud desktops in weak network scenarios such as network jitter.

[0007] The first aspect of this application proposes a cloud desktop streaming collaboration method, applied to a cloud desktop server, the method comprising:

[0008] The server determines the streaming frame rate for streaming cloud desktop images to the client and sends the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client.

[0009] The system receives the desktop input event and the sampling frequency uploaded by the client, and processes the desktop input event based on the sampling frequency of the desktop input event.

[0010] A second aspect of this application proposes another cloud desktop streaming collaboration method, applied to a cloud desktop client, the method comprising:

[0011] The server receives the streaming frame rate sent by the cloud desktop; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client.

[0012] The sampling frequency of desktop input events is determined based on the streaming frame rate;

[0013] The desktop input events are collected based on the sampling frequency and uploaded to the server so that the server processes the desktop input events based on the sampling frequency of the desktop input events.

[0014] A third aspect of this application proposes a cloud desktop streaming collaboration system, the system comprising a server and a client;

[0015] The server is used to provide cloud desktop services, determine the streaming frame rate for streaming cloud desktop images to the client, and send the streaming frame rate to the client; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client.

[0016] The client is used to display the cloud desktop, determine the sampling frequency of desktop input events based on the streaming frame rate, collect the desktop input events based on the sampling frequency, and upload the sampling frequency and the desktop input events to the server.

[0017] The server also processes the desktop input events based on the sampling frequency.

[0018] A fourth aspect of this application provides a cloud desktop streaming collaboration device, applied to a server providing cloud desktop services; the device includes:

[0019] A streaming frame rate determination module is used to determine the streaming frame rate for streaming cloud desktop images to the client; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client.

[0020] A streaming frame rate sending module is used to send the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate;

[0021] An input event processing module is used to receive the desktop input event and the sampling frequency uploaded by the client, and to process the desktop input event based on the sampling frequency.

[0022] The fifth aspect of this application proposes another cloud desktop streaming collaboration device, applied to a client for displaying a cloud desktop, the device comprising:

[0023] A streaming frame rate receiving module is used to receive the streaming frame rate sent by the cloud desktop server; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client.

[0024] A sampling frequency determination module is used to determine the sampling frequency of desktop input events based on the streaming frame rate;

[0025] The input event acquisition module is used to acquire the desktop input events based on the sampling frequency and upload them to the server so that the server can process the desktop input events based on the sampling frequency.

[0026] An embodiment of the sixth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method as described in the first or second aspect.

[0027] An embodiment of the seventh aspect of this application provides a computer-readable storage medium having a computer program stored thereon that is executed by a processor to implement the method as described in the first or second aspect.

[0028] An embodiment of the eighth aspect of this application provides a computer program product including a computer program that is executed by a processor to implement the method described in the first or second aspect.

[0029] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0030] In this embodiment, the cloud desktop server can send the currently determined streaming frame rate to the client, enabling the client to adjust the sampling frequency of desktop input events based on the streaming frame rate. It can also receive desktop input events and sampling frequencies uploaded by the client and process the desktop input events based on the sampling frequency. Thus, by sending the streaming frame rate adopted by the server to the client, allowing the client to adjust the sampling frequency of desktop input events accordingly, and then sending the sampling frequency to the server, the server can process the desktop input events based on this sampling frequency. This achieves a streaming collaboration process between the cloud desktop server and client, enabling the server to provide the client with a smoother and more efficient cloud desktop service.

[0031] Furthermore, given that the streaming frame rate can be dynamically adjusted in real time according to the actual situation, this method can be applied to weak network scenarios. After the server determines the streaming frame rate based on the weak network scenario, the client can adjust the sampling frequency of desktop input events in real time according to the streaming frame rate under the weak network scenario. Then, the server can process the desktop input events according to the adjusted sampling frequency to prevent the desktop input events from being stuttered due to network factors because the client still samples frequently at a high sampling frequency in the weak network scenario, and to prevent the client from further stuttering because the server processes the desktop input events at a high frequency and sends the response image data of the desktop input events to the client at a high frequency.

[0032] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0034] In the attached diagram:

[0035] Figure 1 The following are schematic diagrams illustrating application scenarios of the cloud desktop streaming collaboration method provided in some embodiments of this application;

[0036] Figure 2 The following is a flowchart illustrating a cloud desktop streaming collaboration method applied to a server, provided in some embodiments of this application.

[0037] Figure 3 This application illustrates a schematic diagram of the specific processing flow for desktop input events in some embodiments;

[0038] Figure 4 The following is a schematic diagram illustrating the specific process of the cloud desktop streaming collaboration method provided in some embodiments of this application;

[0039] Figure 5 The following is a flowchart illustrating a cloud desktop streaming collaboration method for a client provided in other embodiments of this application;

[0040] Figure 6 The diagram illustrates the principle and flow of client-side desktop input event collection in some embodiments of this application;

[0041] Figure 7This application illustrates a schematic diagram of the specific processing flow of the client updating desktop image data in some embodiments;

[0042] Figure 8 The diagram shows a schematic representation of the cloud desktop streaming collaboration system provided in some embodiments of this application.

[0043] Figure 9 The present application provides a schematic diagram of the structure of a cloud desktop streaming collaboration device applied to a server, according to some embodiments of this application.

[0044] Figure 10 This application provides a schematic diagram of the structure of a cloud desktop streaming collaboration device for client applications, according to some embodiments of the present application.

[0045] Figure 11 A schematic diagram of an electronic device provided according to an embodiment of this application is shown;

[0046] Figure 12 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0047] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0048] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. All data involved in this application (including but not limited to data used for model training, stored data, and displayed data) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose whether to authorize or refuse authorization.

[0049] First, some of the nouns or terms appearing in the embodiments of this application can be interpreted as follows:

[0050] Cloud desktop service: This is a service that uses cloud computing to place computing and storage in the cloud via transmission protocols, providing virtual desktops. It has advantages such as centralized management, security and reliability, cost savings, and energy efficiency. Users can overcome limitations in time, location, terminal, and application, and access cloud desktops for office work anytime, anywhere.

[0051] The cloud desktop server: This refers to the cloud server that provides the aforementioned cloud desktop services. It may include a data center and / or server cluster located in the cloud, providing the core computing and storage functions of the cloud desktop, as well as network resources. Specifically, it can implement services such as virtualization management, resource allocation, remote access, and data backup and recovery.

[0052] A cloud desktop client refers to the device or software application that a user uses to access cloud desktop services. It can be any type of device, as long as it can connect to the cloud desktop server via a network and has the corresponding client software or browser to access and use the cloud desktop.

[0053] Streaming Protocol (ASP): A communication protocol for cloud desktop services that allows users to access desktop environments or applications running on remote servers in real time over a network, enabling real-time interaction of graphical command streams, image streams, and audio / video streams, thereby providing a smooth and efficient user experience.

[0054] Streaming frame rate, also known as dynamic frame rate or adaptive frame rate, refers to the frame rate at which the cloud desktop image is captured by the server based on the current network capabilities. It is also the frame rate at which the server streams cloud desktop images to the client; in other words, the number of cloud desktop image frames captured and streamed to the client per unit of time. This streaming frame rate can be dynamically adjusted based on the network bandwidth between the server and client, and the client's processing power. For example, when poor network conditions or limited device resources are detected, the streaming frame rate can be reduced to decrease data transmission and alleviate device load, thereby avoiding buffering and stuttering, and maintaining smooth playback. Conversely, if there is a high-speed network connection and high-performance devices between the server and client, a higher streaming frame rate can be used to present a smoother, higher-quality cloud desktop service.

[0055] Streaming collaboration: In this embodiment, it refers to the process in which the cloud desktop server and client work together by using a streaming protocol to transmit the image changes of the cloud desktop to the client in real time, and the client transmitting user input events to the server. This is based on the streaming frame rate provided by the server and the acquisition frequency of the input events collected by the client.

[0056] Desktop input events refer to events triggered by users interacting with various input devices (including but not limited to mouse, keyboard, gamepad, touchpad, etc.) when using a cloud desktop. These events can generate interactive commands in the cloud desktop environment, which are collected by the client and uploaded to the server.

[0057] It is understood that, unless otherwise specified, the images appearing in this embodiment refer to the desktop image content of the cloud desktop.

[0058] In related technologies, streaming protocols such as RDP (Remote Desktop Protocol) and Citrix HDX (High Definition Experience) can be used to implement cloud desktop services. In these services, the server encodes and compresses the image update data of the cloud desktop and then transmits it to the client over the network. The client can transmit user input (such as keyboard and mouse operations) to the server for mirroring. However, these cloud desktop services do not synchronize input event processing with the image streaming frame rate. In weak network scenarios such as those with network jitter, the jitter in the server's reception of user input events can easily lead to unsmooth input event processing, resulting in desktop lag on the client side. Furthermore, under limited network conditions, excessive image updates caused by excessive input event processing can further exacerbate desktop lag.

[0059] To address the aforementioned issues, this application proposes a cloud desktop streaming collaboration method, apparatus, electronic device, storage medium, and program product. In this method, the cloud desktop server dynamically adjusts the streaming frame rate of the cloud desktop image based on actual network and device conditions, and sends the latest determined streaming frame rate to the client, enabling the client to adjust the sampling frequency of desktop input events based on the streaming frame rate. The server also receives desktop input events and sampling frequencies uploaded by the client and processes the desktop input events based on the sampling frequency. Thus, by sending the streaming frame rate adopted by the server to the client, allowing the client to adjust the sampling frequency of desktop input events accordingly, and then sending the sampling frequency back to the server, the server can further process desktop input events based on this sampling frequency. This achieves a streaming collaboration process between the cloud desktop server and client, enabling the server to provide the client with a smoother and more efficient cloud desktop service. Furthermore, given that the streaming frame rate is an image frame rate that can be adjusted in real time according to the actual situation, this method can be applied to weak network scenarios. After the server determines the streaming frame rate based on the weak network scenario, the client can adjust the sampling frequency of desktop input events in real time according to the streaming frame rate under the weak network scenario. Then, the server can process the desktop input events according to the adjusted sampling frequency. This can prevent the desktop input events from being stuttered due to network factors because the client still samples frequently at a high sampling frequency in weak network scenarios, and prevent the server from processing desktop input events at a high frequency and sending the response image data of desktop input events to the client at a high frequency, causing further stuttering on the client side.

[0060] This desktop fluidization collaboration method can be applied to, for example... Figure 1 The desktop streaming collaboration system shown may include a server for providing cloud desktop services and a client for displaying the cloud desktop. The server may be any cloud service platform capable of performing the aforementioned streaming frame rate determination, cloud desktop image acquisition, and desktop input event processing. The client may be any client device or application capable of performing the aforementioned desktop input event acquisition and uploading functions.

[0061] Figure 1 The diagram illustrates an application scenario involving a server and a client. It can be understood that the server and client can interact in the following ways: Figure 1 The one-to-many relationship shown can also be a one-to-one or many-to-one relationship; this embodiment does not specifically limit it. That is, a server can provide cloud desktop services to one or more clients, and a client can also configure and connect to one or more servers to enable working and access in different cloud environments.

[0062] The aforementioned electronic device can be any device capable of executing the aforementioned cloud desktop streaming collaboration method. Specifically, it can be a server-side device, such as a device constituting the server side of the aforementioned cloud desktop, or a client-side device, such as a device constituting the client side of the aforementioned cloud desktop, such as a display. The aforementioned apparatus can be a module on the electronic device specifically used to execute the cloud desktop streaming collaboration method; this embodiment does not specifically limit its specific structure. The aforementioned computer-readable storage medium stores a computer program capable of implementing the cloud desktop streaming collaboration method; it can be any computer-readable storage medium capable of storing a computer program; this embodiment does not specifically limit its specific structure. The aforementioned computer program product can be any product including the aforementioned computer program; this embodiment does not specifically limit its specific implementation form and structure.

[0063] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0064] Example 1

[0065] Please see Figure 2 This is a flowchart illustrating the cloud desktop streaming collaboration method applied to the server side provided in this embodiment. Figure 2 As shown, the cloud desktop streaming collaboration method may include the following steps:

[0066] Step 101: Determine the streaming frame rate for streaming cloud desktop images to the client and send the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate.

[0067] In this context, streaming cloud desktop images can be understood as the process of transmitting cloud desktop images using the streaming protocol described above in this embodiment. The streaming frame rate is the number of frames of cloud desktop image data transmitted per unit of time. When the client can receive the cloud desktop image data in a timely manner, a higher streaming frame rate results in a smoother desktop presentation on the client. If the streaming frame rate is too low, the client cannot receive complete cloud desktop image data in a timely manner, leading to desktop lag on the client side. Conversely, if the streaming frame rate is too high, the client may not be able to receive cloud desktop image data in a timely manner, potentially causing network congestion and preventing the client from uploading desktop input events to the server in a timely manner, also resulting in desktop lag on the client side. The sampling frequency of desktop input events refers to the number of times the client collects desktop input events per unit of time.

[0068] Specifically, when determining the streaming frame rate for streaming cloud desktop images to the client, the network parameters used for communication between the server and the cloud desktop client, as well as the client's image processing performance, can be obtained first. Then, based on the network parameters and image processing performance, the streaming frame rate for streaming cloud desktop images to the client can be determined.

[0069] Among them, the client's image processing performance is used to characterize the client device's ability to process cloud desktop images, which can also be called the client's display rendering capability. Specifically, it can be determined based on the client device's image decoding capability and image display capability.

[0070] In this embodiment, the streaming frame rate of the cloud desktop image can be adaptively adjusted in real time based on the network parameters of the transmission network between the server and the client and the image processing performance of the client, so as to adapt to changes in network status and client device performance and achieve a smoother cloud desktop service.

[0071] It is understood that the above-described scheme of dynamically adjusting the streaming frame rate based on network parameters and the image processing performance of the client device is only one implementation method of this embodiment. This embodiment is not limited to this. For example, a streaming frame rate that is more suitable for most clients can also be determined based on the network parameters of the network transmission device to reduce the amount of data processing and improve the efficiency of cloud desktop services.

[0072] In some optional embodiments, the process of obtaining the client's image processing performance may include the following steps: obtaining the client's image decoding time data and image display time data; and calculating the client's image processing performance based on the image decoding time data and image display time data.

[0073] The image decoding time data may include the number of image frames decoded per unit time, or the time taken to decode a preset number of image frames. The image display time data may include the number of image frames displayed per unit time, or the time taken to display a preset number of image frames.

[0074] The image decoding time data and image display time data mentioned above can be transmitted from the client to the server via a network transmission device. When the data changes, the client can report it to the server in a timely manner so that the server can calculate a more suitable streaming frame rate to provide the client with a smoother and higher quality cloud desktop service.

[0075] This embodiment can effectively reflect the client's image processing performance by using image decoding time data and image display time data, thereby enabling the calculation of a more accurate streaming frame rate.

[0076] It should be noted that the image processing performance of this client can be as shown above, where the client uploads image decoding time data and image display time data, and the server calculates the image processing performance based on the data uploaded by the client. Alternatively, the client can directly calculate the image processing performance and then upload the calculation result to the server. The specific processing procedure can be executed according to the actual situation, and this embodiment does not impose specific limitations on it.

[0077] Specifically, the image processing performance of the client can be calculated using the following formula (1). In formula (1), f represents the display frame rate of the client, that is, the number of display frames per unit time; t1 represents the image decoding time in milliseconds; t2 represents the image display time in milliseconds; and 1000 in the formula represents 1000 milliseconds, that is, 1 second per unit time.

[0078] f = 1000 / (t1+ t2) Formula (1)

[0079] In some alternative embodiments, the process of determining the streaming frame rate of cloud desktop images to the client based on network parameters and image processing performance may include the following steps: determining network transmission performance based on network latency, network bandwidth, and packet loss rate in the network parameters; and determining the streaming frame rate of cloud desktop images to the client based on the network transmission performance and image processing performance.

[0080] Among them, network transmission performance is used to characterize the performance of cloud desktop image data transmitted via network streaming. The higher the value of network transmission performance, the higher the performance of cloud desktop image data transmitted via network streaming. Conversely, the lower the value of network transmission performance, the lower the performance of cloud desktop image data transmitted via network streaming.

[0081] In this embodiment, before determining the streaming frame rate for transmitting cloud desktop images to the client, the network transmission performance is first determined based on network latency, network bandwidth, and packet loss rate in the network parameters. This allows for more accurate real-time determination of the network transmission performance. Then, based on this network transmission performance and image processing performance, the streaming frame rate for transmitting cloud desktop images to the client is determined, resulting in an even more accurate streaming frame rate.

[0082] Specifically, when determining network transmission performance based on network latency, bandwidth, and packet loss rate, one can first determine the weighting coefficients corresponding to each parameter. Then, the network transmission performance is determined by multiplying these parameters by their respective weighting coefficients. Using these three indicators—latency, bandwidth, and packet loss rate—in this way allows for a more accurate calculation of network transmission performance.

[0083] In some alternative embodiments, the step of determining the streaming frame rate for streaming cloud desktop images to the client based on network transmission performance and image processing performance may include the following specific processing steps: determining a first linear parameter; and determining the streaming frame rate for streaming cloud desktop images to the client based on the product of network transmission performance, image processing performance, and the first linear parameter.

[0084] The first linear parameter is used to characterize the linear relationship between the product of network transmission performance and image processing performance and the streaming frame rate.

[0085] In this embodiment, a linear relationship is established between the product of network transmission performance and image processing performance and the streaming frame rate. Based on this linear relationship and the network transmission performance and image processing performance, the streaming frame rate can be quickly calculated, thereby enabling real-time and rapid adjustment of the streaming frame rate and improving the processing efficiency of the cloud desktop streaming collaboration method.

[0086] Specifically, the streaming frame rate can be calculated using the following formula (2).

[0087] F = (a1 * L + a 2 * B + a 3 * D) * k * f Formula (2)

[0088] Where F represents the streaming frame rate, f represents the client's image processing capability, L represents network latency, B represents network bandwidth, and D represents packet loss rate. The calculation result of (a1*L+a2*B+a3*D) is the network transmission performance. a1 to a3 represent weighting coefficients, all of which are greater than 0 and less than 1, and a1+a2+a3=1. k can be understood as the streaming frame rate F having a linear relationship with network transmission performance and the client's image processing capability. The specific value of k can be set according to the actual usage scenario. For example, if the user wants a smoother experience, this value can be increased, similar to adjusting the sensitivity of a mouse.

[0089] It can be understood that in the above formula (2), L represents the positive data of network latency, B represents the positive data of network bandwidth, and D represents the positive data of packet loss rate. That is, the higher the network latency, the lower the value of L; the larger the value of network bandwidth, the higher the value of B; and the larger the value of packet loss rate, the lower the value of D.

[0090] Step 102: Receive desktop input events and sampling frequency uploaded by the client, and process the desktop input events based on the sampling frequency.

[0091] Desktop input events include, but are not limited to, events triggered by operations using devices such as a mouse, keyboard, gamepad, and touchpad. The sampling frequency can be understood as the frequency at which the client collects desktop input events.

[0092] In some optional embodiments, the steps of processing desktop input events based on sampling frequency described above may include the following steps: determining the sampling interval for the client to collect desktop input events based on the sampling frequency; and processing the desktop input events based on the sampling interval and the transmission duration of the desktop input events.

[0093] After determining the sampling interval for desktop input events collected by the client, this embodiment can process the desktop input events based on the sampling interval and the transmission time of the desktop input events. This is to prevent the server from transmitting a large amount of desktop image data to the client before the client has finished collecting the desktop input events, thus preventing the desktop data from being displayed and rendered in time, which could cause network congestion at the client and result in cloud desktop lag.

[0094] Specifically, the process of processing desktop input events based on the sampling interval duration and the transmission duration of desktop input events can include the following: when the transmission duration is greater than or equal to the sampling interval duration, the input processing interface is directly called to process the desktop input events; when the transmission duration is less than the sampling interval duration, the difference between the transmission duration and the sampling interval duration is determined, and after the difference duration, the input processing interface is called to process the desktop input events.

[0095] In this embodiment, as Figure 3 As shown, when the transmission duration T1 of a desktop input event is greater than or equal to the sampling interval T2, the input processing interface can be directly called to process the desktop input event. When the transmission duration T1 of a desktop input event is less than or equal to the sampling interval T2, a delay of (T2-T1) can be made before calling the input processing interface to process the desktop input event. This is to prevent the server from transmitting a large amount of desktop image data to the client, which the client may not have enough time to display and render, thus causing network congestion at the client and resulting in cloud desktop lag.

[0096] It should be noted that the input processing interface is determined by the cloud desktop operating system and is not specifically limited. For example, for the Windows operating system, this embodiment can use the Windows API (Application Programming Interface) or the input filter driver as the input processing interface.

[0097] In practical applications, server-side processing of desktop input events can also include the process of collecting, compressing, encoding, and sending update data of cloud desktop images.

[0098] Specifically, the steps of calling the input processing interface to process desktop input events may include the following: acquiring desktop image update data corresponding to the desktop input event based on the streaming frame rate; determining the target scene type corresponding to the desktop image update data; and compressing and encoding the desktop image update data using the corresponding compression encoding method based on the target scene type.

[0099] The scene types include static scenes and dynamic scenes. Static scenes can include static text images and natural images, where natural images are static images without text. Dynamic scenes can include multiple consecutive video frames. Identification can be performed using neural networks, deep learning, etc. For example, it can be determined by judging whether the updated desktop image contains frequently refreshed areas; if frequently refreshed areas exist, it is identified as a dynamic scene, otherwise it is identified as a static scene.

[0100] In this embodiment, after determining the streaming frame rate, the server can collect desktop image update data corresponding to desktop input events based on the streaming frame rate to adapt to the current network status and client device performance. Then, it can perform corresponding processing on the desktop image update data for different scenario types to improve the processing speed of desktop image update data, thereby improving the streaming transmission efficiency of cloud desktop images.

[0101] In actual processing, such as Figure 4As shown, the server can calculate and update the streaming frame rate based on the network status and the client's image processing performance. After the cloud desktop image is updated, the server can collect images of the desktop image update data according to the currently calculated streaming frame rate, and then perform scene recognition on the currently processed desktop image update data. After that, the server can use the corresponding compression encoding method according to the specific scene type identified.

[0102] Specifically, if the desktop image update data being processed is a static scene, content recognition can be performed first to determine whether the desktop image update data contains text images or natural images. Further, if it is a text image, lossless compression algorithms can be used to compress the desktop image update data losslessly. For example, LZ4 (Extremely Fast Compression) and RLE (Run-Length Encoding) algorithms can be used. If it is a natural image, lossy compression algorithms can be used to compress the desktop image update data lossily. For example, JPEG (Joint Photographic Experts Group), MPEG (Moving Picture Experts Group), and AAC (Advanced Audio Coding) algorithms can be used.

[0103] If the desktop image update data being processed is a dynamic scene, a video encoder can be used directly to compress the desktop image update data. The video encoder can, but is not limited to, use video compression technologies such as H.264, H.265, and AV1.

[0104] The cloud desktop streaming collaboration method provided in this embodiment, after determining the streaming frame rate, can send the latest determined streaming frame rate to the client, enabling the client to adjust the sampling frequency of desktop input events based on the streaming frame rate. It also receives desktop input events and sampling frequencies uploaded by the client, and processes the desktop input events based on the sampling frequency. In this way, by sending the streaming frame rate to the client, allowing the client to adjust the sampling frequency of desktop input events and process them accordingly, a streaming collaboration process between the cloud desktop server and client is realized, enabling the server to provide the client with a smoother and more efficient cloud desktop service. Furthermore, given that the streaming frame rate is an image frame rate that can be adjusted in real time according to the actual situation, this method can be applied to weak network scenarios. After the server determines the streaming frame rate based on the weak network scenario, the client can adjust the sampling frequency of desktop input events in real time according to the streaming frame rate under the weak network scenario. Then, the server can process the desktop input events according to the adjusted sampling frequency. This can prevent the desktop input events from being interrupted due to network factors because the client still samples frequently at a high sampling frequency in weak network scenarios, and prevent the client from experiencing further interruptions due to network congestion because the server processes the desktop input events at a high frequency and sends the response image data of the desktop input events to the client at a high frequency.

[0105] Example 2

[0106] Please refer to Figure 5 This embodiment provides a cloud desktop streaming collaboration method applied to a client, such as... Figure 5 As shown, the method may include the following steps:

[0107] Step 201: Receive the streaming frame rate sent by the cloud desktop server;

[0108] Step 202: Determine the sampling frequency of desktop input events based on the streaming frame rate;

[0109] Step 203: Collect desktop input events based on the sampling frequency and upload them to the server so that the server can process the desktop input events based on the sampling frequency.

[0110] Among them, the streaming frame rate is determined based on network parameters and image processing performance, and represents the frame rate at which the server streams cloud desktop images to the client.

[0111] In some optional embodiments, step 202 above may include the following specific processing procedures: obtaining a second linear parameter; determining the sampling frequency of desktop input events based on the product of the streaming frame rate and the second linear parameter.

[0112] The second linear parameter characterizes the linear relationship between the streaming frame rate and the sampling frequency of desktop input events. This second linear parameter can be a value greater than or equal to 1, and its specific value can be determined according to the actual situation. This embodiment does not specifically limit the specific value of the second linear parameter. For example, it can be set according to the current network speed and stability, as well as the image processing performance of the client device. Specifically, if the image processing performance of the client device allows, the second linear parameter can be set to a larger value to achieve a smoother cloud desktop effect. If the network is unstable and / or the image processing performance of the client device is poor, the second linear parameter can be set to a smaller value to prevent network congestion.

[0113] In this embodiment, a linear relationship is established between the product of network transmission performance and image processing performance and the streaming frame rate. Based on this linear relationship and the network transmission performance and image processing performance, the streaming frame rate can be quickly calculated, thereby enabling real-time and rapid adjustment of the streaming frame rate and improving the processing efficiency of the cloud desktop streaming collaboration method.

[0114] Specifically, the client's sampling frequency can be calculated using the following formula (3). In formula (3), R represents the sampling frequency, F represents the streaming frame rate, and K represents a constant coefficient, K>=1, which can be dynamically adjusted according to the user's actual needs.

[0115] R = K * F Formula (3)

[0116] In addition, in this embodiment, after the client collects desktop input events based on the sampling frequency and uploads them to the server, it can also receive desktop image update data (which may be compressed data) sent by the server, then decode the desktop image update data sent by the server, and call the local display module to render and display it.

[0117] In some alternative embodiments, combined with Figure 6 and Figure 7 The process of implementing this cloud desktop streaming collaboration method on the client side is explained in detail below:

[0118] like Figure 6 As shown, when the client's input device generates a desktop input event, the client can calculate the sampling frequency according to the above formula (3) based on the received streaming frame rate, collect desktop input events according to the calculated sampling frequency, and then send the sampling frequency and the collected desktop input events to the server.

[0119] After that, as Figure 7As shown, the client can receive desktop image update data sent by the server, decode the received desktop image update data, and then render and display the decoded desktop image update data.

[0120] The client can also re-evaluate the display rendering capability according to the above formula (1) after the display rendering is completed, and synchronize the newly evaluated display rendering capability to the server so that the server can dynamically adjust the streaming frame rate according to the newly evaluated display rendering capability.

[0121] In this embodiment, after receiving the streaming frame rate sent by the server, the client can adjust the sampling frequency of desktop input events according to the streaming frame rate, collect desktop input events based on the sampling frequency, and then send the collected desktop input events and sampling frequency to the server so that the server can process the desktop input events according to the sampling frequency. This realizes the streaming collaboration process between the cloud desktop server and the client, enabling the server to provide the client with a smoother and more efficient cloud desktop service. Furthermore, given that the streaming frame rate is an image frame rate that can be adjusted in real time according to the actual situation, this method can be applied to weak network scenarios. After the server determines the streaming frame rate based on the weak network scenario, the client can adjust the sampling frequency of desktop input events in real time according to the streaming frame rate under the weak network scenario. Then, the server can process the desktop input events according to the adjusted sampling frequency. This can prevent the desktop input events from being interrupted due to network factors because the client still samples frequently at a high sampling frequency in weak network scenarios, and prevent the client from experiencing further interruptions due to network congestion because the server processes the desktop input events at a high frequency and sends the response image data of the desktop input events to the client at a high frequency.

[0122] Example 3

[0123] Please refer to Figure 8 This embodiment provides a cloud desktop streaming collaboration system, which includes a server 10 and a client 20. The server 10 provides cloud desktop services, determines the streaming frame rate for streaming cloud desktop images to the client 20, and sends the streaming frame rate to the client 20. The client 20 displays the cloud desktop, determines the sampling frequency of desktop input events based on the streaming frame rate, collects desktop input events based on the sampling frequency, and uploads the sampling frequency and desktop input events to the server 10. The server 10 also processes the desktop input events based on the sampling frequency.

[0124] The cloud desktop streaming collaboration system provided in this embodiment first determines the streaming frame rate in real time through the server 10, and then sends the latest determined streaming frame rate to the client 20. After receiving the streaming frame rate sent by the server 10, the client 20 can adjust the sampling frequency of desktop input events according to the streaming frame rate, and collect desktop input events based on the sampling frequency. Then, it sends the collected desktop input events and sampling frequency to the server 10, which can process the desktop input events according to the sampling frequency. In this way, based on the collaborative processing of streaming frame rate and sampling frequency, the streaming collaboration process between the cloud desktop server 10 and the client 20 is realized, enabling the server 10 to provide the client 20 with a smoother and more efficient cloud desktop service. Furthermore, given that the streaming frame rate is an image frame rate that can be adjusted in real time according to the actual situation, this method can be applied to weak network scenarios. After the server 10 determines the streaming frame rate based on the weak network scenario, the client 20 can adjust the sampling frequency of desktop input events in real time according to the streaming frame rate under the weak network scenario. Then, the server 10 can process the desktop input events according to the adjusted sampling frequency. This can prevent the desktop input events from being interrupted due to network factors because the client 20 still samples frequently at a high sampling frequency in weak network scenarios, and prevent the client 20 from being further interrupted due to network congestion because the server 10 processes the desktop input events at a high frequency and sends the response image data of the desktop input events to the client 20 at a high frequency.

[0125] The server 10 may include an input processing device 11 and an image acquisition device 12. The input processing device 11 can receive desktop input events and sampling frequencies sent by the client 20, and perform reconstruction processing on the received desktop input events based on the sampling frequency. The image acquisition device 12 can calculate the image streaming frame rate according to the current network capabilities, and acquire and compress desktop image update data according to the streaming frame rate, and send the compressed and encoded desktop image update data to the client 20.

[0126] The client 20 may include an input acquisition device 21 and a display rendering device 22. The input acquisition device 21 can calculate the sampling frequency based on the received streaming frame rate, and acquire desktop input events from the local input device based on the sampling frequency. Then, the acquired desktop input events are transmitted to the server 10 via the network, so that the server 10 can process the desktop input events. The input device includes, but is not limited to, a mouse, keyboard, gamepad, touchpad, etc.

[0127] Specifically, when the image acquisition device 12 of the server 10 calculates the image streaming frame rate according to the current network capabilities, it can first obtain the network parameters for communication between the server 10 and the client 20, as well as the image processing performance of the display rendering device 22; then, based on the network parameters and the image processing performance of the display rendering device 22, it can calculate the streaming frame rate.

[0128] The image processing performance of the display rendering device 22 is used to characterize the ability of the display rendering device 22 to process cloud desktop images, which can also be called the display rendering capability. Specifically, it can be determined based on the image decoding capability and image display capability of the display rendering device 22.

[0129] In this embodiment, the image acquisition device 12 can adaptively adjust the streaming frame rate of the cloud desktop image in real time based on the network parameters of the transmission network between the server 10 and the client 20 and the image processing performance of the display rendering device 22, so as to achieve a smoother cloud desktop service.

[0130] In some optional embodiments, the process by which the image acquisition device 12 obtains the image processing performance of the display rendering device 22 may include the following processes: obtaining image decoding time data and image display time data of the display rendering device 22; and calculating the image processing performance of the display rendering device 22 based on the image decoding time data and image display time data.

[0131] The image decoding time data may include the number of image frames decoded per unit time, or the time taken to decode a preset number of image frames. The image display time data may include the number of image frames displayed per unit time, or the time taken to display a preset number of image frames.

[0132] The image decoding time data and image display time data mentioned above can be transmitted from the display rendering device 22 to the image acquisition device 12 via the network. When the data changes, the display rendering device 22 can report to the image acquisition device 12 in a timely manner so that the image acquisition device 12 can calculate a more suitable streaming frame rate to provide users with a smoother and higher quality cloud desktop service.

[0133] Specifically, the image acquisition device 12 can use the following formula (1) to calculate the image processing performance of the display rendering device 22. In formula (1), f represents the display frame rate of the display rendering device 22, that is, the number of display frames per unit time; t1 represents the image decoding time in milliseconds; t2 represents the image display time in milliseconds; 1000 in the formula represents 1000 milliseconds, that is, 1 second per unit time.

[0134] f = 1000 / (t1+ t2) Formula (1)

[0135] It should be noted that the image processing performance of the display rendering device 22 can be as shown above, whereby the display rendering device 22 uploads image decoding time data and image display time data, and the image acquisition device 12 calculates the image processing performance based on the data uploaded by the client 20. Alternatively, the display rendering device 22 can directly calculate the image processing performance and then upload the calculation result to the image acquisition device 12. The specific processing procedure can be executed according to the actual situation, and this embodiment does not impose specific limitations on it.

[0136] In some alternative embodiments, the process by which the image acquisition device 12 determines the streaming frame rate for streaming cloud desktop images to the client 20 based on network parameters and image processing performance may include the following processing: determining network transmission performance based on network latency, network bandwidth, and packet loss rate in the network parameters; and determining the streaming frame rate for streaming cloud desktop images to the client 20 based on the network transmission performance and image processing performance.

[0137] Among them, network transmission performance is used to characterize the performance of cloud desktop image data transmitted via network streaming. The higher the value of network transmission performance, the higher the performance of cloud desktop image data transmitted via network streaming. Conversely, the lower the value of network transmission performance, the lower the performance of cloud desktop image data transmitted via network streaming.

[0138] In this embodiment, before determining the streaming frame rate, the image acquisition device 12 first determines the network transmission performance based on the network latency, network bandwidth, and packet loss rate in the network parameters, which can obtain a more accurate network transmission performance in real time. Then, based on the network transmission performance and image processing performance, the streaming frame rate for streaming the cloud desktop image to the client 20 is determined, which can obtain a more accurate streaming frame rate.

[0139] Specifically, when determining network transmission performance based on network latency, network bandwidth, and packet loss rate among network parameters, the image acquisition device 12 can first determine the weighting coefficients corresponding to network latency, network bandwidth, and packet loss rate, and then determine the network transmission performance based on the product of network latency, network bandwidth, and packet loss rate with their respective weighting coefficients. In this way, by using these three indicators—network latency, network bandwidth, and packet loss rate—the image acquisition device 12 can calculate network transmission performance more accurately.

[0140] Furthermore, when the image acquisition device 12 determines the streaming frame rate for streaming cloud desktop images to the client 20 based on network transmission performance and image processing performance, it can first determine the first linear parameter, and then determine the streaming frame rate based on the product of network transmission performance, image processing performance and the first linear parameter.

[0141] The first linear parameter is used to characterize the linear relationship between the product of network transmission performance and image processing performance and the streaming frame rate.

[0142] Specifically, the streaming frame rate can be calculated using the following formula (2).

[0143] F = (a1 * L + a 2 * B + a 3 * D) * k * f Formula (2)

[0144] Where F represents the streaming frame rate, f represents the image processing capability of client 20, L represents network latency, B represents network bandwidth, D represents packet loss rate, and the calculation result of (a1*L+a2*B+a3*D) is the network transmission performance. a1 to a3 represent weighting coefficients, all of which are greater than 0 and less than 1, and a1+a2+a3=1. k can be understood as the streaming frame rate F having a linear relationship with the network transmission performance and the image processing capability of client 20. The specific value of k can be set according to the actual usage scenario. For example, if the user wants a smoother experience, this value can be increased, similar to adjusting the sensitivity of a mouse.

[0145] It can be understood that in the above formula (2), L represents the positive data of network latency, B represents the positive data of network bandwidth, and D represents the positive data of packet loss rate. That is, the higher the network latency, the lower the value of L; the larger the value of network bandwidth, the higher the value of B; and the larger the value of packet loss rate, the lower the value of D.

[0146] In some alternative embodiments, the input acquisition device 21 of the client 20 is mainly used to: when the input device generates a desktop input event, the input acquisition device 21 can process the desktop input event according to the sampling frequency, and send the sampling frequency and the acquired desktop input event to the server 10. The sampling frequency can be calculated according to the formula (3) above. The display rendering device 22 of the client 20 is mainly used to: receive desktop image update data transmitted from the network transmission device 30, decode the desktop image update data, and then call the display module for rendering and display.

[0147] Furthermore, when the input acquisition device 21 calculates the sampling frequency based on the received streaming frame rate, it can first obtain the second linear parameter; and determine the sampling frequency of the desktop input event based on the product of the streaming frame rate and the second linear parameter.

[0148] The second linear parameter characterizes the linear relationship between the streaming frame rate and the sampling frequency of desktop input events. This second linear parameter can be a value greater than or equal to 1, and its specific value can be determined according to the actual situation. This embodiment does not specifically limit the specific value of the second linear parameter. For example, it can be set according to the current network speed and stability, as well as the image processing performance of the client 20 device. Specifically, if the image processing performance of the client 20 device allows, the second linear parameter can be set to a larger value to achieve a smoother cloud desktop effect. If the network is unstable and / or the image processing performance of the client 20 device is poor, the second linear parameter can be set to a smaller value to prevent network congestion.

[0149] In this embodiment, there is a linear relationship between the product of network transmission performance and image processing performance and the streaming frame rate. The input acquisition device 21 can quickly calculate the streaming frame rate based on this linear relationship and the network transmission performance and image processing performance, thereby enabling real-time and rapid adjustment of the streaming frame rate and improving the processing efficiency of the cloud desktop streaming collaboration method.

[0150] Optionally, the input acquisition device 21 can use the following formula (3) to calculate the sampling frequency of the client 20. In formula (3), R represents the sampling frequency, F represents the streaming frame rate, and K represents a constant coefficient, K>=1, which can be dynamically adjusted according to the user's actual needs.

[0151] R = K * F Formula (3)

[0152] In addition, in this embodiment, the input acquisition device 21 acquires desktop input events based on the sampling frequency and uploads them to the input processing device 11 of the server 10. Afterwards, the display rendering device 22 can receive desktop image update data (which may be compressed data) sent by the image acquisition device 12 of the server 10, decode the desktop image update data, and call the local display module to perform rendering and display. After the display rendering is completed, the display rendering device 22 can also re-evaluate the display rendering capability according to the above formula (1) and synchronize the newly evaluated display rendering capability to the image acquisition device 12 of the server 10, so that the image acquisition device 12 can dynamically adjust the streaming frame rate according to the newly evaluated display rendering capability.

[0153] It is understood that the cloud desktop streaming collaboration system may also include a network transmission device 30, which can be used for communication and data transmission between the client 20 and the server 10. At the same time, it can feed back the current network latency, bandwidth, packet loss rate and other information to the image acquisition device 12 of the server 10.

[0154] Example 4

[0155] Some embodiments of this application also provide a cloud desktop streaming collaboration device, applied to a server providing cloud desktop services. This device is used to execute the aforementioned cloud desktop streaming collaboration method applied to the server. Figure 9 A schematic diagram of the cloud desktop streaming collaboration device is shown, such as... Figure 9 As shown, the cloud desktop streaming collaboration device includes:

[0156] The streaming frame rate determination module is used to determine the streaming frame rate for streaming cloud desktop images to the client.

[0157] The streaming frame rate sending module is used to send the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate;

[0158] The input event handling module is used to receive desktop input events and sampling frequencies uploaded by the client, and to process the desktop input events based on the sampling frequency.

[0159] It is understood that the cloud desktop streaming collaboration device provided in this embodiment and the cloud desktop streaming collaboration method embodiment applied to the server are based on the same inventive concept and have at least the same beneficial effects as the methods they adopt, operate or implement. Furthermore, the various implementation methods of the cloud desktop streaming collaboration method embodiment applied to the server are also applicable to the embodiment of the cloud desktop streaming collaboration device applied to the server.

[0160] Specifically, the streaming frame rate determination module can be used to: obtain the network parameters for communication between the server and the cloud desktop client, as well as the client's image processing performance; and determine the streaming frame rate for streaming cloud desktop images to the client based on the network parameters and image processing performance.

[0161] Furthermore, the streaming frame rate determination module can be specifically used to: obtain the client's image decoding time data and image display time data; and calculate the client's image processing performance based on the image decoding time data and image display time data.

[0162] Optionally, the streaming frame rate determination module can be further used to: determine network transmission performance based on network latency, network bandwidth, and packet loss rate in network parameters; the network transmission performance is used to characterize the performance of streaming cloud desktop image data over the network; and based on the network transmission performance and image processing performance, determine the streaming frame rate for streaming cloud desktop images to the client.

[0163] Optionally, the streaming frame rate determination module can be further used to: determine the weight coefficients corresponding to network latency, network bandwidth, and packet loss rate; and determine network transmission performance based on the product of network latency, network bandwidth, and packet loss rate with their respective weight coefficients.

[0164] Optionally, the streaming frame rate determination module may be further used to: determine a first linear parameter; the first linear parameter is used to characterize the linear relationship between the product of network transmission performance and image processing performance and the streaming frame rate; and determine the streaming frame rate for streaming cloud desktop images to the client based on the product of network transmission performance, image processing performance and the first linear parameter.

[0165] In another optional embodiment, the input event processing module may be specifically used to: determine the sampling interval for the client to collect desktop input events based on the sampling frequency; and process the desktop input events based on the sampling interval and the transmission duration of the desktop input events.

[0166] Optionally, the input event processing module can be further used to: directly call the input processing interface to process desktop input events when the transmission duration is greater than or equal to the sampling interval duration; and determine the difference between the transmission duration and the sampling interval duration when the transmission duration is less than the sampling interval duration, and call the input processing interface to process desktop input events after the difference duration.

[0167] Optionally, the cloud desktop streaming collaboration device may further include an image processing module, which may be specifically used for: acquiring desktop image update data corresponding to desktop input events based on the streaming frame rate; determining the target scene type corresponding to the desktop image update data; the scene type includes dynamic scenes and static scenes; and compressing and encoding the desktop image update data using the corresponding compression encoding method based on the target scene type.

[0168] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0169] Example 5

[0170] Some embodiments of this application also provide a cloud desktop streaming collaboration device, applied to a client for displaying a cloud desktop. This device is used to execute the cloud desktop streaming collaboration method applied to the server described above. Figure 10 A schematic diagram of the cloud desktop streaming collaboration device is shown, such as... Figure 10 As shown, the cloud desktop streaming collaboration device includes:

[0171] The streaming frame rate receiving module is used to receive the streaming frame rate sent by the cloud desktop server. The streaming frame rate is determined based on network parameters and image processing performance, and represents the frame rate at which the server streams cloud desktop images to the client.

[0172] The sampling frequency determination module is used to determine the sampling frequency of desktop input events based on the streaming frame rate;

[0173] The input event acquisition module is used to collect desktop input events based on the sampling frequency and upload them to the server so that the server can process the desktop input events based on the sampling frequency.

[0174] It is understood that the cloud desktop streaming collaboration device provided in this embodiment and the cloud desktop streaming collaboration method embodiment applied to the client are based on the same inventive concept and have at least the same beneficial effects as the methods they adopt, operate or implement. Furthermore, the various implementation methods of the cloud desktop streaming collaboration method embodiment are also applicable to the embodiment of this cloud desktop streaming collaboration device.

[0175] Specifically, the sampling frequency determination module can be used to: obtain a second linear parameter; the second linear parameter is used to characterize the linear relationship between the streaming frame rate and the sampling frequency of desktop input events; and determine the sampling frequency of desktop input events based on the product of the streaming frame rate and the second linear parameter.

[0176] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0177] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0178] Example 6

[0179] This application also provides an electronic device corresponding to the cloud desktop streaming collaboration method provided in the foregoing embodiments, for executing the cloud desktop streaming collaboration method described above.

[0180] Please refer to Figure 11 This illustrates a schematic diagram of the hardware structure of an electronic device provided by some embodiments of this application. For example... Figure 11 As shown, the electronic device includes: a processor 110, a memory 111, a bus 112, and a communication interface 113, wherein the processor 110, the communication interface 113, and the memory 111 are connected via the bus 112; the memory 111 stores a computer program that can run on the processor 110, and when the processor 110 runs the computer program, it executes the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method provided in any of the foregoing embodiments of this application.

[0181] The memory 111 can be any electronic, magnetic, optical, or other physical storage device, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 111 can be RAM (Random Access Memory), flash memory, a storage drive (such as a hard disk drive), any type of storage disk (such as an optical disc, DVD, etc.), or similar storage media, or a combination thereof. Communication between this device network element and at least one other network element is achieved through at least one communication interface 113 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0182] Bus 112 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc.

[0183] Processor 110 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 110 or by instructions in software form. Processor 110 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 111. Processor 110 reads the information in memory 111 and, in conjunction with its hardware, completes the steps of the above method.

[0184] The electronic device provided in this application embodiment and the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0185] Example 7

[0186] This application also provides a computer-readable storage medium corresponding to the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method provided in the foregoing embodiments. Please refer to... Figure 12 The computer-readable storage medium shown is an optical disc 120, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method provided in any of the aforementioned embodiments.

[0187] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0188] Example 8

[0189] This application also provides a computer program product, including a computer program that is executed by a processor to implement the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method of any of the above embodiments.

[0190] The computer-readable storage medium and computer program product provided in the above embodiments of this application are based on the same inventive concept as the cloud desktop streaming collaboration method or cloud desktop streaming collaboration method provided in the embodiments of this application, and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0191] It should be noted that:

[0192] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0193] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than those expressly recited in the various claims. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0194] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0195] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cloud desktop streaming collaboration method, characterized in that, The method, applied to a cloud desktop server, includes: The server determines the streaming frame rate for streaming cloud desktop images to the client and sends the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client. The desktop input event uploaded by the client is received, and the desktop input event is processed based on the sampling frequency of the desktop input event.

2. The method according to claim 1, characterized in that, Determining the streaming frame rate for streaming cloud desktop images to the client includes: Obtain the network parameters used for communication between the server and the cloud desktop client, as well as the image processing performance of the client; Based on the network parameters and the image processing performance, the streaming frame rate for streaming cloud desktop images to the client is determined.

3. The method according to claim 2, characterized in that, The image processing performance is determined based on the client's image decoding time and image display time data.

4. The method according to claim 2, characterized in that, Determining the streaming frame rate for streaming cloud desktop images to the client based on the network parameters and the image processing performance includes: Network transmission performance is determined based on network latency, network bandwidth, and packet loss rate among the network parameters; the network transmission performance is used to characterize the performance of cloud desktop image data transmitted via network streaming. Based on the network transmission performance and the image processing performance, the streaming frame rate for streaming cloud desktop images to the client is determined.

5. The method according to claim 4, characterized in that, The determination of network transmission performance based on the network latency, network bandwidth, and packet loss rate among the network parameters includes: Determine the weighting coefficients corresponding to the network latency, the network bandwidth, and the packet loss rate, respectively. The network transmission performance is determined by multiplying the network latency, the network bandwidth, and the packet loss rate by their respective weighting coefficients.

6. The method according to claim 4, characterized in that, Determining the streaming frame rate for streaming cloud desktop images to the client based on the network transmission performance and the image processing performance includes: A first linear parameter is determined; the first linear parameter is used to characterize the linear relationship between the product of the network transmission performance and the image processing performance and the streaming frame rate; The streaming frame rate for streaming cloud desktop images to the client is determined based on the product of the network transmission performance, the image processing performance, and the first linear parameter.

7. The method according to any one of claims 1-6, characterized in that, The processing of the desktop input event based on the sampling frequency of the desktop input event includes: Based on the sampling frequency, the sampling interval for the client to collect the desktop input events is determined; The desktop input event is processed based on the sampling interval duration and the transmission duration of the desktop input event.

8. The method according to claim 7, characterized in that, The processing of the desktop input event based on the sampling interval duration and the transmission duration of the desktop input event includes: If the transmission duration is greater than or equal to the sampling interval duration, the desktop input event is processed directly by calling the input processing interface. If the transmission duration is less than the sampling interval duration, the difference between the transmission duration and the sampling interval duration is determined, and after the difference duration, the input processing interface is called to process the desktop input event.

9. The method according to claim 8, characterized in that, The process of calling the input processing interface to handle the desktop input event includes: Based on the streaming frame rate, collect desktop image update data corresponding to the desktop input event; Determine the target scene type corresponding to the desktop image update data; the scene type includes dynamic scenes and static scenes. Based on the target scene type, the desktop image update data is compressed and encoded using the corresponding compression encoding method.

10. A cloud desktop streaming collaboration method, characterized in that, The method, applied to a cloud desktop client, includes: The server receives the streaming frame rate sent by the cloud desktop; the streaming frame rate represents the frame rate at which the server streams the cloud desktop image to the client. The sampling frequency of desktop input events is determined based on the streaming frame rate; The desktop input events are collected based on the sampling frequency and uploaded to the server so that the server can process the desktop input events based on the sampling frequency.

11. The method according to claim 10, characterized in that, Determining the sampling frequency of desktop input events based on the streaming frame rate includes: Obtain the second linear parameter; the second linear parameter is used to characterize the linear relationship between the streaming frame rate and the sampling frequency of desktop input events; The sampling frequency of the desktop input event is determined based on the product of the streaming frame rate and the second linear parameter.

12. A cloud desktop streaming collaboration system, characterized in that, The system includes a server and a client; The server is used to provide cloud desktop services, determine the streaming frame rate for streaming cloud desktop images to the client, and send the streaming frame rate to the client. The client is used to display the cloud desktop, determine the sampling frequency of desktop input events based on the streaming frame rate, collect the desktop input events based on the sampling frequency, and upload the desktop input events to the server. The server also processes the desktop input events based on the sampling frequency of the desktop input events.

13. A cloud desktop streaming collaboration device, characterized in that, The device is used as a server for providing cloud desktop services; the device includes: A streaming frame rate determination module is used to determine the streaming frame rate for streaming cloud desktop images to the client; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client. A streaming frame rate sending module is used to send the streaming frame rate to the client so that the client can adjust the sampling frequency of desktop input events based on the streaming frame rate; An input event processing module is used to receive the desktop input event uploaded by the client and process the desktop input event based on the sampling frequency of the desktop input event.

14. A cloud desktop streaming collaboration device, characterized in that, A client device for displaying a cloud desktop, the device comprising: A streaming frame rate receiving module is used to receive the streaming frame rate sent by the cloud desktop server; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client; the streaming frame rate represents the frame rate at which the server streams cloud desktop images to the client. A sampling frequency determination module is used to determine the sampling frequency of desktop input events based on the streaming frame rate; The input event acquisition module is used to acquire the desktop input events based on the sampling frequency and upload them to the server so that the server can process the desktop input events based on the sampling frequency.

15. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method as described in any one of claims 1-11.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method as described in any one of claims 1-11.

17. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method according to any one of claims 1-11.

Images