A cloud desktop multimedia playing method, system, device and storage medium

Abstract

Embodiments of the present application provide a multimedia playing method, system, device and storage medium for cloud desktop. A redirection server is deployed in a cloud desktop protocol server in the cloud desktop, and redirection clients are respectively deployed in various multimedia playing ends that can be used in the cloud desktop. The redirection server and the redirection clients can communicate with each other according to a specified protocol, and the specified protocol is compatible with various players and various browsers. Based on the system architecture, the redirection server can be compatible with various multimedia playing ends in various playing scenarios in the cloud desktop, and the redirection server can redirect undecoded media streams to the cloud desktop terminal, so that the undecoded media streams are decoded and played in the cloud desktop terminal. Thus, in the embodiments of the present application, multimedia redirection can be performed in various playing scenarios in the cloud desktop, and decoding and other operations in the cloud desktop server are no longer needed, so that the resource occupation of the cloud desktop server can be effectively reduced.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a multimedia playback method, system, device and storage medium for cloud desktop. Background Technology

[0002] With the increasing popularity of cloud desktop products and the continuous enrichment of multimedia resources, the demand for multimedia playback within cloud desktops will also grow.

[0003] Currently, the basic multimedia playback solution for cloud desktops is as follows: multimedia playback is performed by a multimedia playback client on the cloud desktop server. During multimedia playback, the cloud desktop is captured as an image, and the captured image is compressed and encoded before being transmitted to the cloud desktop terminal for display. During multimedia playback, all playback-related tasks, such as multimedia decoding, are performed on the cloud desktop server. This consumes a significant amount of CPU, GPU, and network resources on the cloud desktop server, placing a heavy burden on it. Summary of the Invention

[0004] This application provides a multimedia playback method, system, device, and storage medium for cloud desktops, in order to reduce the resource consumption of cloud desktop servers caused by multimedia playback.

[0005] This application provides a multimedia playback system for cloud desktops, including a cloud desktop terminal and a cloud desktop server. The cloud desktop server includes a cloud desktop protocol server and a multimedia playback terminal. The cloud desktop terminal includes a cloud desktop protocol client, and the cloud desktop protocol server includes a redirection server.

[0006] The redirection server is used to receive an undecoded media stream sent by a redirection client running in a multimedia playback client according to a specified protocol. The undecoded media stream is obtained by the redirection client from the multimedia playback client.

[0007] The cloud desktop protocol server is used to send the undecoded media stream received by the redirection server to the cloud desktop protocol client;

[0008] The cloud desktop protocol client is used to decode the undecoded media stream and call the local player to play the decoded multimedia.

[0009] The multimedia playback terminal includes various types of players and browsers, and the specified protocol is compatible with various types of players and browsers, and supports a one-to-many communication mechanism.

[0010] This application also provides a multimedia playback method for a cloud desktop, applicable to a cloud desktop server. The cloud desktop server includes a cloud desktop protocol server and a multimedia playback client. The cloud desktop protocol server includes a redirection server. The method includes:

[0011] Using the redirection server included in the cloud desktop protocol server, the redirection client running in the multimedia playback client receives an undecoded media stream sent by the redirection client according to a specified protocol. The undecoded media stream is obtained by the redirection client from the multimedia playback client.

[0012] The undecoded media stream received by the redirection server is sent to the cloud desktop protocol client in the cloud desktop terminal via the cloud desktop protocol server, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia.

[0013] The multimedia playback terminal includes various types of players and browsers, and the specified protocol is compatible with various types of players and browsers, and supports a one-to-many communication mechanism.

[0014] This application also provides a multimedia playback method for a cloud desktop, applicable to a cloud desktop terminal, wherein the cloud desktop terminal includes a cloud desktop protocol client, and the method includes:

[0015] The cloud desktop protocol client receives an undecoded media stream sent by the cloud desktop protocol server in the cloud desktop server. The undecoded media stream is received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback terminal according to a specified protocol. The redirection client obtains the undecoded media stream from the multimedia playback terminal.

[0016] Decode the undecoded media stream;

[0017] Use the local media player to play the decoded multimedia.

[0018] The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

[0019] This application also provides a cloud desktop server, including a memory and a processor;

[0020] The memory is used to store one or more computer instructions;

[0021] The processor is coupled to the memory and is used to execute the one or more computer instructions for:

[0022] Using the redirection server included in the cloud desktop protocol server, the redirection client running in the multimedia playback client receives an undecoded media stream sent by the redirection client according to a specified protocol. The undecoded media stream is obtained by the redirection client from the multimedia playback client.

[0023] The undecoded media stream received by the redirection server is sent to the cloud desktop protocol client in the cloud desktop terminal via the cloud desktop protocol server, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia.

[0024] The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

[0025] This application also provides a cloud desktop terminal, including a memory and a processor;

[0026] The memory is used to store one or more computer instructions;

[0027] The processor is coupled to the memory and is used to execute the one or more computer instructions for:

[0028] The cloud desktop protocol client receives an undecoded media stream sent by the cloud desktop protocol server in the cloud desktop server. The undecoded media stream is received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback terminal according to a specified protocol. The redirection client obtains the undecoded media stream from the multimedia playback terminal.

[0029] Decode the undecoded media stream;

[0030] Use the local media player to play the decoded multimedia.

[0031] The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

[0032] This application also provides a computer-readable storage medium for storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the aforementioned multimedia playback method for a cloud desktop.

[0033] In this embodiment, improvements are made to the cloud desktop protocol server in the cloud desktop server and the cloud desktop protocol client in the cloud desktop terminal. A redirection server is deployed in the cloud desktop protocol server, and redirection clients are deployed in various multimedia playback terminals that may be used in the cloud desktop. The redirection server and each redirection client can communicate in a one-to-many manner according to a specified protocol, and the specified protocol is compatible with various players and browsers. Based on this system architecture, in various playback scenarios in the cloud desktop, the redirection server is compatible with various multimedia playback terminals and redirects undecoded media streams to the cloud desktop terminal, where they are decoded and played. Thus, in this embodiment, multimedia redirection can cover various playback scenarios in the cloud desktop without requiring decoding operations on the cloud desktop server, effectively reducing the resource consumption of the cloud desktop server. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 A schematic diagram of the structure of a multimedia playback system for a cloud desktop provided as an exemplary embodiment of this application;

[0036] Figure 2 A schematic diagram of the internal exemplary structure of three types of exemplary multimedia playback terminals provided for an exemplary embodiment of this application;

[0037] Figure 3 A schematic diagram illustrating a logic for determining whether to perform multimedia redirection, provided as an exemplary embodiment of this application;

[0038] Figure 4 Another logical diagram illustrating whether to perform multimedia redirection is provided for an exemplary embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the internal exemplary structure of a cloud desktop protocol server provided in an exemplary embodiment of this application;

[0040] Figure 6 A logical diagram illustrating a data interaction method between a cloud desktop server and a cloud desktop terminal, provided as an exemplary embodiment of this application;

[0041] Figure 7 A flowchart illustrating a multimedia playback method for a cloud desktop, provided as another exemplary embodiment of this application;

[0042] Figure 8 A flowchart illustrating another multimedia playback method for a cloud desktop provided as another exemplary embodiment of this application;

[0043] Figure 9 A schematic diagram of the structure of a cloud desktop server is provided as another exemplary embodiment of this application;

[0044] Figure 10 This is a schematic diagram of the structure of a cloud desktop terminal, which is another exemplary embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] Currently, multimedia playback solutions on cloud desktops rely on cloud desktop servers for decoding and other processing, consuming significant resources and placing a heavy burden on them. To address this, some embodiments of this application improve the cloud desktop protocol server in the cloud desktop server and the cloud desktop protocol client in the cloud desktop terminal. A redirection server is deployed in the cloud desktop protocol server, and redirection clients are deployed in various multimedia playback terminals that may be used in the cloud desktop. The redirection server and each redirection client can communicate in a one-to-many manner according to a specified protocol, which is compatible with various players and browsers. Based on this system architecture, in diverse playback scenarios on the cloud desktop, the redirection server can be compatible with various multimedia playback terminals and redirect undecoded media streams to the cloud desktop terminal for decoding and playback. Thus, in these embodiments, multimedia redirection can cover diverse playback scenarios on the cloud desktop without requiring decoding operations on the cloud desktop server, effectively reducing resource consumption on the cloud desktop server.

[0047] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0048] Figure 1 This is a schematic diagram of the structure of a cloud desktop multimedia playback system provided as an exemplary embodiment of this application. For example... Figure 1As shown, the system includes: a cloud desktop server and a cloud desktop terminal.

[0049] In this embodiment, a cloud desktop server can refer to a cloud server used to support the operation of cloud desktops. It should be understood that a cloud desktop server may include components such as CPU, memory, images, and cloud disks, and is a readily available and elastically scalable cloud server. A cloud desktop terminal can refer to a terminal device using cloud desktop technology, which can connect to the cloud desktop server and render the cloud desktop to the front end, as well as redirect input and output data such as mouse operations from the front end to the cloud desktop server.

[0050] refer to Figure 1 In this embodiment, the cloud desktop server and the cloud desktop terminal can communicate via a specific communication protocol. The cloud desktop server may include a cloud desktop protocol server, and the cloud desktop terminal may include a cloud desktop protocol client. The cloud desktop protocol server is used to implement the aforementioned specific communication protocol on the cloud desktop server side, and the cloud desktop protocol client is used to implement the aforementioned specific communication protocol on the cloud desktop terminal side. It should be understood that the cloud desktop protocol server can be a process within the cloud desktop server, and similarly, the cloud desktop protocol client can be a process within the cloud desktop terminal.

[0051] Continue to refer to Figure 1 In this embodiment, a redirection server is also deployed in the cloud desktop protocol server. The redirection server can be a thread contained in the process corresponding to the cloud desktop protocol server.

[0052] In addition, in this embodiment, redirection clients can be deployed in various multimedia playback clients that may be used in the cloud desktop. Each multimedia playback client runs on the cloud desktop server. Similarly, the multimedia playback client can be a process within the cloud desktop server, and the redirection client deployed within it can be a thread within the corresponding process of the multimedia playback client. The multimedia playback client can be an application on the cloud desktop server used for multimedia playback. In this embodiment, the multimedia playback client can be various players or browsers. For example, the multimedia playback client can be VLC player, WMP player, Chrome browser, Edge browser, or Firefox browser, etc. Of course, these are only examples, and this embodiment is not limited to them. This embodiment can support various multimedia playback clients currently used in the cloud desktop, as well as new types of multimedia playback clients that may be added in the future.

[0053] Based on this, refer to Figure 1In this embodiment, the redirection server and each redirection client can communicate in a one-to-many manner according to a specified protocol, and this specified protocol is compatible with various types of players and browsers. Preferably, in this embodiment, the specified protocol can be the WebSocket protocol. Of course, in this embodiment, the specified protocol can also be the pipe protocol, or other protocols that may appear now or in the future that can meet the aforementioned protocol requirements in this embodiment. This embodiment is not limited to these. In this way, based on the specified protocol, the redirection server can communicate with the redirection clients deployed in various multimedia playback terminals to support multimedia redirection for various multimedia playback terminals.

[0054] refer to Figure 1 The redirection client deployed on the multimedia playback terminal can obtain the undecoded media stream from the multimedia playback terminal and send the undecoded media stream to the redirection server according to the aforementioned specified protocol. In this way, the redirection server can receive the undecoded media streams sent by the redirection clients running on various multimedia playback terminals according to the specified protocol. That is, the redirection clients deployed on various multimedia playback terminals that may be used in the cloud desktop can all send the undecoded media streams to the redirection server in this embodiment according to the specified protocol for subsequent processing by the redirection server.

[0055] During their research, the inventors discovered that playback scenarios in cloud desktops are highly diverse. For example, local multimedia is typically played using a media player, while web-based multimedia is usually played using a browser. Therefore, the types of multimedia playback clients that may be used in cloud desktops are numerous. This embodiment proposes a communication mechanism where a redirection server connects to multiple redirection clients according to a specified protocol. This deployment architecture can cover various playback scenarios in cloud desktops. Users can choose the desired multimedia playback client as needed, and all can be redirected by reusing the redirection server. Moreover, new types of multimedia playback clients can be continuously added to the cloud desktop. Simply deploying the redirection client proposed in this embodiment on the new type of multimedia playback client can cover the new type of multimedia playback client, effectively improving the scalability of multimedia playback clients in the cloud desktop.

[0056] In this embodiment, multimedia playback devices can be categorized into at least three types: non-customized players, browsers, and customized redirection-specific players. Figure 2 This document provides schematic diagrams illustrating the internal exemplary structures of three types of exemplary multimedia playback terminals provided for an exemplary embodiment of this application. (Reference) Figure 2 The following will explain the interaction logic between the redirection client and the redirection server for these three exemplary types.

[0057] I. Reference Figure 2 If the multimedia player is any non-custom player, a player plugin can be installed within it. In this case, although the multimedia player doesn't actually play the source multimedia file, it still reads it. The player plugin can then obtain the source file read by the multimedia player and demultiplex it to generate audio and video streams. Based on this, the player plugin can provide the demultiplexed audio and video streams as undecoded media streams to the redirection client in the multimedia player; the redirection client can then send the received audio and video streams to the redirection server. Demultiplexing refers to separating the audio and video streams from the source multimedia file. A non-custom player refers to a third-party developed player.

[0058] Figure 3 A logical diagram illustrating whether to perform multimedia redirection is provided as an exemplary embodiment of this application, with reference to... Figure 3 In this scenario, the non-customized player can preferentially call the player plugin provided in this embodiment. The player plugin can also: if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then demultiplexing and subsequent steps will not be performed; and a rollback instruction will be sent back to the non-customized player to trigger multimedia playback. Specifically, the non-customized player can provide the encoding format corresponding to the multimedia source file to be played to the player plugin in this embodiment. The player plugin can also obtain the encoding formats supported by the cloud desktop terminal. By comparison, the player plugin can determine whether the cloud desktop terminal can support the encoding format corresponding to the multimedia source file to be played. For example, if the encoding format of the multimedia source file to be played is H264, but the cloud desktop terminal does not support decoding this encoding format, then the aforementioned rollback mechanism can be executed. That is, the non-customized server can preferentially call the player plugin provided in this embodiment, and if it is determined that multimedia redirection cannot be performed, the non-customized player can switch to calling the decoding plugin to decode and play the multimedia source file, thereby rolling back to the traditional playback method to ensure normal multimedia playback.

[0059] II. Reference Figure 2If the multimedia playback client is any browser, a browser plugin can be installed within the browser. In this case, the browser plugin can capture the video address in the current webpage and download the multimedia source file to be played from the video address via HTTP requests or other methods. The browser plugin can then provide the downloaded multimedia source file to be played to the redirection client in the browser; the redirection client can then send the multimedia source file to be played to the redirection server in this embodiment. Based on this, the redirection server is also used to: demultiplex the multimedia source file to be played sent by the redirection client in the browser to generate corresponding audio and video streams.

[0060] In this case, considering that browser plugins are usually in JS format and have low processing efficiency, the demultiplexing process is handled by the redirection server in this embodiment.

[0061] In addition, in this case,

[0062] Figure 4 This is another logical diagram illustrating whether to perform multimedia redirection, provided as an exemplary embodiment of this application. (See reference...) Figure 4 In this scenario, the browser can prioritize calling the browser plugin provided in this embodiment. The browser plugin can also be used to: if the video address captured from the current webpage does not support file download, then stop sending the multimedia source file to be played to the redirection server and send a fallback instruction to the browser to trigger multimedia playback. Preferably, in this case, the browser plugin can also be used to: check if a video tag exists in each webpage opened by the browser; if it exists, determine whether the video window specifications meet the specified specifications based on the video tag; if not, execute the aforementioned fallback mechanism to trigger multimedia playback; if it does meet the specifications, then execute the above-mentioned determination of whether the video address captured from the current webpage supports file download. The video tag is existing identification information in the browser, used to identify whether a video to be played exists on the webpage, as well as playback parameters such as video window specifications.

[0063] In some scenarios, due to permission issues, the video URL may not support file download. Therefore, multimedia redirection will not be performed, and the browser will play the multimedia. If the video URL supports download, the multimedia source file to be played can be sent to the redirection server through the redirection client. After receiving the multimedia source file, if the redirection server determines that the cloud desktop terminal cannot support the encoding format of the multimedia source file to be played, it will stop further processing of the multimedia source file to be played and send a fallback instruction to the browser to trigger the browser to play the multimedia. In this way, the redirection server can determine whether the cloud desktop terminal supports the encoding format of the multimedia source file to be played, thereby determining whether to implement the fallback mechanism.

[0064] III. Reference Figure 2 If the multimedia playback client is a customized redirection-specific player, it can read the source file of the multimedia file to be played, demultiplex the file to generate audio and video streams, and then send these streams as undecoded media streams to the redirection server through the redirection client. The redirection-specific player has pre-built program logic for acquiring the multimedia file to be played and for demultiplexing it.

[0065] In this case, the dedicated redirection player has pre-built program logic related to the player plugin in the aforementioned non-custom player. Therefore, there is no need to build a player plugin. Instead, the generated audio and video streams can be directly provided to the redirection client as undecoded media streams according to the pre-built program logic.

[0066] Additionally, in this scenario, the dedicated redirection player can also be used to: if it is determined that the cloud desktop terminal cannot support the codec type corresponding to the source file of the multimedia to be played, then play the multimedia to be played without performing demultiplexing and subsequent processing. In other words, the dedicated redirection player also supports the aforementioned fallback mechanism.

[0067] By using the three types of multimedia playback terminals mentioned above as examples to illustrate the interaction logic between the redirection client and the redirection server, it can be seen that in this embodiment, for different multimedia playback terminals, the transmission of undecoded media streams can be achieved through the cooperation of the redirection client and the redirection server. Furthermore, if it is determined in this step that multimedia redirection cannot be performed, the aforementioned fallback mechanism can be used to fall back to the traditional playback method to ensure that multimedia playback is completed normally.

[0068] For the cloud desktop protocol server, the undecoded media stream received by the redirection server can be sent to the cloud desktop protocol client. As mentioned earlier, the redirection server can essentially be a thread within the cloud desktop protocol server. Therefore, the undecoded media stream obtained by the thread can be obtained by the process and sent to another process in the cloud desktop terminal (i.e., the cloud desktop protocol client). At this point, the undecoded media stream has been successfully sent to the cloud desktop terminal.

[0069] On the cloud desktop terminal side, the cloud desktop protocol client can decode the received undecoded media stream and call the local player to play the decoded multimedia. In this way, the decoding and playback are handled by the cloud desktop terminal, instead of the cloud desktop server, which can effectively reduce the resource consumption of the cloud desktop server for multimedia playback.

[0070] In summary, this embodiment improves upon the cloud desktop protocol server in the cloud desktop server and the cloud desktop protocol client in the cloud desktop terminal. A redirection server is deployed in the cloud desktop protocol server, and redirection clients are deployed in various multimedia playback devices that may be used in the cloud desktop. The redirection server and each redirection client can communicate in a one-to-many manner according to a specified protocol, which is compatible with various players and browsers. Based on this system architecture, in diverse playback scenarios in the cloud desktop, the redirection server is compatible with various multimedia playback devices and redirects undecoded media streams to the cloud desktop terminal, where they are decoded and played. Thus, this embodiment can cover diverse playback scenarios in the cloud desktop for multimedia redirection, eliminating the need for decoding operations on the cloud desktop server and effectively reducing resource consumption on the cloud desktop server.

[0071] In the above or following embodiments, the cloud desktop protocol server may further include a location management component. Figure 5 This is a schematic diagram of the internal exemplary structure of a cloud desktop protocol server provided in an exemplary embodiment of this application. (Refer to...) Figure 5 The location management component in the cloud desktop protocol server can receive the location information corresponding to the playback screen provided by the redirection server; it can also capture the location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop; and calculate the occlusion information of the playback screen based on the location information corresponding to the playback screen and the location information corresponding to each application window.

[0072] Among them, reference Figure 2 The redirection client in the multimedia playback terminal can obtain the location information corresponding to the playback screen and provide it to the redirection server. The redirection server can then pass the received location information corresponding to the playback screen to the location management component in this embodiment. The location information corresponding to the playback screen refers to the location information of the playback screen within the application window of the multimedia playback terminal. For example, for a player, the playback screen only occupies a part of the player's application window; for a browser, the playback screen only occupies a part of the browser's application window.

[0073] In addition, in this embodiment, the location management component can capture the location information of application windows corresponding to various applications in the cloud desktop by calling specified interfaces provided in the operating system. For example, for a cloud desktop using a Windows operating system, the location management component can call the WinEventHook interface to obtain the location information of application windows corresponding to various applications in the cloud desktop. The location information of the application windows may include window coordinates and their display order relative to other windows. An application window displayed at the top may obscure an application window displayed at the bottom. Thus, the location management component can determine the target application window that is displayed before the multimedia playback device, and calculate the obscuring information of the playback screen based on the location information of the target application window, the application window of the multimedia playback device, and the playback screen. It is understood that the obscuring information can be the coordinates of the obscured area in the playback screen, or it can be the coordinates of the unobscured area in the playback screen, etc., and this embodiment does not limit this.

[0074] In this embodiment, the cloud desktop protocol server can obtain the desktop image of the cloud desktop and send the location information and occlusion information corresponding to the desktop image and the playback screen to the cloud desktop protocol client. The desktop image can be obtained by taking a screenshot of the cloud desktop using an existing screenshot tool within the cloud desktop, which will not be explained in detail here. It should be noted that since no multimedia playback is performed on the cloud desktop, there is no playback screen. Therefore, in this embodiment, the playback screen can be displayed as a specified pattern on the cloud desktop. The specified pattern may include, but is not limited to, a pure white pattern, a pure black pattern, or a mosaic pattern.

[0075] In this embodiment, the cloud desktop protocol client can obtain the playback screen of the local player; crop the playback screen according to the occlusion information; stitch the cropped playback screen and the desktop image according to the position information of the playback screen; and display the stitched image on the cloud desktop terminal. It is worth noting that in this embodiment, during multimedia playback on the cloud desktop terminal, the local player is not foregrounded; it only plays multimedia in the background to generate the playback screen. Therefore, the local player on the cloud desktop terminal does not affect the foreground rendering of the cloud desktop. Furthermore, the cloud desktop protocol client in this embodiment can also crop the playback screen generated by the local player according to the occlusion information received from the cloud desktop protocol server, that is, cropping the area of ​​the playback screen obscured by other application windows. Based on this, the cropped playback screen can be stitched to the playback screen position on the desktop image to generate the final rendered image. The cloud desktop terminal will display the rendered image at a certain frame rate to achieve the multimedia playback effect.

[0076] The following will describe the implementation scheme for the redirection client to obtain the location information corresponding to the playback screen, based on the three exemplary multimedia terminal types summarized in the foregoing embodiments.

[0077] If the multimedia player is any non-customized player, it receives the position information corresponding to the playback screen provided by the player plugin built into the multimedia player. This position information is monitored by the multimedia player and provided to the player plugin. In this case, in this embodiment, the multimedia player itself can monitor the position information corresponding to the playback screen and provide it to the player plugin in this embodiment. The player plugin can then provide the received position information corresponding to the playback screen to the redirection client in the multimedia player, and the redirection client will then send the position information corresponding to the playback screen to the redirection server.

[0078] If the multimedia playback client is any browser, it receives the location information corresponding to the playback screen provided by the browser plugin built into the multimedia playback client. This location information is monitored by the browser and provided to the browser plugin. In this case, in this embodiment, the multimedia playback client itself can monitor the location information corresponding to the playback screen and provide it to the browser plugin in this embodiment. The browser plugin can then provide the received location information corresponding to the playback screen to the redirection client in the multimedia playback client, and the redirection client will then send the location information corresponding to the playback screen to the redirection server.

[0079] If the multimedia playback client is a customized redirection-specific player, then the system receives the location information corresponding to the playback screen detected by the redirection-specific player. The redirection-specific player has pre-built program logic for location monitoring and transmission. In this case, the redirection-specific player itself can monitor the location information corresponding to the playback screen and provide it to its internal redirection client, which then sends the location information corresponding to the playback screen to the redirection server.

[0080] In this way, the location information corresponding to the playback screens of various multimedia playback terminals that may be used in the cloud desktop can reach the redirection server. The redirection server can then provide the location information corresponding to the playback screens it obtains to the location management component in this embodiment, as one of the bases for the location management component to calculate occlusion information. Accordingly, in this embodiment, the location management component can accurately calculate the occlusion information corresponding to the playback screen and provide it to the cloud desktop protocol client in the cloud desktop terminal. This allows the cloud desktop protocol client to accurately render the playback screen generated by its local player onto the desktop image of the cloud desktop based on the occlusion information, thereby ensuring the front-end rendering effect during multimedia playback using the multimedia redirection method.

[0081] In the above or following embodiments, in addition to interacting with the undecoded media stream mentioned in the foregoing embodiments, as well as the position information and occlusion information corresponding to the playback screen, the cloud desktop server and the cloud desktop terminal can also interact with the instruction stream involved in the multimedia playback process. The instruction stream contains various control commands that occur during multimedia playback.

[0082] In this embodiment, the cloud desktop protocol server can also obtain a first instruction stream occurring within the multimedia playback terminal from the redirection server. This first instruction stream is obtained by the redirection client in the multimedia playback terminal and provided to the redirection server. The first instruction stream is then sent to the cloud desktop protocol client, allowing the cloud desktop protocol client to control the playback of the local player within the cloud desktop terminal according to the first instruction stream. The first instruction stream may include control instructions such as volume adjustment, play, pause, and end-of-file markers occurring in the multimedia playback terminal.

[0083] In this embodiment, the cloud desktop protocol server can also receive a second instruction stream sent by the cloud desktop protocol client. This second instruction stream is detected by the cloud desktop protocol client from the local player on the cloud desktop terminal. Playback control is then performed on the multimedia playback terminal according to the second instruction stream. The second instruction stream may include control instructions such as search, fast forward, rewind, and stream dragging occurring in the local player on the cloud desktop terminal. These control instructions are redirected to the multimedia playback terminal on the cloud desktop to control playback, thereby ensuring the accuracy of front-end rendering.

[0084] Furthermore, considering the relatively long data link of multimedia redirection and the diverse and complex formats of the transmitted content, this embodiment proposes using serialization and deserialization technologies for data interaction between the cloud desktop protocol server and the cloud desktop protocol client. Using serialization and deserialization technologies enables efficient content transmission between processes and over networks. Serialization can be understood as saving an object into a byte sequence, while deserialization can be understood as converting the byte sequence back into the original object for use.

[0085] Figure 6 This is a logical diagram illustrating a data interaction method between a cloud desktop server and a cloud desktop terminal, provided as an exemplary embodiment of this application. (Reference) Figure 6 In this embodiment, data transmission plugins can be deployed on both the cloud desktop protocol server and the cloud desktop protocol client, and each data transmission plugin can run as a single thread. For example, the data transmission plugin can be a serializer plugin.

[0086] refer to Figure 6In this embodiment, the cloud desktop protocol server can use a data transmission plugin to serialize the undecoded media stream, the position information and occlusion information of the playback screen mentioned in the previous embodiment, and the first instruction stream mentioned in this embodiment, and then send them to the cloud desktop protocol client. The cloud desktop protocol client can then use the data transmission plugin to deserialize the content sent by the cloud desktop protocol server to restore the undecoded media stream, the position information and occlusion information of the playback screen.

[0087] Alternatively, the cloud desktop protocol client can serialize the acquired second command stream using a data transmission plugin before sending it to the cloud desktop protocol server. The cloud desktop protocol server can then use its data transmission plugin to perform serialization processing to reconstruct the second command stream.

[0088] In this way, the cloud desktop protocol server and client processes can send various types of data to each other during remote communication. Regardless of the data type, it will be transmitted over the network as a binary sequence. The sender converts the data into a byte sequence before transmitting it; the receiver can deserialize the byte sequence to reconstruct the data. This enables the transmission of the aforementioned various data types between the cloud desktop protocol server and client.

[0089] It is worth noting that, in this embodiment, in addition to the serialization and deserialization methods described above, the cloud desktop protocol server and the cloud desktop protocol client can also use other methods to perform data interaction, and this embodiment is not limited to these.

[0090] Accordingly, in this embodiment, the instruction streams involved in multimedia playback can also be exchanged between the cloud desktop protocol server and the cloud desktop protocol client. Furthermore, serialization and deserialization can be used to transmit various types of data that need to be exchanged between the cloud desktop protocol server and the cloud desktop protocol client. This allows different types of data that need to be transmitted between the cloud desktop protocol server and the cloud desktop protocol client to be transmitted through appropriate channels, thereby ensuring the timeliness and accuracy of data transmission and improving the smoothness of multimedia playback.

[0091] Figure 7 This is a flowchart illustrating a multimedia playback method for a cloud desktop, provided as another exemplary embodiment of this application. The method can be executed by a data processing device, which can be implemented as software, hardware, or a combination of both. This data processing device can be integrated into a cloud desktop server. The cloud desktop server includes a cloud desktop protocol server and a multimedia playback client. The cloud desktop protocol server includes a redirection server. (Reference) Figure 7 The method may include:

[0092] Step 700: Using the redirection server included in the cloud desktop protocol server, receive the undecoded media stream sent by the redirection client running in the multimedia playback client according to the specified protocol. The undecoded media stream is obtained by the redirection client from the multimedia playback client.

[0093] Step 701: Through the cloud desktop protocol server, send the undecoded media stream received by the redirection server to the cloud desktop protocol client in the cloud desktop terminal, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia.

[0094] The multimedia playback terminal includes various types of players and browsers, and the specified protocol is compatible with various types of players and browsers, and supports one-to-many communication mechanism.

[0095] In one alternative embodiment, the specified protocol adopts the WebSocket communication protocol.

[0096] In an optional embodiment, the method may further include:

[0097] If the multimedia player is any non-custom player, then the redirection client receives the undecoded media stream provided by the player plugin built into the multimedia player. The undecoded media stream is the audio and video stream generated by the player plugin demultiplexing the multimedia source file to be played; and then sends the audio and video streams to the redirection server; or,

[0098] If the multimedia player is any browser, the redirection client receives the undecoded media stream provided by the browser plugin built into the multimedia player. This undecoded media stream is the multimedia source file to be played, downloaded by the browser plugin based on the video address captured from the current webpage. The browser plugin then sends this multimedia source file to the redirection server. The redirection server demultiplexes the multimedia source file sent by the redirection client in the browser to generate the corresponding audio and video streams. Alternatively,

[0099] If the multimedia playback client is a customized redirection-specific player, then the redirection client receives the undecoded media stream provided by the redirection-specific player. The undecoded media stream is the audio and video stream generated by the redirection-specific player after demultiplexing the multimedia file to be played. The audio and video streams are then sent to the redirection server. The redirection-specific player has pre-built program logic for obtaining the multimedia file to be played and for demultiplexing the multimedia file to be played.

[0100] In an optional embodiment, the method may further include:

[0101] If the player plugin determines that the cloud desktop terminal cannot support the encoding format of the multimedia source file to be played, then demultiplexing and subsequent operations will not be performed; instead, a rollback command will be sent to the non-customized player to trigger multimedia playback; the non-customized player will prioritize calling the player plugin, or...

[0102] If the browser plugin determines that the video address captured from the current webpage does not support file download, then the operation of sending the multimedia source file to be played to the redirection server will not be executed, and a fallback command will be sent to the browser to trigger multimedia playback. If, after receiving the multimedia source file, the redirection server determines that the cloud desktop terminal does not support the encoding format corresponding to the multimedia source file to be played, then it will stop further processing of the multimedia source file to be played and send a fallback command to the browser to trigger multimedia playback. The browser will prioritize calling the browser plugin, or...

[0103] It can also use a dedicated redirection player to determine if the cloud desktop terminal cannot support the encoding format of the multimedia source file to be played, and then play the multimedia to be played without performing demultiplexing and subsequent processing.

[0104] In an optional embodiment, the cloud desktop protocol server may further include a location management component, and the method may further include:

[0105] The location management component receives the location information corresponding to the playback screen provided by the redirection server. The location information corresponding to the playback screen is obtained by the redirection client in the multimedia playback client and provided to the redirection server. The location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop is captured through a specified interface. Based on the location information corresponding to the playback screen and the location information corresponding to each application window, the occlusion information of the playback screen is calculated.

[0106] The cloud desktop protocol server sends the location and occlusion information of the desktop image and playback screen of the cloud desktop to the cloud desktop protocol client, so that the cloud desktop protocol client can obtain the playback screen of the local player; the playback screen is cropped according to the occlusion information; the cropped playback screen and the desktop image are stitched together according to the location information of the playback screen; and the stitched screen is displayed on the cloud desktop terminal.

[0107] In an optional embodiment, the method may further include:

[0108] If the multimedia player is any non-custom player, then its built-in redirection client receives the position information corresponding to the playback screen provided by the player plugin embedded in the multimedia player. This position information is monitored by the multimedia player and provided to the player plugin; or...

[0109] If the multimedia player is any browser, then the redirection client within the multimedia player receives the location information corresponding to the playback screen from the browser plugin built into the multimedia player. This location information is monitored by the browser and provided to the browser plugin; or...

[0110] If the multimedia playback client is a customized redirection-specific player, then the redirection client within it receives the location information corresponding to the playback screen detected by the redirection-specific player. The redirection-specific player has pre-set program logic for location monitoring and transmission.

[0111] In an optional embodiment, the method may further include:

[0112] Using the cloud desktop protocol server, the first instruction stream occurring in the multimedia playback client is obtained from the redirection server. The first instruction stream is obtained by the redirection client in the multimedia playback client and provided to the redirection server.

[0113] The first instruction stream is sent to the cloud desktop protocol client so that the cloud desktop protocol client can control the playback of the local player in the cloud desktop terminal according to the first instruction stream;

[0114] Receive the second instruction stream sent by the cloud desktop protocol client. The second instruction stream is detected by the cloud desktop protocol client from the local player.

[0115] Playback control is performed on the multimedia playback terminal according to the second instruction stream.

[0116] In one optional embodiment, the cloud desktop protocol server and the cloud desktop protocol client use serialization and deserialization technologies for data interaction.

[0117] In one alternative embodiment, the playback screen in the desktop image is displayed as a specified pattern, which may include a pure white pattern, a pure black pattern, or a mosaic pattern.

[0118] It is worth noting that the technical details of the above embodiments of the multimedia playback method for cloud desktops can be found in the description of the cloud desktop server in the aforementioned system embodiments. To save space, these details will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0119] Figure 8 This is a flowchart illustrating another exemplary embodiment of a multimedia playback method for a cloud desktop, which can be executed by a data processing device. The data processing device can be implemented as software, hardware, or a combination of both, and can be integrated into a cloud desktop terminal. The cloud desktop terminal may include a cloud desktop protocol client, see reference... Figure 8 The method may include:

[0120] Step 800: Receive the undecoded media stream sent by the cloud desktop protocol server in the cloud desktop server through the cloud desktop protocol client. The undecoded media stream is received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback terminal according to the specified protocol. The redirection client obtains the undecoded media stream from the multimedia playback terminal.

[0121] Step 801: Decode the undecoded media stream;

[0122] Step 802: Use the local media player to play the decoded multimedia;

[0123] The multimedia playback client runs on a cloud desktop server and includes various types of players and browsers. It is compatible with various types of players and browsers by specifying protocols and supports one-to-many communication mechanisms.

[0124] In one alternative embodiment, the specified protocol adopts the WebSocket communication protocol.

[0125] In an optional embodiment, the method may further include:

[0126] The playback screen of the local player is obtained through the cloud desktop protocol client;

[0127] The playback screen is cropped based on the occlusion information;

[0128] Based on the position information of the playback screen, the cropped playback screen and the desktop image are stitched together;

[0129] Display the spliced ​​image on the cloud desktop terminal;

[0130] The occlusion information is generated by the location management component in the cloud desktop protocol server based on the location information corresponding to the playback screen provided by the redirection server and the location information corresponding to the application windows of the multimedia playback terminal and other applications in the cloud desktop.

[0131] In an optional embodiment, the method may further include:

[0132] The second instruction stream is sent from the cloud desktop protocol client to the cloud desktop protocol server so that the cloud desktop protocol server can control the playback of the multimedia playback terminal according to the second instruction stream.

[0133] The second instruction stream is detected by the cloud desktop protocol client from the local player.

[0134] In one optional embodiment, the cloud desktop protocol server and the cloud desktop protocol client use serialization and deserialization technologies for data interaction.

[0135] In one alternative embodiment, the playback screen in the desktop image is displayed as a specified pattern, which may include a pure white pattern, a pure black pattern, or a mosaic pattern.

[0136] It is worth noting that the technical details of the above embodiments of the multimedia playback method for cloud desktops can be found in the descriptions of cloud desktop terminals in the aforementioned system embodiments. To save space, these details will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0137] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The operation numbers, such as 801, 802, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should also be noted that the descriptions such as "first" and "second" in this document are used to distinguish different instruction flows, and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0138] Figure 9 This is a schematic diagram of the structure of a cloud desktop server provided as another exemplary embodiment of this application. For example... Figure 9 As shown, the cloud desktop server includes: memory 90 and processor 91.

[0139] Processor 91, coupled to memory 90, is used to execute computer programs in memory 90 for:

[0140] Using the redirection server included in the cloud desktop protocol server, the redirection client running in the multimedia playback client receives the undecoded media stream sent by the redirection client according to the specified protocol. The undecoded media stream is obtained by the redirection client from the multimedia playback client.

[0141] The cloud desktop protocol server sends the undecoded media stream received by the redirection server to the cloud desktop protocol client in the cloud desktop terminal, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia.

[0142] The multimedia playback client runs on a cloud desktop server and includes various types of players and browsers. It is compatible with various types of players and browsers by specifying protocols and supports one-to-many communication mechanisms.

[0143] In one alternative embodiment, the specified protocol adopts the WebSocket communication protocol.

[0144] In an optional embodiment, processor 91 may also be used for:

[0145] If the multimedia player is any non-custom player, then the redirection client receives the undecoded media stream provided by the player plugin built into the multimedia player. The undecoded media stream is the audio and video stream generated by the player plugin demultiplexing the multimedia source file to be played; and then sends the audio and video streams to the redirection server; or,

[0146] If the multimedia player is any browser, the redirection client receives the undecoded media stream provided by the browser plugin built into the multimedia player. This undecoded media stream is the multimedia source file to be played, downloaded by the browser plugin based on the video address captured from the current webpage. The browser plugin then sends this multimedia source file to the redirection server. The redirection server demultiplexes the multimedia source file sent by the redirection client in the browser to generate the corresponding audio and video streams. Alternatively,

[0147] If the multimedia playback client is a customized redirection-specific player, then the redirection client receives the undecoded media stream provided by the redirection-specific player. The undecoded media stream is the audio and video stream generated by the redirection-specific player after demultiplexing the multimedia file to be played. The audio and video streams are then sent to the redirection server. The redirection-specific player has pre-built program logic for obtaining the multimedia file to be played and for demultiplexing the multimedia file to be played.

[0148] In an optional embodiment, processor 91 may also be used for:

[0149] If the player plugin determines that the cloud desktop terminal cannot support the encoding format of the multimedia source file to be played, then demultiplexing and subsequent operations will not be performed; instead, a rollback command will be sent to the non-customized player to trigger multimedia playback; the non-customized player will prioritize calling the player plugin, or...

[0150] If a browser plugin determines that the video address captured from the current webpage does not support file download, the operation of sending the multimedia source file to be played to the redirection server will not be executed, and a fallback command will be sent to the browser to trigger multimedia playback. If, after receiving the multimedia source file, the redirection server determines that the cloud desktop terminal does not support the encoding format corresponding to the multimedia source file to be played, it will stop further processing of the multimedia source file to be played and send a fallback command to the browser to trigger multimedia playback. The browser will prioritize calling the browser plugin, or...

[0151] It can also use a dedicated redirection player to determine if the cloud desktop terminal cannot support the encoding format of the multimedia source file to be played, and then play the multimedia to be played without performing demultiplexing and subsequent processing.

[0152] In an optional embodiment, the cloud desktop protocol server may further include a location management component, and the processor 91 may also be used for:

[0153] The location management component receives the location information corresponding to the playback screen provided by the redirection server. The location information corresponding to the playback screen is obtained by the redirection client in the multimedia playback client and provided to the redirection server. The location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop is captured through a specified interface. Based on the location information corresponding to the playback screen and the location information corresponding to each application window, the occlusion information of the playback screen is calculated.

[0154] The cloud desktop protocol server sends the location and occlusion information of the desktop image and playback screen of the cloud desktop to the cloud desktop protocol client, so that the cloud desktop protocol client can obtain the playback screen of the local player; the playback screen is cropped according to the occlusion information; the cropped playback screen and the desktop image are stitched together according to the location information of the playback screen; and the stitched screen is displayed on the cloud desktop terminal.

[0155] In an optional embodiment, processor 91 may also be used for:

[0156] If the multimedia player is any non-custom player, then its built-in redirection client receives the position information corresponding to the playback screen provided by the player plugin embedded in the multimedia player. This position information is monitored by the multimedia player and provided to the player plugin; or...

[0157] If the multimedia player is any browser, then the redirection client within the multimedia player receives the location information corresponding to the playback screen from the browser plugin built into the multimedia player. This location information is monitored by the browser and provided to the browser plugin; or...

[0158] If the multimedia playback client is a customized redirection-specific player, then the redirection client within it receives the location information corresponding to the playback screen detected by the redirection-specific player. The redirection-specific player has pre-set program logic for location monitoring and transmission.

[0159] In an optional embodiment, processor 91 may also be used for:

[0160] Using the cloud desktop protocol server, the first instruction stream occurring in the multimedia playback client is obtained from the redirection server. The first instruction stream is obtained by the redirection client in the multimedia playback client and provided to the redirection server.

[0161] The first instruction stream is sent to the cloud desktop protocol client so that the cloud desktop protocol client can control the playback of the local player in the cloud desktop terminal according to the first instruction stream;

[0162] Receive the second instruction stream sent by the cloud desktop protocol client. The second instruction stream is detected by the cloud desktop protocol client from the local player.

[0163] Playback control is performed on the multimedia playback terminal according to the second instruction stream.

[0164] In one optional embodiment, the cloud desktop protocol server and the cloud desktop protocol client use serialization and deserialization technologies for data interaction.

[0165] In one alternative embodiment, the playback screen in the desktop image is displayed as a specified pattern, which may include a pure white pattern, a pure black pattern, or a mosaic pattern.

[0166] Furthermore, such as Figure 9 As shown, the cloud desktop server also includes other components such as communication component 92 and power supply component 93. Figure 9 The diagram only shows a portion of the components and does not imply that cloud desktop server equipment only includes... Figure 9 The components shown.

[0167] It is worth noting that the technical details of the above-mentioned embodiments of the cloud desktop server can be referred to the relevant descriptions of the cloud desktop server in the foregoing system embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0168] Figure 10 This is a schematic diagram of the structure of a cloud desktop terminal, provided as another exemplary embodiment of this application. For example... Figure 10 As shown, the cloud desktop terminal includes: a memory 10 and a processor 11.

[0169] Processor 11, coupled to memory 10, is used to execute computer programs in memory 10 for:

[0170] The cloud desktop protocol client receives undecoded media streams sent by the cloud desktop protocol server in the cloud desktop server. The undecoded media streams are received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback client according to the specified protocol. The redirection client obtains the undecoded media streams from the multimedia playback client.

[0171] Decode the undecoded media stream;

[0172] Use the local media player to play the decoded multimedia.

[0173] The multimedia playback client runs on a cloud desktop server and includes various types of players and browsers. It is compatible with various types of players and browsers by specifying protocols and supports one-to-many communication mechanisms.

[0174] In one alternative embodiment, the specified protocol adopts the WebSocket communication protocol.

[0175] In an alternative embodiment, the processor 11 may also be used for:

[0176] The playback screen of the local player is obtained through the cloud desktop protocol client;

[0177] The playback screen is cropped based on the occlusion information;

[0178] Based on the position information of the playback screen, the cropped playback screen and the desktop image are stitched together;

[0179] Display the spliced ​​image on the cloud desktop terminal;

[0180] The occlusion information is generated by the location management component in the cloud desktop protocol server based on the location information corresponding to the playback screen provided by the redirection server and the location information corresponding to the application windows of the multimedia playback terminal and other applications in the cloud desktop.

[0181] In an alternative embodiment, the processor 11 may also be used for:

[0182] The second instruction stream is sent from the cloud desktop protocol client to the cloud desktop protocol server so that the cloud desktop protocol server can control the playback of the multimedia playback terminal according to the second instruction stream.

[0183] The second instruction stream is detected by the cloud desktop protocol client from the local player.

[0184] In one optional embodiment, the cloud desktop protocol server and the cloud desktop protocol client use serialization and deserialization technologies for data interaction.

[0185] In one alternative embodiment, the playback screen in the desktop image is displayed as a specified pattern, which may include a pure white pattern, a pure black pattern, or a mosaic pattern.

[0186] Furthermore, such as Figure 10 As shown, the cloud desktop terminal also includes other components such as a communication component 12, a power supply component 13, a display component 14, and an audio component 15. Figure 10 The diagram only shows a portion of the components and does not imply that cloud desktop server equipment only includes... Figure 10 The components shown.

[0187] It is worth noting that the technical details of the above-mentioned embodiments of the cloud desktop terminal can be referred to the relevant descriptions of the cloud desktop terminal in the foregoing system embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0188] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the steps described above that can be executed by a cloud desktop server or a cloud desktop terminal.

[0189] The above Figure 9 and 10 The memory in a computer is used to store computer programs and can be configured to store various other data to support operation on a computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disks, or optical disks.

[0190] The above Figure 9 and 10 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0191] The above Figure 10The display components include a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0192] The above Figure 9 and 10 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0193] The above Figure 10 The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0194] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0199] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0200] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0201] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0202] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimedia playback system for cloud desktops, comprising a cloud desktop terminal and a cloud desktop server, wherein the cloud desktop server includes a cloud desktop protocol server and a multimedia playback terminal, the cloud desktop terminal includes a cloud desktop protocol client, and the cloud desktop protocol server includes a redirection server; The redirection server is configured to receive undecoded media streams sent by redirection clients running in a multimedia playback client according to a specified protocol. The undecoded media streams are obtained by the redirection clients from the multimedia playback client and include audio and video streams generated by the multimedia playback client through demultiplexing of the multimedia source file to be played. The redirection server and each redirection client communicate in a one-to-many manner according to the specified protocol. The cloud desktop protocol server is used to send the undecoded media stream received by the redirection server to the cloud desktop protocol client; The cloud desktop protocol client is used to decode the undecoded media stream; Use the local media player to play the decoded multimedia. The multimedia playback terminal includes various types of players and browsers, and the specified protocol is compatible with various types of players and browsers, and supports a one-to-many communication mechanism. If the multimedia playback client is any browser, the browser will preferentially call the browser plugin. The browser plugin is further configured to: if the video address captured from the current webpage does not support file download, then stop sending the multimedia source file to be played to the redirection server, and send a rollback instruction to the browser to trigger the browser to play the multimedia; the redirection server is further configured to: after receiving the multimedia source file, if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then stop the subsequent processing of the multimedia source file to be played; and send a rollback instruction to the browser to trigger the browser to play the multimedia. The cloud desktop protocol server also includes a location management component, which is used to: receive location information corresponding to the playback screen provided by the redirection server, wherein the location information corresponding to the playback screen is obtained by the redirection client in the multimedia playback client and provided to the redirection server; capture the location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop through a specified interface; and calculate the occlusion information of the playback screen based on the location information corresponding to the playback screen and the location information corresponding to each application window. The cloud desktop protocol server is also used to: send the desktop image of the cloud desktop and the position information and occlusion information corresponding to the playback screen to the cloud desktop protocol client; The cloud desktop protocol client is further configured to: acquire the playback screen of the local player; crop the playback screen according to the occlusion information; stitch the cropped playback screen and the desktop image together according to the position information of the playback screen; and display the stitched screen in the cloud desktop terminal.

2. In the system according to claim 1, the specified protocol adopts the WebSocket communication protocol.

3. The system according to claim 1, wherein the redirection client in the multimedia playback terminal is further configured to: If the multimedia playback client is any non-custom player, then the player receives an undecoded media stream provided by the player plugin built into the multimedia playback client. The undecoded media stream is the audio stream and video stream generated by the player plugin after demultiplexing the multimedia source file to be played. The player then sends the audio stream and video stream to the redirection server. or, If the multimedia playback client is any browser, then the browser plugin built into the multimedia playback client receives an undecoded media stream, which is a multimedia source file to be played downloaded by the browser plugin based on a video address captured from the current webpage; the browser plugin sends the multimedia source file to be played to the redirection server; the redirection server is further configured to: demultiplex the multimedia source file to be played sent by the redirection client in the browser to generate corresponding audio and video streams; or, If the multimedia playback terminal is a customized redirection-specific player, then the undecoded media stream provided by the redirection-specific player is received. The undecoded media stream is the audio stream and video stream generated by the redirection-specific player after demultiplexing the multimedia file to be played. The audio stream and video stream are then sent to the redirection server. The redirection-specific player has pre-set program logic for obtaining the multimedia file to be played and for demultiplexing the multimedia file to be played.

4. In the system according to claim 3, the non-customized player preferentially calls the player plugin, and the player plugin is further configured to: if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then no longer perform demultiplexing and subsequent operations; and send a rollback instruction to the non-customized player to trigger the non-customized player to play multimedia; or, The redirection-specific player is further configured to: if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the source file of the multimedia to be played, then play the multimedia to be played without performing the demultiplexing and subsequent processing.

5. The system according to claim 1, wherein the redirection client is further configured to: If the multimedia playback client is any non-customized player, then it receives the position information corresponding to the playback screen provided by the player plugin built into the multimedia playback client. This position information is monitored by the multimedia playback client and provided to the player plugin; or... If the multimedia playback client is any browser, then it receives the location information corresponding to the playback screen provided by the browser plugin built into the multimedia playback client. This location information is monitored by the browser and provided to the browser plugin; or... If the multimedia playback terminal is a customized redirection-specific player, then the position information corresponding to the playback screen detected by the redirection-specific player is received. The redirection-specific player has pre-set program logic for position monitoring and transmission.

6. The system according to claim 1, wherein the cloud desktop protocol server is further configured to: The first instruction stream occurring within the multimedia playback client is obtained from the redirection server. The first instruction stream is obtained by the redirection client in the multimedia playback client and provided to the redirection server. The first instruction stream is sent to the cloud desktop protocol client so that the cloud desktop protocol client can control the playback of the local player in the cloud desktop terminal according to the first instruction stream; Receive a second instruction stream sent by the cloud desktop protocol client, wherein the second instruction stream is detected by the cloud desktop protocol client from the local player; The multimedia playback terminal is controlled to play according to the second instruction stream.

7. In the system according to claim 6, the cloud desktop protocol server and the cloud desktop protocol client use serialization and deserialization technologies for data interaction.

8. The system according to claim 1, wherein the playback screen in the desktop image is displayed as a specified pattern, the specified pattern including a pure white pattern, a pure black pattern, or a mosaic pattern.

9. A multimedia playback method for a cloud desktop, applicable to a cloud desktop server, wherein the cloud desktop server includes a cloud desktop protocol server and a multimedia playback client, the cloud desktop protocol server including a redirection server, the method comprising: The cloud desktop protocol server includes a redirection server, which receives undecoded media streams sent by redirection clients running in a multimedia playback client according to a specified protocol. The undecoded media streams are obtained by the redirection clients from the multimedia playback client. The undecoded media streams include audio and video streams generated by the multimedia playback client through demultiplexing the multimedia source file to be played. The redirection server and each redirection client communicate in a one-to-many manner according to the specified protocol. If the multimedia playback client is any browser, the browser will preferentially call the browser plugin. The browser plugin is also used to: if the video address captured from the current webpage does not support file download, then no longer execute the operation of sending the multimedia source file to be played to the redirection server, and send a rollback instruction to the browser to trigger the browser to play the multimedia. Upon receiving the multimedia source file, if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then the subsequent processing of the multimedia source file to be played is stopped; and a rollback instruction is sent to the browser to trigger the browser to play the multimedia. The undecoded media stream received by the redirection server is sent to the cloud desktop protocol client in the cloud desktop terminal via the cloud desktop protocol server, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia. The cloud desktop protocol server also includes a location management component, which is used to: receive location information corresponding to the playback screen provided by the redirection server, wherein the location information corresponding to the playback screen is obtained by the redirection client in the multimedia playback client and provided to the redirection server; capture the location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop through a specified interface; and calculate the occlusion information of the playback screen based on the location information corresponding to the playback screen and the location information corresponding to each application window. The cloud desktop protocol server is further configured to: send the desktop image of the cloud desktop and the position information and occlusion information corresponding to the playback screen to the cloud desktop protocol client, so that the cloud desktop protocol client can obtain the playback screen of the local player; The playback screen is cropped according to the occlusion information; the cropped playback screen and the desktop image are stitched together according to the position information of the playback screen; the stitched screen is displayed on the cloud desktop terminal. The multimedia playback terminal includes various types of players and browsers, and the specified protocol is compatible with various types of players and browsers, and supports a one-to-many communication mechanism.

10. A multimedia playback method for a cloud desktop, applicable to a cloud desktop terminal, wherein the cloud desktop terminal includes a cloud desktop protocol client, the method comprising: The cloud desktop protocol client receives an undecoded media stream sent by the cloud desktop protocol server in the cloud desktop server. The undecoded media stream is received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback terminal according to a specified protocol. The redirection client obtains the undecoded media stream from the multimedia playback terminal. The undecoded media stream includes the audio stream and video stream generated by the multimedia playback terminal demultiplexing the multimedia source file to be played; the redirection server and each redirection client communicate one-to-many according to the specified protocol. Wherein, if the multimedia playback client is any browser, the browser prioritizes calling the browser plugin. The browser plugin is further configured to: if the video address captured from the current webpage does not support file download, then cease the operation of sending the multimedia source file to be played to the redirection server, and send a rollback instruction to the browser to trigger the browser to play the multimedia; the redirection server is further configured to: after receiving the multimedia source file, if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then stop the subsequent processing of the multimedia source file to be played; and send a rollback instruction to the browser to trigger the browser to play the multimedia. Decode the undecoded media stream; The system receives the desktop image of the cloud desktop and the position information and occlusion information corresponding to the playback screen sent by the cloud desktop protocol server, and obtains the playback screen of the local player; it then crops the playback screen according to the occlusion information; it stitches the cropped playback screen and the desktop image together according to the position information of the playback screen; and displays the stitched screen on the cloud desktop terminal. The location information and occlusion information are received by the location management component in the cloud desktop protocol server from the location information corresponding to the playback screen provided by the redirection server. The location information includes information obtained by the redirection client in the multimedia playback client and provided to the redirection server. The occlusion information of the playback screen is obtained by capturing the location information of the application windows of the multimedia playback client and other applications in the cloud desktop through a specified interface. The occlusion information is calculated based on the location information corresponding to the playback screen and the location information corresponding to each application window. The local player is then called to play the decoded multimedia. The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

11. A cloud desktop server, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: The cloud desktop protocol server includes a redirection server, which receives undecoded media streams sent by redirection clients running in a multimedia playback client according to a specified protocol. The undecoded media streams are obtained by the redirection clients from the multimedia playback client. The undecoded media streams include audio and video streams generated by the multimedia playback client through demultiplexing the multimedia source file to be played. The redirection server and each redirection client communicate in a one-to-many manner according to the specified protocol. in, If the multimedia playback client is any browser, the browser will preferentially call the browser plugin. The browser plugin is also used to: if the video address captured from the current webpage does not support file download, then no longer execute the operation of sending the multimedia source file to be played to the redirection server, and send a rollback instruction to the browser to trigger the browser to play the multimedia. Upon receiving the multimedia source file, if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then the subsequent processing of the multimedia source file to be played is stopped; and a rollback instruction is sent to the browser to trigger the browser to play the multimedia. The undecoded media stream received by the redirection server is sent to the cloud desktop protocol client in the cloud desktop terminal via the cloud desktop protocol server, so that the cloud desktop protocol client can decode the undecoded media stream and call the local player of the cloud desktop terminal to play the decoded multimedia. The cloud desktop protocol server also includes a location management component, which is used to: receive location information corresponding to the playback screen provided by the redirection server, wherein the location information corresponding to the playback screen is obtained by the redirection client in the multimedia playback client and provided to the redirection server; capture the location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop through a specified interface; and calculate the occlusion information of the playback screen based on the location information corresponding to the playback screen and the location information corresponding to each application window. The cloud desktop protocol server is further configured to: send the desktop image of the cloud desktop and the position information and occlusion information corresponding to the playback screen to the cloud desktop protocol client, so that the cloud desktop protocol client can obtain the playback screen of the local player; The playback screen is cropped according to the occlusion information; the cropped playback screen and the desktop image are stitched together according to the position information of the playback screen; the stitched screen is displayed on the cloud desktop terminal. The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

12. A cloud desktop terminal, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: The cloud desktop protocol client receives an undecoded media stream sent by the cloud desktop protocol server in the cloud desktop server. The undecoded media stream is received by the redirection server in the cloud desktop protocol server from the redirection client running in the multimedia playback terminal according to a specified protocol. The redirection client obtains the undecoded media stream from the multimedia playback terminal. The undecoded media stream includes the audio stream and video stream generated by the multimedia playback terminal demultiplexing the multimedia source file to be played; the redirection server and each redirection client communicate one-to-many according to the specified protocol. in, If the multimedia playback client is any browser, the browser will preferentially call the browser plugin. The browser plugin is further configured to: if the video address captured from the current webpage does not support file download, then stop sending the multimedia source file to be played to the redirection server, and send a rollback instruction to the browser to trigger the browser to play the multimedia; the redirection server is further configured to: after receiving the multimedia source file, if it is determined that the cloud desktop terminal cannot support the encoding format corresponding to the multimedia source file to be played, then stop the subsequent processing of the multimedia source file to be played; and send a rollback instruction to the browser to trigger the browser to play the multimedia. Decode the undecoded media stream; Receive the desktop image of the cloud desktop and the location information and occlusion information corresponding to the playback screen sent by the cloud desktop protocol server, and obtain the playback screen of the local player; The playback screen is cropped according to the occlusion information; the cropped playback screen and the desktop image are stitched together according to the position information of the playback screen; the stitched screen is displayed on the cloud desktop terminal. The location information and occlusion information are received by the location management component in the cloud desktop protocol server from the location information corresponding to the playback screen provided by the redirection server. The location information includes information obtained by the redirection client in the multimedia playback client and provided to the redirection server. The occlusion information of the playback screen is obtained by capturing the location information corresponding to the application windows of the multimedia playback client and other applications in the cloud desktop through a specified interface. The occlusion information is calculated based on the location information corresponding to the playback screen and the location information corresponding to each application window. Use the local media player to play the decoded multimedia. The multimedia playback client runs on the cloud desktop server and includes various types of players and browsers. The specified protocol is compatible with various types of players and browsers and supports a one-to-many communication mechanism.

13. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the multimedia playback method of the cloud desktop according to any one of claims 9 or 10.

Images