Screen extension method and device of cloud desktop, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a screen extension method and device of a cloud desktop, electronic equipment and a readable storage medium. The method comprises the following steps: a cloud desktop client receives cloud desktop data sent by a cloud desktop server, wherein the cloud desktop data comprises a first screen type, picture data and audio data; when the first screen type is the auxiliary screen device, the cloud desktop client encodes the audio data into high-level audio encoding data, and encapsulates the high-level audio encoding data and the picture data into a dynamic image expert group transport stream; and the cloud desktop client sends the dynamic image expert group transport stream to the auxiliary screen device, so that the auxiliary screen device decapsulates and decodes the dynamic image expert group transport stream and then displays the dynamic image expert group transport stream. According to the embodiment of the invention, the problem of scene limitation caused by a physical port can be solved, various devices can be connected through a Miracast protocol to serve as auxiliary screen devices, the universality of application scenes is improved, and finally, the data reliability during wireless transmission can be ensured based on coding and packaging.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method, apparatus, electronic device, and readable storage medium for extending the screen of a cloud desktop. Background Technology

[0002] Cloud desktop, also known as cloud-based desktop or Virtual Desktop Infrastructure (VDI), refers to a technology that stores desktop operating systems, applications, and user data on cloud servers and provides remote access to users via a network. Users can access cloud desktops through various terminal devices (such as computers, tablets, and smartphones) and experience an operating environment similar to a local desktop. The terminal device used by the user can be called the cloud desktop client, and the cloud server can be called the cloud desktop server.

[0003] In existing technologies, a secondary screen device can be provided to a cloud desktop client to achieve a dual-screen effect. This can be achieved through a wired connection, such as HDMI (High Definition Multimedia Interface), VGA (Video Graphics Array), or DVI (Digital Visual Interface). The cloud desktop client can send data to the secondary screen device for display, and the secondary screen device can also send commands to the cloud desktop client to upload to the cloud desktop server.

[0004] However, the above solutions have limitations in terms of application scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, electronic device, and readable storage medium for extending the screen of a cloud desktop, so as to solve the problem of limited usage scenarios. The specific technical solution is as follows: In a first aspect of this application, a screen extension method for a cloud desktop is provided, applied to a cloud desktop client. The cloud desktop client runs on a main screen device, connects to a cloud desktop server, and the cloud desktop client and the secondary screen device are directly connected via a wireless local area network and communicate via the Miracast protocol. The method includes: The system receives cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: a first screen type, screen data, and audio data. When the first screen type is a secondary screen device, the audio data is encoded into advanced audio coding data, and the advanced audio coding data and screen data are encapsulated into a Dynamic Picture Experts Group (MPET) transport stream. The MPET transport stream is then sent to the secondary screen device so that the secondary screen device can decapsulate and decode the MPET transport stream before displaying it.

[0006] Optionally, the method further includes: The system mouse is hidden on the main screen, and the second screen type is determined based on the system mouse position. The second screen type includes the main screen device or the secondary screen device. Virtual mouse information is generated based on the second screen type and the system mouse position, and the virtual mouse is displayed based on the virtual mouse information and the second screen type. When the second screen type is the secondary screen device, the virtual mouse information is sent to the secondary screen device through the mouse extension protocol of the wireless LAN display protocol.

[0007] Optionally, the second screen type is determined based on the system mouse position, including: Determine the area on the main screen where the system mouse position is located; determine the second screen type based on the area, with different second screen types corresponding to different areas.

[0008] Optionally, virtual mouse information is generated based on the second screen type and the system mouse position, including: A virtual mouse image is generated based on the second screen type and the screen size corresponding to the second screen type. When the second screen type is the main screen device, the system mouse position is used as the virtual mouse position. When the second screen type is the secondary screen device, the system mouse position is converted into the target position within the area corresponding to the secondary screen device. Based on the ratio between the area size corresponding to the secondary screen device and the secondary screen size, the target position is mapped from the area corresponding to the secondary screen to the secondary screen device to obtain the virtual mouse position.

[0009] Optionally, the method further includes: Upon receiving input, the system sends the virtual mouse information and input content to the cloud desktop server and returns the steps for the cloud desktop client to receive the cloud desktop data sent by the cloud desktop server.

[0010] Optionally, the method further includes: Upon receiving the dual-screen activation command, the client notifies the cloud desktop server to create a secondary screen and scans for secondary screen devices within the target range. Upon detecting a secondary screen device, the client sends a wireless LAN direct connection request to it. Upon receiving success messages for both secondary screen creation and connection, the client performs Miracast protocol integration with the secondary screen device. Once Miracast integration is successful, the client confirms that the wireless LAN direct connection between the cloud desktop client and the secondary screen device has been successfully established.

[0011] In a second aspect of this application, a screen extension device for a cloud desktop is also provided, applied to a cloud desktop client. The cloud desktop client runs on a main screen device, connects to a cloud desktop server, and the cloud desktop client and the secondary screen device are directly connected via a wireless local area network and communicate via the Miracast protocol. The device includes: The data receiving module is used to receive cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: first screen type, screen data and audio data.

[0012] The data processing module is used to encode audio data into advanced audio encoded data and encapsulate the advanced audio encoded data and picture data into a Dynamic Picture Experts Group transport stream when the first screen type is a secondary screen device.

[0013] The data display module is used to send the dynamic image expert group transport stream to the secondary screen device, so that the secondary screen device can decapsulate and decode the dynamic image expert group transport stream before displaying it.

[0014] Optionally, the device further includes: The mouse processing module is used to hide the system mouse on the main screen and determine the second screen type based on the system mouse position. The second screen type includes a main screen device or a secondary screen device. It generates virtual mouse information based on the second screen type and the system mouse position, and displays the virtual mouse based on the virtual mouse information and the second screen type. When the second screen type is a secondary screen device, the virtual mouse information is sent to the secondary screen device through the mouse extension protocol of the wireless LAN display protocol.

[0015] Optionally, the mouse processing module is also used for: Determine the area on the main screen where the system mouse position is located; determine the second screen type based on the area, with different second screen types corresponding to different areas.

[0016] Optionally, the mouse processing module is also used for: A virtual mouse image is generated based on the second screen type and the screen size corresponding to the second screen type. When the second screen type is the main screen device, the system mouse position is used as the virtual mouse position. When the second screen type is the secondary screen device, the system mouse position is converted into the target position within the area corresponding to the secondary screen device. Based on the ratio between the area size corresponding to the secondary screen device and the secondary screen size, the target position is mapped from the area corresponding to the secondary screen to the secondary screen device to obtain the virtual mouse position.

[0017] Optionally, the device further includes: The input processing module is used to send the virtual mouse information and input content to the cloud desktop server when it receives input content, and then return the data to the data receiving module.

[0018] Optionally, the device further includes: The connection establishment module is used to notify the cloud desktop server to create a secondary screen and scan for secondary screen devices within the target range when a dual-screen activation command is received; when a secondary screen device is detected, a wireless LAN direct connection request is sent to the secondary screen device; when a secondary screen creation success message and a connection success message are received, Miracast protocol interface is established with the secondary screen device; when the Miracast protocol interface is successful, the wireless LAN direct connection between the cloud desktop client and the secondary screen device is confirmed to be successfully established.

[0019] In a third aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0020] In a fourth aspect of this application, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0021] In a fifth aspect of this application, a computer program product comprising instructions is also provided, which, when run on a computer, causes the computer to perform the method described in the first aspect above.

[0022] This application provides a cloud desktop screen extension method, device, electronic device, and readable storage medium. The cloud desktop client receives cloud desktop data sent by the cloud desktop server. The cloud desktop data includes a first screen type, screen data, and audio data. When the first screen type is a secondary screen device, the cloud desktop client encodes the audio data into Advanced Audio Coded Data (AADC), and encapsulates the AADC and screen data into a Dynamic Picture Experts Group (MPETG) transport stream. The cloud desktop client sends the MPETG stream to the secondary screen device, allowing the secondary screen device to decapsulate and decode the MPETG stream before displaying it. This application uses a direct wireless LAN connection between the cloud desktop client and the secondary screen device to achieve communication via the Miracast protocol. The cloud desktop data sent to the secondary screen device is encoded and encapsulated. This not only solves the problem of limited scenarios due to physical ports but also allows multiple devices to be connected as secondary screen devices via the Miracast protocol, improving the breadth of application scenarios. Finally, the encoding and encapsulation ensure data reliability during wireless transmission. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a schematic diagram of a scenario according to an embodiment of this application; Figure 2 This is a flowchart illustrating the steps of a cloud desktop screen extension method according to an embodiment of this application. Figure 3 This is a schematic diagram illustrating the mapping relationship between the region and the second screen type in an embodiment of this application; Figure 4 This is a schematic diagram of the system mouse position on the main screen in an embodiment of this application; Figure 5 This is a schematic diagram of data interaction in the cloud desktop system in the embodiments of this application; Figure 6 This is a flowchart illustrating another method for extending the screen of a cloud desktop according to an embodiment of this application. Figure 7 This is a schematic diagram of the structure of a screen extension device for a cloud desktop according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of another cloud desktop screen extension device in an embodiment of this application; Figure 9 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. (Refer to...) Figure 1 As shown, this application embodiment is applied to a cloud desktop system, including a cloud desktop server, a cloud desktop client, and a secondary screen device. The cloud desktop server runs on a remote server. The cloud desktop client runs on the main screen device, which is the terminal device used by the user, and its screen is called the main screen. The secondary screen device, also known as an extended screen device, has its screen called the secondary screen and works in conjunction with the main screen device to display cloud desktop data. The secondary screen device can be any device with a screen, or simply a display screen.

[0027] The connection between the cloud desktop client and the cloud desktop server is typically a wireless network, while the connection between the cloud desktop client and the secondary screen device is wired. The cloud desktop server transmits cloud desktop data to the cloud desktop client using open-source protocols such as Spice or a self-developed protocol. The cloud desktop client receives the cloud desktop data from the transmission channel and displays the data on the main screen device when the transmission channel corresponds to the main screen device, or on the secondary screen device when the transmission channel corresponds to the secondary screen device.

[0028] Understandably, when no secondary screen device is added, there is only one transmission channel between the cloud desktop server and the cloud desktop client to transmit the cloud desktop data of the main screen device. When a secondary screen device is added, the operating system of the cloud desktop server needs to create another transmission channel for the secondary screen device to independently transmit the cloud desktop data of the secondary screen device.

[0029] However, the use cases for this method are limited. For example, the main screen device may not have a spare physical port to connect to the secondary screen device, or the physical port of the main screen device may not be able to connect to the secondary screen device, or the main screen device may be physically disconnected and require an adapter to connect to the secondary screen device. Therefore, screen extension cannot be achieved in these scenarios.

[0030] To address the aforementioned technical issues, this application employs a Wi-Fi Direct connection between the cloud desktop client and the secondary screen device. Communication between the two is achieved via the Miracast protocol, and the cloud desktop data sent to the secondary screen device is encoded and encapsulated. This not only resolves the limitation imposed by physical ports but also allows for the connection of various devices as secondary screen devices via the Miracast protocol, broadening the application scenarios. Finally, the encoding and encapsulation ensure data reliability during wireless transmission.

[0031] The cloud desktop client can be called the Miracast sender, and the secondary screen devices supported by the Miracast protocol can be called Miracast receivers, including but not limited to: TVs, projectors, smartphones, tablets and laptops, screen projectors and other devices.

[0032] The following explains some of the technical terms related to this invention: Wi-Fi Direct is a technology defined by the Wi-Fi (Wireless Fidelity) Alliance that allows Wi-Fi devices to connect directly to each other without relying on traditional home, office, or public networks; it is a peer-to-peer (P2P) connection. This technology enables mobile phones, cameras, printers, PCs (Personal Computers), and gaming devices to create their own Wi-Fi networks without an internet connection, thereby enabling content transmission or display.

[0033] Miracast is a wireless display technology developed by the Wi-Fi Alliance that allows users to wirelessly share multimedia content, including high-resolution images and high-definition (HD) video content, between different Wi-Fi devices. Miracast can function even without a Wi-Fi network.

[0034] MPEG (Moving Pictures Experts Group) 2-TS (Transport Stream): Also known as TS, it is a standard format for transmitting and storing various data containing video, audio, and communication protocols. It is used in digital television broadcasting systems such as DVB (Digital Video Broadcasting), ATSC (Advanced Television Systems Committee), ISDB (Integrated Services Digital Broadcasting), and IPTV (Internet Protocol Television).

[0035] MS-WDHCE (Wi-Fi Display Protocol: Hardware Cursor Extension) is a mouse extension protocol within the Wi-Fi display protocol.

[0036] P2P (peer-to-peer): also known as peer-to-peer network technology, is a new network technology that relies on the computing power and bandwidth of participants in the network, rather than aggregating the reliance on a small number of servers.

[0037] The screen extension method of the cloud desktop of this application will be described in detail below through specific embodiments.

[0038] Figure 2 A flowchart illustrating the steps of a cloud desktop screen extension method provided in this application embodiment is applied to... Figure 1 In the cloud desktop system shown, such as Figure 2 As shown, the method may include the following steps: S101. The cloud desktop client receives cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: first screen type, screen data, and audio data.

[0039] S102, When the cloud desktop client is a secondary screen device, it encodes the audio data into advanced audio encoded data and encapsulates the advanced audio encoded data and the picture data into a Dynamic Picture Experts Group transport stream.

[0040] S103, the cloud desktop client sends the Dynamic Image Expert Group (CAPG) transport stream to the secondary screen device, so that the secondary screen device can decapsulate and decode the CAPG transport stream before displaying it.

[0041] The aforementioned cloud desktop data is generated by the cloud desktop server. The server can capture virtual machine screen and audio data to obtain screen and audio data, which, together with the first screen type, constitute the cloud desktop data. After compression and encoding, the cloud desktop data is transmitted to the cloud desktop client via a wireless communication network. This cloud desktop data can correspond to an application; the screen data is the application's interface, and the audio data is the audio played by the application. The compression and encoding can be achieved using common video compression technologies such as H.264.

[0042] The aforementioned first screen type is used to indicate the screen corresponding to the cloud desktop data, including a main screen device or a secondary screen device. When the first screen type is a main screen device, the cloud desktop client decodes the cloud desktop data and displays it on the main screen device, including displaying screen data on the main screen and playing audio data through the audio module; when the first screen type is a secondary screen device, the above steps S102 to S103 are executed so that the secondary screen device displays the cloud desktop data, including displaying screen data on the secondary screen and playing audio data through the audio module.

[0043] Specifically, the advanced audio coding data in S102 can be implemented using AAC (Advanced Audio Coding), and the Moving Picture Experts Group transport stream can be MPEG2-TS.

[0044] In some possible implementations, in order to save transmission resources, the cloud mobile client can also detect the sound card usage status of the main screen device, the performance parameters of the wireless LAN direct connection, and the performance parameters of the main screen device, so as to determine whether to execute S201 to S202 based on the sound card usage status and these two performance parameters.

[0045] When the sound card usage status indicates that the sound card of the main screen device is idle, and at least one performance parameter is less than or equal to a preset performance threshold, only the screen data and timestamp are sent to the secondary screen device. This allows the audio data to be played on the main screen device while the secondary screen synchronously displays the screen data based on the timestamp. In this way, the cloud desktop client does not need to encode and encapsulate the audio data, reducing the processing complexity of the cloud desktop client and saving transmission resources.

[0046] When the sound card usage status indicates that the sound card of the main screen device is busy, and / or both performance parameters are greater than the preset performance threshold, the above steps S201 to S202 are executed.

[0047] As can be seen from the above, the cloud desktop client receives cloud desktop data and decides whether to display it on the main screen device or the secondary screen device. Furthermore, the cloud desktop client also needs to send mouse information to the secondary screen device to control the cloud desktop data via the mouse. In some possible implementations, the process of sending the mouse information includes: the cloud desktop client hides the system mouse on the main screen and determines the second screen type based on the system mouse position; the second screen type includes either the main screen device or the secondary screen device; the cloud desktop client generates virtual mouse information based on the second screen type and the system mouse position, and displays the virtual mouse according to the virtual mouse information and the second screen type. When the second screen type is a secondary screen device, the virtual mouse information is sent to the secondary screen device via the mouse extension protocol of the wireless LAN display protocol.

[0048] In this embodiment, the system mouse is a display object created by the operating system of the main screen device for the actual mouse on the main screen device, and the system mouse position is the position of the system mouse on the main screen. It can be understood that when the secondary screen is not extended, only the main screen exists. At this time, the cloud desktop client displays the system mouse position and controls the cloud desktop data displayed on the main screen through this system mouse position. When the secondary screen is extended, the main screen device is connected to a real mouse and needs to control the cloud desktop data on both the main and secondary screens. However, there is no secure connection between the main and secondary screen devices, so the main screen device cannot recognize the secondary screen device and therefore cannot move the system mouse from the main screen to the secondary screen. Therefore, this embodiment uses a virtual mouse to display the mouse on both screens and hides the system mouse on the main screen. That is, only the virtual mouse is displayed on the main or secondary screen, and the system mouse is not displayed.

[0049] Specifically, the aforementioned system mouse position is used to generate a virtual mouse. First, the second screen type corresponding to the virtual mouse needs to be determined, representing the screen type where the virtual mouse is displayed, including the main screen or the secondary screen. The system mouse position corresponding to the main screen and the secondary screen is different. For example, when the system mouse position is P1, the second screen type is the main screen; when the system mouse position is P2, the second screen type is the secondary screen. It can be understood that the mapping relationship between the second screen type and the system mouse position can be flexibly set and recorded in a mapping table.

[0050] In some possible implementations, to reduce complexity, the main screen can be divided into at least two regions, with the main screen and the secondary screen corresponding to different regions. In this case, determining the second screen type based on the system mouse position includes: determining the region within the main screen where the system mouse position is located, and then determining the second screen type based on that region; different second screen types correspond to different regions.

[0051] Reference Figure 3As shown, the main screen is divided into area A and area B. Area A corresponds to the main screen device, i.e., the main screen, while area B corresponds to the secondary screen device, i.e., the secondary screen. Therefore, when the system mouse is located in area A, the secondary screen type is the main screen device; when the system mouse is located in area B, the secondary screen type is the secondary screen device.

[0052] The aforementioned area has a relatively regular shape, typically rectangular like the screen itself. Therefore, when recording the mapping relationship between the second screen type and the area, only four coordinates of the area need to be recorded: the minimum x-coordinate, the maximum x-coordinate, the minimum y-coordinate, and the maximum y-coordinate. If the x-coordinate of the system mouse position is greater than or equal to the minimum x-coordinate and less than or equal to the maximum x-coordinate, and the y-coordinate of the system mouse position is greater than or equal to the minimum y-coordinate and less than or equal to the maximum y-coordinate, then the system mouse position is determined to be within this area; otherwise, the system mouse position is determined to be outside this area. In this way, the mapping relationship can be recorded using only a few simple coordinates, helping to reduce the complexity of the mapping relationship and save storage space.

[0053] In some possible implementations, the aforementioned region has the same shape as the corresponding second screen type, and both are scaled proportionally. This ensures that every point in the region can be mapped to a point in the second screen type, preventing situations where some locations in the region lack a corresponding mapped location in the second screen type, or vice versa. In other words, the ratio of the length of the region to the length of the second screen type is equal to the ratio of the width of the region to the width of the second screen type. Specifically, this can be expressed by the following formula: QYC / QYK = PMC / PMK.

[0054] Where QYC is the length of the region, QYK is the width of the region, PMC is the length of the second screen type, and PMK is the width of the second screen type. For example, refer to... Figure 3 As shown, the ratio of the length PMC1 of the main screen to the length QYC1 of the corresponding area A of the main screen is K1, and the ratio of the width PMK1 of the main screen to the width QYK1 of the corresponding area A of the main screen is also K1; the ratio of the length PMC2 of the secondary screen to the length QYC2 of the corresponding area B of the secondary screen is K2, and the ratio of the width PMK2 of the secondary screen to the width QYK2 of the corresponding area B of the secondary screen is also K2.

[0055] In some possible implementations, Figure 3The two areas defined by the main screen are not displayed on the main screen itself, thus avoiding interference with the normal display of the main screen. Furthermore, to assist users in correctly operating the mouse, a tooltip can be displayed on the main screen. When the user interacts with this tooltip, a small sub-area on the main screen displays the two main screen areas and their corresponding second screen types. This allows the user to accurately operate the mouse based on this sub-area.

[0056] Alternatively, the aforementioned prompt box, like the sub-area, can also be hidden by default and displayed when the user triggers a display command.

[0057] In some possible implementations, the cloud desktop client can also determine whether the user is continuously making mouse errors based on the distribution of the system mouse positions. If the user is continuously making mouse errors, the aforementioned sub-area will be displayed. Specifically, the system mouse positions are statistically analyzed over a period of time to obtain the number of mouse movements by the user per unit time and the distribution of the system mouse positions. If the number of mouse movements per unit time is greater than or equal to a preset threshold, and the distribution of the system mouse positions is discrete and irregular, then it is determined that the user is continuously making mouse errors.

[0058] When the aforementioned virtual mouse information includes a virtual mouse image and a virtual mouse position, after determining the second screen type based on the aforementioned area, the cloud desktop client needs to generate virtual mouse information based on the second screen type and the system mouse position. This includes: the cloud desktop client generating a virtual mouse image based on the second screen type and the screen size corresponding to the second screen type; when the second screen type is the primary screen device, the cloud desktop client using the system mouse position as the virtual mouse position; when the second screen type is the secondary screen device, the cloud desktop client converting the system mouse position into a target position within the area corresponding to the secondary screen device, and mapping the target position from the area corresponding to the secondary screen to the secondary screen device based on the proportional relationship between the area size corresponding to the secondary screen device and the secondary screen size, thus obtaining the virtual mouse position.

[0059] The aforementioned virtual mouse image is an object displayed on the screen to indicate the virtual mouse's position. The attributes of the virtual mouse image are related to the second screen type and screen size, including its size, shape, and color. For example, the size of the virtual mouse image is positively correlated with the screen size, while its shape and color are associated with the second screen type to differentiate the mouse display effect on different screens.

[0060] The virtual mouse position mentioned above refers to the display position of the virtual mouse image in the corresponding second screen type, and is associated with the second screen type. When the second screen type is the main screen, the system mouse position is directly used as the virtual mouse position to display the virtual mouse image.

[0061] When the second screen type is a secondary screen, the ratio of the area size of the secondary screen device on the main screen to the size of the secondary screen can be calculated to represent the proportional relationship between the area size of the secondary screen device on the main screen and the size of the secondary screen, also known as the scaling ratio. Therefore, the virtual mouse position can be the ratio of the target position to this scaling ratio, specifically: the horizontal coordinate of the virtual mouse position is the ratio of the horizontal coordinate of the target position to the scaling ratio, and the vertical coordinate of the virtual mouse position is the ratio of the vertical coordinate of the target position to the scaling ratio.

[0062] The target position mentioned above is the position of the system mouse within the designated area. The system mouse position, however, is the position of the system mouse on the main screen. Therefore, the system mouse position needs to be converted to the target position based on the area's position on the main screen. (Refer to...) Figure 4 As shown, region B on the main screen corresponds to the secondary screen device, and the system mouse position is located within region B. When establishing a coordinate system xoy with the main screen, the system mouse position corresponds to coordinates (a1, b1). When establishing a coordinate system uov with region B, the target position of the system mouse position within region B corresponds to coordinates (a2, b2). Here, a1 is the x-axis value, a2 is the u-axis value, b1 is the y-axis value, and b2 is the v-axis value. Therefore, the target position is obtained by subtracting the position of the secondary screen device's corresponding region B on the main screen from the system mouse position, as shown below. Figure 4 As shown, the target position (a2, b2) = system mouse position (a1, b1) - the position of area B corresponding to the secondary screen device in the main screen (a3, b3), where a2 = a1 - a3, b2 = b1 - b3.

[0063] It should be noted that when the second screen type is a secondary screen device, the virtual mouse information is sent to the secondary screen device via the Mouse Extension Protocol (MS-WDHCE) of the Wireless LAN Display Protocol. In this way, there are two independent transmission channels between the cloud desktop client and the secondary screen device, used separately for transmitting cloud desktop data and virtual mouse information. This improves the transmission efficiency of virtual mouse information, reduces transmission latency, and lowers the mouse display latency on the secondary screen.

[0064] After obtaining the aforementioned virtual mouse information, the virtual mouse is displayed according to the virtual mouse information and the second screen type. Specifically, the virtual mouse is displayed in the second screen type based on the virtual mouse information. For example, when the second screen type is the primary screen device, the virtual mouse image is displayed at the virtual mouse position on the primary screen; when the second screen type is the secondary screen device, the virtual mouse image is displayed at the virtual mouse position on both the primary and secondary screens.

[0065] In some possible implementations, users can also input content into the cloud desktop to display the input or control the cloud desktop. Specifically, when the cloud desktop client receives input content, it sends the virtual mouse information and the input content to the cloud desktop server, and returns a step whereby the cloud desktop client receives the cloud desktop data sent by the cloud desktop server. Thus, in this embodiment, input is implemented only through the main screen device. After receiving the virtual mouse information and the input content, the cloud desktop server can update the cloud desktop data according to the virtual mouse information and the input content, and then send it to the cloud desktop client, thereby realizing the updating of cloud desktop data.

[0066] In some examples, it may be sufficient to send only the virtual mouse position and input to the cloud desktop server. The cloud desktop server can then determine the user's operation command for the cloud desktop based on the virtual mouse position and input, and update the cloud desktop data accordingly.

[0067] The input can be done via keyboard, mouse, or a combination of both. The input can be content to be displayed or operation commands.

[0068] Among some possible implementations, refer to Figure 5 As shown, the process of establishing a direct wireless LAN connection between the cloud desktop client and the secondary screen device includes the following steps: When the cloud desktop client receives the dual-screen activation command, it notifies the cloud desktop server to create a secondary screen and scans for secondary screen devices (P2P devices) within the target range. Upon detecting a secondary screen device, it sends a wireless LAN direct connection request to the secondary screen device. When the cloud desktop client receives a creation success message and a connection success message when the secondary screen device successfully establishes a connection, it sends a Miracast protocol interface request to the secondary screen device. When the cloud desktop client receives a protocol interface success message from the secondary screen device, it confirms that the wireless LAN direct connection between the cloud desktop client and the secondary screen device has been successfully established.

[0069] The dual-screen activation command is a command triggered by the user on the cloud desktop client to enable the dual-screen function, which means extending the secondary screen. The dual-screen activation command can be triggered in three ways: The first triggering method involves the cloud desktop client providing a dual-screen activation control, which users interact with to initiate the dual-screen activation command. For example, users can click, long-press, or drag the dual-screen activation control.

[0070] The second triggering method involves the cloud desktop client recognizing the motion state of the main screen device. When the motion state matches the target motion state, the dual-screen activation command is triggered; otherwise, it is not. For example, the target motion state could be swaying in the target direction at the target speed.

[0071] The third trigger method can also be to trigger the dual-screen activation command through the combined operation of one or more physical buttons. For example, pressing and holding both volume buttons simultaneously.

[0072] The triggering method for the dual-screen activation command can be set in the cloud desktop client, including but not limited to: selecting one of the three triggering methods mentioned above and setting the specific triggering parameters for that method. For example, for the first triggering method, the triggering parameters include: the operation method of the dual-screen activation control, the operation duration, etc.; for the second triggering method, the triggering parameters include: the target direction, the target speed; and for the third triggering method, the triggering parameters include: physical buttons, the operation sequence, the operation duration, etc.

[0073] When the cloud desktop client determines that the above dual-screen activation command has been triggered, it needs to perform two processes: notify the cloud desktop server and connect to the secondary screen.

[0074] Specifically, the aforementioned cloud desktop client notifies the cloud desktop server to create a secondary screen, including but not limited to: creating a cloud desktop data acquisition and processing channel for the secondary screen, and creating a corresponding transmission channel for the secondary screen. The cloud desktop data acquisition and processing channel is used to acquire screen and audio data for the secondary screen, and to compress and encode the screen and audio data. The transmission channel is used to transmit the cloud desktop data for the secondary screen, and is different from the transmission channel for the cloud desktop data of the main screen. (See reference...) Figure 5 As shown, when the secondary screen is successfully created, the cloud desktop server sends a secondary screen creation success message to the cloud desktop client; when the secondary screen creation fails, it sends a secondary screen creation failure message to the cloud desktop client.

[0075] Reference Figure 5 As shown, connecting to the secondary screen involves: the cloud desktop client scanning for P2P devices within a target distance range (which can be flexibly configured) to serve as secondary screen devices; when at least one secondary screen device is detected, the user can select one of them as the secondary screen device for this use and send a Wi-Fi direct connection request to that device. Upon receiving the Wi-Fi direct connection request, the secondary screen device authenticates the request, establishes a Wi-Fi direct connection between the secondary screen device and the cloud desktop client, and sends a connection success message to the cloud desktop client. Conversely, if authentication with the cloud desktop client fails, the secondary screen device returns a connection failure message and the reason for the failure to the cloud desktop client.

[0076] In some possible implementations, when scanning for P2P devices, the cloud desktop client can filter them based on whether they have a screen or a sound card, selecting devices with both as secondary screen devices. During this process, secondary screen devices can also be sorted in descending order of screen performance parameters and sound card performance parameters, allowing users to prioritize secondary screen devices with the highest performance parameters in both areas, thereby maximizing the usability of the cloud desktop after screen expansion.

[0077] Reference Figure 5 As shown, when the cloud desktop client receives the success messages for creating and connecting the secondary screen, it initiates a Miracast protocol interface with the secondary screen device. This includes the cloud desktop client sending a Miracast protocol interface request to the secondary screen device. Upon receiving the successful interface message, the cloud desktop client confirms that a direct wireless LAN connection between the cloud desktop client and the secondary screen device has been successfully established. Afterward, the cloud desktop client can communicate with the secondary screen device through this established direct wireless LAN connection.

[0078] After sending a message indicating successful creation of the secondary screen, the cloud desktop server can collect cloud desktop data and send it to the cloud desktop client. Upon receiving the cloud desktop data, the cloud desktop client encodes and encapsulates it to obtain MPEG2-TS data. Then, it can send the MPEG2-TS data to the secondary screen device via a pre-established wireless LAN direct connection, allowing the secondary screen device to decapsulate, decode, and display the data.

[0079] It should be noted that if the cloud desktop server does not detect any P2P devices, and / or receives a secondary screen creation failure message, and / or receives a connection failure message, and / or receives a protocol integration failure message, it indicates that the extended screen has failed. The cloud desktop server can display the failure message and indicate the reason for the failure.

[0080] In summary, the embodiments of this application allow users to trigger dual-screen activation on the cloud desktop client and connect to the secondary screen device while notifying the cloud desktop server to create the secondary screen. This can minimize the latency of the extended screen and improve efficiency.

[0081] Figure 6 This is a flowchart illustrating another method for extending the screen of a cloud desktop according to an embodiment of this application. The following steps are performed by the cloud desktop client, as shown below. Figure 6 As shown, the above-mentioned cloud desktop screen extension method includes the following steps: S201. Upon receiving the dual-screen activation command, notify the cloud desktop server to create a secondary screen and scan for secondary screen devices within the target distance range.

[0082] S202. When a secondary screen device is detected, a wireless LAN direct connection request is sent to the secondary screen device.

[0083] S203. Upon receiving the success message for creating the secondary screen and the success message for connecting, perform Miracast protocol interface with the secondary screen device.

[0084] S204. Upon successful Miracast protocol integration, confirm that the wireless LAN direct connection between the cloud desktop client and the secondary screen device has been successfully established.

[0085] S205. Receive cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: first screen type, screen data, and audio data.

[0086] S206. When the first screen type is the main screen device, display the cloud desktop data on the main screen device.

[0087] S207. When the first screen type is a secondary screen device, the audio data is encoded into advanced audio encoded data, and the advanced audio encoded data and the picture data are encapsulated into a Moving Image Experts Group transport stream.

[0088] S208. Send the Dynamic Image Expert Group (MAG) transport stream to the secondary screen device so that the secondary screen device can decapsulate and decode the MAG transport stream before displaying it.

[0089] S209. Hide the system mouse on the main screen and determine the area of ​​the system mouse position on the main screen.

[0090] S210. Determine the second screen type based on the region, and generate a virtual mouse image based on the second screen type and the screen size corresponding to the second screen type. Different second screen types correspond to different regions. The second screen type includes a main screen device or a secondary screen device.

[0091] S211. When the second screen type is the main screen device, the system mouse position is used as the virtual mouse position.

[0092] S212. When the second screen type is a secondary screen device, the system mouse position is converted into a target position within the area corresponding to the secondary screen device, and the target position is mapped from the area corresponding to the secondary screen to the secondary screen device according to the proportional relationship between the area size corresponding to the secondary screen device and the secondary screen size, so as to obtain the virtual mouse position.

[0093] S213. Display the virtual mouse according to the virtual mouse information and the second screen type. When the second screen type is a secondary screen device, the virtual mouse information is sent to the secondary screen device through the mouse extension protocol of the wireless LAN display protocol.

[0094] S214. Upon receiving input content, send the virtual mouse information and input content to the cloud desktop server and return S205.

[0095] It should be noted that S201 to S214 described above can be referred to the foregoing embodiments, and will not be repeated here. In addition, the order of S201 to S214 can be flexibly adjusted on the basis of mutual independence, and the embodiments of this application do not limit the order of them.

[0096] Figure 7 This is a schematic diagram of the structure of a screen extension device for a cloud desktop according to an embodiment of this application, applied to a cloud desktop client, such as... Figure 7 As shown, the cloud desktop client runs on the main screen device, connects to the cloud desktop server, and the cloud desktop client and secondary screen device are directly connected via a wireless LAN and communicate via the Miracast protocol. The screen extension device 400 of the cloud desktop may include: The data receiving module 401 is used to receive cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: first screen type, screen data and audio data.

[0097] The data processing module 402 is used to encode audio data into advanced audio encoded data and encapsulate the advanced audio encoded data and picture data into a Dynamic Picture Experts Group transport stream when the first screen type is a secondary screen device.

[0098] The data display module 403 is used to send the dynamic image expert group transport stream to the secondary screen device so that the secondary screen device can decapsulate and decode the dynamic image expert group transport stream before displaying it.

[0099] Optionally, refer to Figure 8 As shown, the screen extension device 400 of the cloud desktop also includes: The mouse processing module 404 is used to hide the system mouse on the main screen and determine the second screen type based on the system mouse position. The second screen type includes a main screen device or a secondary screen device. It generates virtual mouse information based on the second screen type and the system mouse position, and displays the virtual mouse based on the virtual mouse information and the second screen type. When the second screen type is a secondary screen device, the virtual mouse information is sent to the secondary screen device through the mouse extension protocol of the wireless LAN display protocol.

[0100] Optionally, the mouse processing module 404 is also used for: Determine the area on the main screen where the system mouse position is located; determine the second screen type based on the area, with different second screen types corresponding to different areas.

[0101] Optionally, the mouse processing module 404 is also used for: A virtual mouse image is generated based on the second screen type and the screen size corresponding to the second screen type. When the second screen type is the main screen device, the system mouse position is used as the virtual mouse position. When the second screen type is the secondary screen device, the system mouse position is converted into the target position within the area corresponding to the secondary screen device. Based on the ratio between the area size corresponding to the secondary screen device and the secondary screen size, the target position is mapped from the area corresponding to the secondary screen to the secondary screen device to obtain the virtual mouse position.

[0102] Optionally, refer to Figure 8 As shown, the cloud desktop screen extension device 400 also includes: The input processing module 405 is used to send the virtual mouse information and input content to the cloud desktop server when it receives input content, and then return the data receiving module.

[0103] Optionally, refer to Figure 8 As shown, the cloud desktop screen extension device 400 also includes: The connection establishment module 406 is used to, upon receiving a dual-screen activation command, notify the cloud desktop server to create a secondary screen and scan for secondary screen devices within the target range; upon detecting a secondary screen device, send a wireless LAN direct connection request to the secondary screen device; upon receiving a secondary screen creation success message and a connection success message, perform Miracast protocol interfacing with the secondary screen device; and upon successful Miracast protocol interfacing, confirm that the wireless LAN direct connection between the cloud desktop client and the secondary screen device has been successfully established.

[0104] The above embodiments are device embodiments corresponding to the method embodiments. They can be referred to the description of the foregoing method embodiments and have the same beneficial effects as the method embodiments.

[0105] This application also provides an electronic device 900, such as... Figure 9 As shown, it includes a processor 9001, a communication interface 9002, a memory 9003, and a communication bus 9004. The processor 9001, communication interface 9002, and memory 9003 communicate with each other via the communication bus 9004. The memory 9003 is used to store computer programs; the processor 9001 is used to implement the screen extension method of the cloud desktop provided in the above embodiment when executing the program stored in the memory 9003.

[0106] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0107] The communication interface is used for communication between the aforementioned terminal and other devices.

[0108] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0109] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0110] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the cloud desktop screen extension methods described in the above embodiments.

[0111] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the cloud desktop screen extension methods in the above embodiments.

[0112] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0113] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] It should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0115] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0116] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for extending the screen of a cloud desktop, characterized in that, The method, applicable to a cloud desktop client running on a primary screen device and connected to a cloud desktop server, includes the following components: Receive cloud desktop data sent by the cloud desktop server, the cloud desktop data including: a first screen type, screen data and audio data; When the first screen type is the secondary screen device, the audio data is encoded into advanced audio encoded data, and the advanced audio encoded data and the screen data are encapsulated into a Moving Image Experts Group transport stream; The Dynamic Image Experts Group (DIP) transport stream is sent to the secondary screen device so that the secondary screen device can decapsulate and decode the DIP transport stream before displaying it.

2. The method according to claim 1, characterized in that, The method further includes: The system mouse is hidden on the main screen, and the second screen type is determined based on the position of the system mouse. The second screen type includes the main screen device or the secondary screen device. Virtual mouse information is generated based on the second screen type and the system mouse position, and the virtual mouse is displayed based on the virtual mouse information and the second screen type. When the second screen type is the secondary screen device, the virtual mouse information is sent to the secondary screen device through the mouse extension protocol of the wireless LAN display protocol.

3. The method according to claim 2, characterized in that, Determining the second screen type based on the system mouse position includes: Determine the area of ​​the system mouse position on the main screen; The second screen type is determined based on the region, and different second screen types correspond to different regions.

4. The method according to claim 3, characterized in that, Virtual mouse information is generated based on the second screen type and the system mouse position, including: A virtual mouse image is generated based on the second screen type and the screen size corresponding to the second screen type. When the second screen type is the main screen device, the system mouse position is used as the virtual mouse position; When the second screen type is the secondary screen device, the system mouse position is converted into a target position within the area corresponding to the secondary screen device, and the target position is mapped from the area corresponding to the secondary screen to the secondary screen device according to the ratio between the area size corresponding to the secondary screen device and the secondary screen size, so as to obtain the virtual mouse position.

5. The method according to claim 2, characterized in that, The method further includes: Upon receiving input, the virtual mouse information and the input are sent to the cloud desktop server, and the cloud desktop client is returned to receive the cloud desktop data sent by the cloud desktop server.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Upon receiving a dual-screen activation command, the cloud desktop server is notified to create a secondary screen and scan for secondary screen devices within the target distance range. When the secondary screen device is detected, a wireless LAN direct connection request is sent to the secondary screen device. Upon receiving success messages for secondary screen creation and connection, the device performs Miracast protocol interface with the secondary screen device. When the Miracast protocol connection is successful, it is confirmed that the wireless LAN direct connection between the cloud desktop client and the secondary screen device has been successfully established.

7. A screen extension device for a cloud desktop, characterized in that, An application for a cloud desktop client, wherein the cloud desktop client runs on a main screen device, the cloud desktop client connects to a cloud desktop server, and the cloud desktop client and a secondary screen device are directly connected via a wireless LAN and communicate via the Miracast protocol, the device comprising: The data receiving module is used to receive cloud desktop data sent by the cloud desktop server. The cloud desktop data includes: a first screen type, screen data, and audio data. The data processing module is used to encode the audio data into advanced audio encoded data when the first screen type is the secondary screen device, and to encapsulate the advanced audio encoded data and the screen data into a Dynamic Picture Experts Group transport stream. The data display module is used to send the dynamic image expert group transport stream to the secondary screen device, so that the secondary screen device can decapsulate and decode the dynamic image expert group transport stream and then display it.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

10. A computer program, characterized in that, The computer program is executed by a computer to implement the method as described in any one of claims 1 to 6.

Images