Display method and device, terminal, and storage medium

Abstract

The display method and device, the terminal, and the storage medium are provided. The method comprises the following steps: obtaining a cache identifier of a shared frame buffer of a virtual machine running on a terminal; in the case that a first display instruction is received, obtaining first data in the shared frame buffer corresponding to the cache identifier; wherein the first display instruction indicates that an image on a virtual display screen of the virtual machine is displayed on a display screen of the terminal; and displaying the first image on the display screen according to the first data. According to the embodiments of the present disclosure, the image on the virtual display screen of the virtual machine can be directly displayed on the display screen of the terminal.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a display method and apparatus, a terminal, and a computer-readable storage medium. Background Technology

[0002] In related technologies, virtual machines can be started on a terminal (i.e., a physical machine) using virtual machine management software (such as Virtual Machine Manager). When the virtual machine uses a virtual Graphics Processing Unit (vGPU) for acceleration, the vGPU can provide a virtual display. However, viewing the content displayed on the virtual display on the physical machine requires processes such as screen capture, encoding, and network transmission. For example, after capturing and encoding the virtual display using screen capture software like vncserver, it is sent over the network to the screen capture software client vncviewer on the physical machine for display. This method results in choppy virtual machine operation. Summary of the Invention

[0003] This disclosure provides a display method and apparatus, a terminal, and a computer-readable storage medium.

[0004] In a first aspect, this disclosure provides a display method applied to a terminal, the display method comprising: acquiring a cache identifier for indicating a shared frame buffer of a virtual machine running on the terminal; upon receiving a first display instruction, acquiring first data in the shared frame buffer corresponding to the cache identifier; wherein the first display instruction instructs the display of an image on a virtual display screen of the virtual machine on the terminal's display screen; and displaying a first image on the display screen according to the first data.

[0005] The embodiments of this disclosure can obtain a cache identifier for a shared frame buffer used to indicate a virtual machine running on a terminal, and upon receiving a first display instruction, obtain first data in the shared frame buffer corresponding to the cache identifier. Then, based on the first data, a first image is displayed on the terminal's display screen, thereby enabling the image (i.e., the first image) on the virtual display screen of the virtual machine to be displayed on the terminal's display screen (i.e., the physical display screen). This realizes the sharing of a physical display screen between the virtual machine and the terminal. Moreover, this method directly obtains the image data displayed by the virtual machine based on the cache identifier of the shared frame buffer of the virtual machine, without going through multiple processes such as screen capture, encoding, and network transmission. Therefore, it can reduce the CPU load and network load of the virtual machine, improve the processing performance of the virtual machine, and thus improve the smoothness of the virtual machine's operation.

[0006] In some possible implementations, obtaining the cache identifier for indicative of a shared frame buffer running on a virtual machine on the terminal includes: determining the physical address of a first frame buffer via the virtual machine and sending the physical address to a first driver for the graphics processor in the terminal; converting the first frame buffer corresponding to the physical address via the first driver to obtain a shared frame buffer and determining the cache identifier corresponding to the shared frame buffer; obtaining the cache identifier from the first driver via the virtual machine and sending the cache identifier to a second driver for the display screen.

[0007] The embodiments of this disclosure can convert the first frame buffer of the virtual machine into a shared frame buffer that supports cross-process access, and transmit the data through the cross-process transmission of the cache identifier corresponding to the shared frame buffer. This achieves zero-copy of the data in the first frame buffer from the virtual machine to the second driver of the terminal's display screen, thereby improving the processing efficiency of the terminal when obtaining the data in the first frame buffer.

[0008] In some possible implementations, the first frame buffer corresponding to the physical address is transformed to obtain a shared frame buffer, including: encapsulating the first frame buffer corresponding to the physical address with direct memory access to obtain a shared frame buffer.

[0009] The embodiments of this disclosure encapsulate the first frame buffer with direct memory access to obtain a shared frame buffer that supports cross-process access, which is fast and convenient, thereby improving the processing efficiency of the first driver when performing frame buffer conversion.

[0010] In some possible implementations, the physical address of the first frame buffer is determined through a virtual machine, including: obtaining the virtual address of the first frame buffer through the virtual machine; and determining the physical address corresponding to the virtual address based on the address mapping relationship.

[0011] The embodiments of this disclosure can obtain the virtual address of the first frame cache through a virtual machine, and determine the physical address corresponding to the virtual address of the first frame cache according to the address mapping relationship. This enables the virtual machine to quickly and accurately determine the physical address of the first frame cache, thereby improving the processing efficiency of the virtual machine when determining the physical address of the first frame cache.

[0012] In some possible implementations, displaying a first image on a display screen based on first data includes: during the process of displaying the first image on the display screen, adding a preset mark to the first image, wherein the preset mark is used to indicate that the first image is a display image of a virtual machine.

[0013] In embodiments of this disclosure, when a first image of a virtual machine is displayed on a terminal's screen, a preset mark is added to the first image, thereby making it easier for users to quickly and accurately identify that the image displayed on the terminal's screen is that of a virtual machine.

[0014] In some possible implementations, the display method further includes: upon receiving a second display instruction, obtaining second data from the second frame buffer of the terminal through a second driver of the display screen; wherein the second display instruction instructs the display of an image of the terminal on the display screen; and displaying the second image on the display screen according to the second data.

[0015] In the embodiments of this disclosure, upon receiving a second display instruction, the second driver of the display screen can obtain the second data in the second frame buffer of the terminal, and display the second image on the display screen according to the second data. Thus, the display image on the display screen can be switched to the image of the terminal according to the second display instruction, thereby realizing the sharing of physical display screen between the virtual machine and the terminal.

[0016] In some possible implementations, the first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key.

[0017] In the embodiments of this disclosure, the first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key, thereby enabling quick switching of the displayed image on the screen via shortcut keys, thus improving the user experience.

[0018] In a second aspect, this disclosure provides a display device applied to a terminal, the display device comprising: a graphics driver module configured to acquire a cache identifier for indicating a shared frame buffer of a virtual machine running on the terminal; and a display driver module configured to acquire first data in the shared frame buffer corresponding to the cache identifier upon receiving a first display instruction; wherein the first display instruction instructs the display of an image on a virtual display screen of the virtual machine on the terminal's display screen; and the first image is displayed on the display screen according to the first data.

[0019] In some possible implementations, the graphics driver module is specifically used to: determine the physical address of the first frame buffer through a virtual machine and send the physical address to the first driver of the graphics processor in the terminal; convert the first frame buffer corresponding to the physical address through the first driver to obtain a shared frame buffer and determine the cache identifier corresponding to the shared frame buffer; obtain the cache identifier from the first driver through a virtual machine and send the cache identifier to the second driver of the display screen.

[0020] In some possible implementations, the display driver module is specifically used to: add a preset mark to the first image during the process of displaying the first image on the display screen to indicate that the first image is a display image of the virtual machine.

[0021] In some possible implementations, the display driver module is further configured to: upon receiving a second display instruction, acquire second data from the second frame buffer of the terminal via a second driver for the display screen; wherein the second display instruction instructs the display of a second image of the terminal on the display screen; and display the second image on the display screen according to the second data.

[0022] Thirdly, this disclosure provides a terminal, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described display method.

[0023] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described display method.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of detailed exemplary embodiments with reference to the accompanying drawings, which are described below.

[0026] Figure 1 A flowchart of a display method provided in an embodiment of this disclosure.

[0027] Figure 2 This is a schematic diagram of a display method provided in an embodiment of the present disclosure.

[0028] Figure 3 This is a schematic diagram of a display method provided in an embodiment of the present disclosure.

[0029] Figure 4 This is a block diagram of a display device provided in an embodiment of the present disclosure.

[0030] Figure 5 This is a block diagram of a terminal provided in an embodiment of the present disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0033] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0035] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0036] To address the aforementioned technical problems, embodiments of this disclosure provide a display method applied to a terminal. The method includes: acquiring a cache identifier for indicating a shared frame buffer of a virtual machine running on the terminal; upon receiving a first display instruction, acquiring first data in the shared frame buffer corresponding to the cache identifier; wherein the first display instruction instructs the display of an image on a virtual display screen of the virtual machine on the terminal's display screen; and displaying a first image on the display screen according to the first data.

[0037] According to the display method of this disclosure, a cache identifier for indicating the shared frame buffer of a virtual machine running on a terminal can be obtained. Upon receiving a first display instruction, first data in the shared frame buffer corresponding to the cache identifier can be obtained. Then, based on the first data, a first image is displayed on the terminal's display screen. This enables the image (i.e., the first image) on the virtual display screen of the virtual machine to be displayed on the terminal's display screen (i.e., the physical display screen). This achieves the sharing of the physical display screen between the virtual machine and the terminal. Moreover, this method directly obtains the image data displayed by the virtual machine based on the cache identifier of the shared frame buffer of the virtual machine, without going through multiple processes such as screen capture, encoding, and network transmission. Therefore, it can reduce the CPU load and network load of the virtual machine, improve the processing performance of the virtual machine, and thus improve the smoothness of the virtual machine's operation.

[0038] The display method according to embodiments of this disclosure can be applied to a terminal. The terminal can be a computing device, a personal computer, a server, etc. The display method of embodiments of this disclosure can be implemented by a processor calling computer-readable program instructions stored in memory, or by a server executing the display method. This disclosure does not limit the specific type of terminal.

[0039] Figure 1 A flowchart illustrating a display method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes steps S11-S13, which are described in detail below.

[0040] In step S11, a cache identifier is obtained to indicate the shared frame buffer of the virtual machine running on the terminal.

[0041] In step S12, upon receiving the first display instruction, the first data in the shared frame buffer corresponding to the cache identifier is obtained.

[0042] The first display instruction instructs the display of an image from the virtual machine's virtual display screen on the terminal's display screen.

[0043] In step S13, a first image is displayed on the screen based on the first data.

[0044] In some possible implementations, the terminal includes a display screen (i.e., a physical display). The terminal also has a graphics processing unit (GPU). During the startup process, the terminal's operating system (OS) loads the first driver for the GPU and the second driver for the display screen.

[0045] The first driver includes the mdev (mediated device) module. The mdev module, or intermediate device module, is the core module used to implement hardware device virtualization. Based on the hardware virtualization capabilities of the GPU, the mdev module can divide a single GPU into multiple independent virtual GPUs (vGPUs) that can be exclusively used by different virtual instances (such as virtual machines, containers, etc.), and allocate independent memory (i.e., video memory), computing units, and other resources to each vGPU, while achieving hardware-level resource isolation.

[0046] In some possible implementations, a virtual GPU device can be created on the terminal via the mdev module in the first driver, and corresponding memory can be allocated to the virtual GPU device. Then, a virtual machine (QEMU process) is started through virtual machine management software (such as Virtual Machine Manager), and the virtual GPU device is passed through to the virtual machine. After the virtual machine starts, it loads the driver for the virtual GPU device and the driver for the virtual display to support graphics processing (such as handling Direct3D graphics applications) and the virtual display.

[0047] In some possible implementations, after running a virtual machine on a terminal, the terminal can be regarded as a physical machine or a host machine, the terminal's operating system as a Host OS, and the virtual machine's operating system as a Guest OS. The virtual machine's operating system can be different from the terminal's operating system. For example, the virtual machine's operating system can be Windows, and the terminal's operating system can be Linux. Alternatively, the virtual machine's operating system can be the same as the terminal's operating system. For example, both the virtual machine and terminal operate systems using Windows. This disclosure does not impose any limitations on this.

[0048] In some possible implementations, in step S11, a cache identifier for indicative of the shared frame buffer running on the terminal can be obtained. The shared frame buffer can be a cache area accessible to multiple processes, used to store first data of images displayed on the virtual display of the virtual machine. The cache identifier is a unique identifier for the shared frame buffer. For example, in the case of a Linux host OS, the shared frame buffer can be a Direct Memory Access Buffer (DMA-Buf), and the cache identifier can be the file descriptor (FD) of the shared frame buffer. There is a one-to-one correspondence between the cache identifier and the shared frame buffer. In practical applications, the first driver can periodically obtain the cache identifier and send it to the second driver of the terminal's display; or, the virtual machine can periodically obtain the cache identifier from the first driver through an interface provided by the first driver and forward it to the second driver of the terminal's display.

[0049] In some possible implementations, the first display instruction can be used to instruct the display of an image from the virtual display screen of the virtual machine on the terminal's display screen. Upon receiving the first display instruction, the second driver of the terminal's display screen can, in step S12, retrieve the first data from the shared frame buffer corresponding to the cache identifier. The second driver can determine the shared frame buffer corresponding to the cache identifier based on the cache identifier, and then read the first data of the image to be displayed on the virtual display screen of the virtual machine from the shared frame buffer. Then, in step S13, based on the first data, the first image, which is the image displayed on the virtual display screen of the virtual machine, is displayed on the terminal's display screen (i.e., the physical display).

[0050] According to the display method of this disclosure, a cache identifier for indicating the shared frame buffer of a virtual machine running on a terminal can be obtained. Upon receiving a first display instruction, first data in the shared frame buffer corresponding to the cache identifier can be obtained. Then, based on the first data, a first image is displayed on the terminal's display screen. This enables the image (i.e., the first image) on the virtual display screen of the virtual machine to be displayed on the terminal's display screen (i.e., the physical display screen). This achieves the sharing of the physical display screen between the virtual machine and the terminal. Moreover, this method directly obtains the image data displayed by the virtual machine based on the cache identifier of the shared frame buffer of the virtual machine, without going through multiple processes such as screen capture, encoding, and network transmission. Therefore, it can reduce the CPU load and network load of the virtual machine, improve the processing performance of the virtual machine, and thus improve the smoothness of the virtual machine's operation.

[0051] The display method according to the embodiments of this disclosure will now be described in detail.

[0052] In some possible implementations, step S11 may include: determining the physical address of the first frame buffer through a virtual machine and sending the physical address to the first driver of the graphics processor in the terminal; converting the first frame buffer corresponding to the physical address through the first driver to obtain a shared frame buffer and determining the cache identifier corresponding to the shared frame buffer; obtaining the cache identifier from the first driver through a virtual machine and sending the cache identifier to the second driver of the display screen.

[0053] After the virtual machine on the terminal starts, it determines the physical address of the first framebuffer in the terminal, which is used to store the first data of the image displayed on the virtual screen. It then sends this physical address to the first driver of the graphics processor in the terminal. Specifically, the virtual machine can send the physical address of the first framebuffer to the driver of the virtual screen; after receiving the physical address, the driver of the virtual screen sends it to the first driver of the graphics processor in the terminal. For example, if the guest OS of the virtual machine is a Windows operating system, the guest OS sends the physical address of the first framebuffer to the display driver (WDDM Display Driver) of the Windows Display Driver Model (WDDM); after receiving the physical address of the first framebuffer, the display driver of the Windows Display Driver Model (WDDM) can send the physical address of the first framebuffer to the mdev module of the first driver of the host OS.

[0054] After receiving a physical address, the first driver of the graphics processor in the terminal determines the first frame buffer corresponding to that physical address, converts the first frame buffer to obtain a shared frame buffer accessible to multiple processes, and then determines the cache identifier corresponding to the shared frame buffer. Subsequently, the virtual machine can obtain this cache identifier from the first driver through the interface provided by the first driver and forward it to the second driver of the terminal's display. For example, after receiving a physical address, the mdev module of the first driver in the terminal determines the first frame buffer corresponding to that physical address, converts the first frame buffer to obtain a shared frame buffer accessible to multiple processes, and then the virtual machine can obtain the cache identifier of the shared frame buffer from the mdev module through the query graphics plane information interface (VFIO_DEVICE_QUERY_GFX_PLANE) and the obtain graphics DMA buffer interface (VFIO_DEVICE_GET_GFX_DMABUF) provided by the vfio-mdev framework of the mdev module, and send the cache identifier to the second driver of the terminal's display. Here, the vfio-mdev framework refers to an intermediate device virtualization framework based on VFIO (Virtual Function I / O).

[0055] In the embodiments of this disclosure, the physical address of the first frame buffer can be determined by a virtual machine and sent to the first driver of the graphics processor in the terminal. Then, the first driver converts the first frame buffer corresponding to the physical address to obtain a shared frame buffer and determines the cache identifier corresponding to the shared frame buffer. The virtual machine then retrieves the cache identifier from the first driver and sends it to the second driver of the display screen. In this way, the first frame buffer of the virtual machine can be converted into a shared frame buffer that supports cross-process access, and cross-process transmission of the cache identifier corresponding to the shared frame buffer achieves zero-copy data transfer from the virtual machine to the second driver of the terminal's display screen, thereby improving the processing efficiency of the terminal when retrieving data from the first frame buffer.

[0056] In some possible implementations, the physical address of the first frame buffer is determined by the virtual machine, including: obtaining the virtual address of the first frame buffer through the virtual machine; and determining the physical address corresponding to the virtual address based on the address mapping relationship.

[0057] After the virtual machine starts, the allocation handle for the first frame buffer can be saved in the virtual GPU device driver (e.g., the vGPU kernel-mode driver). Then, by trapping into the host OS, the physical address (Guest Physical Address, GPA) of the first frame buffer in the virtual machine is translated to the physical address (Host Physical Address, HPA) of the first frame buffer in the terminal. Here, the physical address of the first frame buffer in the virtual machine can be considered as the virtual address of the first frame buffer; the physical address of the first frame buffer in the terminal can be considered as the physical address of the first frame buffer. Afterwards, an address mapping relationship between the virtual address and the physical address of the first frame buffer can be established. The target location of the first frame buffer in the virtual GPU page table can be determined using the allocation handle of the first frame buffer, and the address mapping relationship between the virtual address and the physical address of the first frame buffer can be updated to the target location.

[0058] When determining the physical address of the first frame buffer using a virtual machine, the virtual address of the first frame buffer can first be obtained from the virtual GPU device driver. Then, based on the address mapping relationship in the virtual GPU page table, the physical address corresponding to the virtual address of the first frame buffer (i.e., the physical address of the first frame buffer) can be determined. For example, the Windows virtual machine can obtain the GPA (Gross Address Address) of the first frame buffer (used to store the first data of the image displayed on the virtual screen) from the GPU Driver of the Windows display driver module; then, it passes the virtual address of the first frame buffer to the display driver of the Windows display driver module. The display driver of the Windows display driver module converts the GPA into the physical address HPA of the first frame buffer in the terminal according to the address mapping relationship in the virtual GPU page table. This HPA is the physical address of the first frame buffer.

[0059] In the embodiments of this disclosure, the virtual address of the first frame cache can be obtained through the virtual machine, and the physical address corresponding to the virtual address of the first frame cache can be determined according to the address mapping relationship. This enables the virtual machine to quickly and accurately determine the physical address of the first frame cache, thereby improving the processing efficiency of the virtual machine in determining the physical address of the first frame cache.

[0060] In some possible implementations, the first frame buffer corresponding to the physical address is transformed to obtain a shared frame buffer, including: encapsulating the first frame buffer corresponding to the physical address with direct memory access to obtain a shared frame buffer.

[0061] When converting the first frame buffer corresponding to a physical address into a shared frame buffer, the first frame buffer corresponding to the physical address can be encapsulated using Direct Memory Access (DMA) to obtain the shared frame buffer. For example, when the terminal's operating system is Linux, the first driver of the graphics processor in the terminal can encapsulate the first frame buffer into a DMA buffer through the relevant interface of the DMA buffer and identify this DMA buffer as a shared frame buffer.

[0062] In the embodiments of this disclosure, a shared frame buffer that supports cross-process access is obtained by encapsulating the first frame buffer with direct memory access, which is fast and convenient, thereby improving the processing efficiency of the first driver when performing frame buffer conversion.

[0063] In some possible implementations, step S13 may include: during the process of displaying the first image on the display screen, adding a preset mark to the first image, wherein the preset mark is used to indicate that the first image is a display image of a virtual machine.

[0064] When displaying the first image of a virtual machine on the terminal's screen, a preset marker, such as a virtual machine watermark, can be added to the first image. This preset marker can be used to indicate that the first image is a virtual machine display image, making it easy for users to quickly and accurately identify that the image displayed on the terminal's screen is a virtual machine image.

[0065] In some possible implementations, the display method of this disclosure embodiment further includes: upon receiving a second display instruction, acquiring second data in the second frame buffer of the terminal through a second driver of the display screen; wherein the second display instruction instructs to display an image of the terminal on the display screen; and displaying a second image on the display screen according to the second data.

[0066] The second display instruction can be used to instruct the display of an image from the terminal on the screen. Upon receiving the second display instruction, the second driver of the terminal's display can retrieve second data from the terminal's second frame buffer. This second data is the image data to be displayed by the terminal, i.e., the physical image data, which is stored in the second frame buffer. Then, based on the second data, the second image can be displayed on the terminal's screen.

[0067] In the embodiments of this disclosure, upon receiving a second display instruction, the second data in the second frame buffer of the terminal can be obtained through the second driver of the display screen, and a second image can be displayed on the display screen according to the second data. Thus, the display image on the display screen can be switched to the image of the terminal according to the second display instruction, thereby realizing the sharing of physical display screen between the virtual machine and the terminal.

[0068] In some possible implementations, the first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key. The first shortcut key and the second shortcut key can be different. For example, the first shortcut key is F1, and the second shortcut key is F2. When the first shortcut key F1 is pressed, the first display command is sent, and the image of the virtual machine is displayed on the terminal's screen. When the second shortcut key F2 is pressed, the second display command is sent, and the image of the terminal itself is displayed on the terminal's screen.

[0069] The first shortcut key can also be the same as the second shortcut key. For example, both the first and second shortcut keys can be F1. When the virtual machine's image is displayed on the screen, pressing the F1 key is considered as pressing the second shortcut key, sending a second display command, and displaying the terminal's own image on the screen. When the terminal's own image is displayed on the screen, pressing the F1 key is considered as pressing the first shortcut key, sending a first display command, and displaying the virtual machine's image on the screen.

[0070] In the embodiments of this disclosure, the first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key, thereby enabling quick switching of the displayed image on the screen via shortcut keys, thus improving the user experience.

[0071] Figure 2 This is a schematic diagram illustrating a display method provided in an embodiment of this disclosure. (Refer to...) Figure 2 This display method is applied to a terminal, on which a virtual machine runs. The display method includes steps S201-S208, which are described in detail below.

[0072] In step S201, the virtual address of the first frame cache is obtained through the virtual machine, and the physical address corresponding to the virtual address is determined according to the address mapping relationship.

[0073] In step S202, the physical address is sent to the first driver of the graphics processor in the terminal.

[0074] In step S203, the first driver of the graphics processor performs direct memory access encapsulation on the first frame buffer corresponding to the physical address to obtain the shared frame buffer, and determines the cache identifier corresponding to the shared frame buffer.

[0075] In step S204, the cache identifier is obtained from the first driver through the virtual machine and sent to the second driver of the display screen.

[0076] In step S205, upon receiving a first display instruction, the first data in the shared frame buffer corresponding to the cache identifier is retrieved via the second driver. The first display instruction instructs the display of an image from the virtual machine's virtual display screen on the terminal's screen.

[0077] In step S206, based on the first data, a first image is displayed on the terminal's screen, and a virtual machine watermark is added to the first image.

[0078] In step S207, upon receiving a second display instruction, second data from the terminal's second frame buffer is obtained via a second driver. The second display instruction instructs the display of an image of the terminal on the screen.

[0079] In step S208, a second image is displayed on the terminal's screen based on the second data.

[0080] Figure 3 This is a schematic diagram of a display method provided in an embodiment of the present disclosure. The display method is applied to a terminal 300. The terminal 300 includes a graphics processor 310, a display screen 320, a first driver 330 for the graphics processor 310, a second driver 340 for the display screen 320, and a virtual machine 350. The graphics processor 310 and the display screen 320 are the hardware of the terminal 300; the first driver 330 for the graphics processor GPU 310 and the second driver 340 for the display screen 320 run in the kernel space of the operating system of the terminal 300; the virtual machine 350 runs in the user space of the operating system of the terminal 300. The operating system of the terminal 300 is a Linux operating system, and the operating system of the virtual machine 350 is a Windows operating system.

[0081] The following will combine Figure 3 The processing procedure of the display method according to the present disclosure will be described by way of example. The specific processing procedure of the display method includes steps (1) to (10).

[0082] In step (1), the operating system of terminal 300 loads the first driver 330 of graphics processor 310 and the second driver 340 of display screen 320 during the startup process. The first driver 330 includes an intermediate device module (i.e., the mdev module).

[0083] In step (2), a virtual GPU device is created on the terminal 300 through the mdev module in the first driver 330, and corresponding memory is allocated to the virtual GPU device.

[0084] In step (3), virtual machine 350 is started through virtual machine management software, and the virtual GPU device is passed through to virtual machine 350.

[0085] In step (4), after the virtual machine 350 starts, it loads the drivers for the virtual GPU device and the virtual display to support graphics processing and the virtual display.

[0086] In step (5), after the virtual machine 350 enters the Windows operating system desktop, it saves the allocation handle of the first frame buffer (used to store the first data of the image displayed on the virtual screen) in the kernel-mode driver of the virtual GPU device. Then, by trapping into the Host OS, it translates the physical address of the first frame buffer in the virtual machine into the physical address of the first frame buffer in the terminal. Here, the physical address GPA of the first frame buffer in the virtual machine can be regarded as the virtual address of the first frame buffer; the physical address HPA of the first frame buffer in the terminal can be regarded as the physical address of the first frame buffer. Then, the address mapping relationship between the virtual address and the physical address of the first frame buffer is established, and the target location of the first frame buffer in the vGPU page table is determined through the allocation handle of the first frame buffer. Then, the address mapping relationship between the virtual address and the physical address of the first frame buffer is updated to the target location.

[0087] In step (6), when the virtual machine 350 displays an image on the virtual display screen, the Windows operating system passes the GPA of the first frame cache to the display driver of the Windows display driver module (i.e., the driver for the virtual display screen). The display driver of the Windows display driver module converts the GPA into the physical address HPA of the first frame cache in the terminal according to the address mapping relationship in the virtual GPU page table. This HPA is the physical address of the first frame cache. Then, the display driver of the Windows display driver module passes the physical address of the first frame cache and relevant information about the image to be displayed (such as resolution) to the mdev module in the first driver 330 of the terminal 300, thereby completing the display delivery process of the virtual machine 350.

[0088] In step (7), after the mdev module in the first driver 330 receives the physical address of the first frame buffer and the relevant information of the image to be displayed (such as resolution), it can determine the first frame buffer based on the physical address, and encapsulate the first frame buffer into a shared frame buffer through the relevant interface for direct memory access to the buffer, and determine the cache identifier of the shared frame buffer.

[0089] In step (8), the virtual machine 350 can obtain the cache identifier of the shared frame buffer from the mdev module through the query graphics plane information interface (VFIO_DEVICE_QUERY_GFX_PLANE) and the obtain graphics DMA buffer interface (VFIO_DEVICE_GET_GFX_DMABUF) provided by the vfio-mdev framework of the mdev module.

[0090] In step (9), after the virtual machine 350 obtains the cache identifier, it sends the cache identifier to the second driver 340 of the display screen 320 of the terminal 300.

[0091] In step (10), upon receiving the first display instruction, the second driver 340 of the display screen 320 can obtain the first data in the shared frame buffer corresponding to the cache identifier, and display the first image on the display screen 320 according to the first data. During the process of displaying the first image on the display screen 320, a virtual machine watermark mark can be added to the first image to indicate that the first image is a display image of the virtual machine.

[0092] According to the display method of this disclosure, a cache identifier for indicating the shared frame buffer of a virtual machine running on a terminal can be obtained. Upon receiving a first display instruction, first data in the shared frame buffer corresponding to the cache identifier can be obtained. Then, based on the first data, a first image is displayed on the terminal's display screen. This enables the image (i.e., the first image) on the virtual display screen of the virtual machine to be displayed on the terminal's display screen (i.e., the physical display screen). This achieves the sharing of the physical display screen between the virtual machine and the terminal. Moreover, this method directly obtains the image data displayed by the virtual machine based on the cache identifier of the shared frame buffer of the virtual machine, without going through multiple processes such as screen capture, encoding, and network transmission. Therefore, it can reduce the CPU load and network load of the virtual machine, improve the processing performance of the virtual machine, and thus improve the smoothness of the virtual machine's operation.

[0093] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0094] In addition, this disclosure also provides a display device, a terminal, and a computer-readable storage medium, all of which can be used to implement any of the display methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0095] Figure 4 This is a block diagram of a display device provided in an embodiment of the present disclosure.

[0096] Reference Figure 4 This disclosure provides a display device for use in a terminal, which includes a graphics driver module 41 and a display driver module 42.

[0097] The graphics driver module 41 is used to obtain a cache identifier for indicating the shared frame buffer of the virtual machine running on the terminal.

[0098] The display driver module 42 is used to obtain first data in the shared frame buffer corresponding to the cache identifier when a first display instruction is received; wherein the first display instruction instructs to display an image on the virtual display screen of the virtual machine on the terminal's display screen; and to display the first image on the display screen according to the first data.

[0099] In some possible implementations, the graphics driver module 41 is specifically used to: determine the physical address of the first frame buffer through a virtual machine and send the physical address to the first driver of the graphics processor in the terminal; convert the first frame buffer corresponding to the physical address through the first driver to obtain a shared frame buffer and determine the cache identifier corresponding to the shared frame buffer; obtain the cache identifier from the first driver through a virtual machine and send the cache identifier to the second driver of the display screen.

[0100] In some possible implementations, the graphics driver module 41 is specifically used to: encapsulate the first frame buffer corresponding to the physical address into a direct memory access buffer to obtain a shared frame buffer.

[0101] In some possible implementations, the graphics driver module 41 is specifically used to: obtain the virtual address of the first frame buffer through the virtual machine; and determine the physical address corresponding to the virtual address based on the address mapping relationship.

[0102] In some possible implementations, the display driver module 42 is specifically used to: add a preset mark to the first image during the process of displaying the first image on the display screen, wherein the preset mark is used to indicate that the first image is a display image of a virtual machine.

[0103] In some possible implementations, the display driver module 42 is further configured to: upon receiving a second display instruction, acquire second data from the second frame buffer of the terminal through a second driver program of the display screen; wherein the second display instruction instructs the display of a second image of the terminal on the display screen; and display the second image on the display screen according to the second data.

[0104] In some possible implementations, the first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key.

[0105] Figure 5 This is a block diagram of a terminal provided in an embodiment of the present disclosure.

[0106] Reference Figure 5This disclosure provides a terminal, the electronic device including: at least one processor 701; at least one memory 702, and one or more I / O interfaces 703 connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by at least one processor 701, and the one or more computer programs are executed by at least one processor 701 to enable at least one processor 701 to perform the above-described display method.

[0107] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the above-described display method. The computer-readable storage medium may be volatile or non-volatile.

[0108] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0109] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0110] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0111] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0112] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0114] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A display method, characterized in that, Applied to a terminal, the method includes: Obtain a cache identifier for indicating the shared frame buffer of the virtual machine running on the terminal; Upon receiving a first display instruction, the system retrieves first data from the shared frame buffer corresponding to the cache identifier; wherein the first display instruction instructs the system to display an image from the virtual display screen of the virtual machine on the display screen of the terminal. Based on the first data, a first image is displayed on the display screen.

2. The method according to claim 1, characterized in that, The step of obtaining the cache identifier used to indicate the shared frame buffer of the virtual machine running on the terminal includes: The physical address of the first frame buffer is determined using the virtual machine, and the physical address is sent to the first driver of the graphics processor in the terminal. The first driver program converts the first frame buffer corresponding to the physical address to obtain a shared frame buffer and determines the cache identifier corresponding to the shared frame buffer. The cache identifier is obtained from the first driver through the virtual machine, and the cache identifier is sent to the second driver of the display screen.

3. The method according to claim 2, characterized in that, The step of converting the first frame buffer corresponding to the physical address to obtain a shared frame buffer includes: The first frame buffer corresponding to the physical address is encapsulated by direct memory access to obtain a shared frame buffer.

4. The method according to claim 2, characterized in that, Determining the physical address of the first frame cache through the virtual machine includes: The virtual address of the first frame cache is obtained through the virtual machine; Based on the address mapping relationship, determine the physical address corresponding to the virtual address.

5. The method according to claim 1, characterized in that, The step of displaying the first image on the display screen based on the first data includes: During the process of displaying the first image on the display screen, a preset mark is added to the first image, wherein the preset mark is used to indicate that the first image is the display image of the virtual machine.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Upon receiving a second display instruction, the second data in the second frame buffer of the terminal is obtained through the second driver of the display screen; wherein the second display instruction instructs the display of an image of the terminal on the display screen; Based on the second data, a second image is displayed on the display screen.

7. The method according to claim 6, characterized in that, The first display command is sent via a first shortcut key, and the second display command is sent via a second shortcut key.

8. A display device, characterized in that, Applied to terminals, including: A graphics driver module is used to obtain a cache identifier for indicating a shared frame buffer of a virtual machine running on the terminal; The display driver module is configured to, upon receiving a first display instruction, acquire first data in the shared frame buffer corresponding to the cache identifier; wherein the first display instruction instructs the display screen of the terminal to display an image on the virtual display screen of the virtual machine; and display the first image on the display screen according to the first data.

9. The apparatus according to claim 8, characterized in that, The graphics driver module is specifically used for: The physical address of the first frame buffer is determined using the virtual machine, and the physical address is sent to the first driver of the graphics processor in the terminal. The first driver program converts the first frame buffer corresponding to the physical address to obtain a shared frame buffer and determines the cache identifier corresponding to the shared frame buffer. The cache identifier is obtained from the first driver through the virtual machine, and the cache identifier is sent to the second driver of the display screen.

10. The apparatus according to claim 8, characterized in that, The display driver module is specifically used for: During the process of displaying the first image on the display screen, a preset mark is added to the first image to indicate that the first image is the display image of the virtual machine.

11. The apparatus according to any one of claims 8-10, characterized in that, The display driver module is also used for: Upon receiving a second display instruction, the second data in the second frame buffer of the terminal is obtained through the second driver of the display screen; wherein the second display instruction instructs the display screen to display a second image of the terminal; and the second image is displayed on the display screen according to the second data.

12. A terminal, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to enable the at least one processor to perform the display method as described in any one of claims 1-7.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the display method as described in any one of claims 1-7.

Images