Virtualized screen projection method, hardware equipment for virtualized screen projection and storage medium

Abstract

The invention provides a virtualized screen projection method, hardware equipment for virtualized screen projection and a storage medium, a privileged virtual machine can firstly create a display virtual window corresponding to a client virtual machine, display the display virtual window in a preset screen corresponding to the client virtual machine, and create a transparent virtual window located below the display virtual window. The method comprises the following steps: acquiring a corresponding video memory physical address, further responding to a rendering instruction issued by a client virtual machine, rendering a display image which is the same as a display virtual window for a transparent virtual window, so that the display image is stored in a video memory area corresponding to the video memory physical address, and responding to a display sending instruction issued by the client virtual machine. According to the method, the display image is acquired from the video memory area, and the display image is sent to the projection equipment for projection display, so that the content displayed in the screen of the client virtual machine is projected, the method can flexibly adapt to the projection equipment with different resolutions and hardware interfaces, and each frame of projected image can be ensured not to be distorted.

Description

Technical Field

[0001] The present application relates to the field of projection technology, and in particular to a virtual screen projection method, a hardware device for virtual screen projection, and a storage medium. Background Art

[0002] Currently, hardware virtualization technology allows multiple operating systems or virtual machines to run simultaneously on the same hardware platform, using a hypervisor to create a secure and independent software runtime environment on the underlying hardware. With the hypervisor, multiple operating systems can be accessed from the vehicle computer without integrating two or more sets of hardware devices.

[0003] However, existing methods for projecting content on a single screen typically require the introduction of a display processing unit (DPU), which then adds an interface to connect to the projection device. This approach is costly and difficult to flexibly adapt to different interfaces or different types of projection devices. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a virtualized screen projection method, a virtualized screen projection hardware device and a storage medium to solve the problems in the prior art of high projection cost and difficulty in flexibly adapting to different interfaces or different types of projection devices.

[0005] An embodiment of the present application provides a virtualized screen projection method, which is executed by a privileged virtual machine in a hardware device, wherein the hardware device also includes at least one client virtual machine. The method includes:

[0006] Creating a display virtual window corresponding to the client virtual machine, and displaying the display virtual window in a preset screen corresponding to the client virtual machine;

[0007] Creating a transparent virtual window located below the display virtual window, and obtaining a video memory physical address corresponding to the transparent virtual window;

[0008] In response to a rendering instruction issued by the client virtual machine, rendering a display image identical to that of the display virtual window for the transparent virtual window, so that the display image is stored in a video memory area corresponding to the video memory physical address;

[0009] In response to the display sending instruction issued by the client virtual machine, the display image is obtained from the video memory area corresponding to the video memory physical address, and the display image is sent to the projection device for projection display.

[0010] Optionally, the step of creating a display virtual window corresponding to the client virtual machine includes:

[0011] Obtain the display port number of the preset screen corresponding to the customer virtual machine, and determine the resolution of the preset screen based on the display port number;

[0012] Create a display virtual window corresponding to the customer virtual machine based on the resolution of the preset screen.

[0013] Optionally, creating a transparent virtual window located below the display virtual window includes:

[0014] Determine the resolution of the projection device based on the configuration file, and create a virtual window according to the resolution of the projection device;

[0015] Set the virtual window to be transparently displayed to obtain a transparent virtual window, and place the transparent virtual window below the display virtual window.

[0016] Optionally, in response to the rendering instruction issued by the customer virtual machine, rendering the same display image as the display virtual window for the transparent virtual window includes:

[0017] In response to the rendering instruction issued by the customer virtual machine, call the virtualized graphics processing dynamic library to parse the rendering instruction;

[0018] After the parsing is completed, submit a graphics processing instruction to the graphics processor through the interface in the rendering management library, so that the graphics processor performs rendering synthesis according to the display image of the display virtual window, and makes the transparent virtual window display the same display image as the display virtual window.

[0019] Optionally, in response to the display instruction issued by the customer virtual machine, obtain the display image from the video memory area corresponding to the physical address of the video memory, including:

[0020] In response to the display instruction issued by the customer virtual machine, call the interface in the graphics service subsystem to enable the graphics service subsystem to start direct memory access, and read the display image from the video memory area corresponding to the transparent virtual window according to the physical address of the video memory.

[0021] Optionally, the rendering instruction and the display instruction are generated for the graphics service that calls the virtual graphics device interface of the customer virtual machine.

[0022] Optionally, before creating the display virtual window corresponding to the customer virtual machine, the method further includes:

[0023] Load the driver of the graphics processor and start the graphics service subsystem;

[0024] Start the customer virtual machine so that the customer virtual machine loads the driver of the virtual graphics device interface and starts the graphics service.

[0025] The embodiment of the present application further provides a hardware device for virtualized screen projection. The hardware device includes a privileged virtual machine and at least one guest virtual machine, where;

[0026] The guest virtual machine is used to send a rendering instruction and a display sending instruction to the privileged virtual machine;

[0027] The privileged virtual machine is used to create a display virtual window corresponding to the guest virtual machine, display the display virtual window within a preset screen corresponding to the guest virtual machine, create a transparent virtual window below the display virtual window, obtain the physical memory address corresponding to the transparent virtual window, and in response to the rendering instruction sent by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the memory area corresponding to the physical memory address. In response to the display sending instruction sent by the guest virtual machine, obtain the display image from the memory area corresponding to the physical memory address, and send the display image to a projection device for projection display.

[0028] Optionally, the privileged virtual machine includes a device model and a graphics service subsystem, where;

[0029] The device model is used to call the graphics service subsystem;

[0030] The graphics service subsystem is used to create a display virtual window corresponding to the guest virtual machine, create a transparent virtual window below the display virtual window, and read the display image from the memory area corresponding to the transparent virtual window according to the physical memory address.

[0031] The embodiment of the present application further provides an electronic device, which includes:

[0032] A processor and a memory;

[0033] The processor is used to execute the steps of the virtualized screen projection method provided in any embodiment of the present application by calling a program or instruction stored in the memory.

[0034] The embodiment of the present application further provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the virtualized screen projection method provided in any embodiment of the present application.

[0035] In summary, the present application proposes a virtualized screen projection method, which is executed by a privileged virtual machine in a hardware device. The privileged virtual machine can first create a display virtual window corresponding to a guest virtual machine, display the display virtual window within a preset screen corresponding to the guest virtual machine, and create a transparent virtual window located below the display virtual window, obtain the physical memory address corresponding to the transparent virtual window, and then in response to a rendering instruction issued by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the memory area corresponding to the physical memory address, and in response to a display sending instruction issued by the guest virtual machine, obtain the display image from the display area corresponding to the physical memory address, and send the display image to a projection device for projection display. This method realizes the projection of the content displayed by the guest virtual machine on the screen through the set transparent virtual window and the corresponding physical memory address. Since the transparent virtual window is located below the display virtual window and is transparently displayed, it does not affect the display of the display virtual window on the preset screen, avoiding the projection from affecting the content displayed by the guest virtual machine on the screen. Moreover, the virtual screen projection implemented by this method can flexibly adapt to projection devices with different resolutions and hardware interfaces, without adding new interfaces for the projection device, and can also ensure that each frame of the projected image is not distorted, improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Figure 1 is a flowchart of a virtualized screen projection method provided by an embodiment of the present application;

[0038] Figure 2 is a flowchart of screen projection based on virtualization technology provided by an embodiment of the present application;

[0039] Figure 3 is a system block diagram of a hardware device provided by an embodiment of the present application;

[0040] Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are merely used to explain the related invention and do not limit the invention. Additionally, it should be noted that for the convenience of description, only the parts related to the invention are shown in the drawings.

[0042] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0043] As mentioned in the background art, in view of the problems in the prior art, the present application proposes a virtualized screen projection method. This method can be applicable to project the content displayed within the screen of the guest virtual machine in the in-vehicle computer, or can be applicable to project the content displayed within the screen of the guest virtual machine in the computer, etc.

[0044] The virtualized screen projection method provided by the embodiments of the present application can be executed by a privileged virtual machine in a hardware device, and the hardware device further includes at least one guest virtual machine; the hardware device can be an electronic device capable of implementing hardware virtualization such as an in-vehicle computer, a computer, etc. Among them, a guest virtual machine (Guest Virtual Machine) is a virtual machine running in the virtualized environment of a hardware device. A hardware device can create multiple mutually isolated virtual computer environments through virtualization software, and these virtual computer environments can be called guest virtual machines. They share the hardware resources of the host machine, such as memory, storage, and network, etc. Each guest virtual machine behaves like an independent physical computer at the operating system and application program levels. A privileged virtual machine (Privilege Virtual Machine) is a virtual machine with special permissions in the virtualized environment of a hardware device and can execute some operations crucial for system resource management and security control, etc.

[0045] Figure 1 is a flowchart of a virtualized screen projection method provided by the embodiments of the present application. Refer to Figure 1 The virtualized screen projection method specifically includes:

[0046] S110. Create a display virtual window corresponding to the guest virtual machine and display the display virtual window within a preset screen corresponding to the guest virtual machine.

[0047] Among them, the display virtual window is a virtual window for full-screen displaying the running content of the guest virtual machine. Specifically, a display virtual window corresponding to the guest virtual machine can be created according to the specified display screen of the guest virtual machine, so that the display virtual window can adapt to the resolution of the specified display screen of the guest virtual machine.

[0048] In one example, creating a display virtual window corresponding to a customer virtual machine includes:

[0049] Obtain the display port number of a preset screen corresponding to the customer virtual machine, and determine the resolution of the preset screen based on the display port number; create a display virtual window corresponding to the customer virtual machine based on the resolution of the preset screen.

[0050] Among them, the preset screen corresponding to the customer virtual machine can be a screen that is pre-specified to display the running content of the customer virtual machine. The display port number of the preset screen can be the DP (DisplayPort) port number of the preset screen.

[0051] Exemplarily, the corresponding relationship between different customer virtual machines in the in-vehicle computer and each screen can be written into a table in advance, so as to query the preset screen corresponding to each customer virtual machine and the display port number of the preset screen from the table.

[0052] Furthermore, the resolution of the preset screen can be obtained according to the display port number. Among them, the resolution can refer to the number of pixels contained horizontally and vertically on the screen, usually represented in the form of "horizontal pixel number × vertical pixel number". Different screens in the in-vehicle computer can have their own resolutions. For example, the corresponding relationship between different screens in the in-vehicle computer and the resolutions can be written into a table in advance, so as to query the resolutions of each screen from the table.

[0053] After obtaining the resolution of the preset screen, further, a display virtual window that meets the resolution can be created, and the display virtual window is used to display the running content of the customer virtual machine on the preset screen.

[0054] Through the above example, the resolution can be determined based on the display port number of the preset screen corresponding to the customer virtual machine, so as to create a display virtual window that meets the resolution, which can ensure that the display virtual window can adapt to the preset screen, and further ensure the display effect within the preset screen.

[0055] Specifically, after creating the display virtual window corresponding to the customer virtual machine, the display virtual window can be displayed within the preset screen corresponding to the customer virtual machine.

[0056] S120. Create a transparent virtual window below the display virtual window, and obtain the physical memory address corresponding to the transparent virtual window.

[0057] Specifically, after creating the display virtual window that is displayed within the preset screen, a transparent virtual window corresponding to the projection device can be continued to be created, and the transparent virtual window is located below the display virtual window. Among them, the projection device can be an external projector connected to the hardware device.

[0058] In one example, creating a transparent virtual window located below the display virtual window includes:

[0059] Determine the resolution of the projection device based on the configuration file, and create a virtual window according to the resolution of the projection device; set the virtual window to be transparently displayed to obtain a transparent virtual window, and place the transparent virtual window below the display virtual window.

[0060] Among them, the configuration file can be a file for transmitting parameter information between devices. Specifically, the resolution of the connected projection device can be read from the configuration file.

[0061] Furthermore, a virtual window can be created according to the resolution of the projection device so that the virtual window can adapt to the resolution of the projection device. After creating the virtual window, the attributes of the virtual window can be set to be transparently displayed to obtain a transparent virtual window, and the virtual window can be set to be displayed at the bottom layer and placed below the display virtual window.

[0062] Through the above example, it is possible to create a transparent virtual window located below the display virtual window based on the resolution of the projection device, so that the transparent virtual window meets the resolution of the projection device and can project the display effect of the device.

[0063] It should be noted that in the embodiments of the present application, setting the transparent virtual window below the display virtual window aims to: the display on one screen can be divided into multiple layers, and each layer can have different transparencies. By setting a transparent virtual window located below the virtual window, it is possible to prevent the transparent virtual window corresponding to the projection device from affecting the display of the display virtual window, thereby avoiding the influence of the projection on the display content on the screen and ensuring the display effect of the screen during the projection process.

[0064] After creating the transparent virtual window located below the display virtual window, the physical address of the video memory corresponding to the transparent virtual window can also be obtained. Among them, video memory (display memory) can be pre-allocated for the graphics processing unit (GPU) in the hardware device, and the physical address of the video memory can be the address of a video memory area allocated for the transparent virtual window in the video memory corresponding to the graphics processing unit.

[0065] S130. In response to the rendering instruction issued by the customer virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the video memory area corresponding to the physical address of the video memory.

[0066] In the embodiments of the present application, if the customer virtual machine detects that the projection control is triggered by the user in the preset screen, or the customer virtual machine detects the projection voice instruction of the user, the rendering instruction can be issued to the privileged virtual machine to turn on the projection.

[0067] Specifically, in response to a rendering instruction issued by a customer virtual machine, the images to be displayed in the customer virtual machine application can be rendered and synthesized, so that the transparent virtual window and the display virtual window corresponding to the customer virtual machine display the same content. It should be noted that the display content of the transparent virtual window and the display virtual window can be the same, and the resolutions can be the same or different. The resolution of the transparent virtual window corresponds to the resolution of the projection device, and the resolution of the display virtual window corresponds to the resolution of the preset screen.

[0068] In a specific implementation, in response to a rendering instruction issued by a customer virtual machine, rendering the same display image as the display virtual window for the transparent virtual window includes the following steps:

[0069] Step 11: In response to a rendering instruction issued by a customer virtual machine, call the virtualized graphics processing dynamic library to parse the rendering instruction;

[0070] Step 12: After the parsing is completed, submit a graphics processing instruction to the graphics processor through the interface in the rendering management library, so that the graphics processor performs rendering and synthesis according to the display image of the display virtual window, and makes the transparent virtual window display the same display image as the display virtual window.

[0071] Among them, in step 11, if a rendering instruction issued by a customer virtual machine is received, the virtualized graphics processing dynamic library (Virglrenderer) can be called first to parse the rendering instruction through the virtualized graphics processing dynamic library.

[0072] Specifically, the virtualized graphics processing dynamic library designs a virtual OpenGL rendering device. The customer virtual machine can send the OpenGL rendering instruction to this virtual device. Further, the virtualized graphics processing dynamic library parses the rendering instruction to facilitate subsequent acceleration processing by the graphics processor of the hardware device.

[0073] Further, in step 12, after the parsing is completed, the graphics processor driver can be called through the interface in the rendering management library (libdrm) to submit a graphics processing instruction to the graphics processor, so that the graphics processor performs rendering and synthesis according to the display image of the display virtual window, enabling the transparent virtual window to display the same display image as the display virtual window.

[0074] Through the above steps 11 - step 12, the OpenGL (Open Graphics Library) call in the customer virtual machine can be converted into an instruction that can be understood by the GPU in the hardware device, so that the GPU in the hardware device performs rendering and synthesis, ensuring the rendering efficiency.

[0075] It should be noted that the display image rendered for the transparent virtual window will be stored in the video memory area corresponding to the transparent virtual window, that is, in the video memory area corresponding to the physical address of the video memory of the transparent virtual window. Among them, the video memory area corresponding to the physical address of the video memory can be the storage area pointed to by the physical address of the video memory.

[0076] S140. In response to the display instruction sent by the customer virtual machine, obtain the display image from the video memory area corresponding to the physical address of the video memory, and send the display image to the projection device for projection display.

[0077] Specifically, after completing the rendering and synthesis of the display image of the display virtual window, the customer virtual machine can continue to send a display instruction. In response to this display instruction, the display image in the transparent virtual window can be sent for display, that is, the display image in the transparent virtual window is sent to the projection device for projection display.

[0078] In a specific implementation manner, obtaining the display image from the video memory area corresponding to the physical address of the video memory in response to the display instruction sent by the customer virtual machine includes:

[0079] In response to the display instruction sent by the customer virtual machine, call the interface in the graphics service subsystem to enable the graphics service subsystem to start direct memory access, and read the display image from the video memory area corresponding to the transparent virtual window according to the physical address of the video memory.

[0080] Specifically, if a display instruction sent by the customer virtual machine is received, the interface in the graphics service subsystem can be called, and then the graphics service subsystem will trigger direct memory access (DMA, Direct Memory Access, DMA) for transmission, access the physical address of the video memory, and read the display image in the corresponding video memory area.

[0081] It should be noted that during the process of projecting the content displayed by the customer virtual machine on the preset screen, the above steps of rendering and synthesis, and the steps of reading and transmitting the display image from the video memory area can be executed cyclically. For example, at the first moment of starting the projection, a frame of display image can be obtained from the first address in the physical address of the video memory. At the next moment, the next frame of image can be continuously obtained from the next address in the physical address of the video memory, and so on until the projection ends.

[0082] It can be seen from this that the display frame rate of the projection device is the same as the frame rate of the customer virtual machine application. For example, if the frame rate of the customer virtual machine application is 30 frames per second, the graphics service subsystem will transmit 30 frames of images to the projection device for display per second through DMA.

[0083] In the above embodiments, by invoking the graphics service subsystem for DMA transfer, the video memory area allocated for the transparent virtual window in the GPU video memory can be directly accessed, so as to obtain the display image and perform the transfer without the intervention of the CPU in the hardware device, which can improve the transfer efficiency of the image data. At the same time, the utilization rate of the CPU is greatly improved.

[0084] Specifically, after reading the display image from the video memory area corresponding to the transparent virtual window, further, the display image can be sent to the projection device, and then the projection device can project the received display image, such as projecting it onto the windshield of the vehicle, projecting it onto the top of the vehicle, projecting it onto a curtain in the external environment of the vehicle, etc.

[0085] In the embodiments of the present application, the above rendering instruction and display sending instruction can be instructions that meet the virtual device interface specification (virtio-gpu). In one example, the rendering instruction and display sending instruction are generated by the guest virtual machine calling the graphics service of the virtual graphics device interface.

[0086] Specifically, the guest virtual machine can call the graphics service of the virtual graphics device interface (virtio-gpu) to generate the rendering instruction and display sending instruction. In the embodiments of the present application, both the guest virtual machine and the privileged virtual machine can set the virtual graphics device interface. By calling the virtual graphics device interface to generate instructions, the virtual machine can better handle graphics-related tasks.

[0087] Among them, the above graphics service subsystem, virtual graphics device interface, etc. can all be loaded after the privileged virtual machine is started. Optionally, before creating the display virtual window corresponding to the guest virtual machine, it further includes:

[0088] Loading the driver of the graphics processor and starting the graphics service subsystem; starting the guest virtual machine so that the guest virtual machine loads the driver of the virtual graphics device interface and starts the graphics service.

[0089] Specifically, the privileged virtual machine can first load the driver of the graphics processor in the hardware device and start the graphics service subsystem, so as to facilitate subsequent creation of virtual windows, obtaining the physical address of the video memory, etc. through the graphics service subsystem, and performing rendering and synthesis through the graphics processor.

[0090] Further, the privileged virtual machine can start the guest virtual machine. After the guest virtual machine is started, the guest virtual machine can load the driver of the virtual graphics device interface and then start the graphics service, so as to facilitate subsequent generation of the rendering instruction and display sending instruction through the graphics service.

[0091] Exemplarily, Figure 2 is a flowchart of screen mirroring based on virtualization technology provided by the embodiments of the present application. AsFigure 2 As shown, first, the privileged virtual machine can load the driver of the graphics processor and start the graphics service subsystem. Then, the device model (DM) in the privileged virtual machine starts the guest virtual machine, and the guest virtual machine loads the driver of the virtual graphics device interface (virtio - gpu). Subsequently, the guest virtual machine starts the graphics service.

[0092] After the preparations for the above - mentioned services are completed, refer to Figure 2 , the entire screen mirroring process can include the following steps:

[0093] S1: The device model calls the graphics service subsystem to create a display virtual window for the guest virtual machine;

[0094] S2: The device model calls the graphics service subsystem to create a transparent virtual window for the projection device;

[0095] S3: The graphics service subsystem obtains the physical memory address corresponding to the transparent virtual window;

[0096] S4: The device model waits for instructions sent by the guest virtual machine for processing. Among them, the virtual graphics device interface (virtio - gpu) in the device model receives rendering instructions, and the virtual graphics device interface (virtio - gpu) in the device model receives display instructions;

[0097] S5: The guest virtual machine sends a rendering instruction. After the DM receives the virtio - gpu rendering instruction, it calls Virglrenderer for parsing, and then submits relevant instructions to the GPU through relevant interfaces in libdrm to call the GPU driver for rendering and composition;

[0098] S6: The guest virtual machine sends a rendering instruction. After the DM receives the virtio - gpu display instruction, it calls the interface of the graphics service subsystem for display;

[0099] S7: The graphics service subsystem copies the image to the projection device through DMA, and the projection device projects the image.

[0100] For the above S1 - S7, a display virtual window needs to be created for the guest virtual machine first, and then this window is full - screen displayed on the physical screen. Subsequently, a transparent virtual window is created according to the resolution of the projection device. At this time, the created transparent virtual window has the property of being on the bottom layer and transparent display to avoid any interference with the display virtual window of the guest virtual machine. The transparent virtual window can render the same content as the display virtual window with virtio - gpu. The graphics service subsystem will obtain the physical memory address of the transparent virtual window, and this address is the address where each frame of the rendered image is stored. The graphics service subsystem transfers each frame of the rendered image to the projection device through DMA for screen mirroring.

[0101] The virtualized screen mirroring method provided by the embodiment of the present application is executed by a privileged virtual machine in a hardware device. The privileged virtual machine can first create a display virtual window corresponding to the guest virtual machine, display the display virtual window within a preset screen corresponding to the guest virtual machine, and create a transparent virtual window located below the display virtual window, obtain the physical memory address corresponding to the transparent virtual window, and then, in response to a rendering instruction issued by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the memory area corresponding to the physical memory address, and in response to a display instruction issued by the guest virtual machine, obtain the display image from the memory area corresponding to the physical memory address, and send the display image to a projection device for projection display. By setting the transparent virtual window and the corresponding physical memory address, this method realizes the projection of the content displayed by the guest virtual machine on the screen. Since the transparent virtual window is located below the display virtual window and is transparently displayed, it will not affect the display of the display virtual window on the preset screen, avoiding the projection from affecting the content displayed by the guest virtual machine on the screen. Moreover, the virtual screen mirroring implemented by this method can flexibly adapt to projection devices with different resolutions and hardware interfaces, without adding new interfaces for the projection device, and can also ensure that each frame of the projected image is not distorted, improving the user experience.

[0102] For the same purpose, the embodiment of the present application also provides a hardware device for virtualized screen mirroring. The hardware device includes a privileged virtual machine and at least one guest virtual machine, wherein;

[0103] The guest virtual machine is used to issue a rendering instruction and a display instruction to the privileged virtual machine;

[0104] The privileged virtual machine is used to create a display virtual window corresponding to the guest virtual machine, display the display virtual window within a preset screen corresponding to the guest virtual machine, create a transparent virtual window located below the display virtual window, obtain the physical memory address corresponding to the transparent virtual window, in response to the rendering instruction issued by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the memory area corresponding to the physical memory address, in response to the display instruction issued by the guest virtual machine, obtain the display image from the memory area corresponding to the physical memory address, and send the display image to a projection device for projection display.

[0105] Optionally, the privileged virtual machine includes a device model and a graphics service subsystem. The device model is used to call the graphics service subsystem. The graphics service subsystem is used to create a display virtual window corresponding to the guest virtual machine, create a transparent virtual window below the display virtual window, and read the display image from the video memory area corresponding to the transparent virtual window according to the physical video memory address.

[0106] Exemplarily, the device model is used to obtain the display port number of a preset screen corresponding to the guest virtual machine, determine the resolution of the preset screen based on the display port number, and call the graphics service subsystem to enable the graphics service subsystem to create a display virtual window corresponding to the guest virtual machine based on the resolution of the preset screen.

[0107] Exemplarily, the device model is used to determine the resolution of the projection device based on a configuration file and call the graphics service subsystem to enable the graphics service subsystem to create a virtual window according to the resolution of the projection device; set the virtual window to be transparently displayed to obtain a transparent virtual window, and place the transparent virtual window below the display virtual window.

[0108] Exemplarily, the device model is used to respond to a rendering instruction issued by the guest virtual machine, call a virtualized graphics processing dynamic library to parse the rendering instruction; after the parsing is completed, submit a graphics processing instruction to the graphics processor through an interface in the rendering management library to enable the graphics processor to perform rendering synthesis according to the display image of the display virtual window, so that the transparent virtual window displays the same display image as the display virtual window.

[0109] Exemplarily, the device model is used to respond to a display instruction issued by the guest virtual machine, call an interface in the graphics service subsystem to enable the graphics service subsystem to start direct memory access, and read the display image from the video memory area corresponding to the transparent virtual window according to the physical video memory address.

[0110] Exemplarily, the device model is used to start the guest virtual machine to enable the guest virtual machine to load the driver of the virtual graphics device interface and start the graphics service.

[0111] The hardware device for virtualized screen mirroring provided in the embodiments of the present application can execute the steps in the virtualized screen mirroring method provided in the method embodiments of the present application, and the implementation steps and beneficial effects are not described herein again.

[0112] Figure 3 It is a system block diagram of a hardware device provided in the embodiments of the present application, as Figure 3As shown in the figure, the entire hardware device may include a privileged virtual machine, a guest virtual machine, and a hypervisor.

[0113] Among them, the guest virtual machine can be divided into a user space and a kernel space. In the user space, it can be divided into applications, APIs for graphics rendering, open-source graphics libraries (GL driver, VK driver, gpu-virtio), and a rendering management library (libdrm). In the kernel space, it can include a virtual graphics device interface (virtio-gpu).

[0114] Among them, the privileged virtual machine can be divided into a user space and a kernel space. In the user space, it can be divided into a device model (DM), a graphics service subsystem, a virtualized graphics processing dynamic library (Virglrenderer), and a rendering management library (libdrm). In the kernel space, it can include a graphics processor.

[0115] Specifically, during the process of starting the guest virtual machine, the DM in the privileged virtual machine will, according to the DisplayPort port number that the guest virtual machine needs to display on the specified screen (it is agreed that for the linux running on the privileged virtual machine, when a display screen is connected, relevant functions in the graphics service subsystem can be called to obtain the DisplayPort port), and obtain the resolution of the screen. Then, the DM can call the graphics service subsystem to full-screen display the guest virtual machine on the physical screen, that is, create a virtual display window corresponding to the guest virtual machine. The DM can also call the graphics service subsystem to create a bottom transparent window (i.e., the transparent virtual window corresponding to the projection device) according to the resolution of the projection device configured in the configuration file. The content displayed in this window is the content in the guest virtual machine. Since the window of the projection device is set with bottom and transparent attributes, it will not be displayed on the physical screen and does not affect the display of the guest virtual machine. The graphics service subsystem will obtain the physical memory address of the video memory of the transparent virtual window, and this address is the address where each frame of the rendered image is stored. The guest virtual machine calls virtio-gpu to generate rendering instructions, and the DM calls the graphics processor to complete the rendering and synthesis of each frame of the image. Then, the guest virtual machine calls virtio-gpu to generate display instructions, and the DM calls the relevant interfaces of the graphics service subsystem for display. The graphics service subsystem copies each frame of the image from the physical memory address of the video memory corresponding to the transparent virtual window to the projection device through DMA, and then the projection device performs screen projection.

[0116] Through this architecture, the virtual screen projection function can be realized, flexibly adapting to different resolutions and hardware interfaces of the projection device, and ensuring that each frame of the image is not distorted.

[0117] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. AsFigure 4 As shown, the electronic device 400 includes one or more processors 401 and a memory 402.

[0118] The processor 401 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 400 to perform desired functions.

[0119] The memory 402 can include one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 401 can run the program instructions to implement the virtualized screen projection method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial extrinsic parameters, thresholds, etc. can also be stored in the computer-readable storage media.

[0120] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device 403 can include, for example, a keyboard, a mouse, etc. The output device 404 can output various information to the outside, including warning prompt information, braking force, etc. The output device 404 can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0121] Of course, for simplicity, Figure 4 only some of the components related to the present application in the electronic device 400 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 400 may further include any other appropriate components.

[0122] In addition to the above methods and devices, an embodiment of the present application can also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps of the virtualized screen projection method provided by any embodiment of the present application.

[0123] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, an embodiment of the present application may also be a computer-readable storage medium storing computer program instructions, which, when run by a processor, cause the processor to execute the steps of the virtualized screen mirroring method provided by any embodiment of the present application.

[0125] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] It should be noted that the terms used in the present application are only for describing specific embodiments and do not limit the scope of the present application. As shown in the specification and claims of the present application, unless the context clearly indicates an exception, words such as "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. The term "comprising", "including", or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, or device comprising the element.

[0127] It should also be noted that the orientation or positional relationship indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application. Unless otherwise clearly specified and defined, terms such as "installed", "connected", "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific situations.

[0128] In this article, specific examples are used to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only for helping to understand the method and its core idea of the present application. The above is only the preferred implementation manner of the present application. It should be noted that due to the limited nature of written expression and the objectively infinite specific structures, for those of ordinary skill in the art, without departing from the principle of the present application, several improvements, refinements or changes can be made, or the above technical features can be combined in an appropriate manner; these improvements, refinements, changes or combinations, or directly applying the concept and technical solution of the invention to other occasions without improvement, should all be regarded as the protection scope of the present application.

Claims

1. A virtualized screen mirroring method, characterized in that, Executed by a privileged virtual machine in a hardware device, the hardware device further including at least one guest virtual machine, the method comprising: Create a display virtual window corresponding to the guest virtual machine, and display the display virtual window within a preset screen corresponding to the guest virtual machine; Create a transparent virtual window located below the display virtual window, and obtain the physical memory address corresponding to the transparent virtual window; In response to a rendering instruction issued by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the memory area corresponding to the physical memory address; In response to a display sending instruction issued by the guest virtual machine, obtain the display image from the memory area corresponding to the physical memory address, and send the display image to a projection device for projection display.

2. The method according to claim 1, characterized in that The creating a display virtual window corresponding to the guest virtual machine includes: Obtain the display port number of a preset screen corresponding to the guest virtual machine, and determine the resolution of the preset screen based on the display port number; Create a display virtual window corresponding to the guest virtual machine based on the resolution of the preset screen.

3. The method according to claim 1, wherein The creating a transparent virtual window located below the display virtual window includes: Determine the resolution of the projection device based on a configuration file, and create a virtual window according to the resolution of the projection device; Set the virtual window to be transparently displayed to obtain a transparent virtual window, and place the transparent virtual window below the display virtual window.

4. The method according to claim 1, wherein In response to a rendering instruction issued by the guest virtual machine, rendering the same display image as the display virtual window for the transparent virtual window includes: In response to a rendering instruction issued by the guest virtual machine, call a virtualized graphics processing dynamic library to parse the rendering instruction; After the parsing is completed, submit a graphics processing instruction to a graphics processor through an interface in a rendering management library, so that the graphics processor performs rendering synthesis according to the display image of the display virtual window, and makes the transparent virtual window display the same display image as the display virtual window.

5. The method according to claim 4, characterized in that, In response to a display sending instruction issued by the guest virtual machine, obtaining the display image from the memory area corresponding to the physical memory address includes: In response to a display sending instruction issued by the guest virtual machine, call an interface in a graphics service subsystem, so that the graphics service subsystem starts direct memory access, and reads the display image from the memory area corresponding to the transparent virtual window according to the physical memory address.

6. The method according to claim 5, wherein The rendering instruction and the display sending instruction are generated for a graphics service that calls a virtual graphics device interface by the guest virtual machine.

7. The method according to claim 6, wherein Before the creating a display virtual window corresponding to the guest virtual machine, the method further includes: Load a driver of the graphics processor, and start the graphics service subsystem; Start the guest virtual machine, so that the guest virtual machine loads a driver of the virtual graphics device interface and starts the graphics service.

8. A hardware device for virtualized screen mirroring, characterized in that, The hardware device includes a privileged virtual machine and at least one guest virtual machine, wherein; The guest virtual machine is used to issue a rendering instruction and a display sending instruction to the privileged virtual machine; The privileged virtual machine is used to create a display virtual window corresponding to the guest virtual machine, display the display virtual window within a preset screen corresponding to the guest virtual machine, create a transparent virtual window below the display virtual window, obtain the physical video memory address corresponding to the transparent virtual window, in response to a rendering instruction issued by the guest virtual machine, render the same display image as the display virtual window for the transparent virtual window, so that the display image is stored in the video memory area corresponding to the physical video memory address, and in response to a display sending instruction issued by the guest virtual machine, obtain the display image from the video memory area corresponding to the physical video memory address and send the display image to a projection device for projection display.

9. The hardware device according to claim 8, characterized in that, The privileged virtual machine includes a device model and a graphics service subsystem, where; The device model is used to call the graphics service subsystem; The graphics service subsystem is used to create a display virtual window corresponding to the guest virtual machine, and to create a transparent virtual window below the display virtual window, and to read the display image from the video memory area corresponding to the transparent virtual window according to the physical video memory address.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions, and the program or instructions cause the computer to execute the steps of the virtualized screen projection method according to any one of claims 1 to 7.

Images