A rendering and display method, device and electronic equipment of a multi-virtual machine system

Abstract

The present disclosure relates to a rendering and display method, device and electronic equipment of a multi-virtual machine system. The rendering and display method of the multi-virtual machine system comprises: a virtualization layer, in response to receiving a memory mapping I / O request from a virtio-gpu driver of a first virtual machine, sending rendering instruction arrival information to a virtual graphics library service running in a user state of a second virtual machine by using a virtio device implemented in the virtualization layer; the virtual graphics library service, in response to receiving the rendering instruction arrival information, obtaining first rendering instruction data for executing a target rendering task stored in a shared memory region between the first virtual machine and the second virtual machine by using an ioctl mechanism; and obtaining a rendering result corresponding to the first rendering instruction data by using a hardware GPU device of the second virtual machine according to the first rendering instruction data; and sending image data to be displayed obtained according to the rendering result to the hardware GPU of the second virtual machine by using a GPU driver running in the second virtual machine.

Description

Technical Field

[0001] This disclosure relates to the field of virtual machine technology, and in particular to a rendering and display method, apparatus and electronic device for a multi-virtual machine system. Background Technology

[0002] Virtualization technology is an important technology in the field of computer science. It allows multiple virtual machines to run on a physical computer, thereby improving the utilization of computer hardware. In virtual machine managers, the GPU (Graphics Processing Unit) and display virtualization technology play a crucial role, significantly improving the rendering performance of virtual machines, enhancing the usability of the virtual machine manager, and meeting the needs of the virtual machine manager in rendering and display application scenarios.

[0003] In related technologies, QEMU virtual machine managers can leverage the virtio-gpu protocol of KVM (Kernel-based Virtual Machine) to perform backend rendering of graphics from the virtual machine on the host machine (HOS), and create windows on the host machine using graphical UI frameworks such as SDL and GTK to complete the display task. However, since QEMU is a Type II virtual machine, and Type II virtual machines have different system architectures than Type I virtual machines, the aforementioned QEMU window display virtualization method cannot be applied to Type I virtual machine managers. Furthermore, in the aforementioned QEMU window display virtualization method, the rendered image from the virtual machine needs to be composited again at the host machine's graphics layer to display the virtual machine's image in the host window. Therefore, the amount of data processed is large, resulting in low display performance. Moreover, in embedded applications (e.g., automotive applications), display virtualization typically requires multiple screens to display the image in full screen, while the aforementioned QEMU window display virtualization method only supports single-screen display and is ineffective in full-screen support.

[0004] Therefore, how to efficiently share the computing resources of the hardware GPU and graphics card resources in multiple virtual environments on a Type I virtual machine, while achieving good full-screen support, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a rendering and display method, apparatus and electronic device for a multi-virtual machine system.

[0006] According to a first aspect of the present disclosure, a rendering and display method for a multi-virtual machine system is provided, comprising: the virtualization layer, in response to receiving a memory-mapped I / O request from a virtio-gpu driver of a first virtual machine, sending rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine using a virtio device implemented in the virtualization layer; wherein the second virtual machine is a virtual machine with a hardware GPU device and a hardware graphics card, and the first virtual machine is a virtual machine in the multi-virtual machine system that does not have a hardware GPU device and a hardware graphics card; the virtual graphics library service, in response to receiving the rendering instruction arrival information, using an ioctl mechanism, obtaining first rendering instruction data for executing a target rendering task stored in a shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver; and obtaining a rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data; and sending image data to be displayed obtained based on the rendering result to the hardware graphics card of the second virtual machine using a graphics card driver running on the second virtual machine, so that the hardware graphics card displays the rendering result on a target display device connected to the hardware graphics card.

[0007] In some embodiments, the virtio device includes a virtio proxy device and a virtio stub device; the step of using the virtio device implemented in the virtualization layer to send rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine includes: the virtio proxy device, in response to detecting a memory-mapped I / O request from the virtio-gpu driver, instructing the virtio stub device to send the rendering instruction arrival information to the virtual graphics library service.

[0008] In some embodiments, the virtio stub device sends the rendering instruction arrival information to the virtual graphics library service in the following manner: the virtio stub device triggers an interrupt of a preset type; the driver of the virtio stub device in kernel mode running in the first virtual machine, in response to receiving the interrupt of the preset type, sends the rendering instruction arrival information to the virtual graphics library service.

[0009] In some embodiments, the step of using the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task stored in the shared memory region between the first virtual machine and the second virtual machine by the virtio-gpu driver includes: the virtual graphics library service using the ioctl interface function provided by the driver of the virtio stub device to obtain the first rendering instruction data read from the shared memory region by the driver of the virtio stub device.

[0010] In some embodiments, obtaining the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data includes: generating second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data using a preset algorithm; sending the second rendering instruction data to the hardware GPU device using a GPU driver running in kernel mode, so that the hardware GPU device generates the rendering result of the target rendering task based on the second rendering instruction data; and obtaining the rendering result read from the hardware GPU device by the GPU driver in kernel mode.

[0011] In some embodiments, the first rendering instruction data is TGSI instruction data, and the second rendering instruction data is GLSL instruction data. The step of generating second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data using a preset algorithm includes: the virtual graphics library service using virglrenderer to parse the TGSI instruction data into GLSL instruction data; the step of sending the second rendering instruction data to the hardware GPU device using a GPU driver running in kernel mode includes: using libGLES adapted to the hardware GPU device to parse the GLSL instruction data into control instructions adapted to the GPU driver; calling the interface function provided by the GPU driver to execute the control instructions to write the GLSL instruction data to the hardware GPU device; and the step of obtaining the rendering result read from the hardware GPU device by the kernel-mode GPU driver includes: using EGLContext bound to virglrenderer, and through libGLES adapted to the hardware GPU device, reading the rendering result obtained from the hardware GPU device by the GPU driver.

[0012] According to a second aspect of the present disclosure, a rendering and display apparatus for a multi-virtual machine system is provided, comprising: a sending module, configured to cause the virtualization layer, in response to receiving a memory-mapped I / O request from a first virtual machine via a virtio-gpu driver, to send rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine using a virtio device implemented in the virtualization layer; wherein the second virtual machine is a virtual machine possessing a hardware GPU device and hardware graphics card resources, and the first virtual machine is a virtual machine in the multi-virtual machine system that does not possess a hardware GPU device and hardware graphics card resources; and a first obtaining module, configured to cause the virtual graphics library service, in response to receiving a memory-mapped I / O request from a first virtual machine via a virtio-gpu driver, to send rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine. The rendering instruction arrival information utilizes the ioctl mechanism to obtain first rendering instruction data for executing the target rendering task, stored in the shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver; the second acquisition module is used to obtain the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data; the display module is used to send the image data to be displayed obtained based on the rendering result to the hardware graphics card of the second virtual machine using the graphics card driver running on the second virtual machine, so that the hardware graphics card displays the rendering result on the target display device connected to the hardware graphics card.

[0013] According to a third aspect of the present disclosure, a vehicle is provided that stores a set of instructions, which are executed by the vehicle to implement the rendering and display method of the multi-virtual machine system provided in the first aspect of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the rendering and display method of the multi-virtual machine system provided in the first aspect of the present disclosure.

[0015] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the rendering and display method for a multi-virtual machine system provided in the first aspect of the present disclosure.

[0016] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: A virtualization layer, in response to receiving a memory-mapped I / O request from the virtio-gpu driver of a first virtual machine, uses a virtio device implemented in the virtualization layer to send rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine; the virtual graphics library service, in response to receiving the rendering instruction arrival information, uses an ioctl mechanism to obtain first rendering instruction data for executing a target rendering task stored in a shared memory area between the first and second virtual machines by the virtio-gpu driver; and, based on the first rendering instruction data, uses the hardware GPU device of the second virtual machine to obtain the rendering result corresponding to the first rendering instruction data; and uses the graphics card driver running on the second virtual machine to send the image data to be displayed obtained based on the rendering result to the hardware graphics card of the second virtual machine. Thus, on a Type I virtual machine, the computing resources of the hardware GPU and graphics card resources can be efficiently shared in multiple virtual environments, and a better full-screen support effect can be obtained.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a flowchart illustrating a rendering and display method for a multi-virtual machine system according to an exemplary embodiment.

[0020] Figure 2 This is a flowchart illustrating a method for a virtual graphics library service to receive first rendering instruction data according to an exemplary embodiment.

[0021] Figure 3 This is a flowchart illustrating a method for obtaining rendering results from a virtual graphics library service according to an exemplary embodiment.

[0022] Figure 4 This is a block diagram of a multi-virtual machine system according to an exemplary embodiment.

[0023] Figure 5 This is a schematic diagram illustrating a rendering and display method for a multi-virtual machine system according to an exemplary embodiment.

[0024] Figure 6 This is a block diagram illustrating a rendering and display apparatus for a multi-virtual machine system according to an exemplary embodiment.

[0025] Figure 7This is a block diagram illustrating a vehicle according to an exemplary embodiment.

[0026] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0027] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0028] It should be noted that the relevant embodiments and accompanying drawings are only for describing and illustrating exemplary embodiments provided by this disclosure, and not all embodiments of this disclosure, nor should this disclosure be understood to be limited to the relevant exemplary embodiments.

[0029] It should be noted that the terms "first," "second," etc., used in this disclosure are only used to distinguish different steps, devices, or modules. These terms do not represent any specific technical meaning, nor do they indicate any order or interdependence between them.

[0030] It should be noted that the terms “a,” “a plurality of,” and “at least one” used in this disclosure are illustrative rather than restrictive. Unless otherwise expressly indicated in the context, they should be understood as “one or more.”

[0031] It should be noted that the term "and / or" used in this disclosure is used to describe the relationship between related objects, and generally indicates that there are at least three relationships. For example, A and / or B can at least indicate: the existence of A alone, the existence of both A and B, and the existence of B alone.

[0032] It should be noted that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Unless otherwise specified, the scope of this disclosure is not limited to the order in which the steps are described in the relevant embodiments.

[0033] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0034] Exemplary methods

[0035] Figure 1 This is a flowchart illustrating a rendering and display method for a multi-virtual machine system according to an exemplary embodiment. This rendering and display method for a multi-virtual machine system is applied to, for example... Figure 4In the multi-virtual machine system shown, multiple virtual machines within the system share an on-chip system, including a hardware GPU and a hardware graphics card, through a virtualization layer. In practical implementation, intelligent service robots, as well as intelligent instruments and central infotainment systems in vehicle systems, can use the methods provided in this disclosure to implement GPU virtualization and display virtualization.

[0036] like Figure 1 As shown, the rendering and display method for a multi-virtual machine system includes the following steps:

[0037] In step S110, the virtualization layer, in response to receiving a memory-mapped I / O request from the virtio-gpu driver of the first virtual machine, uses the virtio device implemented in the virtualization layer to send rendering instruction arrival information to the virtual graphics library service running in user mode of the second virtual machine.

[0038] The second virtual machine is a virtual machine with its own hardware GPU and graphics card resources, such as... Figure 5 As shown in dom0, the second virtual machine can directly access and use GPU devices and hardware graphics card resources. For other virtual machines in a multi-virtual machine system that need to use GPUs for graphics processing or computing tasks, GPU resources can be provided to them through the second virtual machine.

[0039] The first virtual machine is a virtual machine in a multi-virtual machine system that does not have its own hardware GPU device or hardware graphics card resources, for example Figure 5 dom1 in the middle.

[0040] The virtio-gpu driver is a GPU driver for virtualization environments. Based on virtio virtualization technology, the virtio-gpu driver provides a general virtual device interface, enabling virtual machines to access GPU devices as if they were ordinary hardware devices.

[0041] In practice, the application running on the first virtual machine can send rendering task instruction data to the backend GPU device through the virtio-gpu driver. In some embodiments, the virtio-gpu driver uses memory mapping to send the rendering task instruction data, generating memory-mapped I / O requests in the process.

[0042] A virtio device is a hardware device simulated using virtualization technology. It can emulate one or more physical devices within a virtual machine, allowing the virtual machine to access these virtual devices as if they were ordinary hardware devices. In practice, virtio devices can be implemented at the virtualization layer based on the virtio specification. For a detailed description of virtio devices, see [link to virtio device documentation]. Figure 2The relevant content will not be repeated here.

[0043] Virtual graphics library services (e.g., Figure 5 The `virgl-service` in the virtual machine provides graphics processing and rendering services, supporting various virtualization technologies such as KVM and VMware. Figure 5 As shown, the virtual graphics library service runs in user mode in the second virtual machine.

[0044] In practical implementation, rendering instruction arrival information can be sent to the user-space virtual graphics library service running on the second virtual machine using virtio devices implemented at the virtualization layer in various ways. For specific embodiments of sending rendering instruction arrival information to the user-space virtual graphics library service running on the second virtual machine using virtio devices implemented at the virtualization layer, please refer to [link to relevant documentation]. Figure 2 The relevant content will not be repeated here.

[0045] In step S120, the virtual graphics library service, in response to receiving the rendering instruction arrival information, uses the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task, which is stored in the shared memory area between the second virtual machine and the first virtual machine by the virtio-gpu driver.

[0046] Target rendering tasks are those that require the use of graphics rendering techniques. Examples of target rendering tasks include scene rendering, character animation, lighting rendering, and special effects compositing.

[0047] In practical implementation, the virtualization layer can allocate a circular buffer in memory as a shared memory area between the first and second virtual machines. When an application running on the first virtual machine calls the virtio-gpu driver to send the first rendering instruction data for executing the target rendering task to the GPU backend device, the virtio-gpu driver can store the first rendering instruction data in this shared memory area. After receiving the rendering instruction arrival information, the virtual graphics library service uses the ioctl mechanism to read data from the virtio-gpu driver. After the virtio-gpu driver reads the first rendering instruction data from this shared memory area, it returns it to the virtual graphics library service. For a detailed description of using the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task stored by the virtio-gpu driver in the shared memory area between the second and first virtual machines, see [link to relevant documentation]. Figure 2 The relevant content will not be repeated here.

[0048] For example only, such as Figure 5As shown, on the first virtual machine, the user-mode rendering stack is configured as MESA. The virtio-gpu dri module of MESA is used to convert the OpenGL interface and instructions into TGSI instructions. MESA uses the kernel-mode virtio-gpu driver running on the first virtual machine to encapsulate the TGSI instructions into an instruction stream that conforms to the virtio-gpu protocol definition and writes the instruction stream into the shared memory area.

[0049] In step S130, the rendering result corresponding to the first rendering instruction data is obtained using the hardware GPU device of the second virtual machine based on the first rendering instruction data.

[0050] In the specific implementation process, the rendering instruction data that the hardware GPU device can execute can be obtained from the first rendering instruction data in a variety of ways. The rendering instruction data is then written into the hardware GPU device, and the hardware GPU device uses its powerful computing and processing capabilities to calculate the rendering result (e.g., lighting data, material data, etc.) corresponding to the first rendering instruction data.

[0051] For a specific embodiment of obtaining the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data, please refer to [link to specific implementation]. Figure 3 The relevant content will not be repeated here.

[0052] In step S140, the graphics card driver running on the second virtual machine is used to send the image data to be displayed obtained from the rendering result to the hardware graphics card of the second virtual machine, so that the hardware graphics card can display the rendering result on the target display device connected to the hardware graphics card.

[0053] In practice, the image data to be displayed may include all information about the image displayed on the display device, such as image size information, pixel data, etc.

[0054] In practice, the virtual graphics library service can utilize the APIs provided by the display framework to send the image data to be displayed to the hardware graphics card and select the target display device. For example, it can use... Figure 5 The libDRM (user-space DRM library) provided by the DRM (DirectRendering Manager) module shown sends the image data to be displayed to the kernel-mode graphics card driver adapted to the target graphics card (e.g., Figure 5 (As shown), the image data to be displayed is then sent to the hardware graphics card by the kernel-mode graphics card driver. DRM is currently the mainstream graphics display framework in Linux.

[0055] The target display device is a display device connected to the hardware graphics card in the multi-virtual machine system, such as an LCD monitor or OLED monitor. The hardware graphics card can connect to the target display device through various interfaces, such as... Figure 5 The MIPI, HDMI, etc. shown.

[0056] In the embodiments provided in this disclosure, a virtualization layer is used to listen for memory-mapped I / O requests from the virtio-gpu driver of the first virtual machine. A virtio device implemented in the virtualization layer is used to send rendering instruction arrival information to the virtual graphics library service running in user mode of the second virtual machine. Upon receiving the rendering instruction arrival information, the virtual graphics library service uses an ioctl mechanism to obtain first rendering instruction data stored in the shared memory area between the first and second virtual machines for executing the target rendering task, thereby transferring the rendering instruction data from the first virtual machine (which does not have hardware GPU resources) to the second virtual machine (which does have hardware GPU resources). On the second virtual machine, based on the first rendering instruction data, the hardware GPU device of the second virtual machine obtains the rendering result corresponding to the first rendering instruction data. The graphics card driver running on the second virtual machine sends the image data to be displayed based on the rendering result to the hardware graphics card of the second virtual machine, so that the hardware graphics card displays the rendering result on the target display device connected to the hardware graphics card. Since the embodiments provided in this disclosure directly display the rendering result using the hardware GPU, compared with related technologies, data processing efficiency is effectively improved, and higher display performance is achieved, meeting the full-screen display effect required for Type I virtual machine management.

[0057] Figure 2 This is a flowchart illustrating a method for a virtual graphics library service to receive first rendering instruction data according to an exemplary embodiment.

[0058] In this embodiment, the virtio device includes a virtio proxy device and a virtio stub device, such as... Figure 2 As shown, the virtual graphics library service can receive the first rendering instruction data using the following steps:

[0059] In step S210, the virtio agent device, in response to detecting a memory-mapped I / O request from the virtio-gpu driver, instructs the virtio stub device to send rendering instruction arrival information to the virtual graphics library service.

[0060] A virtio proxy device is a device used in a virtualization environment that utilizes the virtio protocol to act as a proxy for the GPU backend device of the first virtual machine.

[0061] A virtio stub device is a device used in a virtualization environment that utilizes the virtio protocol to communicate with the operating system in a second virtual machine.

[0062] In the specific implementation process, registration functions or interfaces can be used to associate the shared memory region between the first virtual machine and the second virtual machine with the virtio agent device; listening functions or interfaces provided by the operating system running on the virtual machine can be used to enable the virtio agent device to listen for memory-mapped I / O requests from the virtio-gpu driver.

[0063] In practical implementation, communication between the virtio agent device and the virtio stub device can be achieved according to the virtio protocol. As an example, after the virtio agent device detects a memory-mapped I / O request from the virtio-gpu driver, it encapsulates the request into a virtio message according to the virtio protocol and then sends it to the virtio stub device. Upon receiving the virtio message, the virtio stub device parses the message according to the virtio protocol and then sends rendering instruction arrival information to the virtual graphics library service.

[0064] In some embodiments, the operating system running in the first virtual machine is Linux, and the virtio stub device can send rendering instruction arrival information to the virtual graphics library service through the following steps:

[0065] In step S220, the virtio stub device triggers an interrupt of a preset type; the driver of the virtio stub device in kernel mode running in the first virtual machine, in response to receiving the interrupt of the preset type, sends rendering instruction arrival information to the virtual graphics library service.

[0066] In practice, the operating system running in the first virtual machine can include Linux, Android, or other operating systems that support the virtio-gpu protocol. The driver for the virtio stub device can be the kernel driver for the operating system running in the first virtual machine.

[0067] As an example only, the default interrupt type can be a VQ interrupt, where the interrupt number can be a predefined interrupt number. Upon receiving an interrupt of the default type, the virtio stub device driver can send rendering instruction arrival information to the virtual graphics library service in various ways, such as through message notification.

[0068] In step S230, the virtual graphics library service uses the ioctl interface function provided by the driver of the virtio stub device to obtain the first rendering instruction data read from the shared memory region by the driver of the virtio stub device.

[0069] After receiving the rendering instruction arrival information sent by the driver of the virtio stub device, the virtual graphics library service can use the ioctl interface function provided by the virtio stub device to send a read data instruction to the driver of the virtio stub device. The driver of the virtio stub device then retrieves the first rendering instruction data from the shared buffer area and returns it to the virtual graphics library service according to a preset interface method. The above process may be implemented differently in different operating systems and is not limited to the description in this specification.

[0070] In the embodiments provided in this disclosure, a virtio proxy device implemented at the virtualization layer is used to listen for memory-mapped I / O requests from the virtio-gpu driver. The virtio stub device is then used to send rendering instruction arrival information to the virtual graphics library service. The virtual graphics library service uses the ioctl interface function provided by the virtio stub device driver to obtain the first rendering instruction data read from the shared memory region from the virtio stub device driver. This enables a user-mode application running on a second virtual machine to read rendering instruction data from the kernel-mode GPU driver running on the first virtual machine. Because this process leverages the universal protocol virtio implemented in the Linux ecosystem, it avoids the need for adaptation for different hardware GPU models when performing virtualization at the driver level, effectively improving the versatility of the solution.

[0071] Figure 3 This is a flowchart illustrating a method for obtaining rendering results from a virtual graphics library service according to an exemplary embodiment.

[0072] In step S310, a preset algorithm is used to generate second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data.

[0073] In some embodiments, such as Figure 5 As shown, the first rendering instruction data is TGSI (Tessellation and Geometry Shading Infrastructure) instruction data, and the second rendering instruction data is GLSL (OpenGL Shading Language) instruction data. The virtual graphics library service can use virglrenderer to parse the TGSI instruction data into GLSL instruction data.

[0074] virglrenderer is a framework for achieving GPU acceleration in virtualized environments. It accelerates rendering by passing OpenGL instructions to the actual GPU on the host machine.

[0075] In step S320, the second rendering instruction data is sent to the hardware GPU device using the GPU driver running in kernel mode, so that the hardware GPU device generates the rendering result of the target rendering task according to the second rendering instruction data.

[0076] In some embodiments, such as Figure 5 As shown, libGLES, which is adapted to hardware GPU devices, can be used to parse GLSL instruction data into control instructions adapted to the GPU driver. The interface functions provided by the GPU driver are then called to execute the control instructions, thereby writing the GLSL instruction data to the hardware GPU device. In practice, the control instructions can be GPU IOCTL execution streams, and libGLES can determine the GPU driver interface functions to be called based on the GLSL instruction data to be executed.

[0077] In step S330, the rendering results read from the hardware GPU device are obtained from the GPU driver in kernel mode.

[0078] In the specific implementation process, such as Figure 5 As shown, the EGLContext bound to virglrenderer can be used to read the rendering results obtained by the GPU driver from the hardware GPU device through libGLES, which is adapted to the hardware GPU device.

[0079] In the embodiments provided in this disclosure, a preset algorithm is used to generate second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data; the second rendering instruction data is sent to the hardware GPU device using a GPU driver running in kernel mode; and the rendering results read from the hardware GPU device are obtained from the GPU driver in kernel mode. This achieves the calculation of the rendering instruction data of the first virtual machine using the GPU hardware resources of the second virtual machine.

[0080] Exemplary device

[0081] Figure 6 This is a block diagram illustrating a rendering and display apparatus for a multi-virtual machine system according to an exemplary embodiment. (Refer to...) Figure 6 The device 600 includes a sending module 610, a first acquisition module 620, a second acquisition module 630, and a display module 640.

[0082] The sending module 610 is configured to enable the virtualization layer, in response to receiving a memory-mapped I / O request from the virtio-gpu driver of the first virtual machine, to send rendering instruction arrival information to the virtual graphics library service running in user mode of the second virtual machine using the virtio device implemented in the virtualization layer; wherein the second virtual machine is a virtual machine with hardware GPU device and hardware graphics card resources, and the first virtual machine is a virtual machine in the multi-virtual machine system that does not have hardware GPU device and hardware graphics card resources.

[0083] The first acquisition module 620 is configured to enable the virtual graphics library service to, in response to receiving the rendering instruction arrival information, acquire, using the ioctl mechanism, the first rendering instruction data for executing the target rendering task stored in the shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver.

[0084] The second acquisition module 630 is used to obtain the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data.

[0085] Display module 640 is used to send the image data to be displayed obtained from the rendering result to the hardware graphics card of the second virtual machine using the graphics card driver running on the second virtual machine, so that the hardware graphics card displays the rendering result on the target display device connected to the hardware graphics card.

[0086] In some embodiments, the virtio device includes a virtio proxy device and a virtio stub device; the step of using the virtio device implemented in the virtualization layer to send rendering instruction arrival information to a virtual graphics library service running in user mode of a second virtual machine includes: the virtio proxy device, in response to detecting a memory-mapped I / O request from the virtio-gpu driver, instructing the virtio stub device to send the rendering instruction arrival information to the virtual graphics library service.

[0087] In some embodiments, the virtio stub device sends the rendering instruction arrival information to the virtual graphics library service in the following manner: the virtio stub device triggers an interrupt of a preset type; the driver of the virtio stub device in kernel mode running in the first virtual machine, in response to receiving the interrupt of the preset type, sends the rendering instruction arrival information to the virtual graphics library service.

[0088] In some embodiments, the step of using the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task stored in the shared memory region between the first virtual machine and the second virtual machine by the virtio-gpu driver includes: the virtual graphics library service using the ioctl interface function provided by the driver of the virtio stub device to obtain the first rendering instruction data read from the shared memory region by the driver of the virtio stub device.

[0089] In some embodiments, obtaining the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data includes: generating second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data using a preset algorithm; sending the second rendering instruction data to the hardware GPU device using a GPU driver running in kernel mode, so that the hardware GPU device generates the rendering result of the target rendering task based on the second rendering instruction data; and obtaining the rendering result read from the hardware GPU device by the GPU driver in kernel mode.

[0090] In some embodiments, the first rendering instruction data is TGSI instruction data, and the second rendering instruction data is GLSL instruction data. The step of generating second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data using a preset algorithm includes: the virtual graphics library service using virglrenderer to parse the TGSI instruction data into GLSL instruction data; the step of sending the second rendering instruction data to the hardware GPU device using a GPU driver running in kernel mode includes: using libGLES adapted to the hardware GPU device to parse the GLSL instruction data into control instructions adapted to the GPU driver; calling the interface function provided by the GPU driver to execute the control instructions to write the GLSL instruction data to the hardware GPU device; and the step of obtaining the rendering result read from the hardware GPU device by the kernel-mode GPU driver includes: using EGLContext bound to virglrenderer, and through libGLES adapted to the hardware GPU device, reading the rendering result obtained from the hardware GPU device by the GPU driver.

[0091] In the above embodiments of the rendering and display device for multi-virtual machine systems, the specific processing of each module and its resulting technical effects can be referred to the relevant descriptions in the corresponding method embodiments, and will not be repeated here.

[0092] Exemplary vehicle

[0093] Figure 7 This is a block diagram illustrating a vehicle 700 according to an exemplary embodiment. The vehicle 700 may be a gasoline vehicle, a hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles.

[0094] Reference Figure 7 The vehicle 700 may include multiple subsystems, such as a drive system 710, a control system 720, a sensing system 730, a communication system 740, an information display system 750, and a computing processing system 760. The vehicle 700 may also include more or fewer subsystems, and each subsystem may include multiple components, which will not be described in detail here.

[0095] The drive system 710 includes components that provide power to the vehicle 700. These include, for example, an engine, an energy source, and a transmission.

[0096] The control system 720 includes components that provide control for the vehicle 700. These include, for example, vehicle control, cockpit equipment control, and driver assistance control.

[0097] The perception system 730 includes components that provide the vehicle 700 with perception of its surroundings. These include, for example, a vehicle positioning system, a laser sensor, a voice sensor, an ultrasonic sensor, and camera equipment.

[0098] The communication system 740 includes components that provide communication connectivity for the vehicle 700. These may include, for example, mobile communication networks (e.g., 3G, 4G, 5G networks), WiFi, Bluetooth, and vehicle-to-everything (V2X) connectivity.

[0099] The information display system 750 includes components that provide various information displays for the vehicle 700. These include, for example, vehicle information displays, navigation information displays, and entertainment information displays.

[0100] The computing processing system 760 includes components that provide data computing and processing capabilities for the vehicle 700. The computing processing system 760 may include at least one processor 761 and a memory 762. The processor 761 can execute instructions stored in the memory 762.

[0101] The processor 761 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0102] The memory 762 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0103] In this embodiment of the present disclosure, a set of instructions is stored in the memory 762, and the processor 761 can execute the set of instructions to implement all or part of the steps of the rendering and display method of the multi-virtual machine system described in any of the exemplary embodiments above.

[0104] Exemplary electronic devices

[0105] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. The electronic device 800 may be a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0106] Reference Figure 8 The electronic device 800 may include at least one processor 810 and a memory 820. The processor 810 can execute instructions stored in the memory 820. The processor 810 is communicatively connected to the memory 820 via a data bus. In addition to the memory 820, the processor 810 can also be communicatively connected to an input device 830, an output device 840, and a communication device 850 via the data bus.

[0107] The processor 810 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0108] The memory 820 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0109] In this embodiment of the present disclosure, the memory 820 stores executable instructions, and the processor 810 can read the executable instructions from the memory 820 and execute the instructions to implement all or part of the steps of the rendering and display method of the multi-virtual machine system described in any of the exemplary embodiments above.

[0110] Exemplary computer-readable storage media

[0111] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in any of the methods in the exemplary embodiments described above.

[0112] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0113] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0114] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0115] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A rendering and display method for a multi-virtual machine system, characterized in that, The method for sharing an on-chip system, including a hardware GPU and a hardware graphics card, among multiple virtual machines in the multi-virtual machine system through a virtualization layer includes: The virtualization layer, in response to receiving a memory-mapped I / O request from the virtio-gpu driver of the first virtual machine, uses the virtio device implemented in the virtualization layer to send rendering instruction arrival information to the virtual graphics library service running in user mode of the second virtual machine; wherein, the second virtual machine is a virtual machine with a hardware GPU device and a hardware graphics card, and the first virtual machine is a virtual machine in the multi-virtual machine system that does not have a hardware GPU device and a hardware graphics card. The virtual graphics library service, in response to receiving the rendering instruction arrival information, uses the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task, which is stored in the shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver; and Based on the first rendering instruction data, the rendering result corresponding to the first rendering instruction data is obtained using the hardware GPU device of the second virtual machine; Using the graphics card driver running on the second virtual machine, the image data to be displayed obtained from the rendering result is sent to the hardware graphics card of the second virtual machine, so that the hardware graphics card displays the rendering result on the target display device connected to the hardware graphics card; The virtio devices include virtio agent devices and virtio stub devices; The process of sending rendering instruction arrival information to the user-mode virtual graphics library service running in the second virtual machine using the virtio device implemented in the virtualization layer includes: The virtio proxy device, in response to detecting a memory-mapped I / O request from the virtio-gpu driver, instructs the virtio stub device to send the rendering instruction arrival information to the virtual graphics library service. The virtio proxy device is a device implemented using the virtio protocol and used in a virtualization environment to act as a proxy for the GPU backend device of the first virtual machine. The virtio stub device is a device implemented using the virtio protocol and used in a virtualization environment to communicate with the operating system in the second virtual machine.

2. The method according to claim 1, characterized in that, The virtio stub device sends the rendering instruction arrival information to the virtual graphics library service in the following manner: The virtio stub device triggers an interrupt of a preset type; The driver for the kernel-mode virtio stub device running in the first virtual machine, in response to receiving the preset type of interrupt, sends the rendering instruction arrival information to the virtual graphics library service.

3. The method according to claim 2, characterized in that, The step of using the ioctl mechanism to obtain the first rendering instruction data for executing the target rendering task, which is stored in the shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver, includes: The virtual graphics library service uses the ioctl interface function provided by the driver of the virtio stub device to obtain the first rendering instruction data read from the shared memory region by the driver of the virtio stub device.

4. The method according to claim 3, characterized in that, Based on the first rendering instruction data, and utilizing the hardware GPU device of the second virtual machine, the rendering result corresponding to the first rendering instruction data is obtained, including: Using a preset algorithm, second rendering instruction data adapted to the hardware GPU is generated based on the first rendering instruction data; The second rendering instruction data is sent to the hardware GPU device using a GPU driver running in kernel mode, so that the hardware GPU device generates the rendering result of the target rendering task according to the second rendering instruction data; The rendering results read from the hardware GPU device are obtained from the GPU driver in kernel mode.

5. The method according to claim 4, characterized in that, The first rendering instruction data is TGSI instruction data, and the second rendering instruction data is GLSL instruction data; The step of generating second rendering instruction data adapted to the hardware GPU based on the first rendering instruction data using a preset algorithm includes: The virtual graphics library service uses virglrenderer to parse the TGSI instruction data into GLSL instruction data; The step of sending the second rendering instruction data to the hardware GPU device using a GPU driver running in kernel mode includes: Using libGLES, which is compatible with the hardware GPU device, the GLSL instruction data is parsed into control instructions compatible with the GPU driver; The interface function provided by the GPU driver is invoked to execute the control instructions, so as to write the GLSL instruction data to the hardware GPU device; The step of obtaining the rendering results read from the hardware GPU device by the GPU driver in kernel mode includes: Using the EGLContext bound to virglrenderer, the rendering results obtained by the GPU driver from the hardware GPU device are read from the GPU driver via libGLES, which is adapted to the hardware GPU device.

6. A rendering and display device for a multi-virtual machine system, characterized in that, include: The sending module is configured to enable the virtualization layer, in response to receiving a memory-mapped I / O request from the virtio-gpu driver of the first virtual machine, to send rendering instruction arrival information to the virtual graphics library service in user mode running on the second virtual machine using the virtio device implemented in the virtualization layer; wherein the second virtual machine is a virtual machine with hardware GPU device and hardware graphics card resources, and the first virtual machine is a virtual machine in the multi-virtual machine system that does not have hardware GPU device and hardware graphics card resources. The first acquisition module is used to enable the virtual graphics library service to, in response to receiving the rendering instruction arrival information, use the ioctl mechanism to acquire the first rendering instruction data for executing the target rendering task stored in the shared memory area between the first virtual machine and the second virtual machine by the virtio-gpu driver. The second acquisition module is used to obtain the rendering result corresponding to the first rendering instruction data using the hardware GPU device of the second virtual machine based on the first rendering instruction data. The display module is used to send the image data to be displayed obtained from the rendering result to the hardware graphics card of the second virtual machine using the graphics card driver running on the second virtual machine, so that the hardware graphics card displays the rendering result on the target display device connected to the hardware graphics card. The virtio devices include virtio agent devices and virtio stub devices; The process of sending rendering instruction arrival information to the user-mode virtual graphics library service running in the second virtual machine using the virtio device implemented in the virtualization layer includes: The virtio proxy device, in response to detecting a memory-mapped I / O request from the virtio-gpu driver, instructs the virtio stub device to send the rendering instruction arrival information to the virtual graphics library service. The virtio proxy device is a device implemented using the virtio protocol and used in a virtualization environment to act as a proxy for the GPU backend device of the first virtual machine. The virtio stub device is a device implemented using the virtio protocol and used in a virtualization environment to communicate with the operating system in the second virtual machine.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the rendering and display method of the multi-virtual machine system according to any one of claims 1-5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the rendering and display method for the multi-virtual machine system as described in any one of claims 1-5.

9. A vehicle, characterized in that, A set of instructions is stored, which is executed by the vehicle to implement the rendering and display method of the multi-virtual machine system according to any one of claims 1-5.

Images