Virtual machine image processing method, device, system and electronic device
By directly accessing the video memory using the mechanism provided by the host graphics card driver in a virtual machine, the problem of low image data transmission efficiency in virtual machine image processing is solved, and efficient image processing is achieved that completes operations within the GPU, which is suitable for multiple graphics card types. .
Patent Information
- Application Number
- CN202210217444.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-07
AI Technical Summary
In the existing virtual machine image processing methods, the transmission efficiency of image data is low, especially because the data needs to be copied from video memory to system memory and then copied back to video memory, resulting in poor performance and the GPU's encoding capability cannot be fully utilized.
By utilizing the mechanisms provided by the host graphics card driver, such as EGLStream and DMABUF, directly accessing the video memory of the host graphics card, the image encoding module can operate within the GPU to avoid data copying, thereby improving the transmission efficiency of image data.
It has achieved the improvement of image data transmission efficiency, eliminated the copy operations from video memory to system memory and then to video memory, improved the image processing performance of virtual machines, and is suitable for different types of graphics cards and virtual machine environments.
Smart Images

Figure CN114579255B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, system and electronic equipment for image processing of a virtual machine. Background Art
[0002] At present, the virtual machine running software transmits images through the built-in Simple Protocol for Independent Computing Environment (SPICE) protocol or Virtual Network Console (VNC) protocol. For example, in the current implementation of the virtual machine running software, the host sound card data is encoded by the SPICE server or the VNC server and then transmitted. However, for the existing image data processing method, the transmission efficiency of the image data is low. Summary of the invention
[0003] In view of this, an object of the present invention is to provide an image processing method, device, system and electronic device for a virtual machine, which can improve the transmission efficiency of image data.
[0004] In a first aspect, an embodiment of the present invention provides an image processing method for a virtual machine, which is applied to an image encoding module corresponding to the virtual machine, wherein the virtual machine further corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine, wherein the host machine includes a host graphics card; the method comprises:
[0005] The video memory of the host graphics card is accessed by using a mechanism provided by the host graphics card driver; wherein the video memory stores image data obtained by rendering by the host graphics card based on an image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine running module to the host graphics card according to the image to be rendered of the virtual machine;
[0006] The image data in the video memory is encoded by a graphics processing unit (GPU) of the host graphics card to obtain encoded image data.
[0007] In one implementation, the mechanism provided by the host graphics card driver includes: an enterprise generation language stream mechanism (Enterprise Generation Language Stream, referred to as EGLStream) and / or a direct memory access buffer (Direct Memory Access Buffer, referred to as DMABUF) mechanism.
[0008] In one implementation, the mechanism provided by the host graphics card driver is an EGLStream mechanism; the method further includes:
[0009] Before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, the first EGL interface is initialized based on the EGLStream mechanism and an EGLStream is created to obtain a file handle of a Stream data stream, the first CUDA is initialized based on the EGLStream and a CUeglStreamConnection is created, and the CUeglStreamConnection is connected to the EGLStream, so that the virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism and creates a rendering environment supporting the EGLStream;
[0010] The file handle of the Stream data stream is sent to the virtual machine running module, so that the virtual machine running module uses the file handle of the Stream data stream to create an image area for the image to be rendered.
[0011] In one embodiment, after the step of encoding the image data in the video memory by the GPU of the host graphics card to obtain the encoded image data, the method further includes:
[0012] When the connection of the image encoding module is disconnected, the resources of the EGLStream are deleted.
[0013] In a second aspect, an embodiment of the present invention provides an image processing method for a virtual machine, wherein the virtual machine corresponds to an image encoding module, a virtual machine running module of the virtual machine, and a host machine of the virtual machine, wherein the host machine includes a host graphics card; the method comprises:
[0014] The virtual machine running module obtains the image to be rendered of the virtual machine, and sends an image rendering instruction to the graphics card of the host machine according to the image to be rendered;
[0015] The host graphics card performs rendering based on the image rendering instruction, and stores the image data obtained after rendering in the graphics memory of the host graphics card;
[0016] The image encoding module uses the mechanism provided by the host graphics card driver to access the video memory, and encodes the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0017] In one implementation, the mechanism provided by the host graphics card driver is an EGLStream mechanism;
[0018] Before the step in which the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, the method further includes:
[0019] The image encoding module initializes the first EGL interface based on the EGLStream mechanism and creates an EGLStream, obtains a file handle of a Stream data stream, initializes a first unified computing architecture (Compute Unified Device Architecture, CUDA for short) based on the EGLStream and creates a CUeglStreamConnection (data stream connection), and connects the CUeglStreamConnection to the EGLStream;
[0020] The virtual machine running module initializes the second EGL interface and the first open graphics library (Open Graphics Library, referred to as OpenGL) interface based on the EGLStream mechanism, and creates a rendering environment that supports the EGLStream;
[0021] The image encoding module sends the file handle of the Stream data stream to the virtual machine running module;
[0022] The virtual machine running module creates an image area for the image to be rendered using the file handle of the Stream data flow.
[0023] In one embodiment, after the step of rendering by the host graphics card based on the image rendering instruction, the method further includes:
[0024] The virtual machine running module maps the image data obtained after the host graphics card renders the image data into the image area, and sends a message of image rendering completion to the image encoding module;
[0025] The image encoding module reads the CUeglStreamConnection, and sends the CUeglStreamConnection to the first CUDA application programming interface (Application Programming Interface, API for short), so as to perform encoding operations on the image data based on the API.
[0026] In one embodiment, after the image encoding module encodes the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data, the method further includes:
[0027] When the connection of the image encoding module is disconnected, deleting the resources of the EGLStream;
[0028] The virtual machine running module deletes the image area and the resources of the EGLStream.
[0029] In one embodiment, the mechanism provided by the host graphics card driver is a DMABUF mechanism;
[0030] Before the step in which the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, the method further includes:
[0031] The virtual machine running module initializes the third EGL interface and the second OpenGL interface based on the DMABUF mechanism, creates a first texture object and allocates the video memory to the first texture object, creates a first EGLImage object using the first texture object, calls eglExportDMABUFImageMESA (data stream connection) to obtain the DMABUF descriptor of the first texture object, initializes the second CUDA based on the DMABUF mechanism, and registers the EGLImage object in the second CUDA;
[0032] The image encoding module initializes the fourth EGL interface and the third OpenGL interface based on the DMABUF mechanism;
[0033] The virtual machine running module sends the DMABUF descriptor to the image encoding module;
[0034] The image encoding module creates a second EGLImage object using the DMABUF descriptor, and binds the second EGLImage object to a second texture object, so that the second texture object is associated with the first texture object and both point to the same video memory.
[0035] In one embodiment, after the step of rendering by the host graphics card based on the image rendering instruction, the method further includes:
[0036] The virtual machine running module maps the image data obtained after the host graphics card renders the image data onto the first texture object, and sends a message of image rendering completion to the image encoding module;
[0037] When the host graphics card is an Nvidia graphics card, the image encoding module reads the image data through the CUDA EGL interoperation API and sends the image data to the CUDA encoding library;
[0038] When the host graphics card is an AMD or INTEL open source driver graphics card, the image encoding module reads the image data through the video acceleration interface (Video Acceleration API, Vaapi) EGL interoperation API, and sends the image data to Vaapi for encoding.
[0039] In one embodiment, after the image encoding module encodes the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data, the method further includes:
[0040] When the connection of the image encoding module is disconnected, deleting the resource of the second texture object;
[0041] The virtual machine running module deletes the resource of the first texture object.
[0042] In one implementation, the DMABUF descriptor includes: texture format information of the first texture object and / or a file handle of the first texture object.
[0043] In a third aspect, an embodiment of the present invention provides an image processing device for a virtual machine, which is applied to an image encoding module corresponding to the virtual machine, wherein the virtual machine further corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine, wherein the host machine includes a host graphics card; the device comprises:
[0044] An access module is used to access the video memory of the host graphics card by using a mechanism provided by the host graphics card driver; wherein the video memory stores image data obtained by rendering by the host graphics card based on an image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine running module to the host graphics card according to the image to be rendered of the virtual machine;
[0045] The encoding module is used to encode the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0046] In a fourth aspect, an embodiment of the present invention provides an image processing system for a virtual machine, comprising: an image encoding module corresponding to the virtual machine, a virtual machine running module of the virtual machine, and a host machine of the virtual machine, wherein the host machine includes a host graphics card;
[0047] The virtual machine running module is used to obtain the image to be rendered of the virtual machine, and send an image rendering instruction to the host graphics card according to the image to be rendered;
[0048] The host graphics card is used to render based on the image rendering instruction, and store the image data obtained after rendering in the video memory of the host graphics card;
[0049] The image encoding module is used to access the video memory using a mechanism provided by the host graphics card driver, and encode the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0050] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.
[0051] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.
[0052] The embodiment of the present invention provides a method, device, system and electronic device for image processing of a virtual machine, which are applied to an image encoding module corresponding to the virtual machine. The virtual machine also corresponds to a virtual machine operation module of the virtual machine and a host machine of the virtual machine. The host machine includes a host graphics card, which can access the video memory of the host graphics card by using a mechanism provided by the host graphics card driver, wherein the video memory stores image data obtained by rendering by the host graphics card based on an image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine operation module to the host graphics card according to the image to be rendered of the virtual machine, and then the image data in the video memory is encoded by the GPU of the host graphics card to obtain the encoded image data. The above method enables the image encoding module to directly access the video memory of the host graphics card by using the mechanism provided by the host graphics card driver, and does not need to copy the data from the video memory. The image encoding module can directly access the video memory, and all operations are completed inside the GPU, eliminating the time-consuming operations of image data from the video memory to the system memory and then from the system memory to the video memory, thereby improving the transmission efficiency of image data.
[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0056] Figure 1 A schematic diagram of a prior art solution;
[0057] Figure 2 A schematic diagram of a flow chart of an image processing method for a virtual machine provided by an embodiment of the present invention;
[0058] Figure 3 A schematic diagram of a data interaction mechanism is provided for an embodiment of the present invention;
[0059] Figure 4 A schematic diagram of a flow chart of another image processing method for a virtual machine provided by an embodiment of the present invention;
[0060] Figure 5 A schematic diagram of the structure of an image processing device for a virtual machine provided by an embodiment of the present invention;
[0061] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] Currently, if Figure 1 As shown, the virtual machine running software transmits images through its own SPICE protocol or VNC protocol. In the current implementation of the virtual machine running software, the host graphics card data is encoded by the SPICE server or VNC server and then transmitted. The virtual machine graphics card (Virtio IO GPU, referred to as Virtio-GPU) sends rendering instructions to the host graphics card. After the rendering is completed, the rendered data is copied to the system memory, and the image encoding module is notified through the inter-process communication mechanism. The image encoding module copies the data from the system memory back to the video memory and uses the graphics card encoding API for hardware encoding.
[0064] However, for the existing image data processing method, the transmission capability of the SPICE\VNC protocol is relatively poor and cannot meet the real-time requirements. Moreover, the existing QEMU encoding module is inside QEMU and has high coupling. The SPICE\VNC server needs to copy data from the host GPU video memory, which takes a long time. If software encoding is used, the performance is poor and the encoding capability of the GPU cannot be fully utilized. If GPU hardware encoding is used, the data needs to be copied back to the GPU video memory, and the copying process is also time-consuming. Based on this, an embodiment of the present invention provides an image processing method, device, system and electronic device for a virtual machine, so that the image encoding module can directly access the video memory of the host graphics card using the mechanism provided by the host graphics card driver, thereby improving the transmission efficiency of image data.
[0065] To facilitate understanding of this embodiment, firstly, a virtual machine image processing method disclosed in an embodiment of the present invention is described in detail. The method can be applied to an image encoding module corresponding to the virtual machine. The virtual machine also corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine. The host machine includes a host graphics card. Figure 2 The flowchart of a method for processing an image of a virtual machine is shown, and the method mainly includes the following steps S210 to S220:
[0066] Step S210, using a mechanism provided by the host graphics card driver to access the video memory of the host graphics card.
[0067] The video memory stores image data obtained by rendering by the host graphics card based on image rendering instructions. The image rendering instructions are image rendering instructions sent by the virtual machine running module to the host graphics card according to the image to be rendered of the virtual machine.
[0068] For example, Figure 3 As shown, the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered of the virtual machine, and the video memory stores the image data obtained by the host graphics card rendering based on the image rendering instruction. Since the image encoding module is decoupled from the outside of the virtual machine running module, it is necessary to use the mechanism provided by the host graphics card driver to access the video memory of the host graphics card.
[0069] Step S220 , encoding the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0070] Exemplarily, a graphics card is composed of a GPU and video memory, and the image encoding module corresponding to the virtual machine directly encodes the image data in the video memory through the GPU of the host graphics card. This can be understood as utilizing a "zero copy" method, i.e. utilizing the mechanism provided by the host graphics card driver to enable the virtual machine running module and the image encoding module to share the video memory, thereby enabling the image encoding module to encode the image data in the video memory through the GPU of the host graphics card to directly obtain the encoded image data.
[0071] This method enables the image encoding module to directly access the video memory of the host graphics card by using the mechanism provided by the host graphics card driver. There is no need to copy data from the video memory. The image encoding module can directly access the video memory, and all operations are completed inside the GPU, eliminating the two time-consuming operations of image data from the video memory to the system memory and then from the system memory to the video memory, thereby improving the transmission efficiency of image data.
[0072] In some embodiments, the mechanism provided by the host graphics card driver may include multiple types, so that the method can be more flexibly adapted to different graphics cards according to different mechanisms, so that the image encoding module can directly access the graphics card resources of the host through different mechanisms, thereby improving the transmission efficiency of image data. As an example, the mechanism provided by the host graphics card driver includes: EGLStream mechanism and / or DMABUF mechanism.
[0073] For example, EGLStream is a mechanism provided by Nvidia graphics card driver to efficiently transfer graphics images between different APIs. It is a common existing standard mechanism. These APIs can run on different processes. The DMABUF mechanism is widely applicable and can be applied to, but not limited to, graphics cards such as Nvidia, AMD, and INTEL.
[0074] By making the mechanism provided by the host graphics card driver include multiple types, the method can be adapted to different types of graphics cards, making the application more extensive and improving the transmission efficiency of image data.
[0075] In some embodiments, the method can provide different mechanisms according to the host graphics card driver, flexibly adapt to different host graphics cards, make the application scenarios more extensive, and realize that the image encoding module can directly access the video memory of the host graphics card through multiple mechanisms, thereby improving the transmission efficiency of image data while ensuring data transmission. As an example, the mechanism provided by the host graphics card driver is the EGLStream mechanism; the method can also include the following steps:
[0076] Step S200, before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, initialize the first EGL interface based on the EGLStream mechanism and create EGLStream, obtain the file handle of the Stream data stream, initialize the first CUDA based on EGLStream and create CUeglStreamConnection, connect CUeglStreamConnection to EGLStream, so that the virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism and creates a rendering environment that supports EGLStream;
[0077] Step S202: Send the file handle of the Stream data stream to the virtual machine running module, so that the virtual machine running module creates an image area for the image to be rendered using the file handle of the Stream data stream.
[0078] Among them, EGL is a programming technology that aims to solve the problem of modern, multi-platform application development by providing an abstraction layer above lower-level languages, frameworks and runtime platforms. CUDA is a computing platform for Nvidia graphics cards. It is a general-purpose parallel computing architecture that enables GPUs to solve complex computing problems. EGLStream is a mechanism provided by Nvidia graphics card drivers to efficiently transmit graphics images between different APIs. It is a common existing standard mechanism. These APIs can run on different processes. CUeglStreamConnection is a data stream created by CUDA for connecting to EGLStream. OpenGL is a cross-language, cross-platform interface for rendering 2D and 3D vector graphics. This interface is used to draw complex three-dimensional scenes from simple graphics bits.
[0079] Exemplarily, first, the image coding module (the industry commonly uses Consumer to refer to the image coding module) initializes the first EGL interface based on the EGLStream mechanism, creates an EGLStream, and obtains the file handle of the data stream. Among them, in data transmission, to read data from a file, the application must first call the operating system function and transmit the file name, and select a path to the file to open the file. The function retrieves a sequence number, that is, the file handle, which is the only identification basis for the opened file. After that, the first CUDA is initialized based on the image coding module EGLStream and CUeglStreamConnection is created, and CUeglStreamConnection is connected to EGLStream. The virtual machine running module (the industry commonly uses Producer to refer to the image coding module) initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism, and creates a rendering environment that supports EGLStream, so as to facilitate docking with the image coding module, and then render the image data. After the docking preparation is completed, the image coding module sends the acquired file handle to the virtual machine running module, and the virtual machine running module creates an image area for the image to be rendered according to the file handle of the received data stream.
[0080] By using the EGLStream mechanism provided by the Nvidia graphics card, the image encoding module is connected to the virtual machine running module through an interface, and the virtual machine running module creates an image area for the image to be rendered, so that the image encoding module can render the image data. This method can provide different mechanisms according to the host graphics card driver and flexibly adapt to different host graphics cards, making the application scenarios more extensive. It is realized that through multiple mechanisms, the image encoding module can directly access the video memory of the host graphics card, thereby improving the transmission efficiency of image data.
[0081] Based on the above steps S200 and S202, when the image data encoding is completed, the image encoding module can flexibly process the related resources according to the connection status to avoid the related resources occupying space. For example, when the image encoding module is disconnected, the image encoding module can delete the related resources to avoid occupying space and affecting the performance of the device. As an example, after the above step S220, the method may further include the following steps:
[0082] Step S230, when the connection of the image encoding module is disconnected, delete the resources of EGLStream.
[0083] Exemplarily, when the image encoding module is disconnected, the image encoding module may release related resources of EGLStream by deletion, and the virtual machine running module may also release related resources of the image area and EGLStream by deletion.
[0084] By deleting the data resources related to the image rendered this time as the end of the data encoding process, it is possible to avoid the related resources occupying space and facilitate the rendering of the next image.
[0085] To facilitate understanding of this embodiment, firstly, another virtual machine image processing method disclosed in an embodiment of the present invention is described in detail. The virtual machine corresponds to an image encoding module, a virtual machine running module of the virtual machine, and a host machine of the virtual machine. The host machine includes a host graphics card. Figure 4 The flowchart of a method for processing an image of a virtual machine is shown, and the method mainly includes the following steps S410 to S420:
[0086] Step S410: the virtual machine running module obtains the image to be rendered of the virtual machine, and sends an image rendering instruction to the host machine graphics card according to the image to be rendered.
[0087] For example, Figure 3 As shown, the virtual machine operation module first obtains the image to be rendered of the virtual machine. The virtual machine operation module takes QEMU as an example. QEMU is a general and open source virtual machine monitor that performs hardware virtualization. It directly executes the guest code of the host central processing unit (CPU), so it achieves performance close to that of the native machine when it is used as a virtualizer. After obtaining the image to be rendered of the virtual machine, the virtual machine operation module can send an image rendering instruction to the host graphics card according to the image to be rendered, so that the host graphics card renders the image to be rendered based on the image rendering instruction.
[0088] In step S420 , the host graphics card performs rendering based on the image rendering instruction, and stores the image data obtained after rendering in the graphics memory of the host graphics card.
[0089] For example, Figure 3 As shown, after receiving the image rendering instruction sent by the virtual machine running module, the host graphics card renders the image to be rendered according to the instruction, obtains the rendered image data, and stores the image data in the graphics memory.
[0090] Step S430: the image encoding module uses the mechanism provided by the host graphics card driver to access the video memory, and encodes the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0091] For example, Figure 3 As shown, the image encoding module can directly access the video memory of the host graphics card using the mechanism provided by the host graphics card driver, wherein different host graphics card drivers provide different mechanisms. After the image encoding module accesses the video memory, the GPU of the host graphics card can encode the image data in the video memory to obtain the encoded image data.
[0092] This method enables the image encoding module to directly access the video memory of the host graphics card by using the mechanism provided by the host graphics card driver. There is no need to copy data from the video memory. The image encoding module can directly access the video memory, and all operations are completed inside the GPU, eliminating the two time-consuming operations of image data from the video memory to the system memory and then from the system memory to the video memory, thereby improving the transmission efficiency of image data.
[0093] In some embodiments, the method can provide different mechanisms according to the host graphics card driver, flexibly adapt to different host graphics cards, make the application scenarios more extensive, and realize that the image encoding module can directly access the video memory of the host graphics card through multiple mechanisms, thereby improving the transmission efficiency of image data while ensuring data transmission. As an example, the mechanism provided by the host graphics card driver is the EGLStream mechanism; before the above step S410, the method can also include the following steps:
[0094] Step a), the image encoding module initializes the first EGL interface based on the EGLStream mechanism and creates an EGLStream, obtains the file handle of the Stream data stream, initializes the first CUDA based on the EGLStream and creates a CUeglStreamConnection, and connects the CUeglStreamConnection to the EGLStream.
[0095] Step b), the virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism, and creates a rendering environment supporting EGLStream.
[0096] Step c), the image encoding module sends the file handle of the Stream data stream to the virtual machine running module.
[0097] Step d), the virtual machine operation module uses the file handle of the Stream data stream to create an image area for the image to be rendered.
[0098] Among them, EGL is a programming technology that aims to solve the problem of modern, multi-platform application development by providing an abstraction layer above lower-level languages, frameworks and runtime platforms. CUDA is a computing platform for Nvidia graphics cards. It is a general-purpose parallel computing architecture that enables GPUs to solve complex computing problems. EGLStream is a mechanism provided by Nvidia graphics card drivers to efficiently transmit graphics images between different APIs. It is a common existing standard mechanism. These APIs can run on different processes. CUeglStreamConnection is a data stream created by CUDA for connecting to EGLStream. OpenGL is a cross-language, cross-platform interface for rendering 2D and 3D vector graphics. This interface is used to draw complex three-dimensional scenes from simple graphics bits.
[0099] Exemplarily, first, the image coding module (the industry commonly uses Consumer to refer to the image coding module) initializes the first EGL interface based on the EGLStream mechanism, creates an EGLStream, and obtains the file handle of the data stream. Among them, in data transmission, to read data from a file, the application must first call the operating system function and transmit the file name, and select a path to the file to open the file. The function retrieves a sequence number, that is, the file handle, which is the only identification basis for the opened file. After that, the first CUDA is initialized based on the image coding module EGLStream and CUeglStreamConnection is created, and CUeglStreamConnection is connected to EGLStream. The virtual machine running module (the industry commonly uses Producer to refer to the image coding module) initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism, and creates a rendering environment that supports EGLStream, so as to facilitate docking with the image coding module, and then render the image data. After the docking preparation is completed, the image coding module sends the acquired file handle to the virtual machine running module, and the virtual machine running module creates an image area for the image to be rendered according to the file handle of the received data stream.
[0100] By using the EGLStream mechanism provided by the Nvidia graphics card, the image encoding module is connected to the virtual machine running module through an interface, and the virtual machine running module creates an image area for the image to be rendered, so that the image encoding module can render the image data. This method can provide different mechanisms according to the host graphics card driver and flexibly adapt to different host graphics cards, making the application scenarios more extensive. It is realized that through multiple mechanisms, the image encoding module can directly access the video memory of the host graphics card, thereby improving the transmission efficiency of image data.
[0101] Based on the above steps a), b), c) and d), when the image rendering is completed, the image encoding module can accurately and efficiently perform encoding operations on the image data based on the specified API. For example, the virtual machine running module can send a message that the image rendering is completed to the image encoding module, so that the image encoding module can perform encoding operations on the image data based on the specified API.
[0102] As an example, after the above step S420, the method may further include the following steps:
[0103] Step e), the virtual machine operation module maps the image data obtained after the host graphics card renders it into the image area, and sends a message indicating that the image rendering is completed to the image encoding module.
[0104] Step f), the image encoding module reads CUeglStreamConnection, and sends CUeglStreamConnection to the first CUDA API, so as to perform encoding operations on the image data based on the API.
[0105] Exemplarily, the virtual machine operation module starts the rendering process, uses the host graphics card to render the image, and outputs the image data obtained after the host graphics card is rendered to the image area, and sends a message of image rendering completion to the image encoding module through the data stream interface established in the early stage, notifying the image encoding module that a frame of the picture is rendered. After that, the image encoding module reads CUeglStreamConnection and sends it to the first CUDA API to perform encoding operations on the image data based on the API.
[0106] By enabling the virtual machine running module to map the image data obtained after rendering by the host graphics card to the image area, and sending a message that the image rendering is completed to the image encoding module, the image encoding module reads CUeglStreamConnection and sends CUeglStreamConnection to the first CUDA API to perform encoding operations on the image data based on the API. After the image rendering is completed, the image encoding module can accurately and efficiently perform encoding operations on the image data based on the specified API.
[0107] Based on the above steps a), b), c) and d), when the image data encoding is completed, the image encoding module and the virtual machine operation module can flexibly process the relevant resources according to the connection status to avoid the relevant resources occupying space. For example, when the image encoding module is disconnected, the image encoding module and the virtual machine operation module can delete the relevant resources to avoid occupying space and affecting the performance of the device. As an example, after the above step S430, the method may further include the following steps:
[0108] Step g), when the connection of the image encoding module is disconnected, the resources of EGLStream are deleted.
[0109] Step h), the virtual machine running module deletes the image area and EGLStream resources.
[0110] Exemplarily, when the image encoding module is disconnected, the image encoding module may release related resources of EGLStream by deletion, and the virtual machine running module may also release related resources of the image area and EGLStream by deletion.
[0111] By deleting the data resources related to the image rendered this time as the end of the data encoding process, it is possible to avoid the related resources occupying space and facilitate the rendering of the next image.
[0112] In some embodiments, the method can provide different mechanisms according to the host graphics card driver, flexibly adapt to different host graphics cards, make the application scenarios more extensive, and realize that the image encoding module can directly access the video memory of the host graphics card through multiple mechanisms, thereby improving the transmission efficiency of image data while ensuring data transmission. As an example, the mechanism provided by the host graphics card driver is the DMABUF mechanism; before the above step S410, the method can also include the following steps:
[0113] Step i), the virtual machine running module initializes the third EGL interface and the second OpenGL interface based on the DMABUF mechanism, creates a first texture object and allocates video memory to the first texture object, creates a first EGLImage object using the first texture object, calls eglExportDMABUFImageMESA to obtain the DMABUF descriptor of the first texture object, initializes the second CUDA based on the DMABUF mechanism, and registers the EGLImage object to the second CUDA.
[0114] Step j), the image encoding module initializes the fourth EGL interface and the third OpenGL interface based on the DMABUF mechanism.
[0115] Step k), the virtual machine running module sends the DMABUF descriptor to the image encoding module.
[0116] Step 1), the image encoding module creates a second EGLImage object using the DMABUF descriptor, and binds the second EGLImage object to the second texture object, so that the second texture object is associated with the first texture object and both point to the same video memory.
[0117] Among them, the DMABUF mechanism is widely applicable, and can be applied to, but not limited to, graphics cards such as Nvidia, AMD, and INTEL. EGLImage represents a type of shared resource created by the EGL client API. In theory, applications and related client APIs can create any type of shared data for any purpose.
[0118] Exemplarily, the virtual machine operation module first initializes the third EGL interface and the second OpenGL interface based on the DMABUF mechanism, creates a first texture object and allocates video memory to the first texture object, creates a first EGLImage object using the first texture object, calls eglExportDMABUFImageMESA to obtain the DMABUF descriptor of the first texture object, which contains the format information of the texture (width, height, pixel format, etc.) and a file handle, and then initializes the second CUDA based on the DMABUF mechanism, and registers the EGLImage object to the second CUDA. The image encoding module also initializes the corresponding fourth EGL interface and the third OpenGL interface based on the DMABUF mechanism to facilitate the connection of the interfaces. The virtual machine operation module sends the DMABUF descriptor to the image encoding module, which can use the DMABUF descriptor to create a second EGLImage object and bind the second EGLImage object to the second texture object, so that the second texture object is associated with the first texture object and both point to the same video memory.
[0119] By making the image encoding module and the virtual machine running module establish an EGL interface and an OpenGL interface for connection according to the DMABUF mechanism provided by the graphics card, and then through a series of operations, the image encoding module and the virtual machine running module are both based on the same video memory for data interaction and processing, so that the method can provide different mechanisms according to the host graphics card driver, flexibly adapt to different host graphics cards, make the application scenarios more extensive, and realize that the image encoding module can directly access the video memory of the host graphics card through multiple mechanisms, thereby improving the transmission efficiency of image data.
[0120] Based on the above steps i), j), k) and l), the image encoding module can flexibly select an API for reading image data according to the graphics card type of the host machine, perform different operations according to different graphics card categories, and send the image data to different modules for rendering, thereby achieving efficient transmission of image data. As an example, after the above step S420, the method may further include the following steps:
[0121] In step m), the virtual machine operation module maps the image data obtained after the host graphics card renders the image data onto the first texture object, and sends a message indicating that the image rendering is completed to the image encoding module.
[0122] Step n), when the host graphics card is a Nvidia graphics card, the image encoding module reads the image data through the CUDA EGL interoperation API and sends the image data to the CUDA encoding library.
[0123] Step o), when the host graphics card is an AMD or INTEL open source driver graphics card, the image encoding module reads the image data through the VaapiEGL interoperability API and sends the image data to Vaapi for encoding.
[0124] Among them, Vaapi is a set of open source libraries and standards for video hardware encoding and decoding.
[0125] Exemplarily, the virtual machine running module starts the rendering process, outputs the rendered image to the first texture object created previously, and the rendered picture is still stored in the video memory of the host graphics card, and then sends a message through the EGL interface and the OpenGL interface to notify the image encoding module that a frame of the picture is rendered. Afterwards, the image encoding module can have different operations depending on the type of graphics card. For example, when the image encoding module is an Nvidia graphics card in the host machine, the image encoding module reads the image data through the CUDAEGL interoperation API and sends the image data to the CUDA encoding library; when the image encoding module is an AMD or INTEL open source driver graphics card in the host machine, the image encoding module reads the image data through the Vaapi EGL interoperation API and sends the image data to Vaapi for image / video encoding.
[0126] By making the virtual machine running module map the image data obtained after the host graphics card renders to the first texture object, and sending a message that the image rendering is completed to the image encoding module, the image encoding module can then flexibly select an API for reading image data according to the graphics card type of the host, perform different operations according to different graphics card categories, and send the image data to different modules for rendering, thereby achieving efficient transmission of image data.
[0127] Based on the above steps i), j), k) and l), the image encoding module and the virtual machine operation module can flexibly process related resources according to the connection status to avoid related resources occupying space. For example, when the image encoding module is disconnected, the image encoding module and the virtual machine operation module can delete related resources to avoid occupying space and affecting device performance.
[0128] As an example, after the above step S430, the method may further include the following steps:
[0129] Step p), when the connection of the image encoding module is disconnected, the resources of the second texture object are deleted.
[0130] Step q), the virtual machine running module deletes the resources of the first texture object.
[0131] Exemplarily, similar to the EGLStream mechanism, when the image encoding module is disconnected, the image encoding module can release the related resources of the second texture object by deletion, and the virtual machine running module can also release the related resources of the first texture object by deletion.
[0132] By deleting the data resources related to the image rendered this time as the end of the data encoding process, it is possible to avoid the related resources occupying space and facilitate the rendering of the next image.
[0133] Based on the above steps i), j), k) and l), the DMABUF descriptor may include multiple types, so that the virtual machine operation module can more flexibly send multiple types of descriptors to the image encoding module, so that the image encoding module creates a second EGLImage object according to the DMABUF descriptor, and binds the second EGLImage object to the second texture object, so that the second texture object is associated with the first texture object and both point to the same video memory, thereby achieving efficient transmission of image data. As an example, the DMABUF descriptor includes: texture format information of the first texture object and / or the file handle of the first texture object.
[0134] Exemplarily, the texture format information of the first texture object may include but is not limited to information such as width, height, and pixel format. In data transmission, to read data from a file, the application first needs to call the operating system function and pass the file name, and select a path to the file to open the file. The function retrieves a sequence number, i.e., a file handle, which is a unique identification basis for the opened file.
[0135] By making the DMABUF descriptor include multiple types, the image encoding module and the virtual machine running module can be accurately pointed to the same video memory, thereby achieving efficient transmission of image data.
[0136] In summary, the above-mentioned virtual machine image processing method provided by the embodiment of the present invention includes at least the following features:
[0137] (1) The present invention decouples the image encoding module from the virtual machine running module, which facilitates docking with different video encoding transmission schemes.
[0138] (2) The present invention eliminates the time-consuming operations of transferring image data from video memory to system memory and then from system memory to video memory, thereby improving the transmission efficiency of image data.
[0139] (3) The present invention is applicable to all virtual machines that use Virtio-GPU for 3D rendering. The host graphics card types include NVIDIA, AMD, and INTEL graphics cards. It is applicable to all virtual machines that support Virtio-GPU, including Android and Linux.
[0140] For the image processing method of the virtual machine provided in the above embodiment, the embodiment of the present invention provides an image processing device for a virtual machine, which is applied to the image encoding module corresponding to the virtual machine. The virtual machine also corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine, and the host machine includes a host graphics card. Figure 5 The structure diagram of an image processing device for a virtual machine shown in FIG. 1 mainly includes the following parts:
[0141] The access module 501 is used to access the video memory of the host graphics card by using the mechanism provided by the host graphics card driver; wherein the video memory stores image data obtained by rendering by the host graphics card based on the image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine running module to the host graphics card according to the image to be rendered by the virtual machine;
[0142] The encoding module 502 is used to encode the image data in the video memory through the GPU of the host graphics card to obtain encoded image data.
[0143] In one implementation, the mechanism provided by the host graphics card driver includes: an EGLStream mechanism and / or a DMABUF mechanism.
[0144] In one implementation, the mechanism provided by the host graphics card driver is an EGLStream mechanism; the device further includes:
[0145] A creation module is used for initializing a first EGL interface and creating an EGLStream based on the EGLStream mechanism before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, obtaining a file handle of a Stream data stream, initializing a first CUDA based on the EGLStream and creating a CUeglStreamConnection, and connecting the CUeglStreamConnection to the EGLStream, so that the virtual machine running module initializes a second EGL interface and a first OpenGL interface based on the EGLStream mechanism and creates a rendering environment supporting the EGLStream;
[0146] The sending module is used to send the file handle of the Stream data flow to the virtual machine running module, so that the virtual machine running module uses the file handle of the Stream data flow to create an image area for the image to be rendered.
[0147] In one embodiment, the device further comprises:
[0148] The deleting module is used to delete the resources of the EGLStream when the connection of the image encoding module is disconnected.
[0149] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0150] Regarding the image processing method for a virtual machine provided in the above-mentioned embodiment, an embodiment of the present invention provides an image processing system for a virtual machine, the system comprising:
[0151] An image encoding module corresponding to the virtual machine, a virtual machine running module of the virtual machine, and a host machine of the virtual machine, wherein the host machine includes a host machine graphics card;
[0152] The virtual machine running module is used to obtain the image to be rendered of the virtual machine, and send an image rendering instruction to the host graphics card according to the image to be rendered;
[0153] The host graphics card is used to render based on the image rendering instruction, and store the rendered image data in the video memory of the host graphics card;
[0154] The image encoding module is used to access the video memory using the mechanism provided by the host graphics card driver, and encode the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data.
[0155] The above system enables the image encoding module to directly access the video memory of the host graphics card by utilizing the mechanism provided by the host graphics card driver. There is no need to copy data from the video memory. The image encoding module can directly access the video memory, and all operations are completed inside the GPU, eliminating the two time-consuming operations of transferring image data from the video memory to the system memory and then from the system memory to the video memory, thereby improving the transmission efficiency of image data.
[0156] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes any one of the methods in the above implementation manners.
[0157] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, the electronic device comprises: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.
[0158] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0159] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0160] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any of the embodiments of the above-mentioned embodiments of the present invention can be applied to the processor 60 or implemented by the processor 60.
[0161] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.
[0162] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be referred to the previous method embodiment, which will not be repeated here.
[0163] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0164] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for processing an image of a virtual machine, characterized in that: An image encoding module corresponding to the virtual machine, the virtual machine also corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine, the host machine includes a host graphics card; including: The video memory of the host graphics card is accessed by using a mechanism provided by the host graphics card driver; wherein the video memory stores image data obtained by rendering by the host graphics card based on an image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine running module to the host graphics card according to the image to be rendered of the virtual machine; Encoding the image data in the video memory through the GPU of the host graphics card to obtain encoded image data; The mechanism provided by the host graphics card driver is the EGLStream mechanism; and further includes: Before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, the first EGL interface is initialized based on the EGLStream mechanism and an EGLStream is created to obtain a file handle of a Stream data stream, the first CUDA is initialized based on the EGLStream and a CUeglStreamConnection is created, and the CUeglStreamConnection is connected to the EGLStream, so that the virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism and creates a rendering environment supporting the EGLStream; The file handle of the Stream data stream is sent to the virtual machine running module, so that the virtual machine running module uses the file handle of the Stream data stream to create an image area for the image to be rendered.
2. The method according to claim 1, characterized in that The mechanism provided by the host graphics card driver includes: EGLStream mechanism and / or DMABUF mechanism.
3. The method according to claim 1, characterized in that: After encoding the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data, the method further includes: When the connection of the image encoding module is disconnected, the resources of the EGLStream are deleted.
4. A method for processing an image of a virtual machine, characterized in that: The virtual machine corresponds to an image encoding module, a virtual machine running module of the virtual machine and a host machine of the virtual machine, and the host machine includes a host machine graphics card; including: The virtual machine running module obtains the image to be rendered of the virtual machine, and sends an image rendering instruction to the graphics card of the host machine according to the image to be rendered; The host graphics card performs rendering based on the image rendering instruction, and stores the image data obtained after rendering in the graphics memory of the host graphics card; The image encoding module uses the mechanism provided by the host graphics card driver to access the video memory, and encodes the image data in the video memory through the GPU of the host graphics card to obtain encoded image data; The mechanism provided by the host graphics card driver is the EGLStream mechanism; Before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, the method further includes: The image encoding module initializes a first EGL interface and creates an EGLStream based on the EGLStream mechanism, obtains a file handle of a Stream data stream, initializes a first CUDA based on the EGLStream and creates a CUeglStreamConnection, and connects the CUeglStreamConnection to the EGLStream; The virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism, and creates a rendering environment that supports the EGLStream; The image encoding module sends the file handle of the Stream data stream to the virtual machine running module; The virtual machine running module creates an image area for the image to be rendered using the file handle of the Stream data flow.
5. The method according to claim 4, characterized in that After the host graphics card performs rendering based on the image rendering instruction, the method further includes: The virtual machine running module maps the image data obtained after the host graphics card renders the image data into the image area, and sends a message of image rendering completion to the image encoding module; The image encoding module reads the CUeglStreamConnection and sends the CUeglStreamConnection to the first CUDA API to perform an encoding operation on the image data based on the API.
6. The method according to claim 4, characterized in that After the image encoding module encodes the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data, the method further includes: When the connection of the image encoding module is disconnected, deleting the resources of the EGLStream; The virtual machine running module deletes the image area and the resources of the EGLStream.
7. The method according to claim 4, characterized in that The mechanism provided by the host graphics card driver is a DMABUF mechanism; Before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, it also includes: The virtual machine running module initializes the third EGL interface and the second OpenGL interface based on the DMABUF mechanism, creates a first texture object and allocates the video memory to the first texture object, creates a first EGLImage object using the first texture object, calls eglExportDMABUFImageMESA to obtain the DMABUF descriptor of the first texture object, initializes the second CUDA based on the DMABUF mechanism, and registers the EGLImage object to the second CUDA; The image encoding module initializes the fourth EGL interface and the third OpenGL interface based on the DMABUF mechanism; The virtual machine running module sends the DMABUF descriptor to the image encoding module; The image encoding module creates a second EGLImage object using the DMABUF descriptor, and binds the second EGLImage object to a second texture object, so that the second texture object is associated with the first texture object and both point to the same video memory.
8. The method according to claim 7, characterized in that After the host graphics card performs rendering based on the image rendering instruction, the method further includes: The virtual machine running module maps the image data obtained after the host graphics card renders the image data onto the first texture object, and sends a message of image rendering completion to the image encoding module; When the host graphics card is an Nvidia graphics card, the image encoding module reads the image data through the CUDA EGL interoperation API and sends the image data to the CUDA encoding library; When the host graphics card is an AMD or INTEL open source driver graphics card, the image encoding module reads the image data through the Vaapi EGL interoperation API and sends the image data to Vaapi for encoding.
9. The method according to claim 7, characterized in that: After the image encoding module encodes the image data in the video memory through the GPU of the host graphics card to obtain the encoded image data, the method further includes: When the connection of the image encoding module is disconnected, deleting the resource of the second texture object; The virtual machine running module deletes the resource of the first texture object.
10. The method according to claim 7, characterized in that The DMABUF descriptor includes: texture format information of the first texture object and / or a file handle of the first texture object.
11. An image processing device for a virtual machine, characterized in that: An image encoding module corresponding to the virtual machine, the virtual machine also corresponds to a virtual machine running module of the virtual machine and a host machine of the virtual machine, the host machine includes a host graphics card; including: An access module is used to access the video memory of the host graphics card by using a mechanism provided by the host graphics card driver; wherein the video memory stores image data obtained by rendering by the host graphics card based on an image rendering instruction, and the image rendering instruction is an image rendering instruction sent by the virtual machine running module to the host graphics card according to the image to be rendered of the virtual machine; An encoding module, used for encoding the image data in the video memory through the GPU of the host graphics card to obtain encoded image data; The mechanism provided by the host graphics card driver is the EGLStream mechanism; the device also includes: A creation module is used for initializing a first EGL interface and creating an EGLStream based on the EGLStream mechanism before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, obtaining a file handle of a Stream data stream, initializing a first CUDA based on the EGLStream and creating a CUeglStreamConnection, and connecting the CUeglStreamConnection to the EGLStream, so that the virtual machine running module initializes a second EGL interface and a first OpenGL interface based on the EGLStream mechanism and creates a rendering environment supporting the EGLStream; The sending module is used to send the file handle of the Stream data flow to the virtual machine running module, so that the virtual machine running module uses the file handle of the Stream data flow to create an image area for the image to be rendered.
12. An image processing system for a virtual machine, characterized in that: include: An image encoding module corresponding to the virtual machine, a virtual machine running module of the virtual machine, and a host machine of the virtual machine, wherein the host machine includes a host graphics card; The virtual machine running module is used to obtain the image to be rendered of the virtual machine, and send an image rendering instruction to the host graphics card according to the image to be rendered; The host graphics card is used to render based on the image rendering instruction, and store the image data obtained after rendering in the video memory of the host graphics card; The image encoding module is used to access the video memory by using a mechanism provided by the host graphics card driver, and encode the image data in the video memory through the GPU of the host graphics card to obtain encoded image data; The mechanism provided by the host graphics card driver is the EGLStream mechanism; the image encoding module is also used to: before the virtual machine running module sends an image rendering instruction to the host graphics card according to the image to be rendered, initialize the first EGL interface based on the EGLStream mechanism and create an EGLStream, obtain the file handle of the Stream data stream, initialize the first CUDA based on the EGLStream and create a CUeglStreamConnection, connect the CUeglStreamConnection to the EGLStream, so that the virtual machine running module initializes the second EGL interface and the first OpenGL interface based on the EGLStream mechanism and creates a rendering environment that supports the EGLStream; send the file handle of the Stream data stream to the virtual machine running module, so that the virtual machine running module uses the file handle of the Stream data stream to create an image area for the image to be rendered.
13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 10.
