Graphical rendering method, system, device, medium, and product

Abstract

The present disclosure provides a graphics rendering method based on instruction serialization, a graphics rendering system based on instruction serialization, a client, a host, a computer device, a computer readable storage medium and a computer program product; the method comprises: capturing a first type of instruction sent by an application program in real time; the first type of instruction is a cross-platform general rendering instruction; sequentially storing a plurality of first type of instructions into a ring instruction buffer of the client to generate a first type of instruction sequence; storing rendering data corresponding to the first type of instruction sequence into a shared memory area; the shared memory area is commonly accessed by the host and the client; in the case that the first type of instruction sequence in the ring instruction buffer meets a first trigger condition, submitting the first type of instruction sequence to the host, so that the host renders graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of graphics processing technology, and in particular to a graphics rendering method based on instruction serialization, a graphics rendering system based on instruction serialization, a client, a host, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Mesa is an open-source three-dimensional (3D) graphics library that provides operating systems with implementations of OpenGL, Vulkan, and other graphics application programming interfaces (APIs). LLVMpipe is a software rasterization driver that uses a low-level virtual machine (LLVM) compiler to dynamically compile shader code into processor-executable code, thus achieving hardware-accelerated graphics rendering without a graphics processing unit (GPU). In traditional virtualization environments, guest OS graphics rendering typically uses Mesa's llvmpipe driver—a purely software approach. However, Mesa's llvmpipe driver relies entirely on the central processing unit (CPU) for computation, resulting in extremely low performance and failing to meet the needs of applications such as CAD and games. Therefore, a more efficient graphics rendering method is urgently needed. Summary of the Invention

[0003] In view of the above, the present disclosure provides at least one instruction serialization-based graphics rendering method, instruction serialization-based graphics rendering system, client, host, computer device, computer-readable storage medium, and computer program product.

[0004] The technical solution of this disclosure embodiment is implemented as follows: On one hand, this disclosure provides a graphics rendering method based on instruction serialization, applied to a client machine. The method includes: capturing a first type of instruction sent by an application in real time; the first type of instruction is a cross-platform general rendering instruction; storing multiple first type of instructions sequentially in a circular instruction buffer of the client machine to generate a first type of instruction sequence; storing the rendering data corresponding to the first type of instruction sequence in a shared memory area; the shared memory area is accessible to both the host and the client machine; and when the first type of instruction sequence in the circular instruction buffer meets a first triggering condition, submitting the first type of instruction sequence to the host machine, so that the host machine renders graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area.

[0005] On one hand, this disclosure provides a graphics rendering method based on instruction serialization, applied to a host. The method includes: receiving a first type of instruction sequence including multiple first type instructions sent by a client; the first type of instructions are cross-platform general rendering instructions; the first type of instruction sequence is sent when the client's circular instruction buffer meets a first trigger condition; sequentially converting multiple first type instructions in the first type of instruction sequence to obtain a second type of instruction sequence including multiple second type instructions; the second type of instructions are hardware execution instructions of a graphics processor; obtaining rendering data corresponding to the first type of instruction sequence from a shared memory region; the shared memory region is accessible to both the host and the client; and rendering graphics based on the second type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory region.

[0006] On the other hand, embodiments of this disclosure provide a graphics rendering system based on instruction serialization, the system comprising: The client is configured to capture first-type instructions sent by the application in real time; the first-type instructions are cross-platform general rendering instructions; multiple first-type instructions are stored sequentially in the client's circular instruction buffer to generate a first-type instruction sequence; the rendering data corresponding to the first-type instruction sequence is stored in a shared memory area; the shared memory area is accessible to both the host and the client; when the first-type instruction sequence in the circular instruction buffer meets a first triggering condition, the first-type instruction sequence is submitted to the host; The host is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence; the second-type instructions are hardware execution instructions of the graphics processor; to obtain the rendering data corresponding to the first-type instruction sequence from the shared memory region; and to render graphics based on the second-type instruction sequence and the rendering data corresponding to the first-type instruction sequence in the shared memory region.

[0007] In another aspect, embodiments of this disclosure provide a client, comprising: a capture module configured to capture first type of instructions sent by an application in real time; the first type of instructions being cross-platform general rendering instructions; a storage module configured to sequentially store multiple first type of instructions into a circular instruction buffer of the client, generating a sequence of first type of instructions; the storage module is further configured to store rendering data corresponding to the sequence of first type of instructions into a shared memory area; the shared memory area is accessible to both the host and the client; and a submission module configured to submit the first type of instruction sequence to the host when the sequence of first type of instructions in the circular instruction buffer meets a first triggering condition, so that the host renders graphics based on the first type of instruction sequence and the rendering data corresponding to the sequence of first type of instructions in the shared memory area.

[0008] In another aspect, embodiments of this disclosure provide a host, comprising: a receiving module configured to receive a first type of instruction sequence including multiple first type instructions sent by a client; the first type of instructions are cross-platform general rendering instructions; the first type of instruction sequence is sent when the client's circular instruction buffer meets a first triggering condition; a conversion module configured to sequentially convert multiple first type instructions in the first type of instruction sequence to obtain a second type of instruction sequence including multiple second type instructions; the second type of instructions are hardware execution instructions of a graphics processor; a processing module configured to obtain rendering data corresponding to the first type of instruction sequence from a shared memory region; the shared memory region is accessible to both the host and the client; the processing module is further configured to render graphics based on the second type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory region.

[0009] In another aspect, embodiments of this disclosure provide a computer device, including a host and / or a client.

[0010] In another aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0011] In another aspect, embodiments of this disclosure provide a computer program including computer-readable code, which, when executed in a computer device, causes a graphics processor in the computer device to perform some or all of the steps in the above-described method.

[0012] In another aspect, embodiments of this disclosure provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method.

[0013] In this embodiment, multiple first-type instructions are sequentially stored in the client's circular instruction buffer to generate a first-type instruction sequence, allowing multiple first-type instructions to accumulate in the circular instruction buffer. When the first-type instruction sequence in the circular instruction buffer meets a first triggering condition, the first-type instruction sequence is submitted to the host. This batch submission method reduces notification overhead between the client and host, improves instruction transmission efficiency, and avoids performance degradation caused by frequent submissions. Since the shared memory area is accessed by both the host and client, storing the rendering data corresponding to the first-type instruction sequence in the shared memory area reduces data copying between the client and host, significantly reducing bandwidth usage. Thus, asynchronous decoupling is achieved, allowing the client and host to work in parallel, improving overall rendering performance.

[0014] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0016] Figure 1 A schematic diagram of the implementation process of a graphics rendering method based on instruction serialization provided in this disclosure embodiment. Figure 1 ; Figure 2 A schematic diagram of the implementation process of a graphics rendering method based on instruction serialization provided in this disclosure embodiment. Figure 2 ; Figure 3 A schematic diagram of the composition structure of a graphics rendering system based on instruction serialization provided in this embodiment of the present disclosure; Figure 4 A schematic diagram of the composition structure of a client provided in an embodiment of this disclosure; Figure 5 A schematic diagram of the composition structure of a host provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0018] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] If similar descriptions such as "first / second" appear in the application documents, the following explanation shall be added: the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0022] This disclosure provides a graphics rendering method based on instruction serialization, applied to a guest (also known as a virtual machine). Figure 1 As shown, the method includes the following steps 101 to 104: Step 101: Capture the first type of instructions sent by the application in real time; the first type of instructions are cross-platform general rendering instructions.

[0023] In some embodiments, the host, guest, and GPU can reside in the same computer device and share the same physical graphics card. In this case, the host has absolute control over the GPU and ownership of its drivers. The physical GPU drivers are installed only on the host operating system. Guest machines cannot directly operate the GPU hardware; they must submit requests to the host through virtual machine software or system services.

[0024] In some embodiments, the host and client are located in different computer devices. For example, the host could be a cloud server, and the client could be a laptop, with the GPU installed on the host. In this case, the host owns the GPU and can directly control it. The client may not have a local GPU (or its GPU may be very weak), in which case it needs to send requests to the host over the network.

[0025] An application program (App) is a computer program designed to perform a specific task or achieve a specific function. Applications utilize the hardware and software resources of a computer device to directly provide functional services to the user. Examples of applications include graphics rendering applications, game engine applications, browsers, and design applications.

[0026] The first type of instruction is a cross-platform, general-purpose rendering instruction. For example, the first type of instruction can be an OpenGL instruction, or it can be other graphics API instructions such as OpenGL ES, and this disclosure does not limit this.

[0027] In some implementations, vendor tools such as NVIDIA Nsight and AMD Radeon Profiler can be used to capture Type I instructions through the driver's built-in debugging interface.

[0028] In some implementations, a proxy driver between the application and the GPU driver can be used to capture the first type of instructions.

[0029] In some implementations, the first type of instructions can be captured directly using the tracing tools in Mesa.

[0030] Step 102: Store multiple Class 1 instructions sequentially into the client's circular instruction buffer to generate a Class 1 instruction sequence.

[0031] The first type of instruction sequence refers to a set of multiple first type instructions.

[0032] A circular instruction buffer is an asynchronous queue structure, physically a contiguous block of memory, logically connected end-to-end to form a circular queue. The circular instruction buffer includes a fixed-size contiguous block of memory, a read pointer, and a write pointer. During storage, instructions of type 1 are written from the write pointer; during GPU driver read, instructions are read from the read pointer and executed.

[0033] In some implementations, multiple Type I instructions can be stored sequentially in the client's circular instruction buffer according to the capture order, generating a Type I instruction sequence. The capture order refers to the order in which the client captures Type I instructions.

[0034] In some implementations, each time a first-class instruction is captured, the captured first-class instruction is stored in a circular instruction buffer via a write pointer to generate a first-class instruction sequence.

[0035] In some implementations, when the guest machine starts up, the graphics driver (such as the Virtual OpenGL Driver, or VirGL driver) initializes the circular instruction buffer.

[0036] In some implementations, the initialization of the circular instruction buffer includes: allocating memory directly accessible to the graphics processing unit (GPU), mapping it to the GPU address space to ensure read and write capabilities for both; setting the read pointer (initially 0), write pointer, and instruction alignment rules, and defining circular reuse logic; initializing the fence / semaphore, setting empty / full thresholds to prevent CPU writes from going out of bounds, GPU reads from empty, or read-write conflicts; registering the buffer to the GPU instruction queue-related driver, associating it with instruction parsing rules, and completing availability status verification.

[0037] Step 103: Store the rendering data corresponding to the first type of instruction sequence in the shared memory area; the shared memory area is accessible to both the host and the client.

[0038] Rendering data includes texture data and / or vertex data. Rendering data is the collective term for the data required to render a single frame of an image. Texture data is stored in the form of images (bitmaps) and is used to "paste" onto the surface of a 3D model, giving it rich visual details. Vertex data is a set of data representing the geometry of a 3D model. Each vertex is a point in space; multiple vertices connect to form a triangle, ultimately constituting the 3D outline of the model. Since the shared memory area is accessible to both the host and client machines, the rendering data corresponding to the first type of instruction sequence can be directly stored in the shared memory area for subsequent GPU access. Thus, by using the shared memory area, the copying of video memory data between the client and host machines can be reduced, significantly lowering bandwidth usage.

[0039] In some implementations, the rendering data corresponding to the first type of instruction sequence is stored in a shared memory region based on the first file descriptor; the shared memory region is a memory region that can be accessed by both the host and the client.

[0040] The first file descriptor (fd) is the file descriptor of the shared memory region on the client side, and it serves as the credential for the client to access the shared memory region.

[0041] Step 104: If the first type of instruction sequence in the circular instruction buffer meets the first triggering condition, the first type of instruction sequence is submitted to the host so that the host renders the graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area.

[0042] The first trigger condition is the trigger condition for batch submission of the first type of instructions, which is used to submit the first type of instructions to the host in a sequence.

[0043] In some implementations, the first triggering condition can be determined based on at least one of the following: the number of instructions in the first type of instruction sequence, the execution timing of the first type of instructions, and the empty / full status of the circular instruction buffer.

[0044] In this embodiment, multiple first-type instructions are sequentially stored in the client's circular instruction buffer to generate a first-type instruction sequence, allowing multiple first-type instructions to accumulate in the circular instruction buffer. When the first-type instruction sequence in the circular instruction buffer meets a first triggering condition, the first-type instruction sequence is submitted to the host. This batch submission method reduces notification overhead between the client and host, improves instruction transmission efficiency, and avoids performance degradation caused by frequent submissions. Since the shared memory area is accessed by both the host and client, storing the rendering data corresponding to the first-type instruction sequence in the shared memory area reduces data copying between the client and host, significantly reducing bandwidth usage. Thus, asynchronous decoupling is achieved, allowing the client and host to work in parallel, improving overall rendering performance.

[0045] In some embodiments, the shared memory region is determined by the following steps A1 to A3: Step A1: Register the video memory resources with the host through resource allocation instructions and use the video memory resources as a shared memory region; the resource allocation instructions carry the metadata of the video memory resources.

[0046] Resource allocation instructions are used to register video memory resources.

[0047] In some implementations, the resource allocation instruction carries resource metadata (width, height, format, stride). The stride refers to the total number of bytes occupied by each row of pixel data in the buffer, including valid pixel data and possible alignment padding bytes, to ensure that each row of data meets the alignment requirements of hardware access and improve the efficiency of video memory reading.

[0048] In some implementations, cross-domain shareable physical pages can be allocated via a memory sharing interface, treating these pages as shared memory regions. This allows client and host machines to share physical pages without data copying, achieving zero-copy functionality. Specifically, the client machine can access physical pages through a first file descriptor, and the host machine can access them through a second file descriptor.

[0049] In some implementations, shared memory regions can be negotiated with the Quick Emulator (QEMU) via the Virgil Virtualized Graphics Library (Virgil OpenGL, VirGL driver). VirGL is a core technology stack in the QEMU / KVM ecosystem used to implement hardware acceleration for 3D graphics in virtual machines. Based on a virtual input / output graphics processor (virtio-gpu) semi-virtualized device, it allows virtual machines (guests) to share and use the GPU capabilities of the host machine (host).

[0050] Specifically, during QEMU startup, a shared memory region is allocated and managed via virtio-gpu. After the guest VirGL driver is loaded, it interacts with QEMU through the virtio bus to obtain the address and size of the shared memory. The driver maps this memory into the guest address space, creating a shared region that can be directly accessed by both parties. Subsequent OpenGL instructions / data can be directly written to the shared memory region and passed to QEMU with zero copying.

[0051] Step A2: Allocate a first file descriptor to the client; the first file descriptor is the credential for the client to access the shared memory region.

[0052] After allocating the first file descriptor to the client, the client can directly access the shared memory region through the first file descriptor.

[0053] Step A3: Associate the resource handle of the shared memory region with the first file descriptor and store it in the client's local resource pool.

[0054] Resource handles are abstract identifiers within a process, while the first file descriptor is a system-level global identifier. For example, the first file descriptor can be a Direct Memory Access Buffer (DMA-BUF) file descriptor. A DMA-BUF refers to a shared memory region.

[0055] In this embodiment of the disclosure, video memory resources are registered with the host through resource allocation instructions, and the video memory resources are used as a shared memory region. The resource handle of the shared memory region is associated with the first file descriptor allocated to the client and stored in the client's local resource pool. In this way, the client can directly access the shared memory region through the first file descriptor and store data in the shared memory region.

[0056] In some embodiments, step 103 above can be implemented by the following steps 1031 to 1032: Step 1031: Based on the client's first file descriptor, obtain the resource handle of the shared memory region associated with the first file descriptor from the client's local resource pool.

[0057] After the client's first file descriptor is associated with the resource handle of the shared memory region, it will be stored in the client's local resource pool. Therefore, the resource handle of the shared memory region associated with the first file descriptor can be obtained from the client's local resource pool based on the client's first file descriptor for storing rendering data.

[0058] Step 1032: Based on the resource handle of the shared memory region, store the rendering data corresponding to the first type of instruction sequence into the shared memory region.

[0059] In some implementations, the shared memory region can be accessed based on the resource handle of the shared memory region, and the rendering data corresponding to the first type of instruction sequence can be stored in the shared memory region.

[0060] In this embodiment of the present disclosure, a resource handle of a shared memory region associated with the first file descriptor is obtained from the client's local resource pool based on the client's first file descriptor. Based on the resource handle of the shared memory region, the rendering data corresponding to the first type of instruction sequence is stored in the shared memory region. In this way, the rendering data corresponding to the first type of instruction sequence is stored in the shared memory region through the first file descriptor, which makes it easier for the host to obtain the rendering data corresponding to the first type of instruction sequence from the shared memory region, reduces data copying between the client and the host, and significantly reduces bandwidth usage.

[0061] In some embodiments, the first triggering condition includes at least any one of the following conditions 1 to 3: Condition 1: The storage capacity of the circular instruction buffer is greater than or equal to the first capacity threshold.

[0062] If the storage size of the circular instruction buffer is greater than or equal to the first capacity threshold, it means that there is no free area in the circular instruction buffer.

[0063] In some implementations, the first type of instruction sequence is submitted to the host when there is no free space in the circular instruction buffer.

[0064] Condition 2: A synchronization point is detected at the current moment; the synchronization point is used to control the timing of the execution of the first type of instruction.

[0065] Synchronization points are control nodes used in client-host interactions to coordinate the execution order and mark the completion position of instruction streams. They are implemented through synchronization objects such as fences and semaphores to ensure secure resource access and orderly instruction execution. The detection of a synchronization point at the current moment indicates that it is time to execute the first type of instruction. Therefore, it is determined that the sequence of first-type instructions meets the first triggering condition, and the first type of instructions are submitted in batches.

[0066] In some implementations, if a synchronization point is detected at the current moment, the first type of instruction sequence is submitted to the host.

[0067] Condition 3: The number of instructions in the first type of instruction sequence is greater than or equal to the number threshold.

[0068] The quantity threshold refers to the optimal number of instructions for a single drawing call, which can be obtained through multiple trials.

[0069] In some implementations, if the number of instructions (Draw) in a single drawing call exceeds a certain threshold, the first type of instruction sequence is submitted to the host.

[0070] In this embodiment, the timing of batch instruction submission can be determined through the first triggering condition, thus avoiding low rendering performance due to excessive waiting time. Synchronization points ensure the order of instruction execution, preventing out-of-order execution; asynchronous synchronization between the GPU and CPU ensures rendering is completed before resource usage; resource contention and access conflicts are prevented, avoiding reading and writing incomplete video memory data; rendering completion status is notified for frame display, resource release, and reuse; and system stability is improved, preventing virtual machines from crashing or malfunctioning due to unprepared resources.

[0071] In some embodiments, step 102 above can be implemented by the following steps 1021 to 1023: Step 1021: Using the write pointer, store multiple Type 1 instructions sequentially into the client's circular instruction buffer according to the capture order.

[0072] The write pointer is used to point to the position of the next writable instruction in the buffer.

[0073] In some implementations, after a Type I instruction is written using the write pointer, the write pointer moves forward to point to the next writable instruction; then, the write pointer continues to be used to write the next Type I instruction, and the write pointer continues to move forward to point to the next writable instruction, and so on in a loop.

[0074] Step 1022: Set the read pointer to the first type of instruction captured for the first time.

[0075] The read pointer is used to point to the location of the next instruction to be read and executed in the buffer. Setting the read pointer to the first captured instruction (Type 1) makes it easier for the host to read Type 1 instructions through the read pointer.

[0076] Step 1023: Generate a sequence of first-type instructions based on the first-type instructions stored between the write pointer and the read pointer.

[0077] In this embodiment of the disclosure, the client writes the first type of instruction through the write pointer, and the host reads the first type of instruction to be executed through the read pointer; in this way, the two pointers work together to realize the empty or full judgment of the circular queue, ensuring that the instructions are executed in sequence, and realizing asynchronous decoupling and synchronization between the client and the host.

[0078] In some embodiments, the graphics rendering method based on instruction serialization provided in this disclosure further includes the following steps B1 to B3: Step B1: Receive the file descriptor of the synchronization barrier sent by the host.

[0079] The file descriptor (fence fd) of a synchronization fence is a handle to a synchronization object exposed by the kernel to user space, identifying a fence or synchronization semaphore in the form of fd.

[0080] Step B2: Using the synchronous wait function, determine the host's rendering state based on the file descriptor of the synchronous fence.

[0081] The synchronization wait function is an interface used to wait for the synchronization fence signal. It takes the file descriptor of the synchronization fence (fence fd) as input, blocks the current thread until the GPU instruction is completed and the fence is triggered, thereby achieving execution synchronization between the client and the host.

[0082] In some implementations, the synchronization wait function can be sync_wait. Specifically, the file descriptor and time threshold of the synchronization fence can be used as parameters in the synchronization wait function to obtain sync_wait(fence_fd, timeout); thus, by using sync_wait(fence_fd, timeout), the host rendering is waited for to complete, achieving precise frame synchronization.

[0083] Step B3: If the host's rendering status is determined to be rendering complete, control the compositor to submit the rendered graphics frames.

[0084] In some implementations, the rendering status of the host is "rendering complete" when the fence is triggered, at which point the compositor renders the graphics frames.

[0085] In this embodiment of the disclosure, by using the file descriptor of the synchronization fence and the synchronization wait function, rendering synchronization between the client and the host can be ensured when the GPU instruction execution is completed and the fence is triggered.

[0086] In some embodiments, the "submitting the first type of instruction sequence to the host" in step 104 above can be achieved through the following steps 1041 to 1042: Step 1041: Store the first type of instruction sequence into the unified submission queue.

[0087] The universal queue is used to further cache the first type of instructions.

[0088] Step 1042: If the second triggering condition is met, submit the first type of instruction sequence in the unified submission queue to the host.

[0089] The second triggering condition includes at least: invoking a synchronization operation, and the storage capacity of the unified submission queue being greater than or equal to the second capacity threshold. Of course, the second triggering condition may also include other elements, such as the time interval between the earliest storage time of the first type of instruction sequence in the unified submission queue and the current time being greater than or equal to a time threshold.

[0090] Synchronous operations include at least glFinish and glReadpixel. If the storage capacity of the unified commit queue is greater than or equal to the second capacity threshold, it indicates that the unified commit queue has no free space.

[0091] In some implementations, when switching frame buffers or data master controllers, the data that needs to be kicked can be placed in a queue and then kicked together when it is actually needed (such as when glFinish or glReadpixel is called).

[0092] Frame buffer switching refers to swapping the front buffer currently being displayed with the back buffer currently being drawn in a double-buffered or multi-buffered mechanism, so that the rendered image is displayed on the screen, while the old display buffer is handed over to the next frame for rendering.

[0093] Switching the Data Master (DM) refers to switching the data master hardware or context responsible for managing memory / resource transfer and instruction scheduling in a virtualized graphics (virtio-gpu / VirGL) environment, thereby achieving the transfer of control and data synchronization between multiple channels / engines.

[0094] In this embodiment of the disclosure, by using a unified submission queue to cache the first type of instruction sequence and then issuing it when calling the synchronization operation, the notification overhead between the client and the host can be further reduced, the instruction transmission efficiency can be improved, the performance loss caused by frequent submissions can be avoided, and the frequency of context switching and interruption can be reduced.

[0095] In some embodiments, the graphics rendering method based on instruction serialization provided in this disclosure can also store the first type of instructions into a circular instruction buffer through the following step 1024: Step 1024: When there are target first-class instructions that need to be processed by the compute shader, store the target first-class instructions into the circular instruction buffer using the application programming interface driven by mesa.

[0096] The first type of target instruction refers to rendering instructions that require computation shader processing.

[0097] In some implementations, when writing an application program (APP), the application programming interface (API) can be run using Mesa soft rendering. The Mesa API can be used to save the target first type of instruction to a circular command buffer, and then the circular command buffer can be used to convert it into a second type of instruction.

[0098] In this embodiment of the disclosure, the target first type instruction is saved to a circular command buffer using the Mesa API, which can form a cache queue that can be read repeatedly, thereby improving the processing efficiency of the instruction.

[0099] This disclosure provides a graphics rendering method based on instruction serialization, applied to a host computer. For example... Figure 2 As shown, the method includes the following steps 201 to 204: Step 201: Receive a sequence of first-type instructions sent by the client, which includes multiple first-type instructions; the first-type instructions are cross-platform general rendering instructions; the sequence of first-type instructions is sent when the client's circular instruction buffer meets the first triggering condition.

[0100] In some implementations, the host can use the read pointer in the circular instruction buffer to read the first type of instructions sequentially from the circular instruction buffer to obtain the first type of instruction sequence.

[0101] In some implementations, after reading a first-type instruction using a read pointer, the read pointer moves forward to point to the next readable instruction; then, the read pointer continues to read the next first-type instruction, and the read pointer continues to move forward to point to the next readable instruction, and so on, until the position of the write pointer is reached, thus obtaining the first-type instruction sequence.

[0102] Step 202: Sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence that includes multiple second-type instructions; the second-type instructions are hardware execution instructions of the graphics processor.

[0103] In some implementations, multiple first-type instructions in the first-type instruction sequence can be converted sequentially according to the reading order to obtain the second-type instruction sequence.

[0104] Step 203: Obtain the rendering data corresponding to the first type of instruction sequence from the shared memory region; the shared memory region is accessible to both the host and the client.

[0105] In some implementations, rendering data corresponding to the first type of instruction sequence can be obtained from the shared memory region based on the first file descriptor; the first file descriptor is the file descriptor of the shared memory region on the client side.

[0106] In some implementations, a second file descriptor for the shared memory region on the host side can be determined based on the first file descriptor; and rendering data corresponding to the first type of instruction sequence can be obtained from the shared memory region based on the second file descriptor.

[0107] In some implementations, the first file descriptor can be mapped to the second file descriptor based on the buffer object import function and the general interface for video memory management.

[0108] Step 204: Render graphics based on the rendering data corresponding to the second type of instruction sequence and the first type of instruction sequence in the shared memory region.

[0109] In some implementations, graphics can be rendered using a graphics processor on the host computer, based on rendering data corresponding to a second type of instruction sequence and a first type of instruction sequence in a shared memory region.

[0110] In this embodiment, since the first type of instruction sequence is submitted in batches by the client, the host also retrieves the first type of instruction sequence in batches. This batch retrieval method reduces the notification overhead between the client and the host, improves instruction transmission efficiency, and avoids performance degradation caused by frequent submissions. Because the shared memory area is accessible to both the host and the client, the host can directly retrieve the rendering data corresponding to the first type of instruction sequence from the shared memory area, reducing data copying between the client and the host and significantly reducing bandwidth consumption. Rendering graphics based on the second type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area can achieve near-passthrough graphics performance in a single-GPU shared scenario, while maintaining open-source ecosystem compatibility. Thus, asynchronous decoupling is achieved, allowing the client and host to work in parallel, improving overall rendering performance.

[0111] In some embodiments, step 202 above can be performed via the following step 2021: Step 2021: Call the application programming interface to replay the instructions, and convert multiple first-class instructions in the first-class instruction sequence in turn to obtain the second-class instruction sequence.

[0112] Application Programming Interface (API) replay instructions are used to convert Class 1 instructions into Class 2 instructions through instruction replay. In some implementations, API replay instructions can be standard GL API replay instructions. In some implementations, the application programming interface is invoked to replay the instructions, and multiple first-type instructions in the first-type instruction sequence are converted sequentially according to the reading order to obtain the second-type instruction sequence.

[0113] In this embodiment of the disclosure, by calling the application programming interface to replay the instructions, multiple first-type instructions in the first-type instruction sequence are converted in sequence to obtain a second-type instruction sequence that can be recognized by the graphics processor, so that the graphics processor can render graphics based on the second-type instruction sequence.

[0114] In some embodiments, step 203 above can be implemented by the following steps 2031 to 2033: Step 2031: Determine the first file descriptor corresponding to the first type of instruction sequence; the first file descriptor is the file descriptor of the shared memory region on the client side.

[0115] In some implementations, the first file descriptor corresponding to the first type of instruction sequence can be sent from the client to the host via the regular data transmission channel between the client and the host. Alternatively, the first file descriptor corresponding to the first type of instruction sequence can also be carried in the first type of instruction sequence and provided to the host along with the transmission of the first type of instruction sequence.

[0116] Step 2032: Using the buffer object import function, the first file descriptor is mapped to the second file descriptor through the general interface for video memory management; the second file descriptor is the file descriptor of the shared memory region on the host side.

[0117] Buffer object import functions are used to map client file descriptors (such as dma_fd) to buffer objects accessible to the host graphics processor. In some implementations, the buffer object import function can be gbm_bo_import. The Generic Buffer Manager (GBM) interface is used to uniformly manage memory buffers (BOs), abstracting memory operations across different graphics processors.

[0118] In some implementations, after QEMU receives the first file descriptor (dma_fd), it executes gbm_bo_import, which directly maps the guest video memory (first file descriptor) to a buffer object (second file descriptor) accessible to the host's graphics processor via GBM.

[0119] In some implementations, the Direct Rendering Manager (DRM) driver in the host (such as i915, amdgpu) converts the first file descriptor into a DRM kernel-mode memory object handle (GEM handle, corresponding to the second file descriptor) via DRM_IOCTL_PRIME_FD_TO_HANDLE.

[0120] Step 2033: Based on the second file descriptor, access the shared memory region to obtain the rendering data corresponding to the first type of instruction sequence in the shared memory region.

[0121] In this embodiment of the disclosure, the first file descriptor is mapped to the second file descriptor through the buffer object import function and the general interface for video memory management. The shared memory region is accessed based on the second file descriptor to obtain the rendering data corresponding to the first type of instruction sequence in the shared memory region. In this way, both the client and the host can access the shared memory region without data copying, achieving zero copy.

[0122] In some embodiments, the graphics rendering method based on instruction serialization provided in this disclosure further includes the following steps C1 to C2: Step C1: Call the synchronization object function to create a synchronization barrier.

[0123] The synchronization object function is used to create a synchronization fence (DMA-fence). The synchronization object function can be `eglCreateSyncKHR(EGL_SYNC_NATIVE_FENCE_ANDROID)`.

[0124] In some implementations, the dma-fence can be generated internally by the driver. Each time the driver receives a submit request, it generates a corresponding dma-fence object.

[0125] Step C2: Send the file descriptor of the synchronization barrier to the client through the virtualization input / output channel so that the client can determine the rendering state of the host based on the file descriptor of the synchronization barrier.

[0126] The file descriptor for a synchronization fence can be represented as a fence fd. The core of the Virtual I / O Channel is a standardized data path between the guest and the host / hardware, which is abstracted, isolated, and forwarded by the virtualization layer.

[0127] In some implementations, the fence file descriptor is returned to the guest via the VirtIO channel. In this embodiment of the disclosure, a synchronization object function is called to create a synchronization fence, and the file descriptor of the synchronization fence is sent to the client through a virtualized input / output channel; thus, synchronization fence injection between the host and the client is achieved, thereby realizing precise frame synchronization.

[0128] In some embodiments, the graphics rendering method based on instruction serialization provided in this disclosure further includes the following steps D1 to D2: Step D1: Store the rendered graphics frame in the shared memory area.

[0129] Step D2: After confirming that rendering is complete through the synchronization frame of the synchronization fence, scan the graphics frames in the shared memory region through kernel mode settings and output the graphics frames to the display.

[0130] Kernel Mode Setting (KMS) is used to manage the graphics card's display mode, resolution, refresh rate, and frame buffer.

[0131] In some implementations, a synchronization framework using a synchronization fence (DMA-BUF fence synchronization framework) is employed to ensure that the kernel mode is not scanned for a frame before the GPU has finished rendering. After rendering is complete, the kernel mode is used to scan the shared memory region for graphics frames and output the graphics frames to the display.

[0132] In this embodiment of the disclosure, both the host and the client display the rendered graphics frame after the rendering is completed based on the synchronization fence, which can realize the rendering synchronization between the client and the host.

[0133] This disclosure provides a graphics rendering system based on instruction serialization. The system includes: a client configured to capture first-type instructions sent by an application in real time; the first-type instructions are cross-platform general rendering instructions; multiple first-type instructions are sequentially stored in a circular instruction buffer of the client to generate a first-type instruction sequence; rendering data corresponding to the first-type instruction sequence is stored in a shared memory area; the shared memory area is accessible to both the host and the client; when the first-type instruction sequence in the circular instruction buffer meets a first trigger condition, the first-type instruction sequence is submitted to the host; the host is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence; the second-type instructions are hardware execution instructions of a graphics processor; obtain rendering data corresponding to the first-type instruction sequence in the shared memory area; and render graphics based on the second-type instruction sequence and the rendering data corresponding to the first-type instruction sequence in the shared memory area.

[0134] like Figure 3 As shown, the instruction serialization-based graphics rendering system 300 may include multiple clients and one host. The host and multiple clients transmit data via a Virtual Machine Input / Output Channel (VirtIO). The host and clients share a single graphics processor (GPU), meaning that this embodiment of the disclosure is geared towards a single physical GPU sharing scenario.

[0135] The guest OS includes an improved VirGL driver and an extended VirtIO-GPU front-end; the host user space (QEMU) includes a batch VirGL renderer, a DMA-BUF management module, and a synchronous fence injector; the host kernel provides native support for Direct Memory Access Buffer (DMA-BUF) and Direct Memory Access fence (DMA-fence) through the Direct Render Manager (DRM) / Kernel Mode Settings (KMS).

[0136] The data flow between the host and the client follows a five-stage process: "resource registration → zero-copy mapping → command batch submission → host replay → synchronous display".

[0137] The following describes the application of the instruction serialization-based graphics rendering method provided in this disclosure in a real-world scenario.

[0138] This disclosure proposes a high-performance OpenGL virtualization acceleration system for single-physical GPU shared scenarios (corresponding to the aforementioned instruction serialization-based graphics rendering system), the core idea of ​​which is: The system consists of three parts: The guest OS includes an improved VirGL driver and an extended VirtIO-GPU front end; The host user space (QEMU) includes a batch VirGL renderer, a DMA-BUF management module, and a synchronous fence injector; The host kernel provides native support for Direct Memory Access Buffer (DMA-BUF) and Direct Memory Access Fence (DMA-fence) in the Direct Render Manager (DRM) / Kernel Mode Settings (KMS).

[0139] The data flow between the host and the client follows a five-stage process: "resource registration → zero-copy mapping → command batch submission → host replay → synchronous display".

[0140] The key technical modules involved in the embodiments of this disclosure are described in detail below: (a) Client side: Improved VirGL driver and VirtIO-GPU front end; 1. Zero-copy resource registration mechanism; When a Guest application creates a texture / vertex buffer (such as glGenTextures), the VirGL driver no longer allocates ordinary memory, but instead calls the KVM memory sharing interface to allocate physical pages that can be shared across domains; The specific allocation operation is as follows: Register the video memory resources with the Host through the instruction, and attach the video memory resource metadata (width, height, format, stride). The registered video memory resources are used as a shared memory region. The resource handle of the shared memory region is bound to the DMA-BUF file descriptor (corresponding to the first file descriptor mentioned above) and stored in the local resource pool.

[0141] Advantages: Avoids mapping (memcpy) operations during subsequent data uploads, and reduces CPU usage and time consumption caused by mapping by importing file descriptors (importfd).

[0142] 2. OpenGL instruction batch processing engine; Traditional VirGL submits commands one by one (e.g., triggering a VirtIO notification every time glDrawArrays is called). This disclosure introduces a circular command buffer: The driver accumulates multiple OpenG instructions (corresponding to the first type of instructions mentioned above) into a circular instruction buffer; Batch submission occurs when any of the following conditions are met (corresponding to the first triggering condition mentioned above): The buffer is full (e.g., 4KB). When encountering synchronization points (such as glFinish, eglSwapBuffers); When the number of draws exceeds the maximum value (the value of multiple trials).

[0143] By using the read and write pointers in the circular instruction buffer, the number of instructions sent in the current batch can be determined, thus reducing the number of VirtIO interrupts.

[0144] 3. Introduce the concept of a universal queue. When switching frame buffers or switching data masters (DMs), the data to be sent can be placed in the queue instead of being kicked, and sent together when the kick is actually needed (corresponding to the second triggering condition mentioned above, such as calling glFinish or glReadpixel and other synchronous operations).

[0145] (ii) Host machine side: Optimized VirGL renderer in QEMU; 1. Import the DMA-BUF zero-copy module; After QEMU receives the DMA-BUF file descriptor (corresponding to the first file descriptor mentioned above), it executes gbm_bo_import to directly map the client's video memory to a buffer object accessible to the host's graphics processor via GBM. Physical pages are shared, and there is no data copying, achieving zero copying.

[0146] 2. Batch instruction parsing and replay; QEMU maintains an instruction parsing thread that reads batches of instructions from the VirtIO queue; Use OpenGL context pool to avoid frequent context switching; During playback: Map the client's DMA-BUF file descriptor (corresponding to the first file descriptor mentioned above) to a buffer object accessible to the host GPU (corresponding to the second file descriptor mentioned above). Call standard GL API replay instructions (such as glDrawArrays); If a compute shader is encountered, it is forwarded to the Host Mesa via the Gallium CSO mechanism.

[0147] 3. Cross-domain synchronous fence injector; At critical synchronization points (such as eglSwapBuffers): Each time the driver receives a submit request, it generates a corresponding dma-fence; Return the file descriptor (fence fd) of the synchronization fence to the Guest via the VirtIO channel; The guest can use sync_wait(fence_fd, timeout) to wait for the host to finish rendering, thus achieving precise frame synchronization.

[0148] (iii) Host kernel: DRM / KMS collaborative support; Importing Direct Memory Access Buffer (DMA-BUF): The Direct Render Manager (DRM) driver (such as i915, amdgpu) converts the Guest's DMA-BUF file descriptor into a DRM kernel-mode memory object handle (corresponding to the second file descriptor) through DRM_IOCTL_PRIME_FD_TO_HANDLE, and establishes page table mapping.

[0149] dma-fence synchronization: By utilizing the Linux kernel DMA-BUF fencing framework, the KMS (kernel mode setting) is ensured not to scan the frame before the GPU rendering is complete.

[0150] The workflow of this disclosure embodiment specifically includes (taking one frame rendering as an example): initialization: When the guest starts up (initialized in the first frame of the guest's runtime), the VirGL driver negotiates with QEMU to share a memory region and initializes the circular instruction buffer.

[0151] Application rendering: The application calls glDrawArrays (a standard OpenGL API instruction, a function called by the program to instruct the graphics card to start drawing graphics). The VirGL driver writes the instructions to a local circular buffer. Once 10 instructions have been accumulated (such as rendering instructions for different regions in a single frame of an image), a batch submission is triggered.

[0152] Resource mapping: If a new texture is involved, the Guest allocates a DMA-BUF; QEMU establishes a zero-copy mapping via gbm_bo_import.

[0153] Host playback: QEMU parses the instruction stream and replays it in the Host OpenGL context; The rendering results are written to the shared framebuffer (a portion of the shared memory region).

[0154] Synchronization and display: The guest calls eglSwapBuffers → QEMU to inject dma-fence; After the fence is triggered, the guest notifies the synthesizer to submit the frame. KMS directly scans the shared buffer (a portion of the shared memory area) and outputs the results to the display.

[0155] Based on the foregoing embodiments, this disclosure provides a graphics rendering system based on instruction serialization, characterized in that the system includes: The client is configured to capture first-type instructions sent by the application in real time; the first-type instructions are cross-platform general rendering instructions; multiple first-type instructions are stored sequentially in the client's circular instruction buffer to generate a first-type instruction sequence; the rendering data corresponding to the first-type instruction sequence is stored in a shared memory area; the shared memory area is accessible to both the host and the client; when the first-type instruction sequence in the circular instruction buffer meets a first triggering condition, the first-type instruction sequence is submitted to the host; The host is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence including multiple second-type instructions; the second-type instructions are hardware execution instructions of the graphics processor; the host obtains the rendering data corresponding to the first-type instruction sequence from the shared memory region; and renders graphics based on the second-type instruction sequence and the rendering data corresponding to the first-type instruction sequence in the shared memory region.

[0156] The descriptions of the system embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the system provided by this disclosure can be used to execute the methods described in the method embodiments above. For technical details not disclosed in the system embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0157] Based on the foregoing embodiments, this disclosure provides a client machine, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0158] Figure 4 This is a schematic diagram of the composition structure of a client provided in an embodiment of the present disclosure, such as... Figure 4 As shown, the client 400 includes: a capture module 410, a storage module 420, and a submission module 430, wherein: The capture module 410 is configured to capture first-class instructions sent by the application in real time; the first-class instructions are cross-platform general rendering instructions. The storage module 420 is configured to sequentially store multiple Type I instructions into the client's circular instruction buffer to generate a Type I instruction sequence; Storage module 420 is also configured to store the rendering data corresponding to the first type of instruction sequence in a shared memory area; the shared memory area is accessible to both the host and the client. The submission module 430 is configured to submit the first type of instruction sequence to the host when the first type of instruction sequence in the circular instruction buffer meets the first triggering condition, so that the host can render graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area.

[0159] In some embodiments, the storage module 420 is further configured to: register video memory resources with the host via a resource allocation instruction, and use the video memory resources as a shared memory region; the resource allocation instruction carries metadata of the video memory resources; allocate a first file descriptor to the client; associate the resource handle of the shared memory region with the first file descriptor, and store it in the client's local resource pool; Accordingly, the storage module 420 is also configured to: obtain a resource handle of a shared memory region associated with the first file descriptor from the client's local resource pool based on the client's first file descriptor; and store the rendering data corresponding to the first type of instruction sequence into the shared memory region based on the resource handle of the shared memory region.

[0160] In some embodiments, the first triggering condition includes at least one of the following: the storage capacity of the circular instruction buffer is greater than or equal to a first capacity threshold; a synchronization point is detected at the current moment; the synchronization point is used to control the execution timing of the first type of instruction; the number of instructions in the first type of instruction sequence is greater than or equal to a number threshold.

[0161] In some embodiments, the storage module 420 is further configured to: use a write pointer to sequentially store a plurality of first-type instructions into the client's circular instruction buffer in the order of capture; point the read pointer to the first-type instruction captured for the first time; and generate a sequence of first-type instructions based on the first-type instructions stored between the write pointer and the read pointer.

[0162] In some embodiments, the method further includes: receiving a file descriptor of a synchronization fence sent by the host; determining the rendering state of the host based on the file descriptor of the synchronization fence using a synchronization wait function; and, if the rendering state of the host is determined to be rendering complete, controlling the compositor to submit the rendered graphics frame.

[0163] In some embodiments, the submission module 430 is further configured to: store the first type of instruction sequence in a unified submission queue; and submit the first type of instruction sequence in the unified submission queue to the host when a second triggering condition is met at the current moment; the second triggering condition includes at least: invoking a synchronization operation, and the storage capacity of the unified submission queue being greater than or equal to a second capacity threshold.

[0164] In some embodiments, the storage module 420 is further configured to: when there is a target first-class instruction that needs to be processed by the compute shader, store the target first-class instruction into a circular instruction buffer using the mesa-driven application programming interface.

[0165] Based on the foregoing embodiments, this disclosure provides a host computer, which includes the various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0166] Figure 5 This is a schematic diagram of the composition structure of a host provided in an embodiment of this disclosure, such as... Figure 5 As shown, the host 500 includes: a receiving module 510, a conversion module 520, and a processing module 530, wherein: The receiving module 510 is configured to receive a sequence of first-type instructions sent by the client, which includes multiple first-type instructions; the first-type instructions are cross-platform general rendering instructions; the sequence of first-type instructions is sent when the client's circular instruction buffer meets a first triggering condition; The conversion module 520 is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence including multiple second-type instructions; the second-type instructions are hardware execution instructions of the graphics processor. Processing module 530 is configured to retrieve rendering data corresponding to the first type of instruction sequence from a shared memory region; the shared memory region is accessible to both the host and the client. The processing module 530 is also configured to render graphics based on the rendering data corresponding to the second type of instruction sequence and the first type of instruction sequence in the shared memory region.

[0167] In some embodiments, the conversion module 520 is further configured to: call the application programming interface to replay the instructions, and sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence.

[0168] In some embodiments, the processing module 530 is further configured to: determine a first file descriptor corresponding to a first type of instruction sequence; the first file descriptor is a file descriptor of a shared memory region on the client side; use a buffer object import function to map the first file descriptor to a second file descriptor through a general interface for video memory management; the second file descriptor is a file descriptor of a shared memory region on the host side; and based on the second file descriptor, access the shared memory region to obtain the rendering data corresponding to the first type of instruction sequence in the shared memory region.

[0169] In some embodiments, the processing module 530 is further configured to: call a synchronization object function to create a synchronization barrier; and send the file descriptor of the synchronization barrier to the client through a virtualization input / output channel, so that the client can determine the rendering state of the host based on the file descriptor of the synchronization barrier.

[0170] In some embodiments, the processing module 530 is further configured to: store the rendered graphics frames in a shared memory region; and, if the rendering is determined to be complete by a synchronization frame based on a synchronization fence, scan the graphics frames in the shared memory region through kernel mode settings and output the graphics frames to the display.

[0171] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0172] It should be noted that, in the embodiments of this disclosure, if the above-described graphics rendering method based on instruction serialization is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0173] This disclosure provides a computer device, including a host and / or a client, wherein the client is used to implement... Figure 1The host computer is used to implement some or all of the steps in the method shown. Figure 2 Some or all of the steps of the method shown.

[0174] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a graphics processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0175] This disclosure provides a computer program including computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0176] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0177] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0178] It should be noted that, Figure 6 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 6 As shown, the hardware entity of the computer device 600 includes: a graphics processor 601, a communication interface 602, and a memory 603, wherein: The graphics processor 601 typically controls the overall operation of the computer device 600.

[0179] Communication interface 602 enables computer devices to communicate with other terminals or servers via a network.

[0180] The memory 603 is configured to store instructions and applications executable by the graphics processor 601, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the graphics processor 601 and various modules in the computer device 600. It can be implemented using flash memory or random access memory (RAM). Data transfer between the graphics processor 601, the communication interface 602, and the memory 603 can be performed via bus 604.

[0181] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0182] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0183] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0186] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0187] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0188] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A graphics rendering method based on instruction serialization, characterized in that, Applied to a client, the method includes: Real-time capture of the first type of instructions sent by the application; the first type of instructions are cross-platform general rendering instructions; Multiple instructions of the first type are sequentially stored in the circular instruction buffer of the client machine to generate a sequence of instructions of the first type; The rendering data corresponding to the first type of instruction sequence is stored in a shared memory area; the shared memory area is accessible to both the host and the client. If the first type of instruction sequence in the circular instruction buffer meets the first triggering condition, the first type of instruction sequence is stored in the unified submission queue; When the unified submission queue meets the second triggering condition, the first type of instruction sequence in the unified submission queue is submitted to the host, so that the host renders graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area; the second triggering condition includes at least: calling a synchronization operation, and the storage capacity of the unified submission queue being greater than or equal to a second capacity threshold.

2. The method according to claim 1, characterized in that, The method further includes: The resource allocation instruction registers video memory resources with the host and uses these video memory resources as the shared memory region; the resource allocation instruction carries the metadata of the video memory resources. A first file descriptor is allocated to the client; the first file descriptor serves as the client's credential for accessing the shared memory region. The resource handle of the shared memory region is associated with the first file descriptor and stored in the client's local resource pool.

3. The method according to claim 2, characterized in that, The step of storing the rendering data corresponding to the first type of instruction sequence in a shared memory region includes: Based on the client's first file descriptor, obtain the resource handle of the shared memory region associated with the first file descriptor from the client's local resource pool; Based on the resource handle of the shared memory region, the rendering data corresponding to the first type of instruction sequence is stored in the shared memory region.

4. The method according to any one of claims 1 to 3, characterized in that, The first triggering condition includes at least one of the following: The storage capacity of the circular instruction buffer is greater than or equal to the first capacity threshold; A synchronization point is detected at the current moment; the synchronization point is used to control the execution timing of the first type of instruction. The number of instructions in the first type of instruction sequence is greater than or equal to the number threshold.

5. The method according to any one of claims 1 to 3, characterized in that, The step of sequentially storing multiple instructions of the first type into the circular instruction buffer of the client to generate a sequence of instructions of the first type includes: Using a write pointer, multiple instructions of the first type are sequentially stored in the client's circular instruction buffer according to the capture order; Set the read pointer to the first type of instruction captured for the first time; Based on the first type of instructions stored between the write pointer and the read pointer, a sequence of the first type of instructions is generated.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Receive the file descriptor of the synchronization barrier sent by the host; Using the synchronization wait function, the rendering state of the host is determined based on the file descriptor of the synchronization barrier; If the host's rendering status is determined to be rendering complete, the compositor is controlled to submit the rendered graphics frames.

7. The method according to any one of claims 1 to 3, characterized in that, The step of sequentially storing multiple instructions of the first type into the circular instruction buffer of the client includes: When there are target first-class instructions that need to be processed by the compute shader, the target first-class instructions are stored in the circular instruction buffer using the application programming interface driven by mesa.

8. A graphics rendering method based on instruction serialization, characterized in that, Applied to a host, the method includes: The system receives a sequence of first-type instructions sent by the client, which includes multiple first-type instructions. The first-type instructions are cross-platform general rendering instructions. The sequence of first-type instructions is sent when the client's circular instruction buffer meets a first trigger condition and the unified submission queue meets a second trigger condition. The second trigger condition includes at least: invoking a synchronization operation, and the storage capacity of the unified submission queue is greater than or equal to a second capacity threshold. The first type of instructions in the first type of instruction sequence are converted sequentially to obtain a second type of instruction sequence including multiple second type instructions; the second type of instructions are hardware execution instructions of the graphics processor. Rendering data corresponding to the first type of instruction sequence is obtained from the shared memory region; the shared memory region is accessible to both the host and the client. Render graphics based on the rendering data corresponding to the second type of instruction sequence and the first type of instruction sequence in the shared memory region.

9. The method according to claim 8, characterized in that, The step of sequentially converting multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence including multiple second-type instructions includes: The instructions are replayed by calling the application programming interface, and multiple first-type instructions in the first-type instruction sequence are converted in turn to obtain the second-type instruction sequence.

10. The method according to claim 8, characterized in that, The step of obtaining the rendering data corresponding to the first type of instruction sequence from the shared memory region includes: Determine the first file descriptor corresponding to the first type of instruction sequence; the first file descriptor is the file descriptor of the shared memory region on the client side; Using the buffer object import function, the first file descriptor is mapped to a second file descriptor through the general interface for video memory management; the second file descriptor is the file descriptor of the shared memory region on the host side; Based on the second file descriptor, the shared memory region is accessed to obtain the rendering data corresponding to the first type of instruction sequence in the shared memory region.

11. The method according to any one of claims 8 to 10, characterized in that, The method further includes: Call the synchronization object function to create a synchronization barrier; The file descriptor of the synchronization barrier is sent to the client through a virtualized input / output channel, so that the client can determine the rendering state of the host based on the file descriptor of the synchronization barrier.

12. The method according to claim 8, characterized in that, The method further includes: The rendered graphics frames are stored in the shared memory area; Once the synchronization framework based on the synchronization fence determines that rendering is complete, the graphics frames in the shared memory region are scanned through kernel mode settings, and the graphics frames are output to the display.

13. A graphics rendering system based on instruction serialization, characterized in that, The system includes: A client is configured to capture first-type instructions sent by an application in real time; the first-type instructions are cross-platform general rendering instructions; multiple first-type instructions are sequentially stored in the client's circular instruction buffer to generate a first-type instruction sequence; the rendering data corresponding to the first-type instruction sequence is stored in a shared memory area; the shared memory area is accessible to both the host and the client; if the first-type instruction sequence in the circular instruction buffer meets a first trigger condition, the first-type instruction sequence is stored in a unified submission queue; if the unified submission queue meets a second trigger condition, the first-type instruction sequence in the unified submission queue is submitted to the host; the second trigger condition includes at least: invoking a synchronization operation, and the storage capacity of the unified submission queue being greater than or equal to a second capacity threshold; The host is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence including multiple second-type instructions; the second-type instructions are hardware execution instructions of the graphics processor; obtain rendering data corresponding to the first-type instruction sequence from the shared memory region; and render graphics based on the second-type instruction sequence and the rendering data corresponding to the first-type instruction sequence in the shared memory region.

14. A client machine, characterized in that, The client includes: The capture module is configured to capture the first type of instructions sent by the application in real time; the first type of instructions are cross-platform general rendering instructions. The storage module is configured to sequentially store multiple instructions of the first type into the circular instruction buffer of the client machine, thereby generating a sequence of instructions of the first type. The storage module is further configured to store the rendering data corresponding to the first type of instruction sequence in a shared memory area; the shared memory area is accessible to both the host and the client. The submission module is configured to, when the first type of instruction sequence in the circular instruction buffer satisfies the first trigger condition, The first type of instruction sequence is stored in a unified submission queue; when the unified submission queue meets the second triggering condition, the first type of instruction sequence in the unified submission queue is submitted to the host, so that the host renders graphics based on the first type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory area; the second triggering condition includes at least: calling a synchronization operation, and the storage capacity of the unified submission queue being greater than or equal to a second capacity threshold.

15. A host computer, characterized in that, The host includes: The receiving module is configured to receive a sequence of first-type instructions sent by the client, including multiple first-type instructions; the first-type instructions are cross-platform general rendering instructions; the sequence of first-type instructions is sent when the client's circular instruction buffer satisfies a first trigger condition and the unified submission queue satisfies a second trigger condition; the second trigger condition includes at least: invoking a synchronization operation, and the storage capacity of the unified submission queue is greater than or equal to a second capacity threshold. The conversion module is configured to sequentially convert multiple first-type instructions in the first-type instruction sequence to obtain a second-type instruction sequence including multiple second-type instructions; the second-type instructions are hardware execution instructions of the graphics processor. The processing module is configured to retrieve rendering data corresponding to the first type of instruction sequence from a shared memory region; the shared memory region is accessible to both the host and the client. The processing module is further configured to render graphics based on the second type of instruction sequence and the rendering data corresponding to the first type of instruction sequence in the shared memory region.

16. A computer device, comprising a host and / or a client, characterized in that, The client is used to implement the steps of the method according to any one of claims 1 to 7, and the host is used to implement the steps of the method according to any one of claims 8 to 12.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a graphics processor, it implements the steps of the method according to any one of claims 1 to 12.

18. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when read and executed by a computer device, implements the steps of the method of any one of claims 1 to 12.

Images