A cloud application-based wayland protocol pass-through graphics processing method and related device

Abstract

The application provides a cloud application-based Wayland protocol direct graphics processing method and related equipment, and relates to the technical field of electric digital data processing. The method comprises the following steps: when a target cloud application session is started, a hardware coding service component first queries a capability list of a target cloud hardware encoder, and sends the capability list to a protocol negotiation and forwarding engine component; the engine negotiates with the target cloud application based on the list to determine a compatible target rendering format. After receiving a graphics rendering request of the application and to-be-rendered data, a graphics driver is called to create a target memory access buffer according to the target rendering format; the first frame of to-be-rendered data is rendered in the buffer, and a buffer handle is fed back to the engine. The engine performs compatibility checking on the buffer, and the hardware coding service component further encodes the first frame of to-be-encoded image data according to the checking result. The method can eliminate the redundant synthesis step of a traditional compositor, shorten a data processing link, and reduce delay and resource consumption.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to a Wayland protocol pass-through graphics processing method and related equipment based on cloud applications. Background Technology

[0002] With the rapid development of cloud computing and remote interaction technologies, cloud applications based on Linux systems (such as cloud gaming, remote work, and online education) are becoming increasingly popular. Their core requirement is to efficiently transmit the graphical interface of cloud applications to terminal devices while ensuring a low-latency, high-quality user experience. Wayland, as a next-generation display protocol that directly connects to graphics processing systems, has gradually become the mainstream choice for Linux cloud application graphics processing due to its lightweight architecture and native support for modern GPUs. However, in cloud application scenarios, graphics data needs to undergo multiple processing stages, including rendering, encoding, and transmission. Achieving efficient end-to-end flow of graphics data under the Wayland protocol, while maintaining compatibility with different hardware (such as GPUs from Intel, AMD, and NVIDIA), has become a key challenge in improving the cloud application experience.

[0003] In existing technologies, cloud application graphics processing systems based on the Wayland protocol typically rely on compositors for graphics coordination and management. As a core component of the Wayland protocol, the compositor is responsible for receiving the application's rendering buffer, maintaining the window scene graph, processing window hierarchy and effects, and passing the composited image to the encoding module for remote transmission.

[0004] However, existing compositor-based solutions have significant drawbacks: On the one hand, compositors need to perform redundant operations such as scene management and image compositing. Even in scenarios where cloud applications only run a single program (such as launching only one cloud game), they still need to maintain complete window logic, resulting in wasted CPU / memory resources and adding at least one frame (approximately 16.7ms) of latency, affecting the real-time interactive experience. On the other hand, the graphics buffer format exported by the compositor has poor compatibility with the hardware encoder. Especially in the NVIDIA closed-source driver environment, the DMA-BUF buffer generated by the compositor cannot be directly recognized by the encoder and needs to be converted by CPU readback, further aggravating performance loss and latency, making it difficult to meet the high-efficiency transmission requirements of cloud applications. Summary of the Invention

[0005] This application provides a Wayland protocol pass-through graphics processing method and related equipment for cloud applications, which enables efficient processing of cloud application graphics data from rendering to encoding.

[0006] Firstly, this application provides a Wayland protocol pass-through graphics processing method based on cloud applications, applied to a Wayland protocol pass-through graphics processing system. This system includes a protocol negotiation and forwarding engine component and a hardware encoding service component, replacing the traditional Wayland synthesizer. The method includes: when a session request from a target cloud application is initiated, querying the capability list of the target cloud hardware encoder through the hardware encoding service component, and sending the capability list to the protocol negotiation and forwarding engine component. The capability list includes supported pixel formats, buffer modifiers, and encoding parameters. Based on the capability list, negotiating with the target cloud application through the protocol negotiation and forwarding engine component to confirm... The system determines the target rendering format of the target cloud application that is compatible with the capability list; upon receiving a graphics rendering request initiated by the target cloud application and consecutive frames of data to be rendered, it calls the graphics driver to create a target memory access buffer in the target rendering format; based on the first frame of data to be rendered, it completes single-frame graphics rendering in the target memory access buffer to obtain the first frame of image data to be encoded, and feeds back the handle of the target memory access buffer to the protocol negotiation and forwarding engine component; the protocol negotiation and forwarding engine component performs a compatibility check on the target memory access buffer to obtain the compatibility check result; based on the compatibility check result, the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded.

[0007] By adopting the above technical solution, the hardware encoding service component first queries the target cloud hardware encoder's capability list to clarify key parameters such as supported pixel formats and buffer modifiers, providing a precise basis for subsequent negotiation. Then, the protocol negotiation and forwarding engine negotiates the target rendering format with the target cloud application based on this list, ensuring that the application's rendering direction matches the encoder's capabilities. Subsequently, a buffer is created, rendering data is generated, and a handle is returned. Finally, encoding is performed after a compatibility check. The entire process follows a logical chain of "capability pre-query - targeted negotiation - precise processing," eliminating redundant synthesis steps in traditional compositors, shortening the data processing chain, and reducing the risk of format incompatibility through targeted negotiation. This reduces latency and resource consumption at the architectural level, improving the efficiency and stability of cloud application graphics processing.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the step of performing a compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility check result includes: performing a first compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component, specifically including: verifying whether the pixel format of the target memory access buffer belongs to the pixel format supported by the capability list; verifying whether the buffer modifier of the target memory access buffer is within the range of buffer modifiers supported by the capability list; verifying whether the size parameter of the target memory access buffer conforms to the limitation of the buffer size by the encoding parameter in the capability list; if all the above verifications pass, the compatibility check result is determined to be passed; if any verification fails, the compatibility check result is determined to be failed.

[0009] By adopting the above technical solution, the protocol negotiation and forwarding engine conducts the first compatibility check from three core dimensions: pixel format, buffer modifiers, and size parameters. Pixel format and modifier verification ensures that the buffer data format can be directly recognized by the encoder, avoiding encoding failures due to format incompatibility. Size parameter verification ensures that the buffer size meets the encoder's processing requirements, preventing resource waste or processing anomalies caused by exceeding size limits. These three dimensions of verification complement each other, comprehensively investigating data-level compatibility issues, proactively mitigating the risk of failures in subsequent encoding stages, reducing invalid data flow, and further improving the reliability and efficiency of the graphics processing workflow.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of negotiating a compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility check result further includes: negotiating a second compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component, specifically including: verifying the access rights of the target cloud hardware encoder to the target memory access buffer, the access rights including at least whether the target memory access buffer is allowed to be accessed by the driver of the target cloud hardware encoder; if the verification of the access rights fails, the compatibility check result is determined to be unsuccessful.

[0011] By adopting the above technical solution, the protocol negotiation and forwarding engine adds a second compatibility check on the target cloud hardware encoder's access permissions, focusing on verifying whether the buffer is allowed to be accessed by the encoder driver. This check is designed to address hardware driver barriers (such as NVIDIA's closed-source driver restrictions) and can accurately identify hidden issues such as "format matching but insufficient permissions." If the check fails, the subsequent adaptation process can be initiated promptly; if it passes, it ensures that the encoder can securely and directly access the buffer. This fills the gap left by relying solely on format checks, avoids encoding interruptions caused by driver permission issues, enhances the adaptability of the technical solution to different hardware driver environments, and ensures process continuity.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data to be encoded based on the compatibility check result includes: if the compatibility check result is passed, controlling the protocol negotiation and forwarding engine component to send the handle of the target memory access buffer to the hardware encoding service component; calling the target cloud hardware encoder to encode the first frame of image data to be encoded to generate the first frame of encoded data.

[0013] By adopting the above technical solution, when the compatibility check passes, the protocol negotiation and forwarding engine directly sends the handle of the target memory access buffer to the hardware encoding service component. The handle serves as the unique identifier of the buffer, allowing the encoder to directly access the buffer data without going through a CPU-mediated copy process. This "handle-to-handle direct access" mode achieves end-to-end zero-copy from the application rendering buffer to the encoder, completely avoiding the resource consumption and latency caused by CPU copying, maximizing hardware performance, significantly improving encoding processing speed, and ensuring efficient transmission and real-time rendering of cloud application graphics data.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the protocol negotiation and forwarding engine component has a built-in simplified adaptation unit, which is a lightweight GPU computing context. The step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data to be encoded based on the compatibility check result further includes: if the compatibility check result corresponding to the first compatibility check is unsuccessful, controlling the protocol negotiation and forwarding engine component to start the simplified adaptation unit, calling the GPU to perform format conversion or modifier adjustment on the first frame of image data to be encoded in the target memory access buffer, so as to generate adapted image data and corresponding adapted memory access buffer that are fully compatible with the capability list of the target cloud hardware encoder, and exporting the direct memory access buffer file descriptor of the adapted memory access buffer; controlling the protocol negotiation and forwarding engine component to send the direct memory access buffer file descriptor to the hardware encoding service component, and calling the target cloud hardware encoder to encode the adapted image data to generate the first frame of encoded data.

[0015] By adopting the above technical solution, when the first compatibility check fails, the simplified adaptation unit built into the protocol negotiation and forwarding engine is activated. This unit, acting as a lightweight GPU computing context, can invoke the GPU to perform format conversion or modifier adjustment, avoiding efficiency losses caused by CPU intervention. After adaptation, an adaptation buffer and corresponding file descriptor fully compatible with the encoder capabilities are generated and then passed to the hardware encoding service component for encoding. The entire adaptation process is completed within the GPU with minimal overhead, solving both the format / modifier incompatibility problem and maintaining near-zero-copy performance, ensuring efficient operation of the technical solution in non-ideal format scenarios.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data to be encoded further includes: if the compatibility check result corresponding to the second compatibility check is unsuccessful, controlling the protocol negotiation and forwarding engine component to start the simplified adaptation unit, calling the GPU to internally read the first frame of image data to be encoded from the target memory access buffer, and copying the first frame of image data to be encoded to the internal memory access buffer created by the simplified adaptation unit to obtain an adapted memory access buffer compatible with the driver access requirements; exporting the direct memory access buffer file descriptor of the internal memory access buffer; controlling the protocol negotiation and forwarding engine component to send the direct memory access buffer file descriptor to the hardware encoding service component, and calling the target cloud hardware encoder to encode the adapted image data to generate the first frame of encoded data.

[0017] By adopting the above technical solution, when the second compatibility check fails, the simplified adaptation unit is activated, calling the GPU to internally read the target buffer data and copy it to the internal adaptation buffer. This copying process is performed entirely within the GPU's video memory, bypassing CPU memory and avoiding the high latency and resource consumption of traditional CPU copying. The generated internal adaptation buffer meets the driver access requirements, and its file descriptor can be exported for normal use by the encoder. This bypasses driver permission restrictions, solving the "permission incompatibility" problem without sacrificing performance, and further broadening the technical solution's compatibility with different hardware drivers (especially closed-source drivers).

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded based on the compatibility check result, the method further includes: if the compatibility check result is passed, recording the target rendering format and using the target rendering format as the default rendering format standard in the processing of the image data to be encoded in subsequent frames; if the compatibility check result is failed, recording the adaptation processing information of the first frame of image data to be encoded and reusing the adaptation processing information in the processing of the image data to be encoded in subsequent frames.

[0019] By adopting the above technical solution, the target rendering format or adaptation processing information is recorded after the first frame is processed, and subsequent frames can directly reuse it. If the first frame passes the check, subsequent frames are rendered in the target rendering format by default, without the need for repeated negotiation and checks; if the first frame requires adaptation, subsequent frames can reuse the adaptation parameters without re-executing the adaptation process. This "first frame recording - subsequent reuse" mechanism significantly reduces redundant calculations and interaction steps, lowers the processing overhead of subsequent frames, improves the overall process efficiency, and ensures that the processing logic of subsequent frames is consistent with that of the first frame, guaranteeing the continuity and stability of cloud application graphics display.

[0020] In a second aspect, this application provides a Wayland protocol pass-through graphics processing system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the Wayland protocol pass-through graphics processing system to perform the methods described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a Wayland Protocol pass-through graphics processing system, cause the Wayland Protocol pass-through graphics processing system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product that, when run on a Wayland Protocol pass-through graphics processing system, causes the Wayland Protocol pass-through graphics processing system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By adopting the technical approach of "pre-querying the encoder capability list in the hardware encoding service component + targeted negotiation of the target rendering format by the protocol negotiation and forwarding engine + removing the traditional Wayland compositor", the pre-query clearly defines the encoder capability boundaries, the targeted negotiation ensures rendering format compatibility, and the absence of a compositor eliminates redundant overhead. This effectively solves the technical problems of latency and resource waste, suboptimal data path and poor hardware compatibility introduced by compositors in existing technologies. As a result, it achieves the technical effects of shortening the graphics processing link in cloud applications, reducing resource consumption and improving format compatibility, thus ensuring efficient processing and transmission of graphics data.

[0025] 2. By adopting the technical means of "adding target cloud hardware encoder access permission verification in protocol negotiation and forwarding engine", this verification can accurately identify the hidden problem of "format matching but insufficient driver permissions", effectively solving the technical problem that the existing technology cannot avoid driver permission restrictions (such as NVIDIA closed-source drivers not allowing external buffer access) by only checking the format, which is easy to cause encoding interruption. In this way, the technical solution has enhanced its adaptability to different hardware driver environments, ensuring the continuity and stability of the graphics processing flow in complex driver scenarios.

[0026] 3. By adopting the technical means of "building a lightweight GPU computing context (minimalist adaptation unit) in the protocol negotiation and forwarding engine, and calling the GPU to perform adaptation processing when the format / modifier is incompatible", the minimal adaptation unit avoids CPU intervention and the adaptation overhead within the GPU is minimal. This effectively solves the problem of performance drop caused by the need for CPU readback conversion when the format / modifier is incompatible in the existing technology. As a result, it achieves near-zero copy processing performance in non-ideal format scenarios and ensures the efficient operation of the technical solution under various format requirements. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a system framework for the Wayland protocol pass-through graphics processing method based on cloud applications in this application embodiment;

[0028] Figure 2 This is a flowchart illustrating a Wayland protocol pass-through graphics processing method based on cloud applications in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of a physical device structure of a Wayland protocol pass-through graphics processing system in an embodiment of this application. Detailed Implementation

[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used herein, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the listed items.

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0032] For ease of understanding, the system framework of the method provided in this implementation is described below. Please refer to [link / reference]. Figure 1 This is a schematic diagram of a system framework for a Wayland protocol pass-through graphics processing method based on cloud applications in this application embodiment:

[0033] Figure 1 In this system architecture, the entire system is clearly divided into two core camps: the cloud application client and the server. The server's core consists of a "protocol negotiation and forwarding engine" and a "hardware encoding service." It achieves efficient flow of graphics data through a DMA-BUF buffer and relies on the uinput kernel device for input interaction, completely eliminating the redundant steps of the traditional Wayland compositor and forming an end-to-end direct link of "negotiation-rendering-adaptation-encoding-transmission." The functional descriptions and connection relationships of each component are as follows:

[0034] The cloud application client, which can also be a user interaction terminal, primarily receives encoded video streams transmitted from the server. It then uses a local decoding module to reconstruct these streams into a visual graphical interface, presenting it to the user (e.g., cloud game visuals, remote office windows). It can also capture user input commands (such as clicks, key presses, drags) initiated by the user via mouse, keyboard, touchscreen, etc., encapsulates these commands into a standardized data format, and transmits them to the server over the network. It connects to the server's hardware encoding service components via a video stream channel, receiving the encoded graphical data as the final presentation carrier of the server's processing results. It also connects to the server's uinput kernel device via an input command channel: sending user operation commands to trigger interactive responses from the server's cloud applications (e.g., game character movement, document editing).

[0035] The server-side is the core area for "generation-processing-encoding" of graphics data. It includes five key components: the target Wayland application, the protocol negotiation and forwarding engine, the uinput kernel device, the DMA-BUF buffer, and the hardware encoding service component. The functions and connection logic of each component are as follows:

[0036] The target Wayland application, essentially the source of graphics data generation, serves as a cloud-based business platform (such as cloud-based AAA games or CAD design software). Based on the Wayland protocol, it performs graphics rendering and is the origin of the graphics data to be encoded. It can also receive the target rendering format from the protocol negotiation and forwarding engine, and allocate rendering resources conforming to that format using the local graphics driver. Furthermore, it can translate user interaction logic (such as game operations and window scaling) into graphics rendering instructions, using the GPU to process the data to be rendered for the first and subsequent frames, generating visualized image data, and storing it in a designated DMA-BUF buffer. The target Wayland application and the DMA-BUF buffer are connected via GPU memory mapping: the rendered image data is directly written to the DMA-BUF in the GPU memory, avoiding redundant copying mediated by the CPU.

[0037] The protocol negotiation and forwarding engine component can receive the target cloud hardware encoder capability list (including supported pixel formats, buffer modifiers, encoding resolution / frame rate, etc.) sent by the hardware encoding service via IPC (Inter-Process Communication), and generate a "compatible format list" after parsing. It can also perform targeted protocol negotiation: acting as a Wayland server, it negotiates the rendering format with the target Wayland application by setting the protocol - only providing the application with the options in the "compatible format list", guiding the application to directly render the graphic data that the encoder can process, thus avoiding format incompatibility issues from the source.

[0038] The uinput kernel device is a virtual input device created based on the Linux uinput kernel module. It is responsible for "securely injecting" remote operation commands from cloud application clients into the target Wayland application, achieving real-time and accurate remote interaction. It receives user commands (such as keyboard scan codes and mouse coordinates) forwarded by the protocol negotiation and forwarding engine, encapsulates them into kernel-recognizable input events according to the Linux input event standard, and injects the events into the input queue of the target Wayland application through the input subsystem, ensuring that the application responds to remote operations as if it were processing local input.

[0039] The DMA-BUF buffer can be described as a zero-copy carrier across hardware, storing the image data rendered by the target Wayland application. The physical carrier is GPU memory, avoiding frequent data copying between CPU memory and GPU memory. It supports direct access across components: after the target Wayland application writes data through the GPU, the hardware encoder can directly read the data through its file descriptor without going through protocol negotiation and forwarding engine processing.

[0040] The hardware encoding service component is primarily responsible for converting image data in DMA-BUF into a video stream that can be transmitted over the network and for communicating with the client.

[0041] The above Figure 1 The system shown completely eliminates redundant steps such as scene management and image compositing in traditional Wayland compositors, achieving end-to-end zero-copy from "target Wayland application → DMA-BUF → hardware encoder", thus reducing the latency of graphics processing at the architectural level.

[0042] The following is in conjunction with the above. Figure 1 A detailed process description of the method in this application is provided. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a Wayland protocol pass-through graphics processing method based on cloud applications in an embodiment of this application.

[0043] S201. When the session request of the target cloud application is initiated, the hardware encoding service component queries the capability list of the target cloud hardware encoder and sends the capability list to the protocol negotiation and forwarding engine component. The capability list includes the supported pixel format, buffer modifier and encoding parameters.

[0044] Among them, the target cloud application refers to an application running on a cloud server that requires graphics rendering and is transmitted to the client for display over the network, such as 3D games and video editing software running in the cloud; the session request initiation refers to a request initiated by a user through the client to launch and use the target cloud application, which triggers the cloud system to allocate corresponding resources and establish a communication connection for the user; the hardware encoding service component refers to a software module responsible for managing and scheduling the cloud hardware encoder, whose main functions include querying hardware encoder capabilities, receiving encoding tasks, and controlling the encoding process; the target cloud hardware encoder refers to a dedicated hardware device in the cloud server used for hardware-accelerated encoding of image or video data, such as the encoding unit in a GPU or a dedicated video encoder chip; the capability list refers to the set of technical parameters that the target cloud hardware encoder can support; the buffer modifier refers to an identifier used to describe the attributes of the memory buffer, such as the memory layout, caching strategy, and compression method of the buffer; the encoding parameters refer to the settings that affect the encoding process and the encoding result; the protocol negotiation and forwarding engine component refers to the core component that replaces the traditional Wayland synthesizer, responsible for negotiating protocols with the cloud application, forwarding graphics data and control commands, etc.

[0045] When a user initiates a session request to launch a target cloud application via a client, the cloud system receives the request and prepares to allocate resources and establish a runtime environment for the target cloud application in the initial stage. During the graphics rendering and transmission process of the cloud application, the capabilities of the hardware encoder directly determine the subsequent image data processing method and efficiency. Therefore, it is necessary to obtain its capability list at the beginning of the session to provide a foundation for subsequent protocol negotiation and data processing.

[0046] First, once the cloud system receives and acknowledges the session request from the target cloud application, it triggers the hardware encoding service component to begin operation. As an intermediate module managing the hardware encoder, the hardware encoding service component sends a query command to the target cloud hardware encoder. Upon receiving the query command, the target cloud hardware encoder compiles its supported capabilities into a capability list and returns it to the hardware encoding service component. This capability list contains crucial information. Supported pixel formats determine the image color data format that the hardware encoder can directly process. If the image pixel format rendered by the cloud application is not in this list, format conversion may be required, increasing system overhead. Buffer modifiers relate to whether the hardware encoder can correctly access and process the memory buffer storing image data. Different modifiers correspond to different memory management methods; incompatibility may lead to data access errors. Encoding parameters affect the quality, size, and transmission efficiency of the encoded video stream. For example, a supported high bitrate range ensures high-quality video transmission but increases network bandwidth usage, while a supported high frame rate ensures smooth video playback.

[0047] After obtaining the capability list, the hardware encoding service component sends it to the protocol negotiation and forwarding engine component. As the core component for interacting with the target cloud application, the protocol negotiation and forwarding engine component needs to determine the direction and content of subsequent negotiations with the cloud application based on this capability list. This ensures that the image data rendered by the cloud application can be efficiently processed by the hardware encoder, thus laying a solid foundation for the entire cloud application's graphics processing flow and reducing compatibility issues and performance losses in subsequent data processing.

[0048] S202. Based on the capability list, negotiate with the target cloud application through the protocol negotiation and forwarding engine component to determine the target rendering format of the target cloud application that is compatible with the capability list.

[0049] After the hardware encoding service component sends the capability list of the target cloud hardware encoder to the protocol negotiation and forwarding engine component, but before the target cloud application officially begins graphics rendering, this further ensures that the image data subsequently rendered by the target cloud application can be efficiently and correctly processed by the target cloud hardware encoder, avoiding problems such as encoding failure and performance degradation due to format incompatibility.

[0050] Specifically, after receiving the capability list, the protocol negotiation and forwarding engine component first parses the list to determine which pixel formats the target cloud hardware encoder supports, such as RGB888, YUV420, or other formats; it also determines the range of buffer modifiers, such as which modifiers correspond to memory buffer layouts that the hardware encoder can recognize and access; and it understands the limitations of encoding parameters, such as the maximum supported resolution and frame rate. Subsequently, the protocol negotiation and forwarding engine component proactively establishes a communication connection with the target cloud application and initiates the negotiation process.

[0051] During the negotiation process, the protocol negotiation and forwarding engine component transmits key information from the capability list (such as supported pixel formats and buffer modifier ranges) to the target cloud application. Upon receiving this information, the target cloud application analyzes it in conjunction with its own rendering capabilities and requirements. For example, the target cloud application might originally use RGB565 as its default pixel format, but this format is not on the hardware encoder's support list. In this case, the cloud application will check if it can switch to the RGB888 format from the capability list for rendering. If the cloud application's rendering engine supports multiple buffer modifier settings, it will select the most suitable modifier from those supported by the hardware encoder for its own rendering workflow.

[0052] The protocol negotiation and forwarding engine component continuously receives information from the target cloud application regarding the rendering formats it supports and compares them with the capability list. If a rendering format proposed by the target cloud application fully meets all the requirements in the capability list—that is, the pixel format is within the supported range, the buffer modifiers are within the allowed range, and the relevant parameters also meet the encoding parameter restrictions—then the protocol negotiation and forwarding engine component confirms with the target cloud application that this format is the target rendering format. If the initial rendering format proposed by the target cloud application is incompatible with the capability list, the protocol negotiation and forwarding engine component will provide the cloud application with feedback on the specific reasons for the incompatibility and guide the cloud application to adjust the rendering format parameters. For example, if the buffer modifier proposed by the cloud application is not supported, the protocol negotiation and forwarding engine component will inform the cloud application which modifiers are available, allowing the cloud application to select and adjust its own settings from among them.

[0053] This process may involve multiple rounds of information exchange and parameter adjustments until both parties agree on a target rendering format that is acceptable and compatible with the hardware encoder's capabilities. Once the target rendering format is determined, the protocol negotiation and forwarding engine component records the format and notifies the target cloud application to perform subsequent graphics rendering according to this format. This paves the way for subsequent image data processing, ensuring that the hardware encoder can directly and efficiently encode the rendered image.

[0054] S203. After receiving the graphics rendering request initiated by the target cloud application and the data to be rendered for consecutive frames, call the graphics driver to create a target memory access buffer in the target rendering format.

[0055] Among them, a graphics rendering request refers to an instruction issued by the target cloud application to the system, requesting the rendering of specific graphics data. This request includes various parameters and data information required for rendering. Continuous frames of data to be rendered refer to multiple frames of raw graphics data generated by the target cloud application that need to be rendered sequentially. These data constitute a dynamic image sequence, such as video frames and animation frames. A graphics driver is a software program used to interact with graphics hardware (such as a GPU). It is responsible for translating high-level rendering instructions into operations that the hardware can understand and controlling the graphics hardware to complete the rendering task. The target memory access buffer is a memory area created according to the target rendering format to store the rendered image data. The hardware encoder can directly access the image data through this buffer.

[0056] Specifically, this step is performed after the protocol negotiation and forwarding engine components and the target cloud application have determined the target rendering format, and the target cloud application begins the actual graphics rendering operation. The target cloud application needs to convert the generated consecutive frames of data to be rendered into visualized image data and store it in a suitable memory buffer so that the subsequent hardware encoder can efficiently encode this image data.

[0057] Specifically, when the target cloud application is ready to perform graphics rendering, it will first send a graphics rendering request to the system. This request contains rendering context information, such as the size of the rendering window, the viewpoint, lighting, and other parameters. At the same time, it will submit the data to be rendered for consecutive frames to the system. This data to be rendered may be vertex data, texture data, animation keyframe data, etc., which are the raw materials for generating images.

[0058] After receiving the graphics rendering request and the data to be rendered from consecutive frames from the target cloud application, the system will call the corresponding graphics driver according to the previously determined target rendering format. The graphics driver is the bridge connecting the software application and the graphics hardware. It can understand the requirements of the target rendering format and translate these requirements into instructions that the graphics hardware (such as the GPU) can execute.

[0059] Upon receiving the instruction, the graphics driver is responsible for creating the target memory access buffer. During creation, the graphics driver is configured strictly according to the parameters of the target rendering format. For example, it determines the memory space size and data arrangement of each pixel based on the specified pixel format, and sets the memory allocation method and caching strategy according to the buffer modifiers. For instance, if the target rendering format's pixel format is YUV420, then the color information of each pixel in the buffer will be stored according to the specific proportions and arrangement of the Y, U, and V components; if the buffer modifier specifies a contiguous memory layout, then the graphics driver will ensure that the memory addresses of the buffer are contiguous to facilitate fast access by the hardware encoder.

[0060] Once the target memory access buffer is created, it will serve as the output area for graphics rendering. Image data generated by subsequent rendering of the target cloud application will be stored in this buffer. Because this buffer is created according to a target rendering format compatible with the hardware encoder, the hardware encoder can directly access the data in this buffer for encoding, avoiding data conversion operations required due to format incompatibility, thereby improving the efficiency of the entire graphics processing workflow.

[0061] S204. Based on the first frame of data to be rendered, complete the single-frame graphics rendering in the target memory access buffer to obtain the first frame of image data to be encoded, and feed back the handle of the target memory access buffer to the protocol negotiation and forwarding engine component.

[0062] Among them, the first frame of data to be rendered refers to the first frame of data in the continuous frame data to be rendered generated by the target cloud application. It is the initial material for graphics rendering and contains various original information required for the first frame image. The target memory access buffer is a memory area created according to the target rendering format and used to store the rendered image data.

[0063] Specifically, this step is performed after the target memory access buffer has been created, when the target cloud application needs to start processing the first frame of graphics data and prepare for subsequent encoding operations. The scenario involves completing the rendering and generation of the first frame of image data and passing the buffer information storing that image data to the protocol negotiation and forwarding engine component. This allows the component to perform a compatibility check on the buffer, preparing for subsequent hardware encoding processing.

[0064] Specifically, after the target memory access buffer is created, the system inputs the first frame of data to be rendered into the graphics rendering pipeline. This first frame of data may include 3D model data, texture information, coordinate transformation parameters, color information, etc., and this data is passed to the graphics hardware (such as the GPU). Under the control of the graphics driver, the graphics hardware performs a series of rendering operations according to the requirements of the target rendering format, such as vertex transformation, rasterization, texture mapping, and pixel shading.

[0065] After a series of rendering operations, the first frame of data to be rendered is converted into a complete frame of image data, which is the first frame of image data to be encoded. This data is stored in the target memory access buffer. Since the target memory access buffer is created according to the target rendering format, the pixel format, memory layout, and other aspects of the first frame of image data to be encoded all conform to the requirements of that format.

[0066] After the first frame of image data to be encoded is stored in the target memory access buffer, the graphics driver generates a handle to that buffer. The handle is a unique identifier that other components in the system can use to identify and access the buffer. Subsequently, the graphics driver sends this handle back to the protocol negotiation and forwarding engine component. Upon receiving the handle, the protocol negotiation and forwarding engine component can use it to obtain relevant information about the target memory access buffer (such as the buffer's address, size, and attributes), and then perform a compatibility check on the buffer to determine whether the first frame of image data can be directly processed by the target cloud-based hardware encoder.

[0067] S205. Through this protocol negotiation, the forwarding engine component performs a compatibility check on the target memory access buffer and obtains the compatibility check result.

[0068] This step occurs after the target memory access buffer handle is fed back to the protocol negotiation and forwarding engine component, and before the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded. Its purpose is to ensure that the first frame of image data to be encoded in the target memory access buffer can be processed correctly and efficiently by the target cloud hardware encoder, avoiding encoding failures, data corruption, or performance degradation due to compatibility issues.

[0069] Specifically, after receiving the handle to the target memory access buffer, the protocol negotiation and forwarding engine component uses this handle to obtain detailed attribute information of the buffer, including pixel format, buffer modifiers, size parameters, and memory access permissions. Subsequently, the protocol negotiation and forwarding engine component initiates a compatibility check process, which mainly includes two parts: a first compatibility check and a second compatibility check.

[0070] In the first compatibility check, the protocol negotiation and forwarding engine component compares the pixel format of the target memory access buffer with the pixel formats supported in the target cloud hardware encoder's capability list to verify whether the former falls within the latter's range. For example, if the capability list supports YUV420 and RGB888 pixel formats, but the buffer's pixel format is RGB565, this verification item fails. Next, it checks whether the buffer modifiers are within the range supported by the capability list. Buffer modifiers may involve memory layout (such as contiguous or discrete memory), caching strategies (such as cacheable or non-cacheable), etc. If the modifiers are not supported, this verification item will also fail. In addition, it verifies whether the buffer's size parameters (such as width and height) meet the encoding parameters' limitations on buffer size in the capability list. For example, if the encoder supports a maximum resolution of 1920×1080, but the buffer size is 2560×1440, this verification item fails. Only when all three verifications pass is the first compatibility check considered successful.

[0071] In the second compatibility check, the protocol negotiation and forwarding engine component will focus on verifying the target cloud hardware encoder's access permissions to the target memory access buffer. This includes checking whether the buffer is allowed to be accessed by the encoder's driver. For example, some hardware encoders may only allow access to specific types of memory (such as dedicated video memory rather than system memory). If the memory area where the buffer is located is not allowed, the access permission verification will fail, and the second compatibility check will be considered as failing.

[0072] The protocol negotiation and forwarding engine component integrates the results of the first and second compatibility checks to generate the final compatibility check result. If all verification items pass, the result is "Pass"; if any verification fails, the result is "Fail". This result directly affects the subsequent hardware encoding service component's processing strategy for the first frame of image data to be encoded, and is a crucial step in ensuring the smooth progress of the encoding process.

[0073] S206. Based on the compatibility check result, the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded.

[0074] This step is performed after the protocol negotiation and forwarding engine components have completed and obtained the results of the compatibility check on the target memory access buffer. The scenario is that, based on the compatibility check results, different processing strategies are adopted to ensure that the first frame of image data to be encoded can be correctly encoded, providing a foundation for subsequent image transmission and display.

[0075] Specifically, the protocol negotiation and forwarding engine component sends the compatibility check results to the hardware coding service component, which then takes appropriate action based on the results.

[0076] If the compatibility check result is "passed," it means that all attributes of the target memory access buffer fully meet the requirements of the target cloud hardware encoder. At this time, the protocol negotiation and forwarding engine component will send the handle of the target memory access buffer to the hardware encoding service component. After receiving the handle, the hardware encoding service component will call the driver of the target cloud hardware encoder and directly access the first frame of image data to be encoded in the target memory access buffer through the handle. The encoder will encode the image data according to the encoding parameters supported in the capability list (such as encoding standard, bitrate, frame rate, etc.). For example, if the encoder supports the H.265 encoding standard and the negotiated bitrate is 5Mbps, the encoder will use the H.265 algorithm to compress the first frame of image data into video frame data with a bitrate of approximately 5Mbps, generating the first frame of encoded data.

[0077] If the compatibility check result is "failed," adaptation processing is required based on the specific reason for the failure. If the failure is due to the first compatibility check (e.g., incompatible pixel format, unsupported buffer modifiers, or incompatible size parameters), the protocol negotiation and forwarding engine component will activate the built-in minimalist adaptation unit. This unit is a lightweight GPU computing context that can invoke the GPU to perform format conversion (e.g., converting RGB565 to YUV420) or modifier adjustment (e.g., converting discrete memory layout to contiguous memory layout) on the first frame of image data to be encoded in the target memory access buffer. This generates adapted image data and the corresponding adapted memory access buffer that are fully compatible with the target cloud hardware encoder's capability list, and exports the direct memory access (DMA) file descriptor of this buffer. The protocol negotiation and forwarding engine component sends this file descriptor to the hardware encoding service component, which invokes the target cloud hardware encoder to encode the adapted image data, generating the first frame of encoded data.

[0078] If the second compatibility check fails (i.e., the target cloud hardware encoder lacks access to the target memory access buffer), the protocol negotiation and forwarding engine component will also activate the simplified adaptation unit. This unit will call the GPU to internally read the first frame of image data to be encoded from the target memory access buffer and copy it to the internal memory access buffer created by the simplified adaptation unit. This internal buffer's memory type must meet the encoder driver's access requirements, thus obtaining the adapted memory access buffer. The DMA file descriptor of this internal buffer is then exported and sent by the protocol negotiation and forwarding engine component to the hardware encoding service component. The hardware encoding service component then calls the encoder to encode the adapted image data, generating the first frame of encoded data.

[0079] Regardless of the processing method used, after generating the first frame of encoded data, the hardware encoding service component stores or forwards it to the subsequent transmission module for transmission over the network to the client for decoding and display. Simultaneously, the system records relevant information from this processing session to provide a reference for processing subsequent frames, thereby improving overall processing efficiency.

[0080] In the above embodiment, the hardware encoder capability list is queried in advance, and a compatible target rendering format is determined through negotiation with the cloud application. Redundant operations of traditional compositors are eliminated, and rendering is completed directly based on the target rendering format. Compatibility checks (or minimalist adaptation) ensure that data can be directly processed by the encoder. Therefore, CPU / memory waste in traditional solutions is avoided, latency is reduced, and the problems of poor buffer-encoder compatibility and the need for CPU readback conversion are solved (especially improving compatibility with NVIDIA closed-source drivers). This effectively solves the problems of low efficiency, poor compatibility, and high latency in cloud application graphics data flow under the Wayland protocol, thereby achieving efficient direct pass-through of graphics data rendering and encoding, ensuring a low-latency, high-quality experience, and compatibility with GPUs from multiple vendors.

[0081] In some embodiments, after the hardware encoding service component completes the encoding of the first frame of image data to be encoded, and before the target cloud application begins processing the rendering data of subsequent frames, the following operations can be performed to optimize the processing flow of subsequent frames, reduce repetitive operations and resource consumption, and improve the efficiency and real-time performance of the entire graphics processing system: When the compatibility check result of the first frame processing is "passed," it indicates that the target rendering format fully meets the capability requirements of the target cloud hardware encoder, and the target memory access buffer can be directly accessed and processed by the encoder. At this time, the system will record the target rendering format, including its corresponding pixel format, buffer modifiers, size parameters, and other detailed information. In the processing of subsequent frames, the target cloud application will directly perform graphics rendering according to the recorded target rendering format, without needing to negotiate the format with the protocol negotiation and forwarding engine component. The protocol negotiation and forwarding engine component will also default to using this format as the standard. When performing compatibility checks on the memory access buffer of subsequent frames, it only needs to verify whether it is consistent with the default format, greatly simplifying the check process. This process avoids repeated negotiation and complex checks during each frame processing, reduces system interaction overhead, ensures that subsequent frames can quickly enter the encoding stage, and thus reduces overall processing latency.

[0082] When the compatibility check result for the first frame is "failed," it means that the image data to be encoded in the first frame has undergone adaptation processing (such as format conversion and memory copying) by the simplified adaptation unit to meet the encoder's requirements. At this time, the system records detailed adaptation processing information for the first frame, including format parameters before and after adaptation, conversion algorithms, memory address mapping relationships, processing time, etc. In the processing of subsequent frames, the system directly reuses this adaptation processing information without re-analyzing compatibility issues or reconfiguring adaptation parameters. For example, if the first frame uses an RGB to YUV conversion algorithm due to pixel format incompatibility, subsequent frames will directly call the same algorithm for conversion; if the first frame copies data to a specific type of memory buffer due to memory access permission issues, subsequent frames will directly use that type of buffer and follow the same copying process. This reuse mechanism avoids repeated adaptation analysis and parameter configuration for each frame of data, reduces GPU and CPU resource consumption, and ensures that the adaptation processing of subsequent frames can be carried out efficiently and stably, thereby guaranteeing the processing efficiency and continuity of the entire graphics stream.

[0083] Whether it's recording the target rendering format or adapting processing information, the core purpose of this step is to optimize the processing flow of subsequent frames by reusing the effective information processed in the first frame, reduce redundant operations, improve the overall performance of the system, and ensure that the graphics data of cloud applications can be processed and transmitted in a stable and efficient manner, providing users with a smooth interactive experience.

[0084] The Wayland protocol pass-through graphics processing system in the embodiments of this invention is described below from a hardware processing perspective. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of a physical device structure of a Wayland protocol pass-through graphics processing system in an embodiment of this application.

[0085] It should be noted that, Figure 3 The Wayland protocol pass-through graphics processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0086] like Figure 3As shown, the Wayland protocol pass-through graphics processing system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage section 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0087] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0088] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0089] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0091] Specifically, the Wayland protocol pass-through graphics processing system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the Wayland protocol pass-through graphics processing method based on cloud applications provided in the above embodiment.

[0092] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the Wayland protocol pass-through graphics processing system described in the above embodiments; or it may exist independently and not assembled into the Wayland protocol pass-through graphics processing system. The storage medium carries one or more computer programs, which, when executed by a processor of the Wayland protocol pass-through graphics processing system, cause the Wayland protocol pass-through graphics processing system to implement the cloud application-based Wayland protocol pass-through graphics processing method provided in the above embodiments.

[0093] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0094] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for cloud application based Wayland protocol direct graphics processing, applied to a Wayland protocol direct graphics processing system, and characterized in that, The Wayland protocol pass-through graphics processing system includes a protocol negotiation and forwarding engine component and a hardware encoding service component instead of a traditional Wayland compositor, and the method includes: When a session request of a target cloud application is initiated, querying a capability list of a target cloud hardware encoder through a hardware encoding service component, and sending the capability list to a protocol negotiation and forwarding engine component, the capability list including supported pixel formats, buffer modifiers, and encoding parameters; Based on the capability list, negotiating with the target cloud application through the protocol negotiation and forwarding engine component to determine a target rendering format of the target cloud application compatible with the capability list; After receiving a graphics rendering request initiated by the target cloud application and to-be-rendered data of a continuous frame, invoking a graphics driver to create a target memory access buffer in the target rendering format; Based on single-frame graphics rendering in the target memory access buffer being completed based on the to-be-rendered data of the first frame, obtaining to-be-encoded image data of the first frame, and feeding back a handle of the target memory access buffer to the protocol negotiation and forwarding engine component; Performing compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility checking result; Based on the compatibility checking result, the hardware encoding service component invokes the target cloud hardware encoder to perform encoding processing on the to-be-encoded image data of the first frame; The step of performing compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility checking result includes: Performing first compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component, specifically including: Verifying whether the pixel format of the target memory access buffer belongs to the supported pixel formats in the capability list; Verifying whether the buffer modifier of the target memory access buffer is within the range of the supported buffer modifiers in the capability list; Verifying whether the size parameter of the target memory access buffer conforms to the restriction of the encoding parameter on the buffer size in the capability list; If all the verifications pass, it is determined that the compatibility checking result is passed; If any one of the verifications fails, it is determined that the compatibility checking result is failed; The step of, based on the compatibility checking result, the hardware encoding service component invoking the target cloud hardware encoder to perform encoding processing on the to-be-encoded image data of the first frame includes: If the compatibility checking result is passed, controlling the protocol negotiation and forwarding engine component to send the handle of the target memory access buffer to the hardware encoding service component; Invoking the target cloud hardware encoder to perform encoding processing on the to-be-encoded image data of the first frame to generate first encoded data.

2. The method of claim 1, wherein, The step of performing compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility checking result also includes: Performing second compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component, specifically including: Verifying whether the target memory access buffer is compatible with the target rendering format of the target cloud application; verifying access permission of the target cloud hardware encoder to the target memory access buffer, the access permission at least including whether the target memory access buffer is allowed to be accessed by a driver of the target cloud hardware encoder; if the verification of the access permission fails, determining that the compatibility check result is failed.

3. The method of claim 2, wherein, The protocol negotiation and forwarding engine component is provided with a built-in minimal adaptation unit, which is a lightweight GPU computing context. Based on the compatibility check result, the hardware encoding service component calls the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded. The step further includes: If the compatibility check result corresponding to the first compatibility check is failed, the protocol negotiation and forwarding engine component is controlled to start the minimal adaptation unit, call the GPU to perform format conversion or modifier adjustment on the first frame of image data to be encoded in the target memory access buffer, so as to generate adaptation image data and a corresponding adaptation memory access buffer that are completely compatible with the capability list of the target cloud hardware encoder, and export a direct memory access buffer file descriptor of the adaptation memory access buffer. The protocol negotiation and forwarding engine component is controlled to send the direct memory access buffer file descriptor to the hardware encoding service component, and call the target cloud hardware encoder to perform encoding processing on the adaptation image data to generate a first frame of encoded data.

4. The method of claim 3, wherein, The step of calling the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded by the hardware encoding service component further includes: If the compatibility check result corresponding to the second compatibility check is failed, the protocol negotiation and forwarding engine component is controlled to start the minimal adaptation unit, call the GPU to internally read the first frame of image data to be encoded in the target memory access buffer, and copy the first frame of image data to be encoded to an internal memory access buffer created by the minimal adaptation unit, so as to obtain an adaptation memory access buffer compatible with the access requirement of the driver. A direct memory access buffer file descriptor of the internal memory access buffer is exported. The protocol negotiation and forwarding engine component is controlled to send the direct memory access buffer file descriptor to the hardware encoding service component, and call the target cloud hardware encoder to perform encoding processing on the adaptation image data to generate a first frame of encoded data.

5. The method of claim 1, wherein, After the step of calling the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded based on the compatibility check result by the hardware encoding service component, the step further includes: If the compatibility check result is passed, the target rendering format is recorded, and the target rendering format is taken as a default rendering format standard in the processing of subsequent frames of image data to be encoded. If the compatibility check result is failed, adaptation processing information of the first frame of image data to be encoded is recorded, and the adaptation processing information is reused in the processing of subsequent frames of image data to be encoded.

6. A Wayland protocol pass-through graphics processing system, characterized by, The Wayland protocol pass-through graphics processing system comprises one or more processors and a memory; the memory is coupled with the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, the one or more processors invoke the computer instructions to enable the Wayland protocol pass-through graphics processing system to perform the method in any one of claims 1-5.

7. A computer-readable storage medium comprising instructions, wherein: When the instructions run on the Wayland protocol pass-through graphics processing system, enable the Wayland protocol pass-through graphics processing system to perform the method in any one of claims 1-5.

8. A computer program product, characterised in that, When the computer program product runs on the Wayland protocol pass-through graphics processing system, enable the Wayland protocol pass-through graphics processing system to perform the method in any one of claims 1-5.

Images