Wayland protocol pass-through graph processing method based on cloud application and related equipment

Abstract

The invention provides a Wayland protocol pass-through graph processing method based on cloud application and related equipment, and relates to the technical field of electric digital data processing. The method comprises the steps that when a target cloud application session is started, a hardware coding service component queries a capability list of a target cloud hardware encoder and sends the capability list to a protocol negotiation and forwarding engine component, and an engine negotiates with a target cloud application based on the list and determines a compatible target rendering format. And after a graphic rendering request and to-be-rendered data of an application are received, calling a graphic driver to create a target memory access buffer area according to a target rendering format, rendering the first frame of to-be-rendered data in the buffer area by the application, and feeding back a buffer area handle to the engine. And the engine performs compatibility check on the buffer area, and the hardware encoding service component calls an encoder to encode the first frame of image data to be encoded according to the check result. According to the method, redundant synthesis steps of a traditional synthesizer can be eliminated, a data processing link is shortened, and delay and resource consumption are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, and in particular to a Wayland protocol direct graphics processing method based on a cloud application and related equipment. BACKGROUND

[0002] With the rapid development of cloud computing and remote interaction technology, cloud applications based on Linux systems (such as cloud games, remote offices, online education, etc.) are increasingly popular. The core requirement is to efficiently transmit the graphical interface of cloud applications to terminal devices while ensuring low latency and high-quality user experience. Wayland, as a new generation of display system protocol, has gradually become the mainstream choice for Linux cloud application graphics processing due to its lightweight architecture and native support for modern GPUs. However, in the cloud application scenario, graphics data needs to be processed through multiple links such as rendering, encoding, and transmission. How to achieve efficient end-to-end flow of graphics data under the Wayland protocol while being compatible with different hardware (such as GPUs from Intel, AMD, and NVIDIA manufacturers) has become a key challenge to improve the experience of cloud applications.

[0003] In the prior art, cloud application graphics processing systems based on the Wayland protocol usually rely on a compositor to implement graphics coordination and management. As a core component of the Wayland protocol, the compositor is responsible for receiving the rendering buffer of the application program, maintaining the window scene graph, processing window levels and special effects, and delivering the synthesized picture to the encoding module for remote transmission.

[0004] However, the existing technical solutions that rely on the compositor have significant defects. On the one hand, the compositor needs to perform redundant operations such as scene management and picture synthesis. Even in the scenario where the cloud application only runs a single program (such as starting only one cloud game), the complete window logic still needs to be maintained, resulting in waste of CPU / memory resources and at least 1 frame (about 16.7 ms) of delay, affecting real-time interaction experience. On the other hand, the compatibility of the graphics buffer format exported by the compositor with the hardware encoder is poor. Especially in the NVIDIA closed-source driver environment, the DMA-BUF buffer generated by the compositor cannot be directly recognized by the encoder, and needs to be converted through CPU readback, further exacerbating performance loss and delay, making it difficult to meet the efficient transmission requirements of cloud applications. SUMMARY

[0005] The present application provides a Wayland protocol direct graphics processing method based on a cloud application and related equipment for efficiently processing cloud application graphics data from rendering to encoding.

[0006] In a first aspect, the present application provides a Wayland protocol direct graphics processing method based on a cloud application, applied to a Wayland protocol direct graphics processing system, the Wayland protocol direct graphics processing system comprising a protocol negotiation and forwarding engine component and a hardware encoding service component which replace a traditional Wayland compositor, the method comprising: when a session request of a target cloud application is started, querying a capability list of a target cloud hardware encoder through the hardware encoding service component, and sending the capability list to the protocol negotiation and forwarding engine component, the capability list comprising supported pixel formats, buffer modifiers and encoding parameters; based on the capability list, negotiating with the target cloud application through the protocol negotiation and forwarding engine component to determine a target rendering format of the target cloud application compatible with the capability list; after receiving a graphics rendering request initiated by the target cloud application and continuous frame rendering data, calling a graphics driver to create a target memory access buffer in the target rendering format; based on single-frame graphics rendering of the first frame rendering data in the target memory access buffer, obtaining first frame encoding image data, and feeding back a handle of the target memory access buffer to the protocol negotiation and forwarding engine component; performing compatibility checking on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain a compatibility checking result; based on the compatibility checking result, the hardware encoding service component calls the target cloud hardware encoder to perform encoding processing on the first frame encoding image data.

[0007] By adopting the above technical solution, the hardware encoding service component first queries the capability list of the target cloud hardware encoder to clearly determine the supported pixel formats, buffer modifiers and other key parameters, thereby providing accurate basis for subsequent negotiation; then the protocol negotiation and forwarding engine negotiates the target rendering format with the target cloud application based on the list, thereby ensuring that the application rendering direction matches the encoder capability; subsequently, the buffer is created, the rendering data is fed back, and finally the encoding is performed after compatibility checking. The entire process follows the logic chain of "capability pre-query - directional negotiation - accurate processing", eliminates the redundant synthesis steps of the traditional compositor, shortens the data processing link, reduces the format incompatibility risk through directional negotiation, reduces the delay and resource consumption from the architecture level, and improves the efficiency and stability of cloud application graphics processing.

[0008] In some embodiments of the first aspect, the step of performing compatibility checking on the target memory access buffer by the protocol negotiation and forwarding engine component to obtain a compatibility checking result comprises: performing first compatibility checking on the target memory access buffer by the protocol negotiation and forwarding engine component, specifically comprising: verifying whether the pixel format of the target memory access buffer belongs to the supported pixel formats in the capability list; verifying whether the buffer modifier of the target memory access buffer is within the range of buffer modifiers supported in the capability list; verifying whether the size parameter of the target memory access buffer meets the restriction on buffer size in the encoding parameter in the capability list; if all the above verifications are passed, determining that the compatibility checking result is passed; and if any of the verifications fails, determining that the compatibility checking result is failed.

[0009] By adopting the above technical solution, the protocol negotiation and forwarding engine performs first compatibility checking from three core dimensions of pixel format, buffer modifier and size parameter. Among them, the pixel format and modifier verification ensures that the buffer data format can be directly recognized by the encoder, avoiding encoding failure due to format incompatibility; the size parameter verification ensures that the buffer size meets the processing requirements of the encoder, preventing resource waste or processing abnormalities caused by size overrun. The verification of the three dimensions is complementary, fully investigates the compatibility problems at the data level, avoids the risk of failure in the subsequent encoding link, reduces invalid data flow, and further improves the reliability and efficiency of the graphics processing flow.

[0010] In some embodiments of the first aspect, the step of performing compatibility checking on the target memory access buffer by the protocol negotiation and forwarding engine component to obtain a compatibility checking result further comprises: performing second compatibility checking on the target memory access buffer by the protocol negotiation and forwarding engine component, specifically comprising: verifying the access permission of the target cloud hardware encoder to the target memory access buffer, the access permission at least comprising whether the target memory access buffer is allowed to be accessed by the driver of the target cloud hardware encoder; and if the verification of the access permission fails, determining that the compatibility checking result is failed.

[0011] By adopting the above technical solution, the protocol negotiation and forwarding engine adds second compatibility checking of the access permission of the target cloud hardware encoder, which focuses on verifying whether the buffer is allowed to be accessed by the encoder driver. This check is designed for hardware driver barriers (such as NVIDIA closed-source driver restrictions) and can accurately identify the implicit problem of “format matching but insufficient permission”. If the check fails, the subsequent adaptation process can be started in time; if the check passes, it ensures that the encoder can safely and directly access the buffer. This fills the gap of format checking alone, avoids encoding interruption due to driver permission problems, enhances the adaptation ability of the technical solution to different hardware driver environments, and ensures the continuity of the process.

[0012] In some embodiments of the first aspect, based on the compatibility check result, the step of invoking the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded, comprises: if the compatibility check result is passed, controlling the protocol negotiation and forwarding engine component to send the handle of the target memory access buffer to the hardware encoding service component; and invoking the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded to generate the first frame of encoded data.

[0013] By using the above technical solution, when the compatibility check is passed, the protocol negotiation and forwarding engine directly sends the handle of the target memory access buffer to the hardware encoding service component. The handle is the unique identifier of the buffer, which allows the encoder to directly access the buffer data through the handle without the need for CPU-mediated copy links. This "handle direct transmission-direct access" mode realizes end-to-end zero-copy from the application rendering buffer to the encoder, completely avoids resource occupation and delay caused by CPU copy, maximizes the performance of hardware, significantly improves the encoding processing speed, and ensures efficient transmission and real-time presentation of cloud application graphics data.

[0014] In some embodiments of the first aspect, the protocol negotiation and forwarding engine component is built-in with a minimalist adaptation unit. The minimalist adaptation unit is a lightweight GPU computing context. Based on the compatibility check result, the step of invoking the target cloud hardware encoder to perform encoding processing on the first frame of image data to be encoded, further comprises: if the compatibility check result corresponding to the first compatibility check is not passed, controlling the protocol negotiation and forwarding engine component to start the minimalist adaptation unit, invoke the GPU to perform format conversion or modifier adjustment on the first frame of image data to be encoded in the target memory access buffer to generate adaptation image data and a corresponding adaptation memory access buffer that are fully compatible with the capability list of the target cloud hardware encoder, and export a direct memory access buffer file descriptor of the adaptation memory access buffer; controlling the protocol negotiation and forwarding engine component to send the direct memory access buffer file descriptor to the hardware encoding service component, and invoke the target cloud hardware encoder to perform encoding processing on the adaptation image data to generate the first frame of encoded data.

[0015] By adopting the technical solution, when the first compatibility check fails, a protocol negotiation and forwarding engine built-in minimal adaptation unit is started. The unit serves as a lightweight GPU computing context, and can call the GPU to perform format conversion or modifier adjustment, avoiding efficiency loss caused by CPU intervention. After adaptation, an adapted buffer and a corresponding file descriptor fully compatible with the encoder capability are generated, and then are passed to the hardware encoding service component for encoding. The entire adaptation process is completed in the GPU, with minimal overhead, which not only solves the format / modifier incompatibility problem, but also maintains a near-zero-copy performance level, ensuring efficient operation of the technical solution in non-ideal format scenarios.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the step of calling the target cloud hardware encoder to encode the first frame of image data to be encoded by the hardware encoding service component further includes: if the compatibility check result corresponding to the second compatibility check is a failure, controlling the protocol negotiation and forwarding engine component to start a minimal adaptation unit, calling the GPU to internally read the first frame of image data to be encoded in the target memory access buffer, and copying the first frame of image data to be encoded to an internal memory access buffer created by the minimal adaptation unit to obtain an adapted memory access buffer compatible with the driver access requirement; exporting a direct memory access buffer file descriptor of the internal memory access buffer; and controlling the protocol negotiation and forwarding engine component to send the direct memory access buffer file descriptor to the hardware encoding service component, and calling the target cloud hardware encoder to encode the adapted image data to generate a first frame of encoded data.

[0017] By adopting the technical solution, when the second compatibility check fails, a minimal adaptation unit is started, and the GPU is called to internally read the target buffer data and copy it to an internal adaptation buffer. The copying process is performed entirely in the GPU video memory without passing through the CPU memory, avoiding the high latency and resource consumption of traditional CPU copying. The generated internal adaptation buffer meets the driver access requirement, and after exporting its file descriptor, it can be normally used by the encoder. This bypasses the driver permission restriction, solves the "permission incompatibility" problem without sacrificing performance, and further widens the adaptation range of the technical solution to different hardware drivers (especially closed-source drivers).

[0018] In some embodiments of the first aspect, based on the compatibility check result, after the step of invoking the target cloud hardware encoder to encode the first frame of image data, the hardware encoding service component further comprises: if the compatibility check result is passed, recording the target rendering format, and taking the target rendering format as a default rendering format standard in the processing of subsequent frames of image data; if the compatibility check result is not passed, recording the adaptation processing information of the first frame of image data, and reusing the adaptation processing information in the processing of subsequent frames of image data.

[0019] By using the above technical solution, the target rendering format or the adaptation processing information is recorded after the processing of the first frame, and the subsequent frames can be directly reused. If the first frame is passed, the subsequent frames are rendered by default in the target rendering format, without the need for repeated negotiation and checking; if the first frame needs to be adapted, the subsequent frames can reuse the adaptation parameters, without the need for re-executing the adaptation process. Such a "first frame recording-subsequent reuse" mechanism greatly reduces repeated calculations and interaction links, reduces the processing overhead of subsequent frames, improves the processing efficiency of the overall process, while ensuring that the processing logic of the subsequent frames is consistent with that of the first frame, and guarantees the coherence and stability of the cloud application graphics display.

[0020] In a second aspect, the present application provides a Wayland protocol direct graphics processing system, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code comprising computer instructions, the one or more processors invoking the computer instructions to enable the Wayland protocol direct graphics processing system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0021] In a third aspect, the present application provides a computer readable storage medium comprising instructions that, when executed on a Wayland protocol direct graphics processing system, cause the Wayland protocol direct graphics processing system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product that, when executed on a Wayland protocol direct graphics processing system, causes the Wayland protocol direct graphics processing system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The technical means of "hardware coding service component pre-query encoder capability list + protocol negotiation and forwarding engine directional negotiation target rendering format + remove traditional Wayland synthesizer" is adopted, which explicitly encodes the boundary of the encoder capability through pre-query, ensures the compatibility of the rendering format through directional negotiation, and eliminates the redundant overhead without synthesizer, effectively solving the technical problems of delay and resource waste, non-optimal data path, and poor hardware compatibility introduced by the synthesizer in the prior art, thereby realizing the technical effects of shortening the cloud application graphics processing link, reducing resource consumption, and improving format compatibility, and ensuring efficient processing and transmission of graphics data.

[0024] 2. The technical means of "protocol negotiation and forwarding engine adds target cloud hardware encoder access permission verification" is adopted, which can accurately identify the implicit problem of "format matching but insufficient driver permission", effectively solving the technical problem that only checking the format cannot avoid driver permission restrictions (such as NVIDIA closed-source drivers not allowing external buffer access) and easily leading to encoding interruption in the prior art, thereby realizing the enhanced adaptation capability of the technical solution to different hardware driver environments and ensuring the continuity and stability of the graphics processing flow in complex driver scenarios.

[0025] 3. The technical means of "protocol negotiation and forwarding engine built-in lightweight GPU computing context (extreme simplification adaptation unit), calling GPU for adaptation processing when format / modifier is incompatible" is adopted, which avoids CPU intervention and has minimal GPU internal adaptation overhead, effectively solving the performance degradation problem caused by CPU readback conversion when format / modifier is incompatible in the prior art, thereby realizing near-zero-copy processing performance in non-ideal format scenarios and ensuring efficient operation of the technical solution under diverse format requirements. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a system framework schematic diagram of the Wayland protocol direct-through graphics processing method based on cloud application in the embodiments of the present application; Figure 2 is a flowchart of the Wayland protocol direct-through graphics processing method based on cloud application in the embodiments of the present application; Figure 3 is a schematic diagram of an entity device structure of the Wayland protocol direct-through graphics processing system in the embodiments of the present application. DETAILED DESCRIPTION

[0027] The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in this application, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0028] Hereinafter, the terms "first", "second" are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise specified.

[0029] For the convenience of understanding, the system framework of the method provided by the present embodiment is described below. Please refer to Figure 1 A system framework diagram of the Wayland protocol pass-through graphics processing method based on cloud application in the present embodiment is as follows: Figure 1 In the present embodiment, the entire system architecture is clearly divided into two core camps of cloud application client and server. The server internally takes "protocol negotiation and forwarding engine" and "hardware coding service" as the core, realizes efficient flow of graphics data through DMA-BUF buffer, and relies on uinput kernel device to complete input interaction, completely discards the redundant links of traditional Wayland compositor, and forms an end-to-end pass-through link of "negotiation-rendering-adaptation-coding-transmission". The function description and connection relationship of each component are as follows: The cloud application client, which can also be a user interaction terminal, mainly receives the encoded video stream transmitted by the server, restores it to a visual graphics interface through a local decoding module, and presents it to the user (such as cloud game picture, remote office window). It can also capture the operation instructions (such as click, key, drag) initiated by the user through the mouse, keyboard, touch and other devices, encapsulate the instructions into a standardized data format, and transmit them to the server through the network. Its related components with the hardware coding service of the server are connected through the video stream channel to receive the encoded graphics data as the final presentation carrier of the server processing result; and are connected with the uinput kernel device of the server through the input instruction channel: send user operation instructions to trigger the interactive response of the server cloud application (such as game character movement, document editing).

[0030] The server is the core area of "generation-processing-encoding" of graphics data, which includes five key components: target Wayland application, protocol negotiation and forwarding engine, uinput kernel device, DMA-BUF buffer, and hardware encoding service component. The functions and connection logic of each component are as follows: The target Wayland application can be regarded as the source of graphics data generation. As a cloud-based business carrier (such as cloud 3A games and CAD design software), it completes graphics rendering based on the Wayland protocol and is the source of encoded graphics data. It can also receive the target rendering format issued by the protocol negotiation and forwarding engine, call the local graphics driver to allocate rendering resources that meet the format, and convert user interaction logic (such as game operation and window scaling) into graphics rendering instructions to complete the first frame and subsequent frame rendering data processing through GPU, generate visual image data, and store it in the specified DMA-BUF buffer. The target Wayland application and DMA-BUF buffer are connected through GPU memory mapping: the rendered image data is directly written to the DMA-BUF in the video memory, avoiding redundant copying mediated by CPU.

[0031] The protocol negotiation and forwarding engine component can receive the target cloud hardware encoder capability list (including supported pixel format, buffer modifier, encoding resolution / frame rate, etc.) sent by the hardware encoding service through IPC (inter-process communication), generate a "compatible format list" after parsing, and perform directed protocol negotiation: as a Wayland server, it negotiates the rendering format with the target Wayland application by setting the protocol, providing only the options in the "compatible format list" to the application, guiding the application to directly render the encoder processable graphics data, and avoiding format incompatibility problems from the source.

[0032] The uinput kernel device is a virtual input device created based on the Linux uinput kernel module, which is responsible for "safely injecting" remote operation instructions from the cloud application client into the target Wayland application, achieving real-time and accuracy of remote interaction, receiving user instructions (such as keyboard scan codes and mouse coordinates) forwarded by the protocol negotiation and forwarding engine, packaging them into kernel-recognizable input events according to the Linux input event standard, and injecting the events into the input queue of the target Wayland application through the input subsystem to ensure that the application responds to remote operations as if it were handling local input.

[0033] The DMA-BUF buffer, which can be regarded as a cross-hardware zero-copy carrier, stores the image data rendered by the target Wayland application, and the physical carrier is GPU memory, thereby avoiding frequent copying of data between CPU memory and GPU memory; cross-component direct access is supported: after the target Wayland application writes data through the GPU, the hardware encoder can directly read the data through the file descriptor, without the need for the mediation of the protocol negotiation and forwarding engine.

[0034] The hardware encoding service component is mainly responsible for converting the image data in the DMA-BUF into a video stream that can be transmitted over a network, and completing communication with the client.

[0035] The above Figure 1 The system completely discards the redundant links such as scene management and picture synthesis of the traditional Wayland compositor, and realizes end-to-end zero-copy of "target Wayland application -> DMA-BUF -> hardware encoder", thereby reducing the defect of graphic processing delay from the architecture level.

[0036] The above Figure 1 The method in the present application is described in detail. Please refer to Figure 2 , which is a flowchart of the Wayland protocol pass-through graphic processing method based on a cloud application in the embodiment of the present application.

[0037] S201, when the session request of the target cloud application is started, the capability list of the target cloud hardware encoder is queried through the hardware encoding service component, and the capability list is sent to the protocol negotiation and forwarding engine component, the capability list including supported pixel format, buffer modifier and encoding parameter; The target cloud application refers to an application program running on a cloud server, which needs to be graphically rendered and transmitted to a client for display through a network, such as a 3D game running on a cloud, video editing software, etc. The session request initiation refers to a request initiated by a user through a client, which hopes to start and use the target cloud application. This request triggers the cloud system to allocate corresponding resources for the user and establish a communication connection. The hardware encoding service component refers to a software module responsible for managing and scheduling the invocation of a cloud hardware encoder. Its main functions include querying the hardware encoder capability, receiving the encoding task, and controlling the encoding process. The target cloud hardware encoder refers to a special hardware device in a cloud server used for hardware-accelerated encoding of image or video data, such as an encoding unit in a GPU or a dedicated video encoder chip. The capability list refers to a set of technical parameters supported by the target cloud hardware encoder. The buffer modifier refers to an identifier used to describe the properties of a memory buffer, such as the memory layout, caching strategy, and compression method of the buffer. The encoding parameter refers to a setting that affects the encoding process and the encoding result. The protocol negotiation and forwarding engine component refers to a core component that replaces the traditional Wayland compositor, which is responsible for protocol negotiation with the cloud application, forwarding of graphic data and control instructions, etc.

[0038] When a user initiates a session request to start a target cloud application through a client, the cloud system receives the request and prepares to allocate resources for the target cloud application and establish an initial running environment. During the graphic rendering and transmission of the cloud application, the capability of the hardware encoder directly determines the processing method and efficiency of subsequent image data. Therefore, it is necessary to obtain the capability list at the initial stage of session initiation to provide a basis for subsequent protocol negotiation and data processing.

[0039] Firstly, when the session request of the target cloud application is received and confirmed by the cloud system, the cloud system triggers the hardware encoding service component to start working. As an intermediate module for managing the hardware encoder, the hardware encoding service component sends a query instruction to the target cloud hardware encoder. After receiving the query instruction, the target cloud hardware encoder organizes the supported capability information into a capability list and returns it to the hardware encoding service component. The information contained in the capability list is crucial. The supported pixel format determines the image color data format that the hardware encoder can directly process. If the pixel format of the image rendered by the cloud application is not in the list, format conversion may be needed, which will increase the system overhead. The buffer modifier is related to whether the hardware encoder can correctly access and process the memory buffer storing the image data. Different modifiers correspond to different memory management methods, and if they are not compatible, it may cause data access errors. The encoding parameter affects the quality, size, and transmission efficiency of the encoded video stream. For example, the supported high code rate range can guarantee high-quality video transmission, but will increase the network bandwidth occupancy, while the supported high frame rate can guarantee the smoothness of the video.

[0040] After obtaining the capability list, the hardware encoding service component sends it to the protocol negotiation and forwarding engine component. As the core component for interacting with the target cloud application, the protocol negotiation and forwarding engine component needs to determine the direction and content of subsequent negotiations with the cloud application based on this capability list. This ensures that the image data rendered by the cloud application can be efficiently processed by the hardware encoder, thus laying a solid foundation for the entire cloud application's graphics processing flow and reducing compatibility issues and performance losses in subsequent data processing.

[0041] S202. Based on the capability list, negotiate with the target cloud application through the protocol negotiation and forwarding engine component to determine the target rendering format of the target cloud application that is compatible with the capability list. After the hardware encoding service component sends the capability list of the target cloud hardware encoder to the protocol negotiation and forwarding engine component, but before the target cloud application officially begins graphics rendering, this further ensures that the image data subsequently rendered by the target cloud application can be efficiently and correctly processed by the target cloud hardware encoder, avoiding problems such as encoding failure and performance degradation due to format incompatibility.

[0042] Specifically, after receiving the capability list, the protocol negotiation and forwarding engine component first parses the list to determine which pixel formats the target cloud hardware encoder supports, such as RGB888, YUV420, or other formats; it also determines the range of buffer modifiers, such as which modifiers correspond to memory buffer layouts that the hardware encoder can recognize and access; and it understands the limitations of encoding parameters, such as the maximum supported resolution and frame rate. Subsequently, the protocol negotiation and forwarding engine component proactively establishes a communication connection with the target cloud application and initiates the negotiation process.

[0043] During the negotiation process, the protocol negotiation and forwarding engine component transmits key information from the capability list (such as supported pixel formats and buffer modifier ranges) to the target cloud application. Upon receiving this information, the target cloud application analyzes it in conjunction with its own rendering capabilities and requirements. For example, the target cloud application might originally use RGB565 as its default pixel format, but this format is not on the hardware encoder's support list. In this case, the cloud application will check if it can switch to the RGB888 format from the capability list for rendering. If the cloud application's rendering engine supports multiple buffer modifier settings, it will select the most suitable modifier from those supported by the hardware encoder for its own rendering workflow.

[0044] The protocol negotiation and forwarding engine component continuously receives the rendering format information supported by the target cloud application and compares it with the capability list. If the rendering format proposed by the target cloud application fully meets the requirements in the capability list, i.e., the pixel format is within the supported range, the buffer modifier is within the allowed range, and the related parameters meet the limitations of the encoding parameters, the protocol negotiation and forwarding engine component confirms the format as the target rendering format with the target cloud application. If the initial rendering format proposed by the target cloud application is incompatible with the capability list, the protocol negotiation and forwarding engine component feeds back the specific reasons for the incompatibility to the target cloud application and guides the target cloud application to adjust the rendering format parameters. For example, if the buffer modifier proposed by the target cloud application is not supported, the protocol negotiation and forwarding engine component informs the target cloud application which modifiers are available so that the target cloud application can select and adjust its own settings.

[0045] In this process, multiple rounds of information exchange and parameter adjustment may be required until both parties determine a target rendering format that is acceptable and compatible with the hardware encoder capability. After determining the target rendering format, the protocol negotiation and forwarding engine component records the format and notifies the target cloud application to perform subsequent graphics rendering according to the format, thereby paving the way for subsequent image data processing and ensuring that the hardware encoder can directly and efficiently encode the rendered images.

[0046] S203, after receiving the graphics rendering request initiated by the target cloud application and the continuous frame of to-be-rendered data, invoking a graphics driver to create a target memory access buffer in the target rendering format; The graphics rendering request is an instruction issued by the target cloud application to the system, requiring rendering processing of specific graphics data. The request includes various parameters and data information required for rendering. The continuous frame of to-be-rendered data is a plurality of frames of original graphics data generated by the target cloud application, which need to be rendered in sequence. These data constitute a dynamic image sequence, such as video frames, animation frames, etc. The graphics driver is a software program used to interact with graphics hardware (such as GPU). It is responsible for converting high-level rendering instructions into operations that can be understood by hardware and controlling graphics hardware to complete rendering tasks. The target memory access buffer is a memory area created according to the target rendering format, used to store image data after rendering. The hardware encoder can directly access the image data through the buffer.

[0047] Specifically, the timing of this step is after the protocol negotiation and forwarding engine component and the target cloud application determine the target rendering format, and the target cloud application starts the actual graphics rendering operation. The target cloud application needs to convert the continuous frame of to-be-rendered data generated by the target cloud application into visual image data and store it in a suitable memory buffer, so that the hardware encoder can efficiently encode the image data.

[0048] Specifically, when the target cloud application is ready for graphics rendering, it will first initiate a graphics rendering request to the system, which contains the context information of rendering, such as the size of the rendering window, the viewing angle, the lighting, and other parameters, and will also submit the rendering data of the continuous frames to the system at the same time. These rendering data may be vertex data, texture data, and animation keyframe data of a three-dimensional model, which are the original materials for generating images.

[0049] After the system receives the graphics rendering request and the rendering data of the continuous frames from the target cloud application, it will call the corresponding graphics driver according to the target rendering format determined earlier. The graphics driver is a bridge connecting software applications and graphics hardware, which can understand the requirements of the target rendering format and convert them into instructions that can be executed by the graphics hardware (such as GPU).

[0050] After receiving the instructions, the graphics driver will be responsible for creating the target memory access buffer. During the creation process, the graphics driver will strictly configure according to the parameters of the target rendering format, such as determining the memory space size and data arrangement of each pixel according to the specified pixel format, setting the memory allocation method and caching strategy according to the buffer modifier, etc. For example, if the pixel format of the target rendering format is YUV420, the color information of each pixel in the buffer will be stored according to the specific proportion and arrangement of Y, U, and V components; if the buffer modifier is specified as continuous memory layout, the graphics driver will ensure that the memory address of the buffer is continuous to facilitate fast access by the hardware encoder.

[0051] After the creation of the target memory access buffer, it will serve as the output area for graphics rendering, and the image data generated by subsequent rendering of the target cloud application will be stored in this buffer. Since this buffer is created according to the target rendering format compatible with the hardware encoder, the hardware encoder can directly access the data in this buffer for encoding, avoiding the need for data conversion operations due to format incompatibility, thereby improving the efficiency of the entire graphics processing flow.

[0052] S204, based on the first frame of rendering data, a single frame of graphics rendering is completed in the target memory access buffer, and the first frame of encoded image data is obtained, and the handle of the target memory access buffer is fed back to the protocol negotiation and forwarding engine component; Among them, the first frame of rendering data refers to the first frame of data in the continuous frame of rendering data generated by the target cloud application, which is the initial material for graphics rendering and contains various original information required for the first frame of image; the target memory access buffer is a memory area created according to the target rendering format for storing image data after rendering.

[0053] Specifically, the timing of this step is after the target memory access buffer is created, and the target cloud application needs to start processing the first frame of graphics data and prepare for subsequent encoding operations. The scenario is to complete the rendering and generation of the first frame of image, and pass the buffer information storing the image data to the protocol negotiation and forwarding engine component, so that the component performs compatibility check on the buffer, and prepares for subsequent hardware encoding processing.

[0054] Specifically, after the target memory access buffer is created, the system inputs the first frame of to-be-rendered data into the graphics rendering pipeline. The first frame of to-be-rendered data can include three-dimensional model data, texture information, coordinate transformation parameters, color information, etc., which are passed to the graphics hardware (such as GPU). The graphics hardware, under the control of the graphics driver, performs a series of rendering operations such as vertex transformation, rasterization, texture mapping, and pixel shading according to the requirements of the target rendering format.

[0055] After a series of rendering operations, the first frame of to-be-rendered data is converted into a complete frame of image data, which is the first frame of to-be-encoded image data, which is stored in the target memory access buffer. Since the target memory access buffer is created according to the target rendering format, the pixel format and memory layout of the first frame of to-be-encoded image data conform to the requirements of the format.

[0056] After the first frame of to-be-encoded image data is stored in the target memory access buffer, the graphics driver generates a handle for the buffer. The handle is a unique identifier that other components in the system can use to identify and access the buffer. Subsequently, the graphics driver feeds back the handle to the protocol negotiation and forwarding engine component. After receiving the handle, the protocol negotiation and forwarding engine component can obtain the relevant information of the target memory access buffer (such as the address, size, and properties of the buffer) through the handle, and then perform compatibility check on the buffer to determine whether the first frame of to-be-encoded image data can be directly processed by the target cloud hardware encoder.

[0057] S205, performing compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component, and obtaining a compatibility check result; This step is after the handle of the target memory access buffer is fed back to the protocol negotiation and forwarding engine component, and before the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of to-be-encoded image data. The scenario is to ensure that the first frame of to-be-encoded image data in the target memory access buffer can be correctly and efficiently processed by the target cloud hardware encoder, avoiding encoding failure, data corruption, or performance degradation due to compatibility problems.

[0058] Specifically, after receiving the handle to the target memory access buffer, the protocol negotiation and forwarding engine component uses this handle to obtain detailed attribute information of the buffer, including pixel format, buffer modifiers, size parameters, and memory access permissions. Subsequently, the protocol negotiation and forwarding engine component initiates a compatibility check process, which mainly includes two parts: a first compatibility check and a second compatibility check.

[0059] In the first compatibility check, the protocol negotiation and forwarding engine component compares the pixel format of the target memory access buffer with the pixel formats supported in the target cloud hardware encoder's capability list to verify whether the former falls within the latter's range. For example, if the capability list supports YUV420 and RGB888 pixel formats, but the buffer's pixel format is RGB565, this verification item fails. Next, it checks whether the buffer modifiers are within the range supported by the capability list. Buffer modifiers may involve memory layout (such as contiguous or discrete memory), caching strategies (such as cacheable or non-cacheable), etc. If the modifiers are not supported, this verification item will also fail. In addition, it verifies whether the buffer's size parameters (such as width and height) meet the encoding parameters' limitations on buffer size in the capability list. For example, if the encoder supports a maximum resolution of 1920×1080, but the buffer size is 2560×1440, this verification item fails. Only when all three verifications pass is the first compatibility check considered successful.

[0060] In the second compatibility check, the protocol negotiation and forwarding engine component will focus on verifying the target cloud hardware encoder's access permissions to the target memory access buffer. This includes checking whether the buffer is allowed to be accessed by the encoder's driver. For example, some hardware encoders may only allow access to specific types of memory (such as dedicated video memory rather than system memory). If the memory area where the buffer is located is not allowed, the access permission verification will fail, and the second compatibility check will be considered as failing.

[0061] The protocol negotiation and forwarding engine component integrates the results of the first and second compatibility checks to generate the final compatibility check result. If all verification items pass, the result is "Pass"; if any verification fails, the result is "Fail". This result directly affects the subsequent hardware encoding service component's processing strategy for the first frame of image data to be encoded, and is a crucial step in ensuring the smooth progress of the encoding process.

[0062] S206. Based on the compatibility check result, the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded.

[0063] The timing of this step is after the protocol negotiation and forwarding engine component completes the compatibility check of the target memory access buffer and obtains the result. The scenario is to adopt different processing strategies according to the compatibility check result, to ensure that the first frame of image data to be encoded can be correctly encoded, and to provide a basis for subsequent image transmission and display.

[0064] Specifically, the protocol negotiation and forwarding engine component sends the compatibility check result to the hardware encoding service component, and the hardware encoding service component takes corresponding processing measures according to the result.

[0065] If the compatibility check result is "pass", it means that the properties of the target memory access buffer fully meet the requirements of the target cloud hardware encoder. At this time, the protocol negotiation and forwarding engine component sends the handle of the target memory access buffer to the hardware encoding service component. After receiving the handle, the hardware encoding service component calls the driver of the target cloud hardware encoder to directly access the first frame of image data to be encoded in the target memory access buffer. The encoder encodes the image data according to the encoding parameters (such as encoding standard, code rate, frame rate, etc.) supported in the capability list. For example, if the encoder supports H.265 encoding standard and the negotiated code rate is 5Mbps, the encoder will use H.265 algorithm to compress the first frame of image data into video frame data with a code rate of about 5Mbps, generating the first frame of encoded data.

[0066] If the compatibility check result is "not passed", adaptive processing needs to be performed according to the specific reason for not passing. If it is not passed due to the first compatibility check (such as incompatible pixel format, unsupported buffer modifier, or size parameter not meeting the requirements), the protocol negotiation and forwarding engine component will start the built-in minimal adaptation unit. This unit is a lightweight GPU computing context that can call the GPU to perform format conversion (such as converting RGB565 to YUV420) or modifier adjustment (such as converting discrete memory layout to continuous memory layout) on the first frame of image data to be encoded in the target memory access buffer, generate adaptation image data and corresponding adaptation memory access buffer that are fully compatible with the capability list of the target cloud hardware encoder, and export the direct memory access (DMA) file descriptor of the buffer. The protocol negotiation and forwarding engine component sends the file descriptor to the hardware encoding service component, and the hardware encoding service component calls the target cloud hardware encoder to encode the adaptation image data to generate the first frame of encoded data.

[0067] If the second compatibility check fails (i.e., the target cloud hardware encoder does not have access to the target memory access buffer), the protocol negotiation and forwarding engine component also initiates the minimal adaptation unit, which calls the GPU to internally read the first frame of image data to be encoded in the target memory access buffer and copies it to an internal memory access buffer created by the minimal adaptation unit, which has a memory type that meets the access requirements of the encoder driver, thereby obtaining an adapted memory access buffer. The DMA file descriptor of this internal buffer is then exported and sent by the protocol negotiation and forwarding engine component to the hardware encoding service component, which calls the encoder to encode the adapted image data to generate the first frame of encoded data.

[0068] Regardless of the processing mode, after the hardware encoding service component generates the first frame of encoded data, it stores or forwards it to the subsequent transmission module for transmission to the client over the network for decoding and display. At the same time, the system records relevant information about this processing for reference in the processing of subsequent frames to improve overall processing efficiency.

[0069] In the above embodiment, the hardware encoder capability list is queried in advance, and the target rendering format compatible with the cloud application is determined through negotiation; the redundant operation of the traditional compositor is omitted, and rendering is directly completed based on the target rendering format; and the compatibility check (or minimal adaptation) ensures that the data can be directly processed by the encoder. Therefore, CPU / memory waste in the traditional scheme is avoided, latency is reduced, and the problem of buffer and encoder adaptation difference, CPU re-reading conversion, etc. is solved (especially improving NVIDIA closed-source driver compatibility), effectively solving the problems of low efficiency, poor compatibility, and high latency of cloud application graphics data flow under the Wayland protocol, and realizing efficient direct-throughput of graphics data rendering-encoding, ensuring low latency and high-quality experience, and being compatible with multiple manufacturers' GPUs.

[0070] In some embodiments, after the hardware encoding service component completes the encoding process of the first frame of to-be-encoded image data, before the target cloud application starts processing the to-be-rendered data of the subsequent frame, in order to optimize the processing flow of the subsequent frame, reduce repeated operations and resource consumption, and improve the efficiency and real-time performance of the entire graphics processing system, the following operations can be performed: when the compatibility check result of the first frame processing is "pass", it indicates that the target rendering format fully meets the capability requirements of the target cloud hardware encoder, and the target memory access buffer can be directly accessed and processed by the encoder. At this time, the system records the target rendering format, including its corresponding pixel format, buffer modifier, size parameter and other detailed information. In the processing of the subsequent frame, the target cloud application directly performs graphics rendering according to the recorded target rendering format, without the need to negotiate with the protocol and forward the engine component for format negotiation. The protocol negotiation and forwarding engine component will also use this format as the standard by default, and when checking the compatibility of the memory access buffer of the subsequent frame, it only needs to verify whether it is consistent with the default format, greatly simplifying the checking process. This process avoids repeated negotiation and complex checking for each frame processing, reduces the interaction overhead of the system, ensures that the subsequent frame can quickly enter the encoding stage, and thus reduces the overall processing delay.

[0071] When the compatibility check result of the first frame processing is "not pass", it indicates that the first frame of to-be-encoded image data has been adapted by the minimal adaptation unit (such as format conversion, memory copying, etc.) to meet the requirements of the encoder. At this time, the system records the adaptation processing information of the first frame in detail, including the format parameters before and after adaptation, conversion algorithm, memory address mapping relationship, processing time consumption, etc. In the processing of the subsequent frame, the system directly reuses these adaptation processing information, without the need to analyze compatibility problems or reconfigure adaptation parameters. For example, if the first frame uses the RGB to YUV conversion algorithm due to incompatible pixel formats, the subsequent frame will directly call the same algorithm for conversion; if the first frame copies data to a specific type of memory buffer due to memory access permission problems, the subsequent frame will directly use the type of buffer and follow the same copying process. This reuse mechanism avoids repeated adaptation analysis and parameter configuration for each frame of data, reduces the resource consumption of GPU and CPU, ensures that the adaptation processing of the subsequent frame can be efficiently and stably performed, and thus guarantees the processing efficiency and continuity of the entire graphics stream.

[0072] Whether it is recording the target rendering format or adaptation processing information, the core purpose of this step is to optimize the processing flow of the subsequent frame by reusing the effective information of the first frame processing, reduce redundant operations, improve the overall performance of the system, and ensure that the graphics data of the cloud application can be processed and transmitted in a stable and efficient manner, providing a smooth interactive experience for users.

[0073] The Wayland protocol pass-through graphics processing system in the embodiments of the present application is described below from the perspective of hardware processing. Please refer to Figure 3 FIG. 1 is a schematic diagram of an entity device structure of the Wayland protocol pass-through graphics processing system in the embodiments of the present application.

[0074] It should be noted that Figure 3 The structure of the Wayland protocol pass-through graphics processing system shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0075] As Figure 3 shown, the Wayland protocol pass-through graphics processing system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or programs loaded from a storage portion 308 into a random access memory (RAM) 303, such as performing the methods described in the above embodiments. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0076] The following components are connected to the I / O interface 305: an input portion 306 including an audio input device, a button switch, and the like; an output portion 307 including a liquid crystal display (LCD) and an audio output device, an indicator, and the like; a storage portion 308 including a hard disk and the like; and a communication portion 309 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 310 as needed, so that a computer program read therefrom is installed in the storage portion 308 as needed.

[0077] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising computer programs for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present application are executed.

[0078] Note that specific examples of the computer readable storage medium can include one or more of a bulk storage, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, an optical storage, a portable compact disc read-only memory (CD-ROM), an optical storage, a magnetic storage, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0079] The flowcharts and block diagrams in the attached drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved.

[0080] In particular, the Wayland protocol pass-through graphics processing system of the present embodiment includes a processor and a memory, and the memory stores a computer program, and the computer program is executed by the processor to implement the cloud application-based Wayland protocol pass-through graphics processing method provided by the above-mentioned embodiments.

[0081] As another aspect, the present application also provides a computer readable storage medium, which can be included in the Wayland protocol direct graphics processing system described in the above embodiments, or can exist independently without being assembled into the Wayland protocol direct graphics processing system. The storage medium carries one or more computer programs, which, when executed by a processor of the Wayland protocol direct graphics processing system, enable the Wayland protocol direct graphics processing system to implement the cloud application based Wayland protocol direct graphics processing method provided in the above embodiments.

[0082] The above described and above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0083] In the above embodiments, according to the context, the term "when" can be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting". Similarly, according to the context, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted as meaning "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)".

[0084] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be instructed by a computer program to relevant hardware, and the program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above method embodiments can be included. The foregoing storage medium includes ROM or random storage memory RAM, magnetic disc or optical disc and various program code storage media.

Claims

1. A Wayland protocol pass-through graphics processing method based on cloud applications, applied to a Wayland protocol pass-through graphics processing system, characterized in that, The Wayland protocol pass-through graphics processing system includes a protocol negotiation and forwarding engine component and a hardware encoding service component that replace the traditional Wayland synthesizer. The method includes: When the target cloud application initiates a session request, the hardware encoding service component queries the capability list of the target cloud hardware encoder and sends the capability list to the protocol negotiation and forwarding engine component. The capability list includes supported pixel formats, buffer modifiers, and encoding parameters. Based on the capability list, the target cloud application is negotiated through the protocol negotiation and forwarding engine component to determine the target rendering format of the target cloud application that is compatible with the capability list; Upon receiving the graphics rendering request initiated by the target cloud application and the data to be rendered for consecutive frames, the graphics driver is invoked to create a target memory access buffer in the target rendering format. Based on the first frame of data to be rendered, single-frame graphics rendering is completed in the target memory access buffer to obtain the first frame of image data to be encoded, and the handle of the target memory access buffer is fed back to the protocol negotiation and forwarding engine component. The target memory access buffer is checked for compatibility using the protocol negotiation and forwarding engine component to obtain the compatibility check result. Based on the compatibility check results, the hardware encoding service component calls the target cloud hardware encoder to encode the first frame of image data to be encoded.

2. The method according to claim 1, characterized in that, The step of performing a compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain the compatibility check result includes: The protocol negotiation and forwarding engine component performs a first compatibility check on the target memory access buffer, specifically including: Verify whether the pixel format of the target memory access buffer belongs to the pixel formats supported by the capability list; Verify that the buffer modifier of the target memory access buffer is within the range of buffer modifiers supported by the capability list; Verify that the size parameter of the target memory access buffer conforms to the buffer size limit imposed by the encoding parameters in the capability list; If all the above verifications pass, the compatibility check result is determined to be passed; If any verification fails, the compatibility check result is determined to be unsuccessful.

3. The method according to claim 1, characterized in that, The step of performing a compatibility check on the target memory access buffer through the protocol negotiation and forwarding engine component to obtain the compatibility check result further includes: The second compatibility check of the target memory access buffer is performed by the protocol negotiation and forwarding engine component, specifically including: Verify the access permissions of the target cloud hardware encoder to the target memory access buffer, wherein the access permissions include at least whether the target memory access buffer is allowed to be accessed by the driver of the target cloud hardware encoder; If the access permission verification fails, the compatibility check result is determined to be unsuccessful.

4. The method according to claim 1, characterized in that, The step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data based on the compatibility check result includes: If the compatibility check result is passed, the protocol negotiation and forwarding engine component is controlled to send the handle of the target memory access buffer to the hardware coding service component. The target cloud hardware encoder is invoked to encode the first frame of image data to be encoded, generating the first frame encoded data.

5. The method according to claim 1 or 2, characterized in that, The protocol negotiation and forwarding engine component has a built-in simplified adaptation unit, which is a lightweight GPU computing context. The step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data based on the compatibility check result further includes: If the compatibility check result corresponding to the first compatibility check is not passed, the protocol negotiation and forwarding engine component is controlled to start the simplified adaptation unit, call the GPU to perform format conversion or modifier adjustment on the first frame image data to be encoded in the target memory access buffer, so as to generate adapted image data and corresponding adapted memory access buffer that are fully compatible with the capability list of the target cloud hardware encoder, and export the direct memory access buffer file descriptor of the adapted memory access buffer. The control protocol negotiation and forwarding engine component sends the direct memory access buffer file descriptor to the hardware encoding service component, and calls the target cloud hardware encoder to encode the adapted image data to generate the first frame of encoded data.

6. The method according to claim 1 or 3, characterized in that, The step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data to be encoded also includes: If the compatibility check result corresponding to the second compatibility check is not passed, the protocol negotiation and forwarding engine component is controlled to start the simplified adaptation unit, call the GPU to read the first frame image data to be encoded in the target memory access buffer internally, and copy the first frame image data to be encoded to the internal memory access buffer created by the simplified adaptation unit to obtain an adapted memory access buffer that is compatible with the driver access requirements. Export the direct memory access buffer file descriptor of the internal memory access buffer; The control protocol negotiation and forwarding engine component sends the direct memory access buffer file descriptor to the hardware encoding service component, and calls the target cloud hardware encoder to encode the adapted image data to generate the first frame of encoded data.

7. The method according to claim 1, characterized in that, After the step of the hardware encoding service component calling the target cloud hardware encoder to encode the first frame of image data based on the compatibility check result, the method further includes: If the compatibility check result is passed, the target rendering format is recorded, and the target rendering format is used as the default rendering format standard in the processing of the image data to be encoded in subsequent frames. If the compatibility check result is unsuccessful, the adaptation processing information of the first frame of image data to be encoded is recorded, and the adaptation processing information is reused in the processing of the image data to be encoded in subsequent frames.

8. A Wayland protocol pass-through graphics processing system, characterized in that, The Wayland protocol passthrough graphics processing system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the Wayland protocol passthrough graphics processing system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on a Wayland Protocol passthrough graphics processing system, the Wayland Protocol passthrough graphics processing system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on a Wayland Protocol passthrough graphics processing system, the Wayland Protocol passthrough graphics processing system performs the method as described in any one of claims 1-7.

Images