SPICE protocol image data processing system and method based on hardware unloading

Abstract

The invention provides an SPICE protocol image data processing system and method based on hardware unloading, and relates to the technical field of computers. The virtual machine is used for generating original RGB image data and transmitting the original RGB image data to the server; the server comprises a first customized FPGA network card and is used for receiving original RGB image data and coding based on different coding modes; the first customized FPGA network card comprises a plurality of H.264 hardware coding cores and a color space conversion circuit; a preset static mapping relationship exists between the plurality of H.264 coding cores and a logic block region output by a virtual machine screen; the color space conversion circuit is used for converting the RGB format image data into a YUV format; and the client comprises a second customized FPGA network card and is used for receiving the encoded data from the server and decoding the encoded data to finish image restoration and display. Therefore, the resource consumption of the server is reduced based on the customized FPGA network card and the static mapping relationship, the coding efficiency is improved, and the bandwidth occupation is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a SPICE protocol image data processing system and method based on hardware offloading. Background Technology

[0002] SPICE, as a mainstream virtual desktop communication protocol, meets daily office needs with its rich functionality. Its core data flow logic is as follows: After the Spice Server obtains the virtual machine image data, it encodes and transmits the data to the client for decoding and display, thus realizing remote desktop interaction. In high-frequency scenarios such as video playback and multi-user concurrency, image encoding efficiency directly determines the user experience, becoming a key performance bottleneck for protocol applications.

[0003] In existing technologies, image encoding mainly relies on the server-side CPU to perform JPEG or H264 encoding, which not only consumes a large amount of CPU resources but also leads to frame rate drops and increased latency when multiple devices are connected concurrently. Some optimization solutions introduce the GPU to handle H264 encoding, but the RGB to YUV transcoding still requires CPU participation, and the encoded data needs to be copied to memory before transmission, limiting efficiency. Furthermore, existing solutions encode and transmit the entire screen data regardless of whether the screen image changes locally, resulting in excessive bandwidth consumption and further affecting transmission stability. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a SPICE protocol image data processing system and method based on hardware offloading. By deploying customized FPGA network cards on both the server and client sides, and based on the static mapping relationship between screen areas and hardware encoding cores, hardware offloading and adaptive parallel processing of encoding and decoding tasks are achieved. This effectively reduces the CPU load on the server side and decreases network bandwidth usage, ensuring low latency and high smoothness transmission of the virtual desktop.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a SPICE protocol image data processing system based on hardware offloading, including a server, a client, and a virtual machine running on the server; The virtual machine is used to generate raw RGB image data and transmit it to the server. The server includes a first custom FPGA network card for receiving the original RGB image data and encoding it based on different encoding modes. The first custom FPGA network card includes multiple H.264 hardware encoding cores and a color space conversion circuit. The multiple H.264 encoding cores have a preset static mapping relationship with the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of its corresponding area. The color space conversion circuit is used to convert the RGB format image data to YUV format. The client includes a second custom FPGA network card, which is used to receive encoded data from the server and decode it to complete image restoration and display.

[0006] Preferably, the server writes the original RGB image data to the first custom FPGA network card via PCIe.

[0007] Preferably, the different encoding modes include full-screen parallel encoding and selective encoding of changing regions; In the full-screen parallel encoding mode, the virtual machine outputs the entire frame of raw RGB image data, which is then written to the first customized FPGA network card after being divided into logical block regions. The network card converts the data of each region into YUV format in parallel through a color space conversion circuit, and routes the YUV data of each region to the corresponding H.264 hardware encoding core for synchronous parallel encoding according to the static mapping relationship. After all regions are encoded, the first customized FPGA network card merges and encapsulates the bit streams output by each encoding core and sends them to the client through the network interface.

[0008] Preferably, in the selective encoding mode for the changing region, the server identifies the coordinate information of the screen changes, locates the changing logical block, and captures the image data of the changing logical block; only the original RGB image data of the changing region and its region identifier are written to the first customized FPGA network card. After receiving data, the network card activates only the corresponding H.264 hardware encoding core that has a static mapping relationship with the changed area for encoding based on the area identifier, and sends the encoded area bitstream and its coordinate information on the screen to the client.

[0009] Preferably, the first customized FPGA network card further includes a general data compression core for compressing or decompressing non-video data.

[0010] Secondly, the present invention provides a method for processing SPICE protocol image data based on hardware offloading, comprising: The virtual machine generates the original RGB image data and transmits it to the server. Within the first custom FPGA network card configured on the server, a static mapping relationship is preset between multiple H.264 hardware encoding cores and the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of the corresponding area; the received raw RGB image data is processed based on different encoding modes and the static mapping relationship, first converting the RGB data to YUV, and then performing H.264 encoding; the encoded bitstream is sent to the client through the network interface; The client receives and decodes the bitstream through a second customized FPGA network card to complete image restoration and display.

[0011] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the SPICE protocol image data processing method based on hardware offloading as described in the second aspect.

[0012] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the SPICE protocol image data processing method based on hardware offloading described in the second aspect.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention significantly reduces the host CPU's resource consumption by offloading format transcoding, image encoding, and data transmission functions from the server-side CPU to a custom FPGA network interface card (NIC). This mitigates the risk of frame rate drops caused by CPU performance bottlenecks in multi-user concurrent scenarios. Furthermore, the custom NIC integrates multiple encoding cores to support parallel processing and adaptively employs full-screen segmented encoding or local area encoding modes to adapt to screen variation characteristics. Simultaneously, it directly performs data transcoding, encoding, and transmission in hardware, reducing memory copying and significantly improving encoding efficiency and reducing transmission latency, further lowering bandwidth costs.

[0014] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0016] Figure 1 The main flowchart of a SPICE protocol image data processing method based on hardware offloading is provided in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Example 1 This embodiment discloses a SPICE protocol image data processing system based on hardware offloading, including a server, a client, and a virtual machine running on the server; The virtual machine is used to generate raw RGB image data and transmit it to the server. The server includes a first custom FPGA network card for receiving the original RGB image data and encoding it based on different encoding modes. The first custom FPGA network card includes multiple H.264 hardware encoding cores and a color space conversion circuit. The multiple H.264 encoding cores have a preset static mapping relationship with the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of its corresponding area. The color space conversion circuit is used to convert the RGB format image data to YUV format. The client includes a second custom FPGA network card, which is used to receive encoded data from the server and decode it to complete image restoration and display.

[0019] Next, a detailed description will be given of a SPICE protocol image data processing method based on hardware offloading disclosed in this embodiment.

[0020] The virtual desktop system architecture based on the SPICE protocol and employing hardware encoding / decoding offloading consists of three parts: a server (Host), a client (Client), and a virtual machine (Guest OS) running on the server. This embodiment introduces a customized smart network card to take over the heavy encoding / decoding tasks.

[0021] Within the virtual machine (Guest OS), applications generate image data via the Graphics Device Interface (GDI), while data from peripherals such as audio, camera, and network are also generated. This data is then output via virtual hardware (such as a QXL graphics card, AC97 sound card, Vcamera, and Vnic).

[0022] The SPICE server, acting as the core intermediary, resides on the host machine. Its key responsibility is to receive raw RGB image data from the virtual machine. However, unlike traditional software encoding schemes, in this embodiment, it does not perform encoding calculations but directly sends the raw RGB data to the first custom FPGA network interface card (NIC) equipped on the host machine. This custom FPGA NIC is designed as a hardware device that supports multi-channel concurrent encoding and decoding.

[0023] The client is also equipped with a second custom FPGA network card, whose structure is identical to the first custom FPGA network card. It is responsible for receiving downlink data streams from the server and performing hardware decoding, while simultaneously hardware encoding uplink data from local peripherals (such as cameras). The client application reads the decoded image and audio data from the network card for local rendering and playback, and reads peripheral data and sends it to the network card for uplink transmission.

[0024] Ultimately, users access the virtual desktop through cloud terminal devices. The entire architecture improves processing efficiency and system concurrency by offloading the encoding and decoding workload to dedicated hardware, forming an end-to-end hardware-accelerated data path from the virtual machine to the client.

[0025] Specifically, the customized FPGA network card has encoding and decoding functions, and the network card is connected to the server motherboard via the PCIe bus. Multiple H.264 hardware encoding cores, a general data compression (GZIP) core, and color space conversion circuitry are integrated within the FPGA chip.

[0026] In this embodiment, nine H.264 hardware encoding cores are preferably used. Each encoding core is pre-assigned to process a specific logical block area of ​​the screen, such as dividing the entire screen into nine areas in three rows and three columns. This static mapping relationship between areas and encoding cores ensures that image data, after being written to the network card, can be directly routed to the corresponding encoding core for parallel processing without scheduling delay, thus providing the hardware foundation for low-latency full-screen encoding and efficient local updates.

[0027] The General Data Compression (GZIP) core is a dedicated lossless compression / decompression hardware engine. It is primarily used to process non-video general data streams transmitted in virtual desktops, such as client USB drive file reading and writing, images captured by a webcam (when not encoded in H.264), or network redirected data. For uplink data sent from the client to the server, the GZIP core compresses it before transmission to reduce network bandwidth consumption; for downlink compressed data received by the server, it decompresses it, thus completely offloading the compression / decompression tasks from the server's CPU.

[0028] The color space conversion circuit is a set of parallel processing units specifically designed for high-performance pixel format conversion. It can convert RGB format pixel data (such as RGB24 or ARGB32) written by the SPICE server into the YUV color space format required by H.264 encoding in real time and pipelinedly using a hardware-implemented conversion algorithm. This conversion process is completed before the data enters the encoding core, requiring no CPU intervention and eliminating the performance overhead and latency associated with software conversion.

[0029] In one implementation, the customized FPGA network interface card (NIC) is logically abstracted as a smart NIC supporting advanced functions. To the upper-layer SPICE server, its interface behavior resembles a standard network device, but it possesses the ability to receive raw image data and directly perform encoding and transmission. The application layer only needs to write the data to be encoded and the receiving end's IP information into the device; encoding and transmission are entirely handled by hardware, transparent to the application layer and the terminal. This allows data to be written only once, saving memory and CPU resources and improving data flow efficiency.

[0030] When the SPICE server is running, it first obtains the full-screen image data to be transmitted from the video output buffer of the virtual machine (Guest OS), which is usually in RGB format.

[0031] Unlike traditional methods, the spice-server in this embodiment does not invoke the CPU or GPU for software encoding. Instead, it directly transmits the RGB data to the internal buffer of the custom FPGA network card via PCIe. This achieves direct data transfer from host memory to the dedicated hardware processing unit, avoiding intermediate transfer and copying within CPU memory.

[0032] The specific encoding process employs two optimization modes, either adaptively or selectively, based on the changing characteristics of the screen image: (a) Full-screen parallel encoding The full-screen parallel encoding mode is suitable for scenarios involving large-scale updates or initial transmissions of screen content. In this mode, the spice-server logically divides a complete frame of RGB image data into multiple equally sized rectangular regions; in this embodiment, it is divided into nine regions using a 3x3 grid. These data regions are then sequentially written to the FPGA network card via the PCIe interface.

[0033] The hardware circuitry within the network card first converts the RGB pixel data of each region into the YUV color space in parallel. Then, according to a preset fixed mapping relationship, it allocates the YUV data streams of different regions to the corresponding, independent hardware encoding cores.

[0034] For example, the image data of region 1 is always processed by encoding core 1, region 2 by encoding core 2, and so on. All encoding cores work simultaneously, performing H.264 encoding on their respective assigned data blocks.

[0035] After all regions are encoded, the control logic within the network card merges the bitstreams output by each encoding core in the original region order, encapsulates them into complete frame data packets, and finally sends them directly to the network through the integrated Gigabit Ethernet MAC layer to be transmitted to the client.

[0036] This mode fully utilizes the hardware's inherent parallel processing capabilities, decomposing the originally serial full-frame encoding task into multiple subtasks that are executed concurrently, thereby effectively reducing the total latency of single-frame encoding.

[0037] (ii) Selective coding of change regions The variable region selective encoding mode is suitable for local interface update scenarios common in desktop operations, in order to optimize bandwidth.

[0038] In this mode, the spice-server analyzes the coordinate information of the changing areas in real time. The screen is also logically divided into fixed areas corresponding to the hardware encoding core, such as 9 areas. The spice-server only packages and sends the RGB image data of the area block containing the changing pixels, along with the coordinate information of that area on the screen, to the FPGA network card.

[0039] For example, if the change only occurs in regions 2 and 5, then only data from these two regions is sent. After receiving the data, the FPGA network card activates only the corresponding encoding cores that are fixedly mapped to it (cores 2 and 5 in this example) according to the region number. After encoding is complete, the network card sends the compressed region bitstream along with its coordinate information to the client.

[0040] After receiving this data, client-side software (such as remote-viewer) can update the decoded image blocks to the corresponding positions on the screen based on the coordinate information, achieving efficient local screen refresh. This mode greatly reduces the amount of data to be processed and the network bandwidth usage when updating the screen locally by encoding and transmitting only the changing areas.

[0041] Furthermore, on the client side, a similar or identical custom FPGA network card is deployed for decoding. After receiving network packets from the server, the client's network card first parses the data type: if it's an H.264 stream, the integrated hardware decoding core decodes it back to YUV or RGB data; if it's compressed general-purpose data, such as data from a client's USB flash drive or camera peripheral, the GZIP decoding core decompresses it. The decoded raw data is then submitted to the client's SPICE protocol processing module (remote-viewer) for final display or submitted to the server virtual machine.

[0042] This specific embodiment constructs a complete hardware encoding / decoding offloading path by introducing customized FPGA network cards dedicated to both the server and client. On one hand, it completely transfers the encoding tasks that traditionally rely on the CPU or GPU to the dedicated hardware encoding core and color space conversion circuit within the FPGA, freeing up host computing resources and eliminating the performance loss caused by RGB-YUV conversion and data memory copying, ensuring high frame rates and low latency under multi-channel concurrency. On the other hand, based on the static mapping relationship between screen logical blocks and encoding cores, the system can flexibly support full-screen parallel encoding and selective encoding of changing areas: during large-scale updates, multiple encoding cores process the entire screen in parallel, improving encoding speed; during partial updates, only the changing blocks are encoded and transmitted, significantly saving bandwidth. Thus, the system achieves end-to-end hardware acceleration while balancing encoding efficiency and network transmission efficiency, effectively solving the problems of high CPU load, limited concurrency, and excessive bandwidth consumption in existing solutions.

[0043] Example 2 Based on the SPICE protocol image data processing system based on hardware offloading proposed in Embodiment 1, this embodiment provides a SPICE protocol image data processing method based on hardware offloading, such as... Figure 1 As shown, it includes the following steps: S1: The virtual machine generates the original RGB image data and transmits it to the server; S2: Within the first custom FPGA network card configured on the server, a static mapping relationship is preset between multiple H.264 hardware encoding cores and the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of the corresponding area; the received raw RGB image data is processed based on different encoding modes and the static mapping relationship, first converting the RGB data to YUV, and then performing H.264 encoding; the encoded bitstream is sent to the client through the network interface; S3: The client receives and decodes the bitstream through the second customized FPGA network card to complete image restoration and display.

[0044] This specific embodiment introduces a customized FPGA smart network interface card with integrated multi-encoding cores and deep collaboration with the SPICE protocol stack to establish a hardware-accelerated image processing pipeline that features fixed-region mapping, change perception, and parallel processing. It completely offloads high-load encoding computation tasks from the server CPU to dedicated hardware and integrates multiple steps such as data processing, encoding, and network transmission within the chip, achieving significant performance improvements and resource savings.

[0045] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a hardware-offload-based SPICE protocol image data processing method as described in Embodiment 2 above.

[0046] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the SPICE protocol image data processing method based on hardware offloading as described in Embodiment 2 above.

[0047] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A SPICE protocol image data processing system based on hardware offloading, characterized in that, This includes the server, the client, and the virtual machine running on the server; The virtual machine is used to generate raw RGB image data and transmit it to the server. The server includes a first custom FPGA network card, used to receive the original RGB image data and encode it based on different encoding modes; The first custom FPGA network card includes multiple H.264 hardware encoding cores and a color space conversion circuit; wherein, the multiple H.264 encoding cores have a preset static mapping relationship with the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of the corresponding area; the color space conversion circuit is used to convert RGB format image data into YUV format; The client includes a second custom FPGA network card, which is used to receive encoded data from the server and decode it to complete image restoration and display.

2. The SPICE protocol image data processing system based on hardware offloading as described in claim 1, characterized in that, The server writes the original RGB image data into the first custom FPGA network card via PCIe.

3. The SPICE protocol image data processing system based on hardware offloading as described in claim 1, characterized in that, The different encoding modes include full-screen parallel encoding and variable region selective encoding; In the full-screen parallel encoding mode, the virtual machine outputs the entire frame of raw RGB image data, which is then written to the first customized FPGA network card after being divided into logical block regions. The network card converts the data of each region into YUV format in parallel through a color space conversion circuit, and routes the YUV data of each region to the corresponding H.264 hardware encoding core for synchronous parallel encoding according to the static mapping relationship. After all regions are encoded, the first customized FPGA network card merges and encapsulates the bit streams output by each encoding core and sends them to the client through the network interface.

4. The SPICE protocol image data processing system based on hardware offloading as described in claim 3, characterized in that, In the selective encoding mode for changing regions, the server identifies the coordinate information of screen changes, locates the changing logical blocks, and captures the image data of the changing logical blocks; only the original RGB image data of the changing regions and their region identifiers are written to the first customized FPGA network card. After receiving data, the network card activates only the corresponding H.264 hardware encoding core that has a static mapping relationship with the changed area for encoding based on the area identifier, and sends the encoded area bitstream and its coordinate information on the screen to the client.

5. The SPICE protocol image data processing system based on hardware offloading as described in claim 1, characterized in that, The first custom FPGA network card also includes a general data compression core for compressing or decompressing non-video data.

6. A method for processing SPICE protocol image data based on hardware offloading, based on the SPICE protocol image data processing system based on hardware offloading as described in claim 1, characterized in that, include: The virtual machine generates the original RGB image data and transmits it to the server. Within the first custom FPGA network card configured on the server, a static mapping relationship is preset between multiple H.264 hardware encoding cores and the logical block areas output by the virtual machine screen, so that each encoding core is fixed to process the image data of the corresponding area; the received raw RGB image data is processed based on different encoding modes and the static mapping relationship, first converting the RGB data to YUV, and then performing H.264 encoding; The encoded bitstream is sent to the client via the network interface; The client receives and decodes the bitstream through a second customized FPGA network card to complete image restoration and display.

7. The SPICE protocol image data processing method based on hardware offloading as described in claim 6, characterized in that, The different encoding modes include full-screen parallel encoding and variable region selective encoding; In the full-screen parallel encoding mode, the virtual machine outputs the entire frame of raw RGB image data, which is then written to the first customized FPGA network card after being divided into logical block regions. The network card converts the data of each region into YUV format in parallel through a color space conversion circuit, and routes the YUV data of each region to the corresponding H.264 hardware encoding core for synchronous parallel encoding according to the static mapping relationship. After all regions are encoded, the first customized FPGA network card merges and encapsulates the bit streams output by each encoding core and sends them to the client through the network interface.

8. The SPICE protocol image data processing method based on hardware offloading as described in claim 7, characterized in that, In the selective encoding mode for changing regions, the server identifies the coordinate information of screen changes, locates the changing logical blocks, and captures the image data of the changing logical blocks; only the original RGB image data of the changing regions and their region identifiers are written to the first customized FPGA network card. After receiving data, the network card activates only the corresponding H.264 hardware encoding core that has a static mapping relationship with the changed area for encoding based on the area identifier, and sends the encoded area bitstream and its coordinate information on the screen to the client.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the SPICE protocol image data processing method based on hardware offloading as described in any one of claims 6-8.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the SPICE protocol image data processing method based on hardware offloading as described in any one of claims 6-8.

Images