A USB camera optimization method based on a SPICE protocol

Abstract

The application provides a USB camera optimization method based on a SPICE protocol, and belongs to the technical field of cloud desktops, and comprises the following steps: communication between a host computer and a virtual machine is established; when USB redirection is performed on a SPICE client, a camera device is filtered and ignored; a special USB video channel is established; camera video data is accessed and collected; the camera video data is encoded by using H.264; H.264 encoded data is received by a SPICE server, and is written into a virtual machine virtio device; after H.264 data is read and decoded, the data is written into a virtual camera device; an application program in the virtual machine selects the virtual camera device as a video input source, and acquires virtual camera data. The method can effectively improve the problems of high time delay and high bandwidth occupation when USB camera data is transmitted by using the SPICE protocol under the network isolation environment of the client and the virtual machine.

Description

Technical Field

[0001] This invention relates to a USB camera optimization method, and more particularly to a USB camera optimization method based on the SPICE protocol, belonging to the field of cloud desktop technology. Background Technology

[0002] With the continuous development of virtualization technology, the performance bottlenecks of servers in terms of computing, storage, and networking hardware resources have been significantly improved, driving the widespread deployment of cloud desktop technology across multiple industries. Among numerous remote desktop protocols, SPICE, with its excellent transmission efficiency and scalability, has gradually become one of the mainstream choices. However, in scenarios that heavily rely on real-time audio and video capabilities—such as bank teller services, remote video conferencing, and online education—the stable and efficient transmission of USB cameras still faces challenges. These applications place more stringent demands on the real-time performance, low latency, and high image quality of video data, further driving the continuous optimization and innovation of camera transmission solutions within cloud desktop architectures.

[0003] Currently, the known cloud desktop camera solutions mainly include the following:

[0004] like Figure 2 As shown, Scheme 1 employs the universal USB redirection technology supported by the SPICE protocol. Through collaboration between the client and server, the data stream from the physical USB camera device is transmitted to the virtual machine, and then provided to the application within the virtual machine. SPICE protocol USB redirection achieves seamless mapping of the client's USB camera to the virtual machine by detecting and managing the client's USB device, establishing a dedicated USB redirection channel, and simulating the USB device in the virtual machine. The SPICE client uses libusb to detect the USB camera device and read data from it, then compresses the data using the BOT protocol and transmits it to the SPICE server. The SPICE server writes the data to the virtual USB controller of the virtual machine through spicevmc and usb-redir.

[0005] This approach has the following significant limitations:

[0006] Using the USB BOT (Bulk-Only Transport) protocol for data compression results in low compression efficiency, leading to large video data volumes and significantly increased bandwidth consumption. Furthermore, within the virtual machine, the application needs to convert USB-based data to a standard video format before it can be used. This conversion process introduces additional processing overhead, and the larger the data volume, the higher the latency, severely impacting applications with high real-time requirements.

[0007] like Figure 3As shown, Solution 2 collects USB camera video data on the client side through a proxy program, encodes and compresses it, and then transmits it directly to the virtual machine via the network. The proxy service in the virtual machine is responsible for receiving the data and providing it to the upper-layer application via the virtual camera device. This solution is entirely based on network transmission and does not involve the SPICE protocol. However, this solution has the following significant limitations:

[0008] In typical cloud desktop environments, strict network isolation strategies are usually adopted between clients and virtual machines to achieve security and data isolation. However, this solution relies on direct network communication between the two, which makes it unable to function properly in network isolation scenarios; furthermore, it relies on TCP communication, and interruptions will occur if the virtual machine's IP address changes, thus significantly limiting its practical deployment. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides a USB camera optimization method based on the SPICE protocol to reduce latency and network bandwidth of cloud desktop cameras and improve user experience.

[0010] To achieve the above objectives, the technical solution of the present invention is: a USB camera optimization method based on the SPICE protocol, comprising the following steps:

[0011] S1. Start QEMU on the host machine, configure the character device chardev and virtio device of type spicevmc, and establish communication between the host machine and the virtual machine;

[0012] S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the camera device is actively filtered and ignored at the libusb call level.

[0013] S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission;

[0014] S4. Access the physical USB camera on the terminal through the SPICE client and collect video data from the connected camera.

[0015] S5. The camera video data is encoded using the GStreamer plugin with high efficiency and sent asynchronously via a dedicated USB video channel.

[0016] The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev.

[0017] S7. Deploy the agent service in the virtual machine, create a virtual camera device with the help of v4l2loopback, read H.264 encoded data from the virtio port, decode it and write it directly to the created virtual camera device;

[0018] S8. The application within the virtual machine selects the virtual camera device as the video input source and acquires virtual camera data.

[0019] Furthermore, the virtual machine mentioned in step S1 is a Linux virtual machine.

[0020] Furthermore, the character device added in step S1 is used to communicate with the virtual machine on one end and to receive video streams from the USB video channel on the other end.

[0021] Furthermore, step S2 specifically includes the following: When the client uses spice-gtk, it uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E, it actively filters out the USB camera from the standard USB redirection link.

[0022] Furthermore, step S3 specifically includes the following:

[0023] Add a new channel type to the SPICE protocol;

[0024] The SPICE client initiates a TCP connection to the virtual machine with a channel type.

[0025] After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

[0026] Furthermore, step S4 specifically includes the following:

[0027] Open the USB device using the v4l2src GStreamer plugin;

[0028] Set the acquisition format to obtain the raw video data stream; the acquisition format is video / x-raw,format=YUY2.

[0029] Furthermore, step S5 specifically includes the following:

[0030] Build a GStreamer data processing pipeline to efficiently compress the raw camera video data; use the videoconvert plugin to ensure that the acquisition format matches the encoder;

[0031] An x264enc encoder is used, and parameters are configured to minimize encoding latency; these parameters include tune=zerolatency.

[0032] The generated H.264 encoded data is asynchronously sent to the host server via a specially established USB video channel.

[0033] Furthermore, step S7 specifically includes the following:

[0034] A virtual machine deploys a proxy service called camera-agent, which creates a virtual camera device through v4l2loopback. It uses the epoll mechanism to check if the virtio-serial port is readable, reducing the CPU consumption of the proxy service when it is idle, and reading H.264 encoded data when there is data.

[0035] Construct a GStreamer decoding pipeline to push the H.264 encoded data read by camera-agent from the virtio port to appsrc, use avdec_h264 for decoding, and then output the decoded data to the virtual camera device using v4l2sink.

[0036] Furthermore, the virtual camera device is / dev / video0.

[0037] The beneficial effects of the USB camera optimization method based on the SPICE protocol of the present invention are as follows:

[0038] This invention, based on the SPICE protocol, effectively overcomes the high latency and high bandwidth consumption problems of traditional solutions by optimizing the data transmission process of USB cameras without relying on network communication between the client and the virtual machine. It offers the following significant advantages:

[0039] 1. Significantly improved real-time performance: By adopting efficient H.264 encoding technology and asynchronous data transmission mechanism, the end-to-end transmission time of video data from acquisition to application within the virtual machine is greatly shortened, effectively ensuring the real-time performance of the video stream, which is fully in line with application scenarios with stringent low latency requirements such as bank teller systems and video conferencing.

[0040] 2. Significantly reduced bandwidth usage: By using H.264 encoding to efficiently compress the video stream and transmitting it through a dedicated camera channel, network bandwidth usage is effectively reduced, network resource utilization efficiency is improved, and it can still operate stably in bandwidth-constrained network environments.

[0041] 3. Enhanced system compatibility and deployment convenience: A virtual camera device ( / dev / video0) is created inside the virtual machine using v4l2 loopback technology, enabling applications to directly access the video source in the same way as accessing a local physical camera, without modifying the application code, significantly simplifying the system development and deployment process. Attached Figure Description

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0043] Figure 1 This is a schematic diagram of a USB camera optimization method based on the SPICE protocol according to the present invention.

[0044] Figure 2 This is a schematic diagram of the USB camera redirection process in the existing technology's Scheme 1.

[0045] Figure 3 This is a schematic diagram of the USB camera redirection process in the second scheme of the prior art. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] Example 1

[0048] Reference Figure 1 As shown, the present invention provides a USB camera optimization method based on the SPICE protocol, comprising the following steps:

[0049] S1. Start QEMU on the host machine, configure the character device chardev (of type spicevmc) and the virtio device to establish communication between the host machine and the virtual machine (in this example, a Linux virtual machine); one end of the added character device is used to communicate with the virtual machine, and the other end is used to receive video streams from the USB video channel; the character device starts the virtual machine via the qemu command line, adding the following to the command line:

[0050] -chardev spicevmc,id=camchardev,name=usb-video \

[0051] -device virtio-serial \

[0052] -device virtserialport,chardev=camchardev,name=usb-video-port.

[0053] S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the camera device is actively filtered and ignored at the libusb call level to prevent it from being taken over by the standard SPICE USB redirection channel; specifically...

[0054] When the client uses spice-gtk, it internally uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E (video class), it actively filters out the USB camera from the standard USB redirection link.

[0055] S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission; specifically including the following:

[0056] Add a new channel type to the SPICE protocol;

[0057] The SPICE client initiates a TCP connection to the virtual machine with a channel type.

[0058] After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

[0059] S4. Access the physical USB camera on the terminal via the SPICE client and capture video data from the connected camera; specifically including the following:

[0060] Open the USB device using the v4l2src GStreamer plugin;

[0061] Set the capture format (e.g., video / x-raw,format=YUY2) to obtain the raw video data stream.

[0062] S5. Utilize the GStreamer plugin to efficiently encode the camera video data using H.264, and asynchronously send the encoded data via a dedicated USB video channel; specifically including the following:

[0063] Build a GStreamer data processing pipeline to efficiently compress the raw camera video data; use the videoconvert plugin to ensure that the acquisition format matches the encoder;

[0064] The x264enc encoder is used, and parameters such as tune=zerolatency are configured to minimize encoding latency;

[0065] The generated H.264 encoded data is asynchronously sent to the host server via a specially established USB video channel.

[0066] It should be noted that when GStreamer uses the x264enc encoder, the data it collects is H.264 encoded data. The x264enc encoder is a specific implementation of H.264 encoding.

[0067] The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev.

[0068] S7. Deploy an agent service in the virtual machine, create a virtual camera device using v4l2loopback, read H.264 encoded data from the virtio port, decode it, and directly write it to the created virtual camera device; specifically including the following:

[0069] A virtual machine deploys a proxy service called camera-agent, which creates a virtual camera device (e.g., / dev / video0) through v4l2 loopback. It uses the epoll mechanism to check if the virtio-serial port is readable, reducing the CPU consumption of the proxy service when idle, and reading H.264 encoded data when there is data.

[0070] Construct a GStreamer decoding pipeline, push the H.264 encoded data read by camera-agent from the virtio port to appsrc, decode it using avdec_h264, and then output it to the virtual camera device (e.g., / dev / video0) using v4l2sink.

[0071] S8. The application inside the virtual machine selects a virtual camera device (such as / dev / video0) as the video input source to obtain virtual camera data.

[0072] In summary, the present invention provides a USB camera optimization method based on the SPICE protocol, which eliminates the need for network communication between the client and the virtual machine, reduces latency and network bandwidth of cloud desktop cameras, and improves user experience.

[0073] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A USB camera optimization method based on the SPICE protocol, characterized in that, Includes the following steps: S1. Start QEMU on the host machine, configure the character device chardev and virtio device of type spicevmc, and establish a dedicated video pass-through communication path between the host machine and the virtual machine; one end of the character device is used to communicate with the virtual machine, and the other end is used to receive video streams from the USB video channel; S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the USB camera is actively filtered and ignored at the libusb call level. S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission; S4. Access the physical USB camera on the terminal through the SPICE client and collect video data from the connected camera. Specifically, this includes: using the v4l2src GStreamer plugin to open a USB device; Set the acquisition format to obtain the raw video data stream; the acquisition format is video / x-raw,format=YUY2; S5. Utilize the GStreamer plugin to efficiently encode the camera video data using H.264, and asynchronously send the H.264 encoded data through a dedicated USB video channel; specifically, this includes: building a GStreamer data processing pipeline to efficiently compress the acquired raw camera video data; and using the videoconvert plugin to ensure that the acquisition format matches the encoder. An x264enc encoder is used, and parameters are configured to minimize encoding latency; these parameters include tune=zerolatency. The generated H.264 encoded data is asynchronously sent to the host server via a dedicated USB video channel. The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev. S7. Deploy an agent service in a virtual machine, create a virtual camera device using v4l2loopback, read H.264 encoded data from the virtio port, decode it, and write it directly to the created virtual camera device; specifically, this includes: deploying an agent service camera-agent in the virtual machine, creating a virtual camera device using v4l2loopback, using the epoll mechanism to check if the virtio-serial port is readable to reduce the CPU consumption of the agent service when idle, and reading H.264 encoded data when data is available; Construct a GStreamer decoding pipeline to push the H.264 encoded data read by camera-agent from the virtio port to appsrc, use avdec_h264 to decode it, and then use v4l2sink to output it to the virtual camera device; S8. The application within the virtual machine selects the virtual camera device as the video input source and acquires virtual camera data.

2. The method of claim 1, wherein, The virtual machine mentioned in step S1 is a Linux virtual machine.

3. The method of claim 1, wherein, Step S2 specifically includes the following: When the client uses spice-gtk, it uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E, it actively filters out the USB camera from the standard USB redirection link.

4. The method of claim 1, wherein, Step S3 specifically includes the following: Add a new channel type to the SPICE protocol; The SPICE client initiates a TCP connection to the virtual machine with a channel type. After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

5. The method of claim 1, wherein, The virtual camera device is / dev / video0.

Images