USB camera optimization method based on SPICE protocol

Abstract

The invention provides a USB camera optimization method based on an SPICE protocol, and belongs to the technical field of cloud desktops, and the method comprises the following steps: establishing communication between a host machine and a virtual machine; when the SPICE client side carries out USB redirection, the camera device is filtered and ignored; establishing a special USB video channel; accessing and collecting camera video data; carrying out H.264 coding on the video data of the camera; the SPICE server side receives the H.264 coded data and writes the H.264 coded data into virtual machine virtio equipment; reading H.264 data, decoding the H.264 data, and writing the decoded H.264 data into And the application program in the virtual machine selects the virtual camera equipment as a video input source to obtain virtual camera data. According to the method provided by the invention, the problems of high time delay and high bandwidth occupation when the SPICE protocol transmits the USB camera data under the network isolation environment of the client and the virtual machine can be effectively improved.

Description

Technical Field

[0001] This invention relates to a USB camera optimization method, and more particularly to a USB camera optimization method based on the SPICE protocol, belonging to the field of cloud desktop technology. Background Technology

[0002] With the continuous development of virtualization technology, the performance bottlenecks of servers in terms of computing, storage, and networking hardware resources have been significantly improved, driving the widespread deployment of cloud desktop technology across multiple industries. Among numerous remote desktop protocols, SPICE, with its excellent transmission efficiency and scalability, has gradually become one of the mainstream choices. However, in scenarios that heavily rely on real-time audio and video capabilities—such as bank teller services, remote video conferencing, and online education—the stable and efficient transmission of USB cameras still faces challenges. These applications place more stringent demands on the real-time performance, low latency, and high image quality of video data, further driving the continuous optimization and innovation of camera transmission solutions within cloud desktop architectures.

[0003] Currently, the known cloud desktop camera solutions mainly include the following: like Figure 2 As shown, Scheme 1 employs the universal USB redirection technology supported by the SPICE protocol. Through collaboration between the client and server, the data stream from the physical USB camera device is transmitted to the virtual machine, and then provided to the application within the virtual machine. SPICE protocol USB redirection achieves seamless mapping of the client's USB camera to the virtual machine by detecting and managing the client's USB device, establishing a dedicated USB redirection channel, and simulating the USB device in the virtual machine. The SPICE client uses libusb to detect the USB camera device and read data from it, then compresses the data using the BOT protocol and transmits it to the SPICE server. The SPICE server writes the data to the virtual USB controller of the virtual machine through spicevmc and usb-redir.

[0004] This approach has the following significant limitations: Using the USB BOT (Bulk-Only Transport) protocol for data compression results in low compression efficiency, leading to large video data volumes and significantly increased bandwidth consumption. Furthermore, within the virtual machine, the application needs to convert USB-based data to a standard video format before it can be used. This conversion process introduces additional processing overhead, and the larger the data volume, the higher the latency, severely impacting applications with high real-time requirements.

[0005] like Figure 3As shown, Solution 2 collects USB camera video data on the client side through a proxy program, encodes and compresses it, and then transmits it directly to the virtual machine via the network. The proxy service in the virtual machine is responsible for receiving the data and providing it to the upper-layer application via the virtual camera device. This solution is entirely based on network transmission and does not involve the SPICE protocol. However, this solution has the following significant limitations: In typical cloud desktop environments, strict network isolation strategies are usually adopted between clients and virtual machines to achieve security and data isolation. However, this solution relies on direct network communication between the two, which makes it unable to function properly in network isolation scenarios; furthermore, it relies on TCP communication, and interruptions will occur if the virtual machine's IP address changes, thus significantly limiting its practical deployment. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a USB camera optimization method based on the SPICE protocol to reduce latency and network bandwidth of cloud desktop cameras and improve user experience.

[0007] To achieve the above objectives, the technical solution of the present invention is: a USB camera optimization method based on the SPICE protocol, comprising the following steps: S1. Start QEMU on the host machine, configure the character device chardev and virtio device of type spicevmc, and establish communication between the host machine and the virtual machine; S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the camera device is actively filtered and ignored at the libusb call level. S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission; S4. Access the physical USB camera on the terminal through the SPICE client and collect video data from the connected camera. S5. The camera video data is encoded using the GStreamer plugin with high efficiency and sent asynchronously via a dedicated USB video channel. The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev. S7. Deploy the agent service in the virtual machine, create a virtual camera device with the help of v4l2loopback, read H.264 encoded data from the virtio port, decode it and write it directly to the created virtual camera device; S8. The application within the virtual machine selects the virtual camera device as the video input source and acquires virtual camera data.

[0008] Furthermore, the virtual machine mentioned in step S1 is a Linux virtual machine.

[0009] Furthermore, the character device added in step S1 is used to communicate with the virtual machine on one end and to receive video streams from the USB video channel on the other end.

[0010] Furthermore, step S2 specifically includes the following: When the client uses spice-gtk, it uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E, it actively filters out the USB camera from the standard USB redirection link.

[0011] Furthermore, step S3 specifically includes the following: Add a new channel type to the SPICE protocol; The SPICE client initiates a TCP connection to the virtual machine with a channel type. After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

[0012] Furthermore, step S4 specifically includes the following: Open the USB device using the v4l2src GStreamer plugin; Set the acquisition format to obtain the raw video data stream; the acquisition format is video / x-raw,format=YUY2.

[0013] Furthermore, step S5 specifically includes the following: Build a GStreamer data processing pipeline to efficiently compress the raw camera video data; use the videoconvert plugin to ensure that the acquisition format matches the encoder; An x264enc encoder is used, and parameters are configured to minimize encoding latency; these parameters include tune=zerolatency. The generated H.264 encoded data is asynchronously sent to the host server via a specially established USB video channel.

[0014] Furthermore, step S7 specifically includes the following: A virtual machine deploys an agent service called camera-agent, which creates a virtual camera device through v4l2loopback. It uses the epoll mechanism to check if the virtio-serial port is readable, reducing the CPU consumption of the agent service when it is idle, and reading H.264 encoded data when there is data. Construct a GStreamer decoding pipeline to push the H.264 encoded data read by camera-agent from the virtio port to appsrc, use avdec_h264 for decoding, and then output the decoded data to the virtual camera device using v4l2sink.

[0015] Furthermore, the virtual camera device is / dev / video0.

[0016] The beneficial effects of the USB camera optimization method based on the SPICE protocol of the present invention are as follows: This invention, based on the SPICE protocol, effectively overcomes the high latency and high bandwidth consumption problems of traditional solutions by optimizing the data transmission process of USB cameras without relying on network communication between the client and the virtual machine. It offers the following significant advantages: 1. Significantly improved real-time performance: By adopting efficient H.264 encoding technology and asynchronous data transmission mechanism, the end-to-end transmission time of video data from acquisition to application within the virtual machine is greatly shortened, effectively ensuring the real-time performance of the video stream, which is fully in line with application scenarios with stringent low latency requirements such as bank teller systems and video conferencing.

[0017] 2. Significantly reduced bandwidth usage: By using H.264 encoding to efficiently compress the video stream and transmitting it through a dedicated camera channel, network bandwidth usage is effectively reduced, network resource utilization efficiency is improved, and it can still operate stably in bandwidth-constrained network environments.

[0018] 3. Enhanced system compatibility and deployment convenience: A virtual camera device ( / dev / video0) is created inside the virtual machine using v4l2 loopback technology, enabling applications to directly access the video source in the same way as accessing a local physical camera, without modifying the application code, significantly simplifying the system development and deployment process. Attached Figure Description

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 This is a schematic diagram of a USB camera optimization method based on the SPICE protocol according to the present invention.

[0021] Figure 2This is a schematic diagram of the USB camera redirection process in the existing technology's Scheme 1.

[0022] Figure 3 This is a schematic diagram of the USB camera redirection process in the second scheme of the prior art. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Example 1

[0025] Reference Figure 1 As shown, the present invention provides a USB camera optimization method based on the SPICE protocol, comprising the following steps: S1. Start QEMU on the host machine, configure the character device chardev (of type spicevmc) and the virtio device to establish communication between the host machine and the virtual machine (in this example, a Linux virtual machine); one end of the added character device is used to communicate with the virtual machine, and the other end is used to receive video streams from the USB video channel; the character device starts the virtual machine via the qemu command line, adding the following to the command line: -chardev spicevmc,id=camchardev,name=usb-video \ -device virtio-serial \ -device virtserialport,chardev=camchardev,name=usb-video-port.

[0026] S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the camera device is actively filtered and ignored at the libusb call level to prevent it from being taken over by the standard SPICE USB redirection channel; specifically... When the client uses spice-gtk, it internally uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E (video class), it actively filters out the USB camera from the standard USB redirection link.

[0027] S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission; specifically including the following: Add a new channel type to the SPICE protocol; The SPICE client initiates a TCP connection to the virtual machine with a channel type. After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

[0028] S4. Access the physical USB camera on the terminal via the SPICE client and capture video data from the connected camera; specifically including the following: Open the USB device using the v4l2src GStreamer plugin; Set the capture format (e.g., video / x-raw,format=YUY2) to obtain the raw video data stream.

[0029] S5. Utilize the GStreamer plugin to efficiently encode the camera video data using H.264, and asynchronously send the encoded data via a dedicated USB video channel; specifically including the following: Build a GStreamer data processing pipeline to efficiently compress the raw camera video data; use the videoconvert plugin to ensure that the acquisition format matches the encoder; The x264enc encoder is used, and parameters such as tune=zerolatency are configured to minimize encoding latency; The generated H.264 encoded data is asynchronously sent to the host server via a specially established USB video channel.

[0030] It should be noted that when GStreamer uses the x264enc encoder, the data it collects is H.264 encoded data. The x264enc encoder is a specific implementation of H.264 encoding.

[0031] The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev.

[0032] S7. Deploy an agent service in the virtual machine, create a virtual camera device using v4l2loopback, read H.264 encoded data from the virtio port, decode it, and directly write it to the created virtual camera device; specifically including the following: A virtual machine deploys a proxy service called camera-agent, which creates a virtual camera device (e.g., / dev / video0) through v4l2 loopback. It uses the epoll mechanism to check if the virtio-serial port is readable, reducing the CPU consumption of the proxy service when idle, and reading H.264 encoded data when there is data. Construct a GStreamer decoding pipeline, push the H.264 encoded data read by camera-agent from the virtio port to appsrc, decode it using avdec_h264, and then output it to the virtual camera device (e.g., / dev / video0) using v4l2sink.

[0033] S8. The application inside the virtual machine selects a virtual camera device (such as / dev / video0) as the video input source to obtain virtual camera data.

[0034] In summary, the present invention provides a USB camera optimization method based on the SPICE protocol, which eliminates the need for network communication between the client and the virtual machine, reduces latency and network bandwidth of cloud desktop cameras, and improves user experience.

[0035] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for optimizing a USB camera based on the SPICE protocol, characterized in that, Includes the following steps: S1. Start QEMU on the host machine, configure the character device chardev and virtio device of type spicevmc, and establish communication between the host machine and the virtual machine; S2. In the SPICE client, when using spice-gtk for USB redirection, the bInterfaceClass value of the USB camera is checked, and the USB camera is actively filtered and ignored at the libusb call level. S3. Establish a dedicated USB video channel between the SPICE client and the SPICE server for video data transmission; S4. Access the physical USB camera on the terminal through the SPICE client and collect video data from the connected camera. S5. The camera video data is encoded using the GStreamer plugin with high efficiency and sent asynchronously via a dedicated USB video channel. The S6 SPICE server receives H.264 encoded data from the dedicated channel and writes it to the virtual machine's virtio device via chardev. S7. Deploy the agent service in the virtual machine, create a virtual camera device with the help of v4l2loopback, read H.264 encoded data from the virtio port, decode it and write it directly to the created virtual camera device; S8. The application within the virtual machine selects the virtual camera device as the video input source and acquires virtual camera data.

2. The method according to claim 1, characterized in that, The virtual machine mentioned in step S1 is a Linux virtual machine.

3. The method according to claim 1, characterized in that, The character device added in step S1 is used to communicate with the virtual machine on one end and to receive video streams from the USB video channel on the other end.

4. The method according to claim 1, characterized in that, Step S2 specifically includes the following: When the client uses spice-gtk, it uses libusb to enumerate USB devices. By intercepting libusb's device scan, it checks the device's bInterfaceClass field. If it is 0x0E, it actively filters out the USB camera from the standard USB redirection link.

5. The method according to claim 1, characterized in that, Step S3 specifically includes the following: Add a new channel type to the SPICE protocol; The SPICE client initiates a TCP connection to the virtual machine with a channel type. After receiving a connection request, the SPICE server determines whether it is a USB video channel that needs to establish a connection based on the channel type, and returns a message indicating that the channel connection has been established.

6. The method according to claim 1, characterized in that, Step S4 specifically includes the following: Open the USB device using the v4l2src GStreamer plugin; Set the acquisition format to obtain the raw video data stream; the acquisition format is video / x-raw,format=YUY2.

7. The method according to claim 1, characterized in that, Step S5 specifically includes the following: Build a GStreamer data processing pipeline to efficiently compress the raw camera video data; use the videoconvert plugin to ensure that the acquisition format matches the encoder; An x264enc encoder is used, and parameters are configured to minimize encoding latency; these parameters include tune=zerolatency. The generated H.264 encoded data is asynchronously sent to the host server via a specially established USB video channel.

8. The method according to claim 1, characterized in that, Step S7 specifically includes the following: A virtual machine deploys an agent service called camera-agent, which creates a virtual camera device through v4l2loopback. It uses the epoll mechanism to check if the virtio-serial port is readable, reducing the CPU consumption of the agent service when it is idle, and reading H.264 encoded data when there is data. Construct a GStreamer decoding pipeline to push the H.264 encoded data read by camera-agent from the virtio port to appsrc, use avdec_h264 to decode it, and then use v4l2sink to output it to the virtual camera device.

9. The method according to claim 8, characterized in that, The virtual camera device is / dev / video0.

Images