Video data transmission method of UVC device and UVC device

Abstract

The invention provides a video data transmission method of UVC equipment and the UVC equipment. The video data transmission method comprises the steps that S1, an image sensor converts an image collected by a lens into video data; s2, processing the video data in an image processing module, storing the processed video data in a memory, recording a storage address and a data length of the processed video data, and adding a UVC protocol header in front of the storage address and the data length; and S3, a data packet containing the UVC protocol head, the processed video data storage address and the data length is sent to USB hardware, and the USB hardware sends the stored video data with the UVC protocol head to an external terminal through a USB protocol. According to the UVC equipment, repeated carrying of complete video data is avoided, system resources can be effectively saved, and the data processing and transmission speed is increased.

Description

Technical Field

[0001] The present invention belongs to the field of video processing, and particularly relates to a method for transmitting video data of a UVC device and a UVC device. Background Art

[0002] In recent years, the miniaturization and portability of electronic devices have become the key development directions of major manufacturers, and UVC devices (abbreviation of USB Video Camera, i.e., an external camera that transmits video using a USB interface) have also been widely used.

[0003] The image processing process of a UVC device can be divided into three links: image acquisition, image processing, and data transmission. The image acquisition link refers to the process of converging the light emitted by the external scene by the lens, decomposing the external image into pixels by the image sensor and converting it into an electrical signal, and then converting it into a digital signal through an analog-to-digital converter; the image processing link is to process the digital signal transmitted by the image sensor by the ISP (abbreviation of Image Signal Processor, i.e., the image signal processing unit) and JPEG (abbreviation of Joint Photographic Experts Group, i.e., the image coding unit), encoding the digital signal into a specified format, improving the image quality and video compression; the data transmission link is to transmit the encoded video data generated in the image processing link to the external terminal through the USB. After receiving the video data through the USB interface, the external terminal decodes, displays, and stores the video data.

[0004] Meanwhile, UVC (abbreviation of USB Video Class, i.e., USB Video Class) is a protocol standard defined for USB video capture devices. It is a protocol standard jointly launched by Microsoft and several other device manufacturers for USB video capture devices and has become one of the USB org standards. Today's mainstream operating systems (such as Windows XP SP2 and later, Linux 2.4.6 and later, MacOS 10.5 and later) already provide UVC device drivers. Therefore, hardware devices compliant with the UVC specification can be used normally in terminal devices without installing any driver programs. Devices using UVC technology include cameras, digital cameras, analog video converters, TV sticks, and still image cameras, etc. The latest UVC version is UVC 1.5, which is defined by the USB Implementers Forum to include the basic protocol and payload format. Using UVC technology makes the operation of hardware between various programs smoother and more convenient, and also omits the link of driver program installation. For example, a USB camera is a camera using a USB interface, which is plug-and-play, with a foolproof operation, no need for a capture card, no need for power supply, no need to disassemble the chassis, and supports laptop computers. It has a lower cost, is convenient and practical, and has a simple operation compared with traditional surveillance cameras.

[0005] In UVC devices such as USB cameras, to achieve the processing and transmission of image data, it is necessary to first transfer the video data from the image signal processing unit or the image encoding unit to the cache of the application responsible for image data transmission, and then transfer it from the cache of the application to the USB endpoint cache. A total of two transfers are required, and then the USB hardware will transmit the image data to the external terminal.

[0006] In view of this, the existing UVC devices have the following problems:

[0007] 1. In order for the application running in the embedded system of the UVC device to process the image data, it is necessary to first transfer the image data to the application cache, and then, in order to send the image data in multiple data packets, it is necessary to transfer the image data from the application cache to the USB endpoint cache. These two transfers require a large amount of memory;

[0008] 2. The multiple transfers of image data occupy the CPU performance and RAM bandwidth, and at the same time, the power consumption is also higher;

[0009] 3. The multiple transfers of image data result in a longer data transmission delay.

[0010] In view of this, there is an urgent need in the current market for a UVC device that can solve the above problems such as large resource occupation, high power consumption, and long data transmission delay, so as to meet the market requirements. Summary of the Invention

[0011] In order to solve the above problems, the object of the present application is to provide a method for transmitting video data of a UVC device and a UVC device, which can effectively avoid technical problems such as large resource consumption and long delay caused by multiple transfers of video data through the optimization of the video data transmission method.

[0012] Specifically, the technical solution of the present invention provides a method for transmitting video data of a UVC device, and the method includes:

[0013] S1, an image sensor converts the image captured by the lens into video data;

[0014] S2, the video data is processed in an image processing module, and the processed video data is stored in a memory. The storage address and data length of the processed video data are recorded, and a UVC protocol header is added before the storage address and data length.

[0015] S3, a data packet containing the UVC protocol header, the storage address and data length of the processed video data is sent to a USB hardware, and the USB hardware sends the stored video data with the UVC protocol header to an external terminal through a USB protocol.

[0016] According to a preferred embodiment, in step S3, the USB hardware sends the stored video data with the UVC protocol header to an external terminal through a USB protocol, which further includes: the driver program of the UVC device obtains the video data according to the storage address and data length, splits a frame of video data according to the payload size, writes a UVC protocol header at the front of the original video data, and tells the USB hardware the address of the split video data with the UVC protocol header in sequence. The USB hardware sends it out according to the USB protocol until all of this frame of video data is transmitted to the external terminal, and then obtains the next frame of video data and repeats the above steps.

[0017] According to a preferred embodiment, the size of the driver buffer is equal to or less than the payload size required by the UVC protocol.

[0018] According to a preferred embodiment, the image processing module includes an image signal processing unit. The image signal processing unit receives the video data transmitted from the image sensor and processes the video data to generate video source data; the processed video data is the video source data processed by the image signal processing unit.

[0019] According to a preferred embodiment, the image processing module includes an image signal processing unit and an image encoding unit. The image signal processing unit processes video data to generate video source data, and the image encoding unit encodes the video source data to generate video image data. The processed video data is the video image data generated after being processed by the image signal processing unit and the image encoding unit.

[0020] According to a preferred embodiment, the image encoding unit stores the output video image data in a random access memory, and an application running on the UVC device caches the storage address and data length of the video image data.

[0021] According to a preferred embodiment, when starting to encode a frame of video source data, the image encoding unit records the storage address, and when the encoding of the frame is completed, it records the data length of the video image data.

[0022] According to a preferred embodiment, the image encoding unit has at least two caches. While one cache is transmitting video image data through the USB hardware, the image encoding unit encodes the next frame of video source data and outputs it to another idle cache.

[0023] Specifically, the technical solution of the present invention provides a UVC device configured to implement the video data transmission method described in any one of the above.

[0024] Therefore, the remarkable effect of this application is that the video data transmission of the present invention avoids the multiple transfers of complete video data in the prior art. Instead, by only sending the storage address and data length to the USB hardware, the data transmission within the UVC device is achieved, reducing the delay of video data transmission and also avoiding the writing of complete video data into the cache, effectively reducing resource occupation and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0026] Figure 1 It is a schematic diagram of the video data transmission method of the UVC device of this application.

[0027] Figure 2 It is a schematic diagram of the structure of the UVC device of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings.

[0029] Please refer to Figure 1Schematic diagram of the video data transmission method of the UVC device shown. On the video path of the UVC device of the present invention, the following parts are further included:

[0030] S1. The image sensor converts the image captured by the lens into video data.

[0031] S2. The said video data is processed in the image processing module, and the processed video data is stored in the memory. The storage address and data length of the processed video data are recorded, and a UVC protocol header is added to the storage address and data length. Further, S2 at least includes:

[0032] S2.1. The image signal processing unit receives the video data transmitted by the image sensor through MIPI (abbreviation of Mobile Industry Processor Interface, i.e., Mobile Industry Processor Interface), and processes the video data such as image effect optimization to generate video source data;

[0033] S2.2. The image coding unit encodes the video source data output by the image signal processing unit into mjpeg - format video data and stores it in the Random Access Memory (RAM). The application program caches the storage address and data length of the video data.

[0034] S2.3. After receiving the instruction to transmit the video data, the application program adds a UVC protocol header to the front of the video data by the driver.

[0035] S3. The data packet containing the UVC protocol header, the storage address and data length of the processed video data is sent to the USB hardware, and the USB hardware sends the stored video data with the UVC protocol header to the external terminal through the USB protocol.

[0036] In this application, it can be understood that the UVC device may not perform image encoding unit and mjpeg encoding. After the image signal processing unit finishes processing, the video source file is output to the Random Access Memory. Then the application program stores the storage address and data length of the video source file in the cache, adds a UVC protocol header to the storage address and data length in the cache and packs them. The packed data is sent to the USB hardware. The USB hardware retrieves the video source data according to the storage address and data length and transmits it to the external terminal.

[0037] In S3, during the mjpeg format encoding process, the video source data is stored in the memory of the UVC device. The image encoding unit will save the memory address and data length for each encoding. Preferably, the storage address of the encoded video image data is saved at the start of each mjpeg encoding, and the data length is saved when the mjpeg encoding is completed. Each mjpeg encoding is performed on one frame of the image.

[0038] The encoded mjpeg video image data and the video source data are stored frame by frame respectively. The storage address and data length are also the information after encoding one frame of the image. After the image data or video source data of this frame of the image is sent to the external terminal through the USB hardware, the storage address and data length stored in the buffer will be released, and the storage address and data length of the new frame of the image will be stored. The above storage address and data length are not only saved in the buffer of the image encoding unit, but also saved by the application program after it obtains the storage address and data length.

[0039] In S4, after receiving the instruction to transmit the video image data, the driver first obtains the storage address and data length, then obtains the video source data or image data according to the storage address and data length, and finally splits and packs the video source data or image data. The sizes of the multiple data packets generated after splitting and packing do not exceed the payload size required by the UVC protocol, and each data packet contains a UVC protocol header.

[0040] The application program sets a specified buffer area in the buffer, and the application program stores the storage address and data length of the video source data or image data in this specified buffer area. The driver adds a UVC protocol header to the specified buffer area, and then controls the USB hardware to obtain the video source data or image data according to the USB protocol, based on the specified storage address and data length, and transmits the video source data or image data to the external terminal through the USB interface. After transmitting one frame of video data or image data, the storage address and data length of the new frame of video source data or image data will be stored in the specified buffer area frame by frame in a covering manner.

[0041] The driver is set with a driver buffer, and the size of the driver buffer is the same as the size of the payload data required by the UVC protocol, generally between 1 Kbyte and 3 Kbytes. After the driver obtains the storage address and data length of the video source data or image data, it fills in the UVC protocol header at the front of the driver buffer, and writes the video source data or image data into the remaining space after the UVC protocol header. For the data packets after the first payload packet, the UVC protocol header is directly written in front of the address of the original video data, and there is no transfer of image data. Generally, the size of the video source data or image data is larger than the size of the driver buffer. Therefore, when sending a frame of video source data or image data, the video source data or image data will be split, and the split video source data or image data will be placed after the UVC protocol header in the driver buffer. After the driver buffer is full or a frame of data is completely written, the data in the driver buffer is sent to the external terminal through the USB hardware. After the data transmission is completed, the driver puts the remaining part of the video source data or image data that has not been sent into the driver buffer after the UVC protocol header until a frame of video source data or image data is completely sent, and then the next frame of video source data or image data is sequentially placed after the UVC protocol header in the driver buffer and sent.

[0042] The JPEG encoder of the image coding unit has at least two caches. While one cache is transmitting image data through the USB hardware, the JPEG encoder encodes the next frame of video source data and outputs it to another idle cache.

[0043] After a frame of image data is transmitted through the USB hardware, the application program notifies the image coding unit that the transmission of this frame of image data is completed, starts transmitting the next frame of image data through the USB hardware, and stores the image data of the next frame on the cache where the image data that has completed USB transmission is located.

[0044] Please refer to Figure 2 The structural schematic diagram of the UVC device shown. As can be seen from the figure, after the UVC device of the present invention obtains video data from the image sensor (which can be through the MIPI interface), it is processed by the image signal processing unit, and then encoded by the image coding unit and stored in the random access memory. The application program of the UVC device stores the storage address and data length in the cache, adds the UVC protocol header and provides it to the USB hardware. Then, the USB hardware splits the image data encoded by the image coding unit and sequentially adds the UVC protocol header to the split data and sends it to the external terminal for storage or playback by the external terminal. It can be understood that the UVC device of the present invention can also not perform image coding unit processing, directly store the video source data output by the image signal processing unit in the random access memory, record the storage address and data length, and then send it to the external terminal in the same way, and the external terminal performs image coding processing on the video source data.

[0045] In the present invention, in the whole process from the image sensor generating raw data, to the image signal processing unit generating video source data, then to the image encoding unit generating video data, and finally to the USB hardware sending, there is only one transfer of the payload size. By using the payload to achieve the direct pass-through of the video source data or video data, the CPU and RAM occupancy of the UVC device can be significantly reduced and the video latency can be decreased.

[0046] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A video data transmission method for a UVC device, characterized in that: The following steps are included in the UVC device: S1, the image sensor converts the image captured by the lens into video data; S2, the video data is processed in the image processing module, and the processed video data is stored in the memory, the storage address and data length of the processed video data are recorded, and a UVC protocol header is added before the storage address and data length; S3, sending a data packet containing a UVC protocol header, a storage address of processed video data and a data length to the USB hardware, and the USB hardware sends the stored video data with the UVC protocol header to an external terminal through the USB protocol.

2. The video data transmission method of a UVC device according to claim 1, characterized in that: The USB hardware in step S3 sends the stored video data with the UVC protocol header to the external terminal via the USB protocol, further comprising: The driver of the UVC device obtains video data according to the storage address and the data length, splits a frame of video data according to the payload size, writes a UVC protocol header in the front of the video data, and tells the USB hardware the address of the split video data with the UVC protocol header in turn, which is sent out by the USB hardware according to the USB protocol until the frame of video data is completely transmitted to the external terminal, and then obtains the next frame of video data and repeats the above steps.

3. The video data transmission method of a UVC device according to claim 1, characterized in that: The driver buffer size is equal to or smaller than the load size required by the UVC protocol.

4. The video data transmission method of a UVC device according to claim 1, characterized in that: The image processing module includes an image signal processing unit, which receives video data transmitted from the image sensor and processes the video data to generate video source data; the processed video data is the video source data processed by the image signal processing unit.

5. The video data transmission method of a UVC device according to claim 1, characterized in that: The image processing module includes an image signal processing unit and an image encoding unit. The image signal processing unit processes video data to generate video source data, and the image encoding unit encodes the video source data to generate image data. The processed video data is the image data generated after being processed by the image signal processing unit and the image encoding unit.

6. The video data transmission method of a UVC device according to claim 5, characterized in that: The image encoding unit stores the output image data in a random access memory, and the application program running on the UVC device caches the storage address and data length of the image data.

7. The video data transmission method of a UVC device according to claim 5, characterized in that: The image encoding unit records the storage address when starting to encode a frame of video source data, and records the data length of the image data when the encoding of the frame is completed.

8. The video data transmission method of a UVC device according to claim 5, characterized in that: The image encoding unit has at least two buffers. While one of the buffers is transmitting image data through USB hardware, the image encoding unit encodes the next frame of video source data and outputs it to another idle buffer.

9. A UVC device, characterized in that: The UVC device is configured to implement the video data transmission method described in any one of claims 1 to 8.

Images