Video direct connection method for USB camera

Abstract

The invention provides a video direct connection method for a USB (Universal Serial Bus) camera, which comprises the following steps: S1, adding a synchronization mechanism between an ISP (Internet Service Provider) and a JPEG (Joint Photographic Experts Group) in advance, and adding a UDMA hardware module; the JPEG supports dynamic coding control; s2, outputting image data by the Sensor, and transmitting the image data to a video processing chip through an MIPI (Mobile Industry Processor Interface); s3, the ISP processes the image data, if the output is in the MJPEG format, the ISP outputs the image data in the YUY2 / NV12 format to the OCM and provides a synchronizing signal for the JPEG, the JPEG automatically reads image source data in the YUY2 / NV12 format from the OCM after receiving the synchronizing signal and encodes the image source data, and then the encoded image data in the MJPEG format is transmitted to the UDMA through a hardware through path; if the output is in the YUY2 format, the ISP directly outputs the image data in the YUY2 format to the UDMA through a hardware through path; s4, the UDMA packages the received image data according to the UVC protocol requirement, adds a UVC protocol header, then sends the packaged data to the OCM, and generates an interrupt signal; and S5, the software responds to the UDMA interrupt signal, and controls the USB to carry the image data packaged according to the UVC protocol from the OCM to the USB FIFO and send the image data to the host.

Description

Technical Field

[0001] The invention belongs to the technical field of video processing chips, and in particular relates to a video pass-through method for a USB camera. Background Art

[0002] First of all, in the prior art, the high-definition video image processing process can be divided into image acquisition, image processing at the sending end, signal transmission, image processing at the receiving end, image display and other links. Each link requires the support of a video chip with specific functions to be realized. The image acquisition link refers to the light emitted by the external scene gathered by the lens, the external image is decomposed into pixels and converted into electrical signals through the sensor, and converted into digital signals through the analog-to-digital converter. The image processing link at the sending end refers to the preliminary processing of the digital signal transmitted by the sensor by the image processing chip and the video processing chip, and image processing such as format processing, image quality improvement and video compression encoding. The signal transmission link is to transmit the video signal through a specific transmission medium. The image processing link at the receiving end refers to the decoding and playback of the video signal by various functional chips after the display terminal receives it. The video path mainly refers to the process from image acquisition to image processing at the sending end and then to signal transmission.

[0003] Secondly, USB cameras are cameras with USB interfaces, plug and play, fool-style operation, no acquisition card, no power supply, no need to disassemble the chassis, and support laptops. Compared with traditional surveillance cameras, they are cheaper, more convenient and practical, and easier to operate.

[0004] UVC stands for USB Video Class, which is a protocol standard defined for USB video capture devices. It is a protocol standard defined for USB video capture devices jointly launched by Microsoft and several other device manufacturers and has become one of the USB org standards.

[0005] Today's mainstream operating systems (such as Windows XP SP2 and later, Linux 2.4.6 and later, MacOS10.5 and later) have provided UVC device drivers, so hardware devices that meet UVC specifications can be used normally in the host without installing any drivers. Devices that use UVC technology include cameras, digital cameras, analog image converters, TV sticks, and still image cameras.

[0006] The latest UVC version is UVC 1.5, which is defined by the USB Implementers Forum and includes basic protocols and payload formats.

[0007] Webcams are the first and most common UVC devices. All operating systems after Windows XP SP2 can support UVC, not to mention Vista. Linux kernels after 2.4 support a large number of device drivers and can support UVC devices.

[0008] The benefits of using UVC: After USB becomes a standard in the video field, the hardware will run more smoothly between various programs, and the driver installation step is omitted.

[0009] In the existing USB camera, each module on the video data path completes the work independently, and the image data is transferred to each module for processing in the memory in frames, and the processing related to the UVC protocol is completed and transmitted in the memory.

[0010] However, there are the following problems in the prior art:

[0011] (1) The image data is cached and processed as a whole frame, which requires a large amount of memory space;

[0012] (2) The UVC protocol header needs to be processed by software, which consumes CPU and memory resources;

[0013] (3) The image data processing process involves a lot of memory transfer and software processing, and the delay from video to the host is large.

[0014] In addition, common terms in the prior art include:

[0015] (1) Sensor: Sensor, specifically refers to image sensor in this patent;

[0016] (2) MIPI: Mobile Industry Processor Interface;

[0017] (3) ISP: Image Signal Processor, image signal processing unit;

[0018] (4) JPEG: Joint Photographic Experts Group, image coding unit;

[0019] (5)UDMA: UVC Direct Memory Access, UVC data pass-through unit;

[0020] (6) USB: Universal Serial Bus;

[0021] (7) OCM: On Chip Memory, on-chip memory;

[0022] (8) UVC: USB Video Camera, a camera that uses a USB interface to transmit video;

[0023] (9)TIZIANO: the specific name of the chip ISP of Beijing Ingenic Integrated Circuit Co., Ltd. (hereinafter referred to as Ingenic), which is the same concept as ISP;

[0024] (10) YUY2: An image data format where one pixel occupies 2 bytes of space;

[0025] (11)CPU:Center Processing Unit, central processing unit;

[0026] (12)FIFO: First Input First Output, first-in, first-out cache. Summary of the invention

[0027] In order to solve the above problems, the purpose of this application is to enable:

[0028] (1) Save memory usage;

[0029] (2) Save CPU and memory resources;

[0030] (3) Reduce the delay in video transmission to the host.

[0031] Specifically, the present invention provides a method for video pass-through of a USB camera, the method comprising the following steps:

[0032] S1, add a synchronization mechanism between ISP and JPEG in advance, the synchronization mechanism is implemented by hardware, ISP / JPEG is synchronized directly through hardware electrical signals, and a UDMA hardware module is added to facilitate UVC protocol processing and automatically output the processed data to OCM; JPEG supports dynamic encoding control, that is, support for this function is added to JPEG, and the encoding quantization parameters can be controlled according to the complexity of the current environment to ensure that the size of the encoded image is controlled within a stable range; the dynamic encoding control function is completed by the cooperation of software and hardware. First, the hardware supports dynamic adjustment of the quantization parameters, but the magnitude of the specific next frame quantization parameters needs to be configured to the hardware by the software program; that is, the software program will compare the size of the current frame after encoding with the target size, calculate the quantization parameter level to be used for the next frame encoding based on the comparison result, and configure the value to the JPEG hardware module to achieve the purpose of controlling the size of the encoded image and prevent data overflow or consumption of additional cache;

[0033] S2, the sensor outputs image data and transmits it to the video processing chip through the MIPI interface; S3, the ISP in the video processing chip processes the image data, and the data format entering the ISP is RAW format / YUY2 format. The ISP output is YUY2 / NV12 format. The image format after encoding is MJPEG format or YUY2 format for processing. The MJPEG format needs to rely on the output of the JPEG encoding module:

[0034] If the data format output to the PC is MJPEG format, it needs to be encoded:

[0035] Output the processed image data to OCM and provide synchronization signals to JPEG every 16 lines. JPEG requires at least 16 lines of data for one encoding. After receiving the ISP synchronization signal, JPEG automatically reads the image source data in YUY2 / NV12 format from OCM and encodes it. Then, the encoded image data is transmitted to UDMA through the hardware pass-through path.

[0036] If the data format output to the PC is YUY2 format, the YUY2 format data is directly sent to UDMA through the direct data path:

[0037] Output image data in YUY2 format to UDMA through hardware pass-through;

[0038] S4, UDMA unpacks the received image data into the length required by the UVC protocol, adds a UVC protocol header to the data, then sends the packaged image data to OCM and generates an interrupt signal;

[0039] S5, the UDMA driver responds to the UDMA interrupt signal, controls the USB to move the image data packaged according to the UVC protocol from the OCM to the USB FIFO and sends it to the host.

[0040] In step S1, the added synchronization mechanism is to set the ISP to output 16 lines of image data to the on-chip memory OCM each time to meet the basic needs of the JPEG MCU encoding unit, and then send a signal to JPEG. After receiving the signal, JPEG reads the image data from OCM for encoding processing, and sends a feedback signal to the ISP after the processing is completed, prompting the ISP to continue writing subsequent image data in the OCM address.

[0041] In step S1, the UDMA module receives the image data of the previous ISP / JPEG through a data direct path without passing through the memory, that is, there is a direct data path between UDMA and JPEG / ISP, and the image data can be sent directly from JPEG / ISP to UDMA without passing through OCM. After the image data enters UDMA, UDMA will automatically split the image data and add the UVC protocol header, and send the repackaged data to OCM.

[0042] In step S1, the size of each frame of the image is required to be no larger than 800KB / frame. The goal is to control the size after encoding. The root cause of the requirement is that the upper limit of the synchronous transmission bandwidth of USB 2.0 is 24MB / s. If the JPEG image is too large, it cannot be sent to the PC through the USB bus within 1 frame. At this time, the data of the next frame will be flushed out after the data of the previous frame is received. In this case, only an additional memory cache can be opened to avoid it. After supporting dynamic encoding, the size of the JPEG encoded image can be controlled at a reasonable target, which may be 400KB / frame or 200KB / frame. The determination of the specific target size depends on the actual needs including comprehensive considerations of resolution / frame rate / encoding quality. Dynamic encoding control will compare the size of the current frame with the target size to determine whether the encoding complexity of the next frame needs to be increased or vice versa.

[0043] In the step S3, the image data can be cached by line rather than by frame. This is achieved through hardware design. The original hardware design is to cache one frame and then notify the outside, while this method is to notify the outside after caching 16 lines. There is no need for additional UVC protocol processing and CPU data transfer. The entire process is implemented through hardware pass-through and UDMA. The UDMA is a hardware module in the video path, which is responsible for reorganizing the data according to the requirements of the UVC protocol and adding a protocol header.

[0044] Therefore, the advantages of this application are: this method improves data transmission efficiency, reduces video delay, saves memory usage, and saves CPU and memory resources. Specifically, it includes:

[0045] (1) Reduce the memory space occupied by image cache;

[0046] Existing technology: A single frame of a 1920*1080 YUY2 image format requires about 4MB of memory space. Considering that the data path pipeline needs to cache at least two frames, at least about 8MB of space is actually required;

[0047] This application: The YUY2 image format of 1920*1080 only needs to cache 32 lines of data, which requires 1920*2*32 approximately 0.12MB of memory space.

[0048] (2) Automated UVC protocol processing avoids secondary transport and software processing of image data, improves efficiency, and saves resources;

[0049] Prior art: The software divides the image data of the whole frame into the length required by the UVC protocol (*~3072Bytes), and then adds the UVC protocol header before each packet of UVC data. This application: Through the UDMA module, it automatically receives the image data transmitted by ISP / JPEG, and packages the data and adds the UVC protocol header, without consuming additional memory and CPU resources.

[0050] (3) Improve the efficiency of data processing and transmission and reduce the delay of video data to the host;

[0051] Existing technology: Image data flows in frames, and the modules of the video channel are connected mainly by software processing, and the software needs to add a UVC protocol header. The measured delay is about 200ms;

[0052] This application: Image data flows in lines. Theoretically, the USB can transfer data to the host in a relatively short time after the sensor outputs the complete frame data. And the whole process is mainly completed by hardware, reducing the delay caused by software processing. The measured video delay is about 140ms. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0054] Figure 1 This application involves a schematic diagram of the design principle.

[0055] Figure 2 It is a flow chart of the present application method. DETAILED DESCRIPTION

[0056] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0057] The embodiment of the present application is a method for video pass-through of a USB camera, such as Figure 1 As shown in the figure, the design principle, in the video path, from the image sensor to the host, mainly includes the following parts:

[0058] Sensor: collects video data and outputs it to the video processing chip;

[0059] Mobile Industry Processor Interface MIPI: receives data output by the sensor and converts it into digital signals and outputs them to the image signal processing unit ISP;

[0060] Image signal processing unit ISP: receives video data from MIPI and optimizes the image effect; (In the embodiment, Ingenic TIZIANO image processor is used);

[0061] Image encoding unit JPEG: encodes the video source data output by the ISP into MJPEG format; Universal Serial Bus USB: outputs the encoded video data to the host via USB;

[0062] UDMA is also specifically added in this application.

[0063] like Figure 2 As shown, the main steps implemented in this application are:

[0064] S1, add a synchronization mechanism between ISP and JPEG in advance, the synchronization mechanism is implemented by hardware, ISP / JPEG is synchronized directly through hardware electrical signals, and a UDMA hardware module is added to facilitate UVC protocol processing and automatically output the processed data to OCM; JPEG supports dynamic encoding control, that is, support for this function is added to JPEG, and the encoding quantization parameters can be controlled according to the complexity of the current environment to ensure that the size of the encoded image is controlled within a stable range; the dynamic encoding control function is completed by the cooperation of software and hardware. First, the hardware supports dynamic adjustment of the quantization parameters, but the specific magnitude of the next frame quantization parameters needs to be configured to the hardware by the software program; the software program will compare the size of the current frame after encoding with the target size, calculate the quantization parameter level to be used for the next frame encoding based on the comparison result, and configure the value to the JPEG hardware module to prevent data overflow or additional cache consumption;

[0065] The added synchronization mechanism is to set the ISP to output 16 lines of image data to the on-chip memory OCM every time to meet the basic needs of the JPEGMCU encoding unit, and then send a signal to JPEG. After receiving the signal, JPEG reads the image data from OCM for encoding processing, and after the processing is completed, it gives a feedback signal to ISP, prompting ISP to continue writing subsequent image data in the OCM address; the UDMA module receives the image data of the previous stage isp / jpeg through a data direct path, without passing through the memory, that is, there is a direct data path between UDMA and JPEG / ISP, and the image data can be directly sent from JPEG / ISP to UDMA without passing through OCM. After the image data enters UDMA, UDMA will automatically split the image data and add the UVC protocol header, and send the repackaged data to OCM;

[0066] In the step S1, the size of each frame of the picture is required to be no larger than 800KB / frame, and the goal is to control the size after encoding. The root cause of the requirement is that the upper limit of the synchronous transmission bandwidth of USB 2.0 is 24MB / s. If the JPEG picture is too large, it cannot be sent to the PC through the USB bus within 1 frame. At this time, the data of the next frame will flush the data that has not been output in the previous frame. In this case, only an additional memory cache can be opened to avoid it. After supporting dynamic encoding, the size of the JPEG encoded picture can be controlled at a reasonable target, which may be 400KB / frame or 200KB / frame. The determination of the specific target size depends on the actual needs including the comprehensive consideration of resolution / frame rate / encoding quality. 800KB / frame is obtained based on the theoretical bandwidth upper limit of USB 2.0 synchronous transmission 24MB / s divided by 30 fps. It is an extreme case and is generally not used in practice. The dynamic encoding control compares the size of the current frame with the target size to determine whether the encoding complexity of the next frame needs to be increased or vice versa.

[0067] The quality design of JPEG dynamic encoding: after inputting the expected target image size, the JPEG hardware will automatically adjust the encoding quantization parameters according to the complexity of the scene to make it close to the expected target size. This design is only effective in the MJPEG encoding process, and other formats or other links do not need to be paid attention to;

[0068] S2, the sensor outputs image data and transmits it to the video processing chip through the MIPI interface; S3, the ISP in the video processing chip processes the image data, and the data format entering the ISP is RAW format (main format) / YUY2 format (format in rare cases), and the ISP output is YUY2 / NV12 format. The image format after encoding is MJPEG format or YUY2 format for processing. The MJPEG format needs to rely on the output of the JPEG encoding module:

[0069] If the data format output to the PC is MJPEG format, it needs to be encoded:

[0070] Output the processed image data to OCM and provide synchronization signals to JPEG every 16 lines. JPEG requires at least 16 lines of data for one encoding. After receiving the ISP synchronization signal, JPEG automatically reads the image source data in YUY2 / NV12 format from OCM and encodes it. Then, the encoded image data is transmitted to UDMA through the hardware pass-through path.

[0071] If the data format output to the PC is YUY2 format, the YUY2 format data is directly sent to UDMA through the direct data path:

[0072] Output image data in YUY2 format to UDMA through hardware pass-through;

[0073] In step S3, the image data can be cached by line instead of by frame, which is achieved through hardware design. The original hardware design is to cache one frame and then notify the outside, while this method is to notify the outside after caching 16 lines; there is no need for additional UVC protocol processing and CPU to carry out data transfer, and the entire process is implemented through hardware pass-through and UDMA. The UDMA is a hardware module in the video path, which is responsible for reorganizing the data according to the requirements of the UVC protocol and adding the protocol header, replacing the original software task, so UDMA can only be regarded as a key design in hardware pass-through, but cannot represent the hardware pass-through itself;

[0074] S4, UDMA unpacks the received image data into the length required by the UVC protocol, adds a UVC protocol header to the data, then sends the packaged image data to OCM and generates an interrupt signal;

[0075] S5, the UDMA driver responds to the UDMA interrupt signal, controls the USB to move the image data packaged according to the UVC protocol from the OCM to the USB FIFO and sends it to the host.

[0076] Through the method of the present application, data can be cached by row rather than by frame, and no additional UVC protocol processing and CPU are required for data movement. The entire process is implemented through hardware pass-through and UDMA, which can significantly improve efficiency and reduce memory consumption.

[0077] Specifically, if it is MJPEG format data processing flow:

[0078] 1. The sensor outputs image data and transmits it to the video processing chip through the MIPI interface;

[0079] 2. ISP processes the image data, outputs the processed image data to OCM, and provides synchronization signals to JPEG;

[0080] 3. After receiving the ISP synchronization signal, JPEG automatically reads the image source data in YUY2 format from OCM and encodes it, and then transmits the encoded image data to UDMA through a direct path;

[0081] 4. UDMA unpacks the received image data into the length required by the UVC protocol, adds a UVC protocol header to the data, then sends the packaged image data to OCM and generates an interrupt signal;

[0082] 5. The software responds to the UDMA interrupt signal, controls the USB to move the image data packaged according to the UVC protocol from the OCM to the USB FIFO and sends it to the host.

[0083] Specifically, if the data is in YUY2 format, the processing flow is as follows:

[0084] 1. The sensor outputs image data and transmits it to the video processing chip through the MIPI interface;

[0085] 2. ISP processes the image data and outputs the image data in yuy2 format to UDMA through a direct path;

[0086] 3. UDMA unpacks the received image data into the length required by the UVC protocol, adds a UVC protocol header to the data, then sends the packaged image data to OCM and generates an interrupt signal;

[0087] 4. The software responds to the UDMA interrupt signal, controls the USB to move the image data packaged according to the UVC protocol from the OCM to the USB FIFO and sends it to the host.

[0088] In summary, this method mainly involves:

[0089] (1) Design of UDMA module:

[0090] a. Receive image data from the previous ISP / JPEG through the data pass-through path without passing through the memory;

[0091] b. Automatically package the image data according to the UVC protocol requirements and add the UVC protocol header.

[0092] (2)JPEG dynamic coding quality design:

[0093] After entering the expected target image size, the JPEG hardware automatically adjusts the encoding quantization parameters based on the complexity of the scene to make it close to the expected target size.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for video pass-through of a USB camera, characterized in that: The method comprises the following steps: S1, add a synchronization mechanism between ISP and JPEG in advance, the synchronization mechanism is implemented by hardware, ISP / JPEG is synchronized directly through hardware electrical signals, and a UDMA hardware module is added to facilitate UVC protocol processing and automatically output the processed data to OCM; JPEG supports dynamic encoding control, that is, support for this function is added to JPEG, and the encoding quantization parameters can be controlled according to the complexity of the current environment to ensure that the size of the encoded image is controlled within a stable range; the dynamic encoding control function is completed by the cooperation of software and hardware. First, the hardware supports dynamic adjustment of the quantization parameters, but the magnitude of the specific next frame quantization parameters needs to be configured to the hardware by the software program; that is, the software program will compare the size of the current frame after encoding with the target size, calculate the quantization parameter level to be used for the next frame encoding based on the comparison result, and configure the value to the JPEG hardware module to achieve the purpose of controlling the size of the encoded image and prevent data overflow or consumption of additional cache; S2, the sensor outputs image data and transmits it to the video processing chip through the MIPI interface; S3, the ISP in the video processing chip processes the image data. The data entering the ISP is in RAW format or YUY2 format. The ISP output is in YUY2 / NV12 format. The image is processed according to whether the output image format of the encoded data is MJPEG format or YUY2 format. The MJPEG format needs to rely on the output of the JPEG encoding module: If the data format output to the PC is MJPEG format, it needs to be encoded: Output the processed image data to OCM and provide synchronization signals to JPEG every 16 lines. JPEG requires at least 16 lines of data for one encoding. After receiving the ISP synchronization signal, JPEG automatically reads the image source data in YUY2 / NV12 format from OCM and encodes it. Then, the encoded image data is transmitted to UDMA through the hardware pass-through path. If the data format output to the PC is YUY2 format, the YUY2 format data is directly sent to UDMA through the direct data path: Output image data in YUY2 format to UDMA through hardware pass-through; S4, UDMA unpacks the received image data into the length required by the UVC protocol, adds a UVC protocol header to the data, then sends the packaged image data to OCM and generates an interrupt signal; S5, the UDMA driver responds to the UDMA interrupt signal, controls the USB to move the image data packaged according to the UVC protocol from the OCM to the USB FIFO and sends it to the host.

2. A method for video pass-through of a USB camera according to claim 1, characterized in that: In step S1, the added synchronization mechanism is to set the ISP to output 16 lines of image data to the on-chip memory OCM each time to meet the basic needs of the JPEGMCU encoding unit, and then send a signal to JPEG. After receiving the signal, JPEG reads the image data from OCM for encoding processing.

3. A method for video pass-through of a USB camera according to claim 1, characterized in that: In step S1, the UDMA module receives the image data of the previous ISP / JPEG through a data direct path without passing through the memory, that is, there is a direct data path between UDMA and JPEG / ISP, and the image data can be sent directly from JPEG / ISP to UDMA without passing through OCM. After the image data enters UDMA, UDMA will automatically split the image data and add the UVC protocol header, and send the repackaged data to OCM.

4. A method for video pass-through of a USB camera according to claim 1, characterized in that: In step S1, the size of each frame of the image is required to be no larger than 800KB / frame. The purpose is to control the size after encoding. The reason for this requirement is that the upper limit of the synchronous transmission bandwidth of USB 2.0 is 24MB / s. If the MJPEG image is too large, it cannot be sent to the PC via the USB bus within 1 frame. At this time, the data of the next frame will be flushed out after it arrives. In this case, only an additional memory cache can be opened to avoid this. With support for dynamic encoding, the size of MJPEG encoded content can be controlled at a reasonable target, which may be 400KB / frame or 200KB / frame. The specific target size depends on actual needs, including comprehensive considerations of resolution / frame rate / encoding quality. Dynamic encoding control will compare the size of the current frame with the target size to determine whether the encoding complexity of the next frame needs to be increased or vice versa.

5. The method for video pass-through of a USB camera according to claim 1, characterized in that: In the step S3, the image data can be cached by line rather than by frame. This is achieved through hardware design. The original hardware design is to cache one frame and then notify the outside, while this method is to notify the outside after caching 16 lines. There is no need for additional UVC protocol processing and CPU data transfer. The entire process is implemented through hardware pass-through and UDMA. The UDMA is a hardware module in the video path, which is responsible for reorganizing the data according to the requirements of the UVC protocol and adding a protocol header.

Images