Processing circuit and method for RAW data based on MIPI CSI2 protocol

Abstract

The invention relates to a processing circuit and method for RAW data based on an MIPI CSI2 protocol. The processing circuit comprises a RAW type video effective pixel data extraction unit, a horizontal direction resolution extraction and shaping unit, an effective pixel data splicing and combining unit, a double-buffer control unit and an AXI bus write interface, RAW data are respectively accessed to the RAW type video effective pixel data extraction unit and the horizontal direction resolution extraction and shaping unit, and the RAW data are respectively accessed to the effective pixel data splicing and combining unit and the AXI bus write interface. The RAW type video effective pixel data extraction unit and the horizontal direction resolution extraction shaping unit are respectively connected to the effective pixel data splicing combination unit, and the effective pixel data splicing combination unit is sequentially connected to the double-buffer control unit and the AXI bus write interface. According to the method, the functions of RAW type video effective pixel data extraction, horizontal direction resolution extraction and shaping, effective pixel data splicing and combination, double-buffer control, AXI bus interface writing and the like are integrated, and the method has rapid pipeline processing performance.

Description

Technical Field

[0001] The present invention belongs to integrated circuit design technology, and in particular to a processing circuit based on MIPI CSI2 protocol RAW data. Background Art

[0002] Typically, the RAW type raw data information output from digital cameras, webcams, remote sensing images or multispectral camera sensors is written to storage by the host through software. This processing method has a slow transmission speed and cannot meet the high-speed transmission processing requirements of video image processing. Summary of the invention

[0003] In order to solve the above technical problems existing in the background technology, the present invention proposes a processing circuit and method based on MIPI CSI2 protocol RAW data, which integrates the functions of RAW type video effective pixel data extraction, horizontal resolution extraction and shaping, effective pixel data splicing combination, double buffer control and AXI bus write interface, and has fast pipeline processing performance. The original video image information RAW type data output by the camera sensor of the digital camera can be cleverly spliced ​​and combined, and quickly and pipelined in the original transmission format and completely stored in the storage controller module, thereby achieving an ideal video image processing effect.

[0004] The technical solution of the present invention is: the present invention is a processing circuit based on MIPI CSI2 protocol RAW data, and its special feature is: the processing circuit includes a RAW type video effective pixel data extraction unit, a horizontal resolution extraction and shaping unit, an effective pixel data splicing and combination unit, a double buffer control unit and an AXI bus write interface, the RAW data are respectively connected to the RAW type video effective pixel data extraction unit and the horizontal resolution extraction and shaping unit, the RAW type video effective pixel data extraction unit and the horizontal resolution extraction and shaping unit are respectively connected to the effective pixel data splicing and combination unit, and the effective pixel data splicing and combination unit is sequentially connected to the double buffer control unit and the AXI bus write interface.

[0005] Furthermore, the RAW type video effective pixel data extraction unit divides the MIPI CSI2 protocol RAW data into two categories for processing, and fills the high bit of the effective data width of RAW6 and RAW7 with 0 according to the 8-bit processing of the RAW8 data type; fills the high bit of the effective data width of RAW10, RAW12 and RAW14 with 0 according to the 16-bit processing of the RAW16 data type; according to the RAW data output by the MIPI CSI2 protocol front-end processing unit, one pixel clock cycle includes 3 RAW data, that is, P n = {RAW1, RAW2, RAW3}, swap the high and low bits of the three RAW data in each pixel clock cycle, that is, Pn ={RAW3,RAW2,RAW1}.

[0006] Furthermore, the horizontal resolution extraction and shaping unit sets a register counter to count the number of input video data in real time by line. The counter increases by 1 for each transmission of valid video data. The counter value at the end of a line of video data transmission is the horizontal resolution value of the RAW data. In order to facilitate the subsequent modules to splice and combine valid pixel data, the horizontal resolution of the extracted RAW6, RAW7 and RAW8 type video data is shaped to an integer multiple of 32, and the horizontal resolution of the extracted RAW10, RAW12, RAW14 and RAW16 type video data is shaped to an integer multiple of 16.

[0007] Furthermore, the effective pixel data splicing and combining unit is used for splicing and combining RAW_8bit effective pixel data and RAW_16bit effective pixel data; wherein:

[0008] RAW_8bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 31, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 11 levels of register delay; when the counter value is 10, {RAW 31 RAW 30 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 21, {RAW 63 RAW 62 ...RAW 33 RAW 32} to form the second group of 256-bit data; when the counter value is 31, {RAW 95 RAW 94 ...RAW 65 RAW 64} forms the third group of 256-bit data, which is exactly 32 clock pixel data P n The complete splicing combination is completed, and the subsequent clock pixel data is processed in the same way as described above;

[0009] RAW_16bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 15, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 5-level register delay; when the counter value is 5, {RAW 15 RAW14 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 10, {RAW 31 RAW 30 ...RAW 17 RAW 16} to form the second group of 256-bit data; when the counter value is 15, {RAW 47 RAW 46 ...RAW 33 RAW 32} forms the third group of 256-bit data, at this time, there are exactly 16 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above.

[0010] Furthermore, the double buffer control unit performs double buffering control on the video data in units of lines, and sets an asynchronous FIFO to control two DPRAMs to perform ping-pong operations, wherein one DPRAM performs a read operation while the other DPRAM performs a write operation.

[0011] Furthermore, the AXI bus write interface initiates a data read operation to the double buffer control unit in units of lines, and then outputs the RAW type video data to the storage control unit according to the timing requirements of the AXI bus write interface.

[0012] A method for implementing the above-mentioned MIPI CSI2 protocol RAW data processing circuit is special in that the method comprises the following steps:

[0013] 1) The RAW type video effective pixel data extraction unit divides the MIPI CSI2 protocol RAW data into two categories for processing. The effective data bit width of RAW6 and RAW7 is padded with 0 in the high bit and processed according to the 8-bit RAW8 data type; the effective data bit width of RAW10, RAW12 and RAW14 is padded with 0 in the high bit and processed according to the 16-bit RAW16 data type; according to the RAW data output by the MIPI CSI2 protocol front-end processing unit, one pixel clock cycle includes 3 RAW data, that is, P n = {RAW1, RAW2, RAW3}, swap the high and low bits of the three RAW data in each pixel clock cycle, that is, P n ={RAW3,RAW2,RAW1};

[0014] 2) The horizontal resolution extraction and shaping unit sets a register counter to count the number of input video data in real time according to the line. The counter increases by 1 for each transmission of valid video data. The counter value at the end of a line of video data transmission is the horizontal resolution value of the RAW data. In order to facilitate the subsequent modules to splice and combine valid pixel data, the horizontal resolution of the extracted RAW6, RAW7 and RAW8 type video data is shaped to an integer multiple of 32, and the horizontal resolution of the extracted RAW10, RAW12, RAW14 and RAW16 type video data is shaped to an integer multiple of 16.

[0015] 3) RAW_8bit effective pixel data splicing combination and RAW_16bit effective pixel data splicing combination;

[0016] The RAW_8bit effective pixel data splicing combination sets a register counter with an initial value of 0. Each clock pixel P is output. n The counter increases by 1, and when the counter counts to 31, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 11 levels of register delay; when the counter value is 10, {RAW 31 RAW 30 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 21, {RAW 63 RAW 62 ...RAW 33 RAW 32} to form the second group of 256-bit data; when the counter value is 31, {RAW 95 RAW 94 ...RAW 65 RAW 64} forms the third group of 256-bit data, which is exactly 32 clock pixel data P n The complete splicing combination is completed, and the subsequent clock pixel data is processed in the same way as described above;

[0017] RAW_16bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 15, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 5-level register delay; when the counter value is 5, {RAW 15 RAW 14 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 10, {RAW 31 RAW 30...RAW 17 RAW 16} to form the second group of 256-bit data; when the counter value is 15, {RAW 47 RAW 46 ...RAW 33 RAW 32} forms the third group of 256-bit data, at this time, there are exactly 16 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above;

[0018] 4) The double buffer control unit performs double buffering control on the video data in units of lines, and sets an asynchronous FIFO to control two DPRAMs to perform ping-pong operation, one DPRAM performs a read operation, and the other DPRAM performs a write operation at the same time;

[0019] 5) The AXI bus write interface initiates a data read operation to the double buffer control unit in units of lines, and then outputs the RAW type video data to the storage control unit according to the timing requirements of the AXI bus write interface.

[0020] The advantages of the present invention are: the present invention proposes a processing circuit and method based on MIPI CSI2 protocol RAW data, first extracts the effective pixel data of RAW type video, extracts and shapes the horizontal resolution at the same time, then splices and combines the effective pixel data, and finally uses a double buffer control mechanism to store the video image data in a storage control module through the write interface of the AXI bus. Its advantage is that when performing RAW type data video image processing, the use of hardware digital logic design circuit can completely and quickly store the RAW data in the storage control module according to the original data format. The design circuit can be logically integrated, the logic algorithm is simple, and has fast pipeline performance, thereby achieving an ideal video image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a circuit structure diagram of the present invention;

[0022] Figure 2 It is a RAW_8bit effective pixel data splicing combination image of the present invention;

[0023] Figure 3 This is a spliced ​​combination diagram of the RAW_16bit effective pixel data of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 .

[0025] See also Figure 1The present invention proposes a structure of a specific embodiment of a processing circuit based on MIPI CSI2 protocol RAW data, which integrates the functions of RAW type video effective pixel data extraction, horizontal resolution extraction and shaping, effective pixel data splicing combination, double buffer control and AXI bus write interface, and has fast pipeline processing performance, and is composed of the following modules:

[0026] RAW type video valid pixel data extraction unit: MIPI CSI2 protocol RAW data supports seven types, including RAW6, RAW7, RAW8, RAW10, RAW12, RAW14 and RAW16. RAW8 and RAW16 are commonly used, so they are divided into two categories for processing. The high bit of the effective data width of RAW6 and RAW7 is padded with 0 and processed according to the 8-bit data type of RAW8; the high bit of the effective data width of RAW10, RAW12 and RAW14 is padded with 0 and processed according to the 16-bit data type of RAW16. According to the RAW type data output by the MIPI CSI2 protocol front-end processing module, one pixel clock cycle includes 3 RAW data, that is, P n = {RAW1, RAW2, RAW3}, in order to facilitate the processing of the following modules, the three RAW data of each pixel clock cycle are swapped in high and low order, that is, P n ={RAW3,RAW2,RAW1}.

[0027] Horizontal resolution extraction and shaping unit: Video resolution information includes the number of valid pixels in the horizontal direction and the number of valid lines in the vertical direction. Video data is processed in units of lines. Since the RAW type data output by the front-end processing module includes 3 RAW data in one pixel clock cycle, the actual horizontal resolution length is the horizontal length of the standard resolution divided by 3. In order to simplify the complexity of hardware logic digital circuit processing, a register counter is set to count the number of input video data in real time according to the line. The counter increases by 1 for each transmission of valid video data. The counter value at the end of a line of video data transmission is the horizontal resolution value of the RAW data. In order to facilitate the subsequent modules to splice and combine valid pixel data, the horizontal resolution of the extracted RAW6, RAW7 and RAW8 type video data is shaped to an integer multiple of 32, and the horizontal resolution of the extracted RAW10, RAW12, RAW14 and RAW16 type video data is shaped to an integer multiple of 16.

[0028] See also Figure 2 , RAW_8bit effective pixel data splicing combination: The data bit width of each clock cycle of the AXI bus is 256 bits. For the RAW_8bit data type, the bit width is 8 bits, so 32 effective pixel data need to be spliced ​​into 256 bits, and the splicing order is RAW 31RAW 30 RAW 29...... RAW2RAW1RAW0. Since the RAW type data output by the MIPI CSI2 protocol front-end processing module includes three RAW data in one pixel clock cycle, namely P n ={RAW1,RAW2,RAW3}, so 32 clock pixel data P is required n That is, 96 RAW data are divided into 3 groups, each group has exactly 32 RAW data groups to form 256 bits. Set a register counter with an initial value of 0. Each time a clock pixel P is output n The counter increases by 1, and when the counter counts to 31, it is cleared to 0 and starts counting again, and so on. n Use 11 levels of register delay. When the counter value is 10, {RAW 31 RAW 30 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 21, {RAW 63 RAW 62 ...RAW 33 RAW 32} to form the second group of 256-bit data; when the counter value is 31, {RAW 95 RAW 94 ...RAW 65 RAW 64} forms the third group of 256-bit data, which is exactly 32 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above.

[0029] See also Figure 3 , RAW_16bit effective pixel data splicing combination: The data bit width of each clock cycle of the AXI bus is 256 bits. For the RAW_16bit data type, the bit width is 16 bits, so 16 effective pixel data need to be spliced ​​into 256 bits, and the splicing order is RAW 15 RAW 14 RAW 13...... RAW2RAW1RAW0. Since the RAW type data output by the MIPI CSI2 protocol front-end processing module includes three RAW data in one pixel clock cycle, namely P n ={RAW1,RAW2,RAW3}, so 16 clock pixel data P is required n That is, 48 ​​RAW data are divided into 3 groups, each group has exactly 16 RAW data groups to form 256 bits. Set a register counter with an initial value of 0. Each time a clock pixel P is output nThe counter increases by 1, and when the counter counts to 15, it is cleared to 0 and starts counting again, and so on. n Use 5-level register delay. When the counter value is 5, {RAW 15 RAW 14 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 10, {RAW 31 RAW 30 ...RAW 17 RAW 16} to form the second group of 256-bit data; when the counter value is 15, {RAW 47 RAW 46 ...RAW 33 RAW 32} forms the third group of 256-bit data, at this time, there are exactly 16 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above.

[0030] Double buffer control unit: Video data has fast pipeline transmission performance. In order to avoid overwriting errors or blockages when processing video data, and to perform cross-clock domain processing, the video data is double buffered in units of lines. An asynchronous FIFO is set to control two DPRAMs for ping-pong operation, one DPRAM performs read operation, and the other DPRAM performs write operation.

[0031] AXI bus write interface: RAW data processing ultimately requires the video data to be stored in the storage controller module through the AXI bus, so that subsequent modules can process the video image data. First, the AXI bus write interface initiates a read data operation to the double buffer control module in units of lines, and then outputs the RAW type video data to the storage control module according to the AXI bus write interface timing requirements.

[0032] The content of the present invention and the technical content not specifically described in the above embodiments are the same as the prior art.

[0033] The above are only specific embodiments disclosed in the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A processing circuit based on MIPI CSI2 protocol RAW data, characterized in that: The processing circuit includes a RAW type video effective pixel data extraction unit, a horizontal resolution extraction and shaping unit, an effective pixel data splicing and combining unit, a double buffer control unit and an AXI bus write interface. The RAW data are respectively connected to the RAW type video effective pixel data extraction unit and the horizontal resolution extraction and shaping unit. The RAW type video effective pixel data extraction unit and the horizontal resolution extraction and shaping unit are respectively connected to the effective pixel data splicing and combining unit. The effective pixel data splicing and combining unit is sequentially connected to the double buffer control unit and the AXI bus write interface.

2. The processing circuit based on MIPI CSI2 protocol RAW data according to claim 1, characterized in that: The RAW type video effective pixel data extraction unit processes the MIPI CSI2 protocol RAW data into two categories, and fills the high bit of the effective data width of RAW6 and RAW7 with 0 according to the 8-bit processing of the RAW8 data type; The effective data bit width of RAW10, RAW12 and RAW14 is padded with 0 in the high bit and processed as 16 bits of RAW16 data type; according to the RAW data output by the front-end processing unit of the MIPI CSI2 protocol, one pixel clock cycle includes 3 RAW data, that is, P n = {RAW1, RAW2, RAW3}, swap the high and low bits of the three RAW data in each pixel clock cycle, that is, P n ={RAW3,RAW2,RAW1}.

3. The processing circuit based on MIPI CSI2 protocol RAW data according to claim 2, characterized in that: The horizontal resolution extraction and shaping unit sets a register counter to count the number of input video data in real time by line. The counter increases by 1 for each transmission of valid video data. The counter value at the end of a line of video data transmission is the horizontal resolution value of the RAW data. In order to facilitate the subsequent modules to splice and combine effective pixel data, the horizontal resolution of the extracted RAW6, RAW7 and RAW8 type video data is shaped to an integer multiple of 32, and the horizontal resolution of the extracted RAW10, RAW12, RAW14 and RAW16 type video data is shaped to an integer multiple of 16.

4. The processing circuit based on MIPI CSI2 protocol RAW data according to claim 3, characterized in that: The effective pixel data splicing and combining unit is used for splicing and combining RAW_8bit effective pixel data and RAW_16bit effective pixel data; wherein: RAW_8bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 31, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 11 levels of register delay; when the counter value is 10, {RAW 31 RAW 30 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 21, {RAW 63 RAW 62 ...RAW 33 RAW 32 } to form the second group of 256-bit data; when the counter value is 31, {RAW 95 RAW 94 ...RAW 65 RAW 64 } forms the third group of 256-bit data, which is exactly 32 clock pixel data P n The complete splicing combination is completed, and the subsequent clock pixel data is processed in the same way as described above; RAW_16bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 15, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 5-level register delay; when the counter value is 5, {RAW 15 RAW 14 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 10, {RAW 31 RAW 30 ...RAW 17 RAW 16 } to form the second group of 256-bit data; when the counter value is 15, {RAW 47 RAW 46 ...RAW 33 RAW 32 } forms the third group of 256-bit data, at this time, there are exactly 16 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above.

5. The processing circuit based on MIPI CSI2 protocol RAW data according to claim 4, characterized in that: The double buffer control unit performs double buffer control on the video data in units of lines, and sets an asynchronous FIFO to control two DPRAMs to perform ping-pong operation, one DPRAM performs a read operation, and the other DPRAM performs a write operation at the same time.

6. The processing circuit based on MIPI CSI2 protocol RAW data according to claim 5, characterized in that: The AXI bus write interface initiates a data read operation to the double buffer control unit in units of lines, and then outputs the RAW type video data to the storage control unit according to the timing requirements of the AXI bus write interface.

7. A method for implementing the MIPI CSI2 protocol RAW data processing circuit according to claim 1, characterized in that: The method comprises the following steps: 1) The RAW type video effective pixel data extraction unit divides the MIPI CSI2 protocol RAW data into two categories for processing. The effective data bit width of RAW6 and RAW7 is padded with 0 in the high bit and processed according to the 8-bit RAW8 data type; the effective data bit width of RAW10, RAW12 and RAW14 is padded with 0 in the high bit and processed according to the 16-bit RAW16 data type; according to the RAW data output by the MIPI CSI2 protocol front-end processing unit, one pixel clock cycle includes 3 RAW data, that is, P n = {RAW1, RAW2, RAW3}, swap the high and low bits of the three RAW data in each pixel clock cycle, that is, P n ={RAW3,RAW2,RAW1}; 2) The horizontal resolution extraction and shaping unit sets a register counter to count the number of input video data in real time according to the line. The counter increases by 1 for each transmission of valid video data. The counter value at the end of a line of video data transmission is the horizontal resolution value of the RAW data. In order to facilitate the subsequent modules to splice and combine valid pixel data, the horizontal resolution of the extracted RAW6, RAW7 and RAW8 type video data is shaped to an integer multiple of 32, and the horizontal resolution of the extracted RAW10, RAW12, RAW14 and RAW16 type video data is shaped to an integer multiple of 16. 3) RAW_8bit effective pixel data splicing combination and RAW_16bit effective pixel data splicing combination; The RAW_8bit effective pixel data splicing combination sets a register counter with an initial value of 0. Each clock pixel P is output. n The counter increases by 1, and when the counter counts to 31, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 11 levels of register delay; when the counter value is 10, {RAW 31 RAW 30 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 21, {RAW 63 RAW 62 ...RAW 33 RAW 32 } to form the second group of 256-bit data; when the counter value is 31, {RAW 95 RAW 94 ...RAW 65 RAW 64 } forms the third group of 256-bit data, which is exactly 32 clock pixel data P n The complete splicing combination is completed, and the subsequent clock pixel data is processed in the same way as described above; RAW_16bit effective pixel data splicing combination: set a register counter with an initial value of 0, and output a clock pixel P n The counter increases by 1, and when the counter counts to 15, it is cleared to 0 and starts counting again, and so on; at the same time, the clock pixel data P n Use 5-level register delay; when the counter value is 5, {RAW 15 RAW 14 ...RAW1RAW0} form the first group of 256-bit data; when the counter value is 10, {RAW 31 RAW 30 ...RAW 17 RAW 16 } to form the second group of 256-bit data; when the counter value is 15, {RAW 47 RAW 46 ...RAW 33 RAW 32 } forms the third group of 256-bit data, at this time, there are exactly 16 clock pixel data P n The complete splicing combination is completed. The subsequent clock pixel data is processed in the same way as described above; 4) The double buffer control unit performs double buffering control on the video data in units of lines, and sets an asynchronous FIFO to control two DPRAMs to perform ping-pong operation, one DPRAM performs a read operation, and the other DPRAM performs a write operation at the same time; 5) The AXI bus write interface initiates a data read operation to the double buffer control unit in units of lines, and then outputs the RAW type video data to the storage control unit according to the timing requirements of the AXI bus write interface.

Images