Real-time reordering collection method for output data of a low-light imaging sensor

Abstract

The application discloses a real-time sequence adjusting and collecting method for output data of a low-light imaging sensor, and a hardware system of the method is composed of a GSENSE400BSI image sensor, an FPGA chip, a DDR3 SDRAM memory, a USB3.0 data transmission interface and a PC upper computer. The method alternately buffers data for the sensor Channel by designing a FIFO circuit to obtain row data with correct pixel sequences. Then, the row data is buffered by using the DDR3 SDRAM, and frame data with correct pixel sequences is obtained by adjusting the write address of the memory. Finally, the frame data is sequentially transmitted to the PC upper computer through the USB3.0 data, and data storage and image display are performed. The application realizes real-time sequence adjusting and collecting of image data based on the FPGA, facilitates synchronous performance of data transmission, storage and image display, simplifies design and development of a program of the upper computer, and reduces performance requirements of the upper computer.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of imaging, in particular to a real-time sorting collection method of micro-light night vision imaging sensor output data. BACKGROUND

[0002] With the development of science and technology, micro-light high-resolution scientific cameras are widely used in biological imaging, military, aerospace, medical treatment and astronomy, etc. Due to the monopoly of foreign countries in CMOS and CCD camera technology, most of the micro-light high-resolution scientific cameras used in China are imported. However, the foreign cameras have strict export restrictions for many application scenarios in China, and the imported complete cameras have low developability and high price, which are difficult to meet certain specific requirements. The GSENSE400BSI is a CMOS image sensor with low noise, high sensitivity and high dynamic range, which can meet the domestic high-end micro-light imaging application requirements. The image sensor can sample the image twice, and the combination of the scene information of the two sampling can greatly improve the dynamic range and improve the shooting effect in the micro-light environment. The GSENSE400BSI image sensor supports HDR and STD modes. In the HDR mode, the image sensor samples two groups of image data of high gain (HG) and low gain (LG) of the picture at the same time, and outputs the data through 8 channels, each channel alternately outputs the row data of high gain and low gain images in adjacent pixel periods. In the STD mode, the image sensor samples two consecutive pixel rows of a frame of image at the same time, and outputs through 8 channels, each channel alternately outputs different row data in adjacent pixel periods. In both modes, the output data of the image sensor needs to be reordered to obtain the image with correct pixel order. One way is to directly transmit the data to the host computer, and write a software program on the host computer to reorder the data and obtain the image with correct pixel order. The defect of this method is that the sensor pixels are many, the amount of image data collected is large, the software reordering consumes a long time, and it is difficult to achieve image acquisition and display at the same time. At the same time, the frame rate of the sensor is 24fps in the HDR mode and 48fps in the STD mode, and the software needs to realize synchronous execution of data sorting, image display and data storage, which requires high performance of the host computer, and ordinary PC cannot meet the requirements. SUMMARY

[0003] To solve the above problems, the purpose of the present application is to provide a real-time image data sorting method based on FPGA hardware, design FPGA hardware sorting circuit, reorder the image data in real time before transmitting to the host computer, and transmit to the host computer according to the pixel order of the image. Simplify the host computer program, improve the performance of the host computer, and enable the host computer to display the image in real time.

[0004] To achieve the above purpose, the technical scheme of the present application is:

[0005] A kind of micro-light imaging sensor output data acquisition system based on FPGA is proposed, including GSENSE400BSI image sensor, FPGA chip, DDR3 SDRAM memory, USB3.0 data transmission interface and PC host computer.The FPGA of the described FPGA acquires the image data output of GSENSE400BSI image sensor, carries out data sorting in FPGA interior, and is buffered in the described DDR3 SDRAM memory, then is transmitted to the PC host computer through the described USB3.0 interface for long-term storage and image display.

[0006] The FPGA chip realizes the FIFO buffer circuit of acquisition data, DDR3 storage control circuit and USB3.0 data transmission control circuit.

[0007] The DDR3 SDRAM memory includes two DDR3 storage chips, and frame data collected by FPGA is stored using ping-pong operation.When the DDR3 storage control circuit stores image data to one storage chip, the USB3.0 data transmission control circuit reads out image data from another storage chip and transmits it.The read data clock of the DDR3 memory is faster than the write data clock, i.e., the read data speed is faster than the write data speed.

[0008] A data real-time sequencing method based on FPGA is proposed, including the following contents:

[0009] Each data output Channel of GSENSE400BSI image sensor alternately outputs two rows of pixel data of frame image in adjacent pixel periods.In the STD mode, it alternately outputs pixel data of adjacent two rows of the same frame image, and in the HDR mode, it alternately outputs two rows of pixel data of high-gain frame and low-gain frame image.

[0010] For Channel i (i=0,1…7), four data buffer FIFOs are designed, marked as FIFO i*4 , FIFO i*4+1 , FIFO i*4+2 and FIFO i*4+3 , FIFO is divided into A and B groups, wherein FIFO i*4 and FIFO i*4+2 are A group, and FIFO i*4+1 and FIFO i*4+3 are B group.The data real-time sequencing method of GSENSE400BSI image sensor is as follows:

[0011] S1: define 0 / 1 binary variable Flag_Pixel, change the value of Flag_Pixel according to the pixel period; define 0 / 1 binary variable Flag_Line, count the line period, change the value of Flag_Line according to the line period. Since the GSENSE400BSI image sensor outputs two rows of data at the same time, the line period is actually the time for the sensor to output two rows of pixel data.

[0012] S2: according to the values of Flag_Pixel and Flag_Line, alternately cache the data output of Channel i , Channel i*4 , Channel i*4+1 and Channel i*4+2 to FIFO i*4+3 .

[0013] When Flag_Line=0 is executed: cache the row data of the output of the GSENSE400BSI image sensor in the A group FIFO. When Flag_Pixel=0, cache the data output of Channel i to FIFO i*4 ; when Flag_Pixel=1, cache the data output of Channel i to FIFO i*4+2 .

[0014] When Flag_Line=1 is executed: cache the row data of the output of the GSENSE400BSI image sensor in the B group FIFO. When Flag_Pixel=0, cache the data output of Channel i to FIFO i*4+1 ; when Flag_Pixel=1, cache the data output of Channel i to FIFO i*4+3 .

[0015] After the line period, two rows of pixel data of the frame image are continuously cached in the FIFO. When Flag_Line=0, one row of data is continuously cached in FIFO i*4 (i=0, 1…7), and the other row is continuously cached in FIFO i*4+2 (i=0, 1…7). When Flag_Line=1, one row of data is continuously cached in FIFO i*4+1 (i=0, 1…7), and the other row is continuously cached in FIFO i*4+3 (i=0, 1…7).

[0016] S3: After two rows of frame image data are cached, the row data is read out from the FIFO in order, and the specific method is as follows:

[0017] When Flag_Line = 0, first read out the data in FIFO i*4+1 (i = 0, 1…7), then read out the data in FIFO i*4+3 (i = 0, 1…7) in turn.

[0018] When Flag_Line = 1, first read out the data in FIFO i*4 (i = 0, 1…7), then read out the data in FIFO i*4+2 (i = 0, 1…7) in turn.

[0019] S4: cache the data read out in step S3 to DDR3 SDRAM. The read and write of DDR3 SDRAM adopts burst read and write mode, 64 words are read and written each time, and the length of address space of each burst read and write operation is 2 10 . Since the FIFO caches the data of two adjacent lines of the same frame in STD mode, the data is written to SDRAM in sequence to obtain the image with correct pixel sequence. In HDR mode, the FIFO caches the data of different frames of high-gain frame and low-gain frame. In order to separate the line data of high-gain frame and low-gain frame, the data is cached to DDR3 SDRAM by changing the write address of SDRAM to adjust the data sequence again. The specific method is as follows:

[0020] In STD mode: the base address WriteAddrBase of the first burst write operation is set to 0, and before each burst write operation, the base address is increased by 2 based on the base address of the last write operation, i.e. “WriteAddrBase = WriteAddrBase + 2 10 ”. 10

[0021] In HDR mode: the storage space of high-gain frame and low-gain frame of two images in SDRAM is calculated, the storage space of low-gain image is 0~2 23 -1, and the storage space of high-gain image is 2 23 ~2 24 -1. Define the current write operation base address WriteAddrLowBase of low-gain image address space and the current write operation base address WriteAddrHighBase of high-gain image address space.

[0022] ​When low-gain image row data is read from the FIFO, it is written to the SDRAM low-gain image storage space. Before the first burst write, "WriteAddrLowBase = 0, WriteAddrBase = WriteAddrLowBase" is executed. Before each subsequent low-gain address space burst write operation, "WriteAddrLowBase = WriteAddrLowBase + 2" is executed. 10 ,WriteAddrBase=WriteAddrLowBase”.

[0023] When high-gain image row data is read from the FIFO, it is written to the SDRAM high-gain image storage space. Before the first burst write, "WriteAddrHighBase=2" is executed. 23 WriteAddrBase = WriteAddrHighBase. Before each subsequent high-gain address space burst write operation, execute "WriteAddrHighBase = WriteAddrHighBase + 2". 10 ,WriteAddrBase=WriteAddrHighBase”.

[0024] After storing two frames of image data on one SDRAM chip (in STD mode, two consecutive frames are stored; in HDR mode, two high-gain and low-gain images of the same scene are stored), the process switches to another SDRAM chip to cache the image data in the same way.

[0025] S5: After two frames of images are written to the SDRAM chip, data is read sequentially from the chip. The base address ReadAddrBase of the first burst read operation is set to 0. Before each subsequent burst read operation, "ReadAddrBase = ReadAddrBase + 2" is executed. 10 The read data is then buffered again through a FIFO, converting the data word length from 128 bits to 32 bits, and finally sent to the host computer for storage and display via a USB 3.0 data transfer circuit.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The FPGA is used to receive the output data of the GSENSE400BSI image sensor, and the hardware circuit in the FPGA performs real-time data reordering to ensure that the data transmitted to the host PC is image data with the correct pixel order. This allows the host computer program to directly display the image and store the data, simplifying the design and development of the host computer program.

[0028] The real-time sequencing of the image sensor output data facilitates the synchronous performance of image data transmission, storage and image display, and reduces the performance requirement of the host PC. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The micro-light imaging sensor output data real-time sequencing acquisition system of the embodiment of the present application;

[0030] Figure 2 The data sequencing method based on FIFO data buffering of the embodiment of the present application;

[0031] Figure 3 The DDR3 SDRAM data write address calculation method in the STD and HDR modes of the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0033] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims.

[0034] One specific embodiment of the micro-light imaging sensor output data real-time sequencing acquisition system provided by the present application is shown in Figure 1 The system includes a GSENSE400BSI image sensor 1, an FPGA chip 2, a DDR3 SDRAM memory 3, a USB3.0 data transmission module 4, and a PC host computer 5. The functional circuit in the FPGA includes: 32 FIFO data receiving buffers, a multiplexer MUX circuit, a DDR3 SDRAM control circuit, a SDRAM read data buffer FIFOtoPC, and a USB3.0 data transmission control circuit.

[0035] The 8 data output channels of the GSENSE400BSI image sensor 1 are shown in Figure 2 Channel 0 sequentially outputs the sample values of the 0-255 pixels of an image row, Channel 1 sequentially outputs the sample values of the 256-511 pixels, and Channel 7 sequentially outputs the sample values of the 1792-2047 pixels. Each channel alternately outputs the pixel data of different two rows in adjacent pixel periods. In the STD mode, the pixel data of adjacent two rows of the same image frame is alternately output, and in the HDR mode, the pixel data of two rows of the high-gain frame and the low-gain frame of the same image is alternately output.

[0036] The 32 FIFO buffers in FPGA chip 2 are used for the output data of the 8 channels of GSENSE400BSI image sensor 1. For example... Figure 2 As shown, where FIFO 4i ~FIFO 4i+3 Used for caching Channel i Output data for (i = 0, 1, ..., 7). Divide all FIFOs into two groups, FIFO... 4i and FIFO 4i+2 Group A, FIFO 4i+1 and FIFO 4i+3 This is group B. The two groups of FIFOs alternately cache image data according to a row cycle. After one group of FIFOs caches two rows of data, the other group of FIFOs is switched to cache the next two rows of data.

[0037] A further use of the 32 FIFO buffers in FPGA chip 2 is to adjust intersecting row pixel data into two consecutive row pixel data. For example... Figure 2 As shown, within a row cycle, group A FIFOs cache data. Taking Channel 0's data caching as an example, FIFO0 and FIFO2 alternately cache data according to the pixel cycle. After a row cycle, FIFO0 and FIFO2 have cached consecutive row pixel data from 0 to 255 respectively. Then, group B FIFOs switch to cache data, with FIFO1 and FIFO3 alternately caching Channel 0's data according to the pixel cycle. The same working method is applied to other FIFOs and channels; group A FIFOs all cache consecutive row pixel data, including FIFO0, FIFO4, ... FIFO... 28 (FIFO 4i (i = 0, 1, ..., 7) caches continuous pixel data for the entire row, FIFO2, FIFO6, ... FIFO 30 (FIFO 4i+2 (i = 0, 1, ..., 7) caches another complete row of consecutive pixel data; specifically, in STD mode, FIFO... 4i and FIFO 4i+2 (i = 0, 1, ..., 7) caches two consecutive rows of pixel data from the same frame image, FIFO in HDR mode. 4i and FIFO 4i+2 (i = 0, 1, ..., 7) respectively cache two rows of pixel data for the low-gain frame and the high-gain frame of the same screen.

[0038] After a row cycle, FIFO A stores two complete rows of data, FIFO B is switched to buffer data, and the multiplexer circuit controls the sequential reading of the FIFOs. 4i (i=0,1,…,7) and FIFO 4i+2The data is in the range (i = 0, 1, ..., 7). Note that the write word length for the FIFO is 16 bits, and the read word length is 128 bits. Furthermore, the FIFO is asynchronous; the read FIFO clock is at least twice the write FIFO clock to ensure that when switching to write to another FIFO after writing to one FIFO, all data in the other FIFO's buffer has been read.

[0039] The data read from the FIFO is sent to the DDR3 SDRAM memory 3 for a second data buffering. Its main purpose is to further adjust the output data order of the GSENSE400BSI image sensor 1 in HDR mode, separating low-gain frame data from high-gain frame data. The DDR3 SDRAM memory 3 consists of two memory chips, using a ping-pong method for data buffering to ensure that data sorting operations do not reduce the overall data transfer rate. The DDR3 SDRAM uses a burst read / write method, reading or writing 64 words at a time, with each word being 128 bits long. Therefore, the address space length for each burst read / write operation is 2... 10 .

[0040] In STD mode, the FIFO buffers two adjacent rows of data within the same frame. Writing the data sequentially to SDRAM yields an image with the correct pixel order. The address calculation method for write operations in STD mode is as follows: Figure 3 As shown in Figure a, the storage space for the two frames of images is calculated to be 0 to 2. 24 -1, the base address WriteAddrBase for the first burst write operation is set to 0, and before each subsequent burst write operation, the base address is increased by 2 based on the base address of the previous write operation. 10 , that is, "WriteAddrBase=WriteAddrBase+2 10 ".

[0041] In HDR mode, the FIFO buffers two lines of data from different frames (high gain and low gain). When buffering data in DDR3 SDRAM, the data order is adjusted again by changing the write address of the SDRAM. The address calculation method for write operations in HDR mode is as follows: Figure 3 As shown in b, calculate the storage space of the high-gain and low-gain images in the SDRM. The storage space of the low-gain image is 0~2. 23 -1, high-gain image storage space is 2 23 ~2 24-1. Define the current write operation base address of low gain image address space WriteAddrLowBase, and the current write operation base address of high gain image address space WriteAddrHighBase. When reading low gain image row data from FIFO, write low gain image storage space of SDRAM. Before the first burst write, execute "WriteAddrLowBase=0, WriteAddrBase=WriteAddrLowBase". Before each low gain address space burst write operation later, execute "WriteAddrLowBase=WriteAddrLowBase+2, WriteAddrBase=WriteAddrLowBase". 10 When reading high gain image row data from FIFO, write high gain image storage space of SDRAM. Before the first burst write, execute "WriteAddrHighBase=2, WriteAddrBase=WriteAddrHighBase". Before each high gain address space burst write operation later, execute "WriteAddrHighBase=WriteAddrHighBase+2, WriteAddrBase=WriteAddrHighBase". 23 10

[0042] When a piece of SDRAM chip stores 2 frames of image data, switch to another piece of SDRAM storage chip to continue caching data. Specifically, in STD mode, the 2 frames of image stored by SDRAM chip are two consecutive frames of image, and in HDR mode, the 2 frames of image stored are two frames of high and low gain image of the same picture.

[0043] After writing two frames of image into SDRAM chip, sequentially read data from the storage chip. The base address of the first burst read operation is set to 0, and before each burst read operation later, execute "ReadAddrBase=ReadAddrBase+2". 10

[0044] FIFOtoPC caches data read from SDRAM, and its main purpose is to convert the data word length from 128 bits to 32 bits, which is convenient for subsequent data transmission according to the protocol of USB3.0.

[0045] The USB3.0 data transmission circuit sends data to the host PC, and the host caches and counts the received data. When the data amount of a frame of image is reached, store and display the cached data frame, and continue to receive the image data of the next frame.

[0046] ​​​The micro-light imaging sensor output data real-time sequence collection method has the characteristics that:

[0047] The FIFO cache-based sequence adjustment circuit is added in the FPGA for collecting the GSENSE400BSI image sensor output data, without increasing the additional hardware cost, the real-time sequence function of the image data is realized, and the data transmitted to the host PC is the image data with correct pixel sequence.

[0048] Compared with the method for adjusting the pixel sequence by the PC host software program, the pixel data real-time sequence method based on the FPGA hardware is more convenient for the synchronization of the image data transmission, storage and image display, and reduces the performance requirement of the host PC.

[0049] The above embodiment only describes the preferred embodiment of the present application, and does not limit the concept and scope of the present application. Various modifications and improvements of the technical solutions of the present application without departing from the design concept of the present application all belong to the protection scope of the present application.

Claims

1. A method for real-time sequencing and acquisition of output data from a low-light imaging sensor, characterized in that, The hardware system upon which this method is based includes a GSENSE400BSI image sensor, an FPGA chip, a DDR3 SDRAM memory, a USB 3.0 data transfer interface, and a PC host computer. The FPGA acquires image data output from the GSENSE400BSI image sensor, sorts the data internally, and caches it in the DDR3 SDRAM memory. This data is then transmitted to the PC host computer via the USB 3.0 interface for long-term storage and image display. The internal circuitry implemented by the FPGA chip includes a FIFO buffer circuit, a DDR3 storage control circuit, and a USB 3.0 data transfer control circuit. The DDR3 SDRAM memory comprises two DDR3 memory chips, using a ping-pong operation to store the frame data acquired by the FPGA. While the DDR3 storage control circuit stores image data on one memory chip, the USB 3.0 data transfer control circuit reads image data from the other memory chip and transmits it. The specific method includes the following: Each data output channel of the GSENSE400BSI image sensor alternately outputs two rows of pixel data of the frame image in adjacent pixel periods; when the GSENSE400BSI image sensor is working in STD mode, it alternately outputs two rows of pixel data of adjacent rows of the image, and in HDR mode, it alternately outputs two rows of pixel data of high-gain frame and low-gain frame images. For Channel i Design four data buffer FIFOs (i = 0, 1, ..., 7), labeled as FIFOs. i*4 FIFO i*4+1 FIFO i*4+2 and FIFO i*4+3 The FIFOs are divided into two groups, A and B, where the FIFOs are... i*4 and FIFO i*4+2 Group A, FIFO i*4+1 and FIFO i*4+3 For Group B; the real-time data sequencing method for the GSENSE400BSI image sensor is as follows: S1: Define a 0 / 1 binary variable Flag_Pixel, and change the value of Flag_Pixel alternately according to the pixel cycle; define a 0 / 1 binary variable Flag_Line, and change the value of Flag_Line alternately according to the line cycle; since the GSENSE400BSI image sensor outputs two lines of data simultaneously, the line cycle is actually the time it takes for the sensor to output two lines of pixel data. S2: Based on the values ​​of Flag_Pixel and Flag_Line, adjust the Channel... i Data output is alternately buffered to FIFO i*4 FIFO i*4+1 FIFO i*4+2 and FIFO i*4+3 middle; When Flag_Line = 0: The output line data of the GSENSE400BSI image sensor is buffered in FIFO group A; when Flag_Pixel = 0, the Channel... i Data output buffered to FIFO i*4 In the middle; when Flag_Pixel=1, the Channel i Data output buffered to FIFO i*4+2 middle; When Flag_Line = 1: The line data output from the GSENSE400BSI image sensor is buffered in group B FIFO; when Flag_Pixel = 0: Channel... i Data output buffered to FIFO i*4+1 In the middle; when Flag_Pixel=1, the Channel i Data output buffered to FIFO i*4+3 middle; After a line period, two lines of pixel data from the frame image are continuously buffered in a FIFO; when Flag_Line = 0, one line of data is continuously buffered in the FIFO. i*4 In (i = 0, 1…7), another row is cached consecutively in FIFO. i*4+2 In (i = 0, 1…7); when Flag_Line = 1, one row of data is continuously cached in the FIFO. i*4+1 In (i = 0, 1…7), another row is cached consecutively in FIFO. i*4+3 (i=0,1...7); S3: After the two rows of frame image data are buffered in the FIFO, the row data is read out of the FIFO in a certain order, as follows: When Flag_Line = 0, first read the FIFO sequentially. i*4+1 The data in (i = 0, 1…7) is then read out sequentially from the FIFO. i*4+3 Data in (i = 0, 1…7); When Flag_Line = 1, first read the FIFO sequentially. i*4 The data in (i = 0, 1…7) is then read out sequentially from the FIFO. i*4+2 Data in (i = 0, 1…7); S4: Cache the data read in step S3 into DDR3 SDRAM; DDR3 SDRAM reads and writes use burst read / write mode, with each read / write operation handling 64 words, each word being 128 bits long. Therefore, the address space length for each burst read / write operation is 2... 10 In STD mode, the FIFO caches two adjacent rows of data within the same frame, and writing the data sequentially to SDRAM yields an image with the correct pixel order. However, in HDR mode, the FIFO caches different frame data for high-gain and low-gain frames. To separate the row data of high-gain and low-gain frames, when caching the data in DDR3 SDRAM, the write address of the SDRAM needs to be changed, and the data order needs to be adjusted again. The specific method is as follows: In STD mode: the base address WriteAddrBase for the first burst write operation is set to 0, and before each subsequent burst write operation, the base address is increased by 2 based on the base address of the previous write operation. 10 , that is, "WriteAddrBase=WriteAddrBase+2 10 ”; In HDR mode: Calculate the storage space of the high-gain and low-gain images in the SDRM, with the low-gain image having a storage space of 0 to 2. 23 -1, high-gain image storage space is 2 23 ~2 24 -1; Defines the current write operation base address WriteAddrLowBase for the low-gain image address space and the current write operation base address WriteAddrHighBase for the high-gain image address space. When reading low-gain image row data from the FIFO, write to the SDRAM low-gain image storage space; before the first burst write, execute "WriteAddrLowBase = 0, WriteAddrBase = WriteAddrLowBase"; before each subsequent low-gain address space burst write operation, execute "WriteAddrLowBase = WriteAddrLowBase + 2". 10 ,WriteAddrBase=WriteAddrLowBase”; When high-gain image row data is read from the FIFO, it is written to the SDRAM high-gain image storage space; before the first burst write, "WriteAddrHighBase=2" is executed. 23 WriteAddrBase = WriteAddrHighBase; Before each subsequent high-gain address space burst write operation, execute "WriteAddrHighBase = WriteAddrHighBase + 2". 10 ,WriteAddrBase=WriteAddrHighBase”; After storing two frames of image data on one storage chip, the process switches to another storage chip and caches the image data in the same way. S5: After two frames of images are written to the memory chip, data is read sequentially from the chip; the base address ReadAddrBase of the first burst read operation is set to 0, and before each subsequent burst read operation, "ReadAddrBase = ReadAddrBase + 2" is executed. 10 The read data is buffered again through a FIFO, and the data word length is converted from 128 bits to 32 bits. Finally, the data is sent to the host computer for storage and display through the USB 3.0 data transmission circuit.

Images