High-frame-frequency output system and method based on large-area-array CMOS

Abstract

The invention discloses a high-frame-frequency output system and method based on a large-area-array CMOS, and belongs to the technical field of image processing. The system comprises a bus communication module, a CMOS configuration read-back module, a CMOS detector control module, a CMOS detector, an image data processing and caching module and a data transmission output module, through an imaging time sequence pipeline design and a ping-pong caching design, the imaging frame frequency is greatly improved, and under the condition that only one high-speed data transmission interface exists, when one hundred million pixels are output in a full format, the imaging frame frequency is greatly improved. The frame frequency can reach 1Hz, and the frame frequency can be increased to 5Hz in a windowing mode of the detector.

Description

Technical Field

[0001] The invention relates to a high frame rate output system and method based on a large-array CMOS, belonging to the technical field of image processing. Background Art

[0002] In the field of aerospace applications, the use of large-array CMOS image detectors is still in its infancy, and will be widely popularized as the main detector in the future. As the demand for high temporal and high spatial resolution of aerospace remote sensing cameras becomes more and more urgent, the CMOS image sensor module of the imaging system needs to have extremely high pixels, so the output image data is not only huge in volume, but also very fast. Most of the existing aerospace remote sensing cameras in China have tens of millions of pixels. The size of the imaging system CMOS array in the document "Design of Large-array CMOS Real-time Synchronous Imaging and Display System" is 1280×1024, and the pixel output frequency is 108MHz. The imaging system CMOS pixel in the document "Design of Large-array High-frame-rate CMOS Imaging Electronics System" is 19.54 million, and the pixel output rate is 300MHz. In the document "Design and Implementation of Video Circuit of Staring Camera of "Gaofen-4" Satellite", the staring camera system carried by the "Gaofen-4" satellite uses a 10k*10k pixel array CMOS, which can capture image information with a resolution of 50m in the geostationary orbit of 36,000km, but the fastest frame rate is 1Hz. Existing CMOS area arrays are difficult to meet the requirements of high pixels and high frame rates. Summary of the invention

[0003] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art, and to propose a high frame rate output system and method based on a large array CMOS, which can realize high frame rate output of image data through system module design and method design.

[0004] The technical solution of the present invention is:

[0005] A high frame rate output system based on a large array CMOS includes a bus communication module, a CMOS configuration readback module, a CMOS detector control module, a CMOS detector, an image data processing and buffering module and a data transmission output module;

[0006] The bus communication module communicates with the satellite platform via the bus, receives the standby mode to imaging mode conversion instruction, remote control instruction and GPS second pulse signal sent by the satellite platform, forwards the standby mode to imaging mode conversion instruction to the CMOS configuration readback module, forwards the remote control instruction to the CMOS configuration readback module and the image data processing and cache module; generates an internal frame synchronization signal according to the remote control instruction and sends it to other modules;

[0007] The CMOS configuration readback module receives the instruction to convert the standby mode to the imaging mode, configures the detector and sends the configuration data to the CMOS detector control module, then reads back the data of each register of the CMOS detector and outputs it to the image data processing and cache module; receives the remote control instruction, sends the remote control instruction codeword to the image data processing and cache module, converts the remote control instruction codeword into an I2C configuration codeword and sends it to the CMOS detector;

[0008] The CMOS detector control module receives the configuration data, sends an exposure timing control signal to the CMOS detector, and controls the CMOS detector imaging;

[0009] The CMOS detector performs imaging according to the exposure timing control signal and outputs corresponding image data according to the I2C configuration codeword;

[0010] The image data processing and buffering module receives the image data output by the CMOS detector, performs channel integration and arrangement on the image data, and buffers the processed image data and the register data into the DDR according to the remote control command codeword;

[0011] The digital transmission output module reads the image data from the DDR and sends it to the external digital transmission receiving subsystem through the serial interface.

[0012] Furthermore, the image data processing and cache module is provided with two DDR chips, adopts a ping-pong cache design, and stores the two frames of output images of the detector respectively, and reads out the image data of the previous frame for output while storing the current frame image data.

[0013] Furthermore, during the first frame synchronization, the CMOS detector control module outputs an exposure signal of the first frame to the CMOS detector to control the image exposure of the first frame; during the second frame synchronization, the image of the first frame after exposure is output from the CMOS detector to the image data processing and caching module, and at the same time, the CMOS detector control module outputs an exposure signal of the second frame to the CMOS detector to control the image exposure of the second frame; when the third frame synchronization is generated internally, the data transmission output module reads the processed image of the first frame from a DDR and outputs it through the serial data transmission interface, and at the same time, the CMOS detector control module outputs an exposure signal of the third frame to the CMOS detector to control the image exposure of the third frame, and the image of the second frame after exposure is output from the CMOS detector to the image data processing and caching module, and the image data processing and caching module stores the processed image data in another DDR, and so on, to achieve pipeline timing design.

[0014] Furthermore, the system is deployed on an FPGA, which includes at least 6 GTH channels and has external interfaces including but not limited to the following:

[0015] 422 bus interface: used for bus communication module to receive satellite platform remote control instructions;

[0016] CMOS control signal interface: used for CMOS configuration readback module to configure CMOS, and CMOS detector control module to drive CMOS;

[0017] CMOS data interface: used for image data processing and buffer module to receive CMOS image data;

[0018] DDR interface: used for image data processing and cache module to store and read processed image data;

[0019] Data sending interface: used for the data transmission output module to output the final image data.

[0020] Furthermore, the CMOS detector control module receives the configuration data and sends an exposure timing control signal to the CMOS detector, including a CMOS detector Power Down signal, a reset signal and an exposure drive signal.

[0021] Furthermore, the CMOS detector receives an I2C configuration codeword converted by a remote control command. If the I2C configuration codeword is a full-frame image mode, all image data is output; if the I2C configuration codeword specifies that the detector starts to open a window from the nth row and opens a window for m rows, image data from the nth row to the n+m-1th row is output.

[0022] Furthermore, the image data processing and caching module receives a remote control command codeword. If the remote control command codeword is a full-frame image mode, all processed image data stitching register data is cached in the DDR; if the remote control command codeword specifies output starting from the pth column and outputting the qth column, the image data processing and caching module reads out a row of data of the processed image, then intercepts the image data stitching register data from the pth column to the p+q-1th column and caches it in the DDR.

[0023] Furthermore, the CMOS configuration readback module performs data transmission via an I2C bus, and a configuration rate is not less than 100 KHz.

[0024] The imaging method based on a large-array CMOS high frame rate output system comprises:

[0025] Step S1: The bus communication module generates an internal frame synchronization signal with a variable period according to a control instruction, which is used to trigger the exposure drive of the CMOS detector and the storage and reading of the detector pixels;

[0026] Step S2: After receiving the remote control instruction of switching from standby mode to imaging mode, the CMOS configuration readback module starts to configure and read back the registers of CMOS;

[0027] Step S3: After the configuration is completed, the CMOS detector control module outputs an exposure timing control signal of the detector according to the internal frame synchronization signal, and controls the CMOS detector to start exposure;

[0028] Step S4: After the exposure is completed, the CMOS detector outputs serial data, and the image data processing and buffer module simultaneously receives the serial pixel data of each channel and converts it into parallel pixel data;

[0029] Step S5: the image data processing and buffering module selects whether to perform binning processing on the pixel data according to the remote control instruction;

[0030] Step S6: The CMOS configuration readback module configures the detector according to the remote control command to select the full-frame image mode or the arbitrary windowing mode. If it is the arbitrary windowing mode, the remote control command codeword stipulates that the detector starts to open the window from the nth row, opens the window m rows, and the detector outputs the image from the nth row to the n+m-1th row;

[0031] Step S7: the image data processing and buffering module stores and processes the pixel data according to the internal frame synchronization signal;

[0032] Step S8: the image data processing and caching module selects full-frame or arbitrary window size pixel output according to the remote control instruction. The remote control instruction stipulates that the output starts from the pth column and outputs the qth column. After the image data processing and caching module reads out a row of data, it intercepts the pixels from the pth column to the p+q-1th column.

[0033] Step S9: the data transmission output module sends the image processed by the image data processing and buffering module through the serial data transmission interface.

[0034] The advantages of the present invention compared with the prior art are:

[0035] (1) The present invention adopts an imaging timing pipeline design and designs a standby mode in the imaging process to reduce the frequency of configuring and reading back the CMOS. Compared with the traditional imaging serial timing design, the imaging frame rate is greatly improved. In the case of only one high-speed data transmission interface, the frame rate can reach 1 Hz when the full-frame output is 100 million pixels, and the frame rate can be increased to 5 Hz in the detector window mode.

[0036] (2) The present invention can dynamically open windows on the detector and arbitrarily select the pixel size of the output image.

[0037] (3) The present invention can configure the large array detector to switch to different working modes in real time, and read back the working status of the detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 This is a schematic diagram of a high frame rate output system based on a large array CMOS according to an embodiment of the present invention;

[0040] Figure 2 This is a flow chart of high frame rate imaging based on a large-array CMOS high frame rate output system according to an embodiment of the present invention;

[0041] Figure 3 Timing diagram for traditional imaging systems;

[0042] Figure 4 This is a timing design diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] The invention proposes a high frame rate output system based on large array CMOS, comprising a bus communication module, a CMOS detector control module, a CMOS configuration readback module, an image data processing and buffering module, and a data transmission output module.

[0045] The bus communication module communicates with the satellite platform through the bus, receives the control instructions of the camera to control the imaging of the detector, and at the same time, it composes the imaging status (standby, imaging) and CMOS status (gain, exposure time) collected inside the imaging circuit into a telemetry data frame, and finally returns the imaging circuit telemetry data frame to the satellite platform.

[0046] The CMOS detector control module sets the exposure time of the detector according to the bus instructions, controls the detector imaging, and provides the necessary working conditions for the CMOS image sensor, including outputting the Power Down signal, reset signal, and exposure drive signal to the CMOS device in sequence.

[0047] The CMOS configuration readback module configures the detector through the I2C bus, changes the CMOS working mode (HDR mode or non-HDR mode), gain, windowing, etc. Read back the various registers of the CMOS image sensor, splice the register content and image data, and output them through the high-speed serial data transmission interface. The traditional configuration method is to configure the CMOS before each frame detector integration, and read back the CMOS after configuration. The configuration readback needs to avoid the time of detector exposure and pixel output, so it will increase the frame cycle time. In order to increase the frame rate, this system designs a standby mode, and switches between the standby mode and the imaging mode through bus remote control commands. In standby mode, the imaging circuit does not configure CMOS, does not drive CMOS imaging, and has no data transmission output. It is suitable for scenes where there is no imaging demand to save power consumption. If the CMOS needs to be reconfigured, first send the standby mode remote control command to configure the CMOS, and then send the imaging mode command to continue imaging. Therefore, the detector is configured only once when the standby mode is switched to the imaging mode. During the imaging process, if the CMOS is not reconfigured, the status of other registers of the CMOS will not change except for the temperature. Therefore, the CMOS status register is only read back when the standby mode is switched to the imaging mode, and it is no longer read back during the imaging process, which greatly shortens the frame period.

[0048] The image data processing and cache module receives the image data output by CMOS, integrates and arranges the image data in 6 channels, and splices the 6 channels of image data received simultaneously into one data channel in the order of 1-6, and then splices the detector readback data and other contents, and caches the integrated image data into DDR. The timing of the traditional imaging system is to control the detector integration first, receive the detector image after integration, and then output the detector image through the high-speed data transmission interface after reception. The advantage of this design is that the timing design and memory control are simple, such as Figure 3 In order to increase the frame rate, this system adopts pipeline design for imaging timing. While the detector is integrating, the previous detector image is output through the high-speed data transmission interface. Figure 4As shown, when the first frame synchronization is generated internally, the exposure signal of the first frame is output to CMOS to control the exposure of the image of the first frame; when the second frame synchronization is generated internally, the image of the first frame after exposure is output from CMOS to FPGA and stored in DDR_1 outside the FPGA chip. At the same time, the exposure signal of the second frame is output to CMOS to control the exposure of the image of the second frame; when the third frame synchronization is generated internally, the image of the first frame after exposure is read out from DDR_1 and output through the high-speed serial data transmission interface. At the same time, the exposure signal of the third frame is output to CMOS to control the exposure of the image of the third frame. The image of the second frame after exposure is output from CMOS to FPGA and stored in DDR_2 outside the FPGA chip, and so on. Due to the pipeline design, it is necessary to read out the previous frame of pixels while storing the detector pixels and output them through the high-speed data transmission interface. Therefore, two DDRs are designed to store the output images of the two frames before and after the detector respectively. While storing the image data of the current frame, the image data of the previous frame is read out and output, so that the imaging process is streamlined.

[0049] The digital transmission output module reads the image data from the DDR and sends it to the digital transmission receiving subsystem through a high-speed serial interface.

[0050] The present invention uses a FPGA to deploy a high frame rate output system, requiring the FPGA to contain at least 6 GTH channels, which can simultaneously receive 6-channel images of the detector, and the receiving rate of each channel is 3.125 Gbps. Figure 1 As shown, FPGA has the following interfaces for external connection:

[0051] Second pulse interface: The bus communication module receives the second pulse signal of the satellite platform GPS through the RS422 bus (master / backup) for time management;

[0052] 422 bus interface: The bus communication module uses A and B dual asynchronous serial communication buses to exchange information, receive satellite platform remote control commands, and return telemetry information;

[0053] CMOS control signal interface: including exposure drive interface and I2C configuration interface. The CMOS configuration readback module configures the CMOS through the I2C configuration interface to change the working mode (HDR mode or non-HDR mode), gain, windowing, etc. of the CMOS. The CMOS detector control module drives the CMOS through the exposure drive interface.

[0054] CMOS data interface: The image data processing and cache module receives CMOS high-speed data signals through the CMOS data interface, and the pixel output rate is 3.125Gbps;

[0055] DDR interface: The image data processing and cache module stores and reads the pixel data after CMOS imaging through the DDR interface;

[0056] High-speed data transmission interface: The data transmission output module arranges and outputs the CMOS image data through the high-speed data transmission interface;

[0057] FLASH interface: FLASH is used to store the working parameters required for CMOS imaging. The CMOS configuration readback module reads the working parameters through the FLASH interface.

[0058] Refresh chip interface: The bus communication module receives remote control instructions and controls the refresh chip through the refresh chip interface to dynamically refresh the FPGA and load the program.

[0059] The ultra-large area array CMOS with a pixel output rate of 3.125GHz receives 6-channel high-speed data output by the detector through the high-speed serial port of the FPGA, realizes dynamic configuration and readback of the CMOS registers, performs 2*2 binning and windowing on the image data output by the CMOS, and the processed pixels are passed through the DDR ping-pong cache and finally output line by line through the high-speed serial interface, achieving a high frame rate output of 200ms, which can realize the capture of 20m resolution image information in the geostationary orbit of 36,000km.

[0060] The imaging method using the high frequency frame output system proposed by the present invention is as follows: Figure 2 As shown, including:

[0061] Step S1: Internal frame synchronization generation: The bus communication module generates an internal frame synchronization signal with a variable period (the shortest is 200ms) according to the remote control instruction, which is used to trigger the exposure drive of the CMOS detector, the storage and reading of the detector pixels.

[0062] Step S2: According to the remote control command of switching the standby mode to the imaging mode, the CMOS configuration readback module starts to configure and read back the CMOS registers. The configuration readback of the CMOS is completed through the I2C bus, and the configuration rate is 100KHz;

[0063] Step S3: After the configuration is completed, the CMOS detector control module outputs the exposure timing control signal of the detector according to the internal frame synchronization signal to control the CMOS to start exposure;

[0064] Step S4: After the exposure is completed, the CMOS starts to output serial data, and the image data processing and cache module uses the high-speed serial interface inside the FPGA to simultaneously receive 6-channel serial pixel data, each channel at 3.125GHz, and converts them into parallel 14-bit quantized pixels;

[0065] Step S5: the image data processing and buffering module selects whether to bin the 14-bit quantized pixels according to 2*2 according to the remote control instruction;

[0066] Step S6: The CMOS configuration readback module selects the full-frame image mode or the arbitrary windowing mode according to the remote control command to configure the detector. If it is the arbitrary windowing mode, the remote control command codeword stipulates that the detector starts to open the window from the nth row and opens the window for m rows. The CMOS configuration readback module sends the detector the detector windowing command, and the detector outputs the image from the nth row to the n+m-1th row.

[0067] Step S7: the image data processing and buffering module stores the windowed pixel data into DDR_1 according to the internal frame synchronization signal;

[0068] Step S8: The image data processing and caching module reads pixel data from DDR_2 row by row, and selects full frame or arbitrary window size pixel output according to the remote control command. The remote control command stipulates that the output starts from the pth column and outputs the qth column. After the image data processing and caching module reads out a row of data, it intercepts the pth column to the p+q-1th column pixel output;

[0069] Step S9: The data transmission output module finally sends out the image processed by the image data processing and buffering module through the high-speed serial data transmission interface.

[0070] The above-described embodiments are only preferred specific implementations of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A high frame rate output system based on large array CMOS, characterized in that: It includes bus communication module, CMOS configuration readback module, CMOS detector control module, CMOS detector, image data processing and buffer module and data transmission output module; The bus communication module communicates with the satellite platform via the bus, receives the standby mode to imaging mode conversion instruction and remote control instruction sent by the satellite platform, forwards the standby mode to imaging mode conversion instruction to the CMOS configuration readback module, forwards the remote control instruction to the CMOS configuration readback module and the image data processing and cache module; generates an internal frame synchronization signal according to the remote control instruction and sends it to other modules; The CMOS configuration readback module receives the instruction to convert the standby mode to the imaging mode, configures the detector and sends the configuration data to the CMOS detector control module, then reads back the data of each register of the CMOS detector and outputs it to the image data processing and cache module; receives the remote control instruction, sends the remote control instruction codeword to the image data processing and cache module, converts the remote control instruction codeword into an I2C configuration codeword and sends it to the CMOS detector; The CMOS detector control module receives the configuration data, sends an exposure timing control signal to the CMOS detector, and controls the CMOS detector imaging; The CMOS detector performs imaging according to the exposure timing control signal and outputs corresponding image data according to the remote control command codeword; The image data processing and buffering module receives the image data output by the CMOS detector, performs channel integration and arrangement on the image data, and buffers the processed image data and the register data into the DDR according to the remote control command codeword; The digital transmission output module reads the image data from the DDR and sends it to the external digital transmission receiving subsystem through the serial interface.

2. The large-array CMOS high frame rate output system according to claim 1, characterized in that: The image data processing and cache module is provided with two DDR chips, adopts a ping-pong cache design, and stores the two frames of output images of the detector respectively. While storing the current frame image data, the image data of the previous frame is read out for output.

3. The large-array CMOS high frame rate output system according to claim 2, characterized in that: During the first frame synchronization, the CMOS detector control module outputs the exposure signal of the first frame to the CMOS detector to control the image exposure of the first frame; during the second frame synchronization, the image of the first frame after exposure is output from the CMOS detector to the image data processing and cache module, and at the same time, the CMOS detector control module outputs the exposure signal of the second frame to the CMOS detector to control the image exposure of the second frame; when the third frame synchronization is generated internally, the data transmission output module reads the processed image of the first frame from a DDR and outputs it through the serial data transmission interface, and at the same time, the CMOS detector control module outputs the exposure signal of the third frame to the CMOS detector to control the image exposure of the third frame, and the image of the second frame after exposure is output from the CMOS detector to the image data processing and cache module, and the image data processing and cache module stores the processed image data in another DDR, and so on, to realize the imaging timing pipeline design.

4. The large-array CMOS high frame rate output system according to claim 1, characterized in that: The system is deployed on an FPGA, which contains at least 6 GTH channels and has external interfaces including but not limited to the following: 422 bus interface: used for bus communication module to receive satellite platform remote control instructions; CMOS control signal interface: used for CMOS configuration readback module to configure CMOS, and CMOS detector control module to drive CMOS; CMOS data interface: used for image data processing and buffer module to receive CMOS image data; DDR interface: used for image data processing and cache module to store and read processed image data; Data sending interface: used for the data transmission output module to output the final image data.

5. The large-array CMOS high frame rate output system according to claim 1, characterized in that: The CMOS detector control module receives the configuration data and sends an exposure timing control signal to the CMOS detector, including a CMOS detector PowerDown signal, a reset signal and an exposure drive signal.

6. The large-array CMOS high frame rate output system according to claim 1, characterized in that: The CMOS detector receives the I2C configuration codeword converted by the remote control command. If the I2C configuration codeword is a full-frame image mode, all image data are output; if the I2C configuration codeword specifies that the detector starts to open a window from the nth row and opens a window for m rows, image data from the nth row to the n+m-1th row are output.

7. The high frame rate output system based on large array CMOS according to claim 1, characterized in that: The image data processing and caching module receives the remote control command codeword. If the remote control command codeword is a full-frame image mode, all the processed image data splicing register data is cached in the DDR; if the remote control command codeword specifies to start output from the pth column and output the qth column, the image data processing and caching module reads out a row of data of the processed image, then intercepts the image data splicing register data from the pth column to the p+q-1th column and caches it in the DDR.

8. The large-array CMOS high frame rate output system according to claim 1, characterized in that: The CMOS configuration readback module performs data transmission via the I2C bus, and the configuration rate is not less than 100KHz.

9. The imaging method based on the large-array CMOS high frame rate output system according to claim 1, characterized in that: include: Step S1: The bus communication module generates an internal frame synchronization signal with a variable period according to a control instruction, which is used to trigger the exposure drive of the CMOS detector and the storage and reading of the detector pixels; Step S2: After receiving the remote control instruction of switching from standby mode to imaging mode, the CMOS configuration readback module starts to configure and read back the registers of CMOS; Step S3: After the configuration is completed, the CMOS detector control module outputs an exposure timing control signal of the detector according to the internal frame synchronization signal, and controls the CMOS detector to start exposure; Step S4: After the exposure is completed, the CMOS detector outputs serial data, and the image data processing and buffer module simultaneously receives the serial pixel data of each channel and converts it into parallel pixel data; Step S5: the image data processing and buffering module selects whether to perform binning processing on the pixel data according to the remote control instruction; Step S6: The CMOS configuration readback module configures the detector according to the remote control command to select the full-frame image mode or the arbitrary windowing mode. If it is the arbitrary windowing mode, the remote control command codeword stipulates that the detector starts to open the window from the nth row, opens the window m rows, and the detector outputs the image from the nth row to the n+m-1th row; Step S7: the image data processing and buffering module stores and processes the pixel data according to the internal frame synchronization signal; Step S8: the image data processing and caching module selects full-frame or arbitrary window size pixel output according to the remote control instruction. The remote control instruction stipulates that the output starts from the pth column and outputs the qth column. After the image data processing and caching module reads out a row of data, it intercepts the pixels from the pth column to the p+q-1th column. Step S9: the data transmission output module sends the image processed by the image data processing and buffering module through the serial data transmission interface.

Images