A high frame rate output system and method based on large-area CMOS array

Abstract

This invention discloses a high frame rate output system and method based on a large-area CMOS array, belonging to the field of image processing technology. The system includes a bus communication module, a CMOS configuration readback module, a CMOS detector control module, a CMOS detector, an image data processing and buffering module, and a data transmission output module. Through imaging timing pipeline design and ping-pong buffer design, the imaging frame rate is greatly improved. With only one high-speed data transmission interface, the frame rate can reach 1Hz when outputting 100 million pixels in full-frame mode, and can be increased to 5Hz in detector windowing mode.

Description

Technical Field

[0001] This invention relates to a high frame rate output system and method based on a large-area CMOS array, belonging to the field of image processing technology. Background Technology

[0002] In the aerospace application field, the use of large-area CMOS image detectors is still in its early stages, but it will be widely adopted as the main detector in the future. With the increasingly urgent need for high temporal and spatial resolution in aerospace remote sensing cameras, the CMOS image sensing module of the imaging system needs to have extremely high pixel counts. Therefore, the output image data is not only massive in volume but also very high in speed. Currently, most existing aerospace remote sensing cameras in China have tens of millions of pixels. The paper "Design of a Large-Area CMOS Real-Time Synchronous Imaging and Display System" describes an imaging system with a CMOS array size of 1280×1024 and a pixel output frequency of 108MHz. The paper "Design of a Large-Area High-Frame-Frequency CMOS Imaging Electronic System" describes an imaging system with 19.54 million CMOS pixels and a pixel output rate of 300MHz. The paper "Design and Implementation of Video Circuit for the Staring Camera of the 'Gaofen-4' Satellite" describes a staring camera system on the "Gaofen-4" satellite using a 10k*10k pixel CMOS array, capable of capturing 50m resolution images from a geostationary orbit of 36,000km, but with a maximum frame rate of 1Hz. Existing CMOS arrays are insufficient to meet the requirements of high pixel count and high frame rate. Summary of the Invention

[0003] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a high frame rate output system and method based on a large-area CMOS array. Through system module design and method design, high frame rate output image data can be achieved.

[0004] The technical solution of the present invention is:

[0005] A high frame rate output system based on a large-area CMOS array includes a bus communication module, a CMOS configuration readback module, a CMOS detector control module, a CMOS detector, an image data processing and buffering module, and a data transmission output module.

[0006] The bus communication module communicates with the satellite platform via the bus, receives standby mode to imaging mode conversion instructions, remote control instructions, and GPS second pulse signals sent by the satellite platform, forwards the standby mode to imaging mode conversion instructions to the CMOS configuration readback module, and forwards the remote control instructions to the CMOS configuration readback module and the image data processing and caching module; and generates internal frame synchronization signals based on the remote control instructions and sends them to other modules.

[0007] The CMOS configuration readback module receives the instruction to switch from standby mode to imaging mode, configures the detector and sends the configuration data to the CMOS detector control module, then reads back the data of each register of the CMOS detector and outputs it to the image data processing and caching module; it receives remote control instructions, sends remote control instruction codewords to the image data processing and caching module, converts the remote control instruction codewords into I2C configuration codewords and sends them to the CMOS detector.

[0008] The CMOS detector control module receives configuration data, sends exposure timing control signals to the CMOS detector, and controls the CMOS detector to image.

[0009] The CMOS detector performs imaging based on the exposure timing control signal and outputs corresponding image data according to the I2C configuration codeword;

[0010] The image data processing and caching module receives image data output from the CMOS detector, performs channel integration and arrangement on the image data, and caches the processed image data splicing register data into DDR according to the remote control command codeword.

[0011] The data transmission output module reads image data from the DDR and sends it to the external data transmission receiving subsystem via a serial interface.

[0012] Furthermore, the image data processing and caching module is equipped with two DDR chips and adopts a ping-pong cache design to store the output images of the detector in the previous and next frames respectively. While storing the image data of the current frame, it reads out the image data of the previous frame and outputs it.

[0013] Furthermore, during the first frame synchronization, the CMOS detector control module outputs the exposure signal of the first frame to the CMOS detector to control the exposure of the first frame image. During the second frame synchronization, the image of the first frame after exposure is output from the CMOS detector to the image data processing and buffering module. At the same time, the CMOS detector control module outputs the exposure signal of the second frame to the CMOS detector to control the exposure of the second frame image. When the third frame synchronization is generated internally, the data transmission output module reads the processed image of the first frame from a DDR chip and outputs it through the serial data transmission interface. At the same time, the CMOS detector control module outputs the exposure signal of the third frame to the CMOS detector to control the exposure of the third frame image. The image of the second frame after exposure is output from the CMOS detector to the image data processing and buffering module. The image data processing and buffering module stores the processed image data in another DDR chip, and so on, to realize the pipelined timing design.

[0014] Furthermore, the system is deployed on an FPGA, which contains at least 6 GTH channels and has external interfaces including but not limited to the following:

[0015] 422 bus interface: used for the bus communication module to receive remote control commands from the satellite platform;

[0016] CMOS control signal interface: used by the CMOS configuration readback module for CMOS configuration and by the CMOS detector control module for CMOS driving;

[0017] CMOS data interface: used by the image data processing and buffering module to receive CMOS image data;

[0018] DDR interface: used for image data processing and caching modules to store and retrieve processed image data;

[0019] Data transmission interface: used for the data transmission output module to output the final image data.

[0020] Furthermore, the CMOS detector control module receives configuration data and sends exposure timing control signals to the CMOS detector, including a CMOS detector Power Down signal, a reset signal, and an exposure drive signal.

[0021] Furthermore, the CMOS detector receives the I2C configuration codeword converted from the remote control command. If the I2C configuration codeword is in full-frame image mode, then all image data is output; if the I2C configuration codeword specifies that the detector starts to open a window from the nth row and opens a window for m rows, then the image data from the nth row to the (n+m-1)th row is output.

[0022] Furthermore, the image data processing and caching module receives remote control command codes. If the remote control command code is in full-frame image mode, then all processed image data splicing register data is cached in DDR. If the remote control command code specifies that output starts from column p and outputs column q, the image data processing and caching module reads one line of data from the processed image and then extracts the image data splicing register data from column p to column p+q-1 and caches it in DDR.

[0023] Furthermore, the CMOS configuration readback module transmits data via an I2C bus, with a configuration rate of not less than 100kHz.

[0024] Imaging methods based on large-area CMOS high-frame-rate output systems include:

[0025] Step S1: The bus communication module generates an internal frame synchronization signal with a variable period according to the control command, which is used to trigger the exposure drive of the CMOS detector, the storage and reading of detector pixels;

[0026] Step S2: After receiving the remote control command to switch from standby mode to imaging mode, the CMOS configuration readback module begins to configure and read back the CMOS registers.

[0027] Step S3: After configuration, based on the internal frame synchronization signal, the CMOS detector control module outputs the detector's exposure timing control signal to control the CMOS detector to start exposure;

[0028] Step S4: After exposure, the CMOS detector outputs serial data, and the image data processing and buffering module simultaneously receives serial image data from each channel and converts it into parallel image data.

[0029] Step S5: The image data processing and caching module selects whether to perform binning processing on the image data according to the remote control command;

[0030] Step S6: The CMOS configuration readback module configures the detector according to the selection of full-frame image mode or arbitrary windowing mode based on the remote control command. If it is arbitrary windowing mode, the remote control command code specifies that the detector starts to open the window from the nth row, opens the window for m rows, and the detector outputs the image from the nth row to the n+m-1th row.

[0031] Step S7: The image data processing and caching module stores and processes image data according to the internal frame synchronization signal;

[0032] Step S8: The image data processing and caching module selects full-frame or arbitrary window size pixel output according to the remote control command. The remote control command specifies that the output starts from column p and outputs column q. After reading a line of data, the image data processing and caching module extracts pixels from column p to column p+q-1.

[0033] Step S9: The data transmission module sends the image processed by the image data processing and caching module out through the serial data transmission interface.

[0034] The advantages of this invention compared to the prior art are:

[0035] (1) The present invention adopts an imaging timing pipeline design and a standby mode is designed in the imaging process to reduce the frequency of configuration and CMOS readback. Compared with the traditional imaging serial timing design, the imaging frame rate is greatly improved. When there is only one high-speed data transmission interface, the frame rate can reach 1Hz when the full-frame output is 100 million pixels. The frame rate can be increased to 5Hz in the detector windowing mode.

[0036] (2) The present invention can dynamically open the window of the detector and arbitrarily select the pixel size of the output image.

[0037] (3) The present invention can configure the large array detector to switch different working modes in real time, and read back the working status of the detector. Attached Figure Description

[0038] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0039] Figure 1 This is a schematic diagram of a high frame rate output system based on a large-area CMOS array, according to an embodiment of the present invention.

[0040] Figure 2 This is a flowchart illustrating high frame rate imaging based on a large-area CMOS high frame rate output system, as described in an embodiment of the present invention.

[0041] Figure 3 This is a timing design diagram for a traditional imaging system;

[0042] Figure 4 This is a timing design diagram for an embodiment of the present invention. Detailed Implementation

[0043] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] This invention proposes a high frame rate output system based on a large-area CMOS array, including a bus communication module, a CMOS detector control module, a CMOS configuration readback module, an image data processing and caching module, and a data transmission output module.

[0045] The bus communication module communicates with the satellite platform via the bus, receives control commands from the camera to control the detector to image, and combines information such as the imaging status (standby, imaging) and CMOS status (gain, exposure time) collected inside the imaging circuit into telemetry data frames. Finally, the telemetry data frames of the imaging circuit are returned to the satellite platform.

[0046] The CMOS detector control module sets the detector's exposure time according to bus instructions, controls the detector's imaging, and provides the necessary operating conditions for the CMOS image sensor, including sequentially outputting Power Down signals, reset signals, and exposure drive signals to the CMOS device.

[0047] The CMOS configuration readback module configures the detector via the I2C bus, changing the CMOS operating mode (HDR or non-HDR), gain, windowing, etc. It reads back the various registers of the CMOS image sensor, stitches the register contents with the image data, and outputs the data through a high-speed serial data transmission interface. Traditional configuration methods involve configuring the CMOS before detector integration for each frame and then reading it back. This readback needs to avoid the detector exposure and pixel output times, thus increasing the frame cycle length. To improve the frame rate, this system incorporates a standby mode, switching between standby and imaging modes via bus remote control commands. In standby mode, the imaging circuit does not configure the CMOS, does not drive CMOS imaging, and there is no data transmission output. This is suitable for scenarios where there is no imaging requirement and power saving is desired. If CMOS reconfiguration is needed, a standby mode remote control command is first sent to configure the CMOS, followed by an imaging mode command to continue imaging. Therefore, the detector is configured only once when switching from standby mode to imaging mode. During imaging, if the CMOS is not reconfigured, the states of the CMOS registers, except for the temperature register, will not change. Thus, the CMOS status register is read back only when switching from standby mode to imaging mode, and is not read back during imaging, which greatly shortens the frame period.

[0048] The image data processing and caching module receives image data output from the CMOS sensor, performs 6-channel integration and arrangement of the image data, and stitches the simultaneously received 6-channel image data into a single data stream in sequence 1-6. This data stream is then inlaid with detector readback data and other information, and the integrated image data is cached in DDR memory. Traditional imaging systems follow a timing sequence of first controlling detector integration, then receiving the detector image, and finally outputting the image via a high-speed data transmission interface. The advantage of this design is its simplicity in timing design and memory control. Figure 3 As shown. To improve the frame rate, this system employs a pipelined design for the imaging timing, outputting the previous frame's detector image via a high-speed data transmission interface while the detector is integrating. The timing design is as follows: Figure 4As shown, during the first frame synchronization generated internally, the exposure signal of the first frame is output to the CMOS to control the exposure of the first frame image. During the second frame synchronization generated internally, the image of the first frame after exposure is output from the CMOS to the FPGA and stored in DDR_1 outside the FPGA. At the same time, the exposure signal of the second frame is output to the CMOS to control the exposure of the second frame image. During the third frame synchronization generated internally, the image of the first frame after exposure is read from DDR_1 and output through the high-speed serial data transmission interface. At the same time, the exposure signal of the third frame is output to the CMOS to control the exposure of the third frame image. The image of the second frame after exposure is output from the CMOS to the FPGA and stored in DDR_2 outside the FPGA, and so on. Due to the pipelined design, it is necessary to read out the previous frame pixel and output it through the high-speed data transmission interface while storing the detector pixel. Therefore, two DDRs are designed to store the output images of the detector for the previous and next frames respectively. While storing the current frame image data, the image data of the previous frame is read out and output, making the imaging process pipelined.

[0049] The data transmission outgoing module reads image data from the DDR and sends it to the data transmission receiving subsystem via a high-speed serial interface.

[0050] This invention employs a single FPGA to deploy a high frame rate output system. The FPGA must contain at least six GTH channels, capable of simultaneously receiving images from all six channels of the detector, with each channel having a receiving rate of 3.125 Gbps. Figure 1 As shown, the FPGA has the following interfaces for external connections:

[0051] Second pulse interface: The bus communication module receives GPS second pulse signals from the satellite platform via RS422 bus (master / backup) for time management;

[0052] 422 Bus Interface: The bus communication module uses A and B dual asynchronous serial communication buses to exchange information, receive remote control commands from the satellite platform, and return telemetry information;

[0053] CMOS control signal interface: including exposure drive interface and I2C configuration interface. The CMOS configuration readback module configures the CMOS through the I2C configuration interface, changing the CMOS operating mode (HDR mode or non-HDR mode), gain, windowing, etc. The CMOS detector control module drives the CMOS through the exposure drive interface.

[0054] CMOS data interface: The image data processing and caching module receives high-speed CMOS data signals through the CMOS data interface, with a pixel output rate of 3.125Gbps;

[0055] DDR interface: The image data processing and caching module stores and reads image data after CMOS imaging through the DDR interface;

[0056] High-speed data transmission interface: The data transmission output module outputs CMOS image data after arranging it through the high-speed data transmission interface;

[0057] FLASH interface: FLASH is used to store the working parameters required for CMOS imaging. The CMOS configuration readback module reads the working parameters through the FLASH interface.

[0058] Refresh chip interface: The bus communication module receives remote control commands and controls the refresh chip to dynamically refresh and load programs on the FPGA through the refresh chip interface.

[0059] The ultra-large CMOS array with a pixel output rate of 3.125GHz receives 6 channels of high-speed data from the detector through the high-speed serial port of the FPGA, enabling dynamic configuration and readback of the CMOS registers. The image data output by the CMOS undergoes 2*2 binning and windowing processing. The processed pixels are buffered by DDR ping-pong and finally output line by line through the high-speed serial interface, achieving a high frame rate of 200ms. This enables the capture of 20m resolution image information from geostationary orbit at 36,000km.

[0060] The method for imaging using the high-frequency frame output system proposed in this invention is as follows: Figure 2 As shown, it includes:

[0061] Step S1: Internal frame synchronization generation: The bus communication module generates an internal frame synchronization signal with a variable period (minimum 200ms) according to the remote control command, which is used to trigger the exposure drive of the CMOS detector, the storage and reading of detector pixels.

[0062] Step S2: According to the remote control command to switch imaging mode from standby mode, the CMOS configuration readback module starts to configure and read back the CMOS registers. The CMOS configuration readback is completed through the I2C bus, and the configuration rate is 100KHz.

[0063] Step S3: After configuration, based on the internal frame synchronization signal, the CMOS detector control module outputs the detector exposure timing control signal to control the CMOS to start exposure;

[0064] Step S4: After exposure, the CMOS starts to output serial data. The image data processing and caching module uses the high-speed serial interface inside the FPGA to receive 6 channels of serial pixel data, each channel 3.125GHz, and converts it into parallel 14-bit quantized pixels.

[0065] Step S5: The image data processing and caching module selects whether to perform binning processing on the 14-bit quantized pixels in a 2*2 manner according to the remote control command;

[0066] Step S6: The CMOS configuration readback module configures the detector according to the selection of full-frame image mode or arbitrary windowing mode based on the remote control command. If it is arbitrary windowing mode, the remote control command code specifies that the detector starts to open the window from the nth row and opens the window for m rows. The CMOS configuration readback module sends the detector windowing command to the detector, and the detector outputs the image from the nth row to the n+m-1th row.

[0067] Step S7: The image data processing and caching module stores the windowed image data into DDR_1 according to the internal frame synchronization signal;

[0068] Step S8: The image data processing and caching module reads pixel data row by row from DDR_2, and selects full-frame or arbitrary window size pixel output according to the remote control command. The remote control command specifies that the output starts from column p and outputs column q. After reading a row of data, the image data processing and caching module extracts and outputs pixels from column p to column p+q-1.

[0069] Step S9: The data transmission module finally sends the image processed by the image data processing and caching module out through the high-speed serial data transmission interface.

[0070] The embodiments described above are merely preferred embodiments of the present invention. Ordinary variations and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A high frame rate output system based on a large-area CMOS array, characterized in that, It includes a bus communication module, a CMOS configuration readback module, a CMOS detector control module, a CMOS detector, an image data processing and caching module, and a data transmission output module; The bus communication module communicates with the satellite platform via the bus, receives standby mode to imaging mode conversion instructions and remote control instructions sent by the satellite platform, forwards the standby mode to imaging mode conversion instructions to the CMOS configuration readback module, and forwards the remote control instructions to the CMOS configuration readback module and the image data processing and caching module; generates internal frame synchronization signals according to the remote control instructions and sends them to other modules; The CMOS configuration readback module receives the instruction to switch from standby mode to imaging mode, configures the detector and sends the configuration data to the CMOS detector control module, then reads back the data of each register of the CMOS detector and outputs it to the image data processing and caching module; it receives remote control instructions, sends remote control instruction codewords to the image data processing and caching module, converts the remote control instruction codewords into I2C configuration codewords and sends them to the CMOS detector. The CMOS detector control module receives configuration data, sends exposure timing control signals to the CMOS detector, and controls the CMOS detector to image. The CMOS detector performs imaging based on the exposure timing control signal and outputs corresponding image data according to the remote control command code. The image data processing and caching module receives image data output from the CMOS detector, performs channel integration and arrangement on the image data, and caches the processed image data splicing register data into DDR according to the remote control command codeword. The data transmission output module reads image data from the DDR and sends it to the external data transmission receiving subsystem via a serial interface.

2. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The image data processing and caching module is equipped with two DDR chips and adopts a ping-pong cache design to store the output images of the detector in the previous and next frames respectively. While storing the image data of the current frame, it reads out the image data of the previous frame and outputs it.

3. The high frame rate output system based on a large-area CMOS array according to claim 2, characterized in that, During the first frame synchronization, the CMOS detector control module outputs the exposure signal of the first frame to the CMOS detector to control the exposure of the first frame image. During the second frame synchronization, the image of the first frame after exposure is output from the CMOS detector to the image data processing and buffering module. At the same time, the CMOS detector control module outputs the exposure signal of the second frame to the CMOS detector to control the exposure of the second frame image. When the third frame synchronization is generated internally, the data transmission output module reads the processed image of the first frame from a DDR chip and outputs it through the serial data transmission interface. At the same time, the CMOS detector control module outputs the exposure signal of the third frame to the CMOS detector to control the exposure of the third frame image. The image of the second frame after exposure is output from the CMOS detector to the image data processing and buffering module. The image data processing and buffering module stores the processed image data in another DDR chip, and so on, to realize the imaging timing pipeline design.

4. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The system is deployed on an FPGA, which contains at least 6 GTH channels and has external interfaces including but not limited to the following: 422 bus interface: used for the bus communication module to receive remote control commands from the satellite platform; CMOS control signal interface: used by the CMOS configuration readback module for CMOS configuration and by the CMOS detector control module for CMOS driving; CMOS data interface: used by the image data processing and buffering module to receive CMOS image data; DDR interface: used for image data processing and caching modules to store and retrieve processed image data; Data transmission interface: used for the data transmission output module to output the final image data.

5. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The CMOS detector control module receives configuration data and sends exposure timing control signals to the CMOS detector, including the CMOS detector PowerDown signal, reset signal, and exposure drive signal.

6. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The CMOS detector receives the I2C configuration codeword converted from the remote control command. If the I2C configuration codeword is in full-frame image mode, then all image data is output. If the I2C configuration codeword specifies that the detector should start opening a window from the nth row and open a window for m rows, then the image data from the nth row to the (n+m-1)th row is output.

7. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The image data processing and caching module receives remote control command codes. If the remote control command code is in full-frame image mode, the processed image data splicing register data is cached in DDR. If the remote control command code specifies that the output starts from column p and outputs column q, the image data processing and caching module reads one line of data from the processed image and then extracts the image data splicing register data from column p to column p+q-1 and caches it in DDR.

8. The high frame rate output system based on a large-area CMOS array according to claim 1, characterized in that, The CMOS configuration readback module transmits data via the I2C bus, with a configuration rate of not less than 100KHz.

9. The method for imaging based on the large-area CMOS high frame rate output system as described in claim 1, characterized in that, include: Step S1: The bus communication module generates an internal frame synchronization signal with a variable period according to the control command, which is used to trigger the exposure drive of the CMOS detector, the storage and reading of detector pixels; Step S2: After receiving the remote control command to switch from standby mode to imaging mode, the CMOS configuration readback module begins to configure and read back the CMOS registers. Step S3: After configuration, based on the internal frame synchronization signal, the CMOS detector control module outputs the detector's exposure timing control signal to control the CMOS detector to start exposure; Step S4: After exposure, the CMOS detector outputs serial data, and the image data processing and buffering module simultaneously receives serial image data from each channel and converts it into parallel image data. Step S5: The image data processing and caching module selects whether to perform binning processing on the image data according to the remote control command; Step S6: The CMOS configuration readback module configures the detector according to the selection of full-frame image mode or arbitrary windowing mode based on the remote control command. If it is arbitrary windowing mode, the remote control command code specifies that the detector starts to open the window from the nth row, opens the window for m rows, and the detector outputs the image from the nth row to the n+m-1th row. Step S7: The image data processing and caching module stores and processes image data according to the internal frame synchronization signal; Step S8: The image data processing and caching module selects full-frame or arbitrary window size pixel output according to the remote control command. The remote control command specifies that the output starts from column p and outputs column q. After reading a line of data, the image data processing and caching module extracts pixels from column p to column p+q-1. Step S9: The data transmission module sends the image processed by the image data processing and caching module out through the serial data transmission interface.

Images