An infrared array detector image multi-mode fusion system

Abstract

This invention discloses a multi-mode image fusion system for an infrared array detector, comprising: a clock and reset control module, an exposure adjustment control module, an infrared array detector driver module, an access control module, an image fusion processing module, and a data transmission module. This invention utilizes an FPGA to design an image fusion imaging software system for a large-area infrared array detector under both fixed and variable frame frequencies. It employs two sets of high-reliability, high-capacity DDR3 memory ping-pong buffers to achieve high-speed, multi-frame image caching for the large-area infrared array detector. The system dynamically adjusts the frame frequency, exposure time, and exposure count in real-time via commands, automatically identifies the functional mode, and switches between different fusion modes in real-time. While improving target image capabilities, this system addresses and achieves compatibility with different requirements for image fusion from satellite-based infrared array detectors, promoting enhanced camera system scalability and improving application efficiency.

Description

Technical Field

[0001] This invention relates to a multi-mode image fusion system for an infrared array detector, belonging to the field of on-board data integration and application technology. Background Technology

[0002] Image fusion technology is a technique that processes and calculates image data from a unified target source to obtain high-quality target information to the greatest extent possible. In the aerospace field, due to the superior performance of infrared array detectors, its application is becoming increasingly widespread.

[0003] By continuously capturing images at a high frame rate along the on-orbit pushbroom direction using an array remote sensing detector, and performing high-frame-rate continuous exposure imaging, multiple highly overlapping images of the same ground feature can be obtained, thus acquiring richer raw data. Based on images of the same scene, image fusion can yield a response curve with a large dynamic range, enabling the output of target images with a high signal-to-noise ratio.

[0004] Faced with different working orbit parameters, different working modes, and different satellite platform attitude control capabilities, the traditional form of image fusion processing after frame-pushing imaging is relatively simple, that is, image data acquisition and fusion processing is completed with a fixed frame rate and a fixed number of exposures. The application conditions are relatively simple and fixed, and it is not possible to select and adjust the exposure time and number of exposures and the image data to be processed more precisely, which is slightly insufficient in more complex application scenarios. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a multi-mode fusion system for infrared array detector images.

[0006] The technical solution of this invention is:

[0007] This invention discloses a multi-mode image fusion system for an infrared array detector, comprising: a clock and reset control module, an exposure adjustment control module, an infrared array detector driving module, an access control module, an image fusion processing module, and a data transmission module; wherein,

[0008] The clock and reset control module receives external pin input reset signals and reference clock signals, and generates local synchronous reset signals and operating clocks.

[0009] The exposure adjustment control module generates the number of exposures, the integration time code value corresponding to each exposure, the imaging control signal, the status register value, and auxiliary data according to the external bus control commands; after determining the validity of the imaging control signal, the number of exposures, and the integration time code value, it sends them to the infrared array detector driver module; and sends the imaging control signal, the status register value, and the auxiliary data to the data transmission module.

[0010] The infrared array detector driver module divides the clock signal from the working clock sent by the reset control module and outputs a pixel clock signal to the infrared array detector. Based on the pixel clock signal, the number of exposures and the integration time code value sent by the exposure adjustment control module, it generates an integration time signal and sends it to the infrared array detector to control the infrared array detector to complete the exposure. After performing complete row arrangement processing on the raw image data output by the infrared array detector, it generates image data. The generated image data and the imaging control signal sent by the exposure adjustment control module are sent to the access control module.

[0011] The access control module includes two sets of storage units; based on the imaging control signal, the generated image data is stored in the BRAM storage area using a ping-pong storage method.

[0012] The image fusion processing module parses the fusion count according to the external bus control command; based on the fusion count, it reads the generated image data from the BRAM storage area, performs fusion processing, and sends the fused image data to the data transmission module.

[0013] The data transmission module, based on the imaging control signal sent by the exposure adjustment control module, arranges the status register value and auxiliary data with the fused image data sent by the image fusion processing module to generate the arranged image data, which is then output to the outside through the data transmission interface.

[0014] Furthermore, in the above-mentioned fusion system, the ping-pong storage method specifically involves: triggering a ping-pong marker according to an imaging control signal, writing the image data from the BRAM storage area into one set of storage units, and reading the image data from the other two sets of storage units, and transmitting them to the image fusion processing module.

[0015] Furthermore, in the above-mentioned fusion system, the row arrangement processing of the array image data and its storage in the storage area is specifically performed as follows: according to the channel order of the output image of the infrared array detector, the received array image data of each channel are arranged into a complete row in sequence and stored in the storage area.

[0016] Furthermore, in the aforementioned fusion system, the determination of the validity of the imaging control signal, the number of exposures, and the integration time code value specifically involves:

[0017] The current imaging period is obtained based on the imaging control signal;

[0018] Calculate the total time for the number of exposures and the integration time;

[0019] Determine if the total time is less than or equal to the current imaging cycle; if so, it is considered valid data, and the imaging control signal, integration time code value, and number of exposures are output; otherwise, it is considered invalid data.

[0020] Furthermore, in the aforementioned fusion system, the method for generating the integral time signal based on the pixel clock signal, the number of exposures sent by the exposure adjustment control module, and the integral time code value is as follows:

[0021] The counter value is generated based on the period of the pixel clock signal;

[0022] The number of pixel clocks is obtained based on the integral time code value;

[0023] When the counter's count value is less than or equal to the number of pixel clocks, a high or low level signal that meets the detector's operating requirements is generated, which is the integration time signal.

[0024] Furthermore, in the aforementioned fusion system, based on the imaging control signal sent by the exposure adjustment control module, the status register value and auxiliary data are arranged with the fused image data sent by the image fusion processing module. The specific method is as follows:

[0025] At each rising edge of the imaging control signal, auxiliary data and status register values ​​are acquired to obtain the acquired data;

[0026] The collected data is arranged in the order of collection and framed with the image data to form the edited image data.

[0027] Furthermore, in the aforementioned fusion system, the specific method for performing the fusion process is as follows:

[0028] S61. Divide the storage area into a first partition Bram1 and a second partition Bram2;

[0029] S62. Store the data of the nth row of the i-th frame of the image to be fused into a partition Bram1; where i≤N, n≤M, N is the maximum number of frames, and M is the maximum number of rows in each frame.

[0030] S63. When the data of the nth row of the (i+1)th frame arrives, the image data in the first partition Bram1 is read out, and the average of each pixel in the data of the nth row of the (i+1)th frame is added to it and then merged into a frame image and stored in the second partition Bram2.

[0031] S64. When the data of the nth row of the (i+2)th frame arrives, the image data in the second partition Bram2 is read out, added to the data of the nth row of the (i+2)th frame, averaged, and then merged into a single frame image, which is then stored in the first partition Bram1.

[0032] S65. Repeat steps S62 to S64 until the required number of fusions is reached or all image data fusion processing is completed.

[0033] Furthermore, in the above-mentioned fusion system, the number of fusions is less than or equal to the number of exposures.

[0034] Furthermore, in the above-mentioned fusion system, the external bus control command controls the image fusion mode to be turned on or off; if the image fusion mode is off, the data transmission module outputs the original image data sent from the infrared array detector; if the image fusion mode is on, the data transmission module outputs the fused image data.

[0035] The advantages of this invention over the prior art are as follows:

[0036] (1) This invention utilizes FPGA to design an image fusion imaging software system for a large-area infrared detector under both fixed and variable frame frequencies. It employs two sets of high-reliability, high-capacity DDR3 memory ping-pong buffers to achieve multi-frame image caching for the high-speed, large-area infrared detector. The frame frequency, exposure time, and exposure count are dynamically adjusted in real-time via instructions, and the function mode is automatically identified, allowing for real-time switching between different fusion modes. While enhancing target image capabilities, this invention addresses and achieves compatibility with different image fusion requirements of satellite-based large-area infrared detectors, promoting improved camera system functionality and application efficiency.

[0037] (2) The present invention can realize multi-frame image fusion or original image output under multi-frame frequency and multi-exposure modes, and the mode can be switched in real time. It has high controllability and is more in line with the actual application effect.

[0038] (3) The two sets of DDR3 memory ping-pong caches of the present invention realize massive image data caching, the selection of image frames is flexible, and data access does not conflict.

[0039] (4) The system of the present invention has high overall controllability, high compatibility, and is easy to expand, which is conducive to software architecture and product development.

[0040] (5) This invention utilizes FPGA to implement the design of an infrared array detector image fusion imaging software system, making full use of on-chip storage resources and external memory to complete image fusion and downlink in real time, saving ground processing time. The exposure and fusion modes are dynamically adjusted through instructions, and the functional modules automatically identify the fusion switch, fusion mode, completion flag, and image selection. The system is more in line with the actual application environment, and its functions are easy to expand and develop, which helps to improve the overall architecture and productization of the software. Attached Figure Description

[0041] Figure 1 This is a diagram showing the system composition and relationships of the present invention;

[0042] Figure 2 This is a schematic diagram of the detector exposure mode of the present invention;

[0043] Figure 3 This is a schematic diagram of the fusion processing flow of the present invention;

[0044] Figure 4 This is a schematic diagram of the detector configuration of the present invention;

[0045] Figure 5 This is a schematic diagram of the fusion operation process of the present invention;

[0046] Figure 6 This is a schematic diagram of the data transmission module of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] This invention discloses a multi-mode image fusion system for an infrared array detector, comprising: a clock and reset control module, an exposure adjustment control module, an infrared array detector driving module, an access control module, an image fusion processing module, and a data transmission module; wherein,

[0049] The clock and reset control module receives external pin input reset signals and reference clock signals, and generates local synchronous reset signals and operating clocks.

[0050] The exposure adjustment control module generates the number of exposures, the integration time code value corresponding to each exposure, the imaging control signal, the status register value, and auxiliary data according to the external bus control commands; after determining the validity of the imaging control signal, the number of exposures, and the integration time code value, it sends them to the infrared array detector driver module; and sends the imaging control signal, the status register value, and the auxiliary data to the data transmission module.

[0051] The infrared array detector driver module divides the clock signal from the working clock sent by the reset control module and outputs a pixel clock signal to the infrared array detector. Based on the pixel clock signal, the number of exposures and the integration time code value sent by the exposure adjustment control module, it generates an integration time signal and sends it to the infrared array detector to control the infrared array detector to complete the exposure. After performing complete row arrangement processing on the raw image data output by the infrared array detector, it generates image data. The generated image data and the imaging control signal sent by the exposure adjustment control module are sent to the access control module.

[0052] The access control module includes two sets of storage units; based on the imaging control signal, the generated image data is stored in the BRAM storage area using a ping-pong storage method.

[0053] The image fusion processing module parses the fusion count according to the external bus control instructions; based on the fusion count, it reads the generated image data from the BRAM storage area, performs fusion processing, and sends the fused image data to the data transmission module.

[0054] The data transmission module, based on the imaging control signal sent by the exposure adjustment control module, arranges the status register value and auxiliary data with the fused image data sent by the image fusion processing module to generate arranged image data, which is then output externally through the data transmission interface. For example... Figure 6 As shown.

[0055] Preferably, the ping-pong storage method specifically involves: triggering a ping-pong marker according to the imaging control signal, writing the image data from the BRAM storage area into one set of storage units, and reading the image data from the other two sets of storage units, and transmitting them to the image fusion processing module.

[0056] Preferably, the array image data is processed by row arrangement and stored in the storage area. Specifically, the received array image data of each channel is arranged into a complete row according to the channel order of the output image of the infrared array detector and stored in the storage area.

[0057] Preferably, the validity of the imaging control signal, the number of exposures, and the integration time code value is determined, specifically as follows:

[0058] The current imaging period is obtained based on the imaging control signal;

[0059] Calculate the total time for the number of exposures and the integration time;

[0060] Determine if the total time is less than or equal to the current imaging cycle; if so, it is considered valid data, and the imaging control signal, integration time code value, and number of exposures are output; otherwise, it is considered invalid data.

[0061] Preferably, an integral time signal is generated based on the pixel clock signal, the number of exposures sent by the exposure adjustment control module, and the integral time code value. The specific method is as follows:

[0062] The counter value is generated based on the period of the pixel clock signal;

[0063] The number of pixel clocks is obtained based on the integral time code value;

[0064] When the counter's count value is less than or equal to the number of pixel clocks, a high or low level signal that meets the detector's operating requirements is generated, which is the integration time signal.

[0065] Preferably, based on the imaging control signal sent by the exposure adjustment control module, the status register value and auxiliary data are arranged with the fused image data sent by the image fusion processing module. The specific method is as follows:

[0066] At each rising edge of the imaging control signal, auxiliary data and status register values ​​are acquired to obtain the acquired data;

[0067] The collected data is arranged in the order of collection and framed with the image data to form the edited image data.

[0068] Preferably, a fusion process is performed, specifically as follows:

[0069] S61. Divide the storage area into a first partition Bram1 and a second partition Bram2;

[0070] S62. Store the data of the nth row of the i-th frame of the image to be fused into a partition Bram1; where i≤N, n≤M, N is the maximum number of frames, and M is the maximum number of rows in each frame.

[0071] S63. When the data of the nth row of the (i+1)th frame arrives, the image data in the first partition Bram1 is read out, and the average of each pixel in the data of the nth row of the (i+1)th frame is added to it and then merged into a frame image and stored in the second partition Bram2.

[0072] S64. When the data of the nth row of the (i+2)th frame arrives, the image data in the second partition Bram2 is read out, added to the data of the nth row of the (i+2)th frame, averaged, and then merged into a single frame image, which is then stored in the first partition Bram1.

[0073] S65. Repeat steps S62 to S64 until the required number of fusions is reached or all image data fusion processing is completed.

[0074] Preferably, the number of fusions is less than or equal to the number of exposures.

[0075] Preferably, an external bus control command controls whether the image fusion mode is turned on or off; if the image fusion mode is off, the data transmission module outputs the original image data sent from the infrared array detector; if the image fusion mode is on, the data transmission module outputs the fused image data.

[0076] Example

[0077] First, the detector is exposed multiple times according to the imaging control signal, and multiple frames of images are sequentially stored in DDR3. The partitioning of the storage area corresponds to the images under different exposure times and is fixed and adjusted by instructions. When image fusion mode is enabled, the image data frames that meet the requirements are read out row by row, and the average is taken by overlaying each pixel to fuse them into a single image frame. When fusion mode is disabled, the raw data of a specific frame can be selectively output, and then arranged and output through the data transmission module. During the data access process, two sets of DDR3 ping-pong operations are used to ensure complete image frames and avoid image access conflicts. If image fusion mode is disabled, the raw image data of a specific frame can be selectively output. The data processing flow is as follows: Figure 5 As shown.

[0078] like Figure 1As shown, this invention discloses an infrared array detector image multi-mode fusion system. Through bus commands, it dynamically adjusts the driving timing of the detector's multi-frame rate and different exposure times, adjusts image storage selection and image fusion, and achieves real-time switching between different fusion modes. The system composition is as follows: Figure 1 As shown.

[0079] In image fusion mode, the detector is first exposed multiple times according to the imaging control signal, and multiple frames of images are sequentially stored in the external DDR3 memory to ensure that the memory area division corresponds to the images at different exposure times. Next, the multiple frames that meet the command requirements are read out, and the images are fused into a single frame by overlaying and averaging the pixels. When this mode is enabled, the two sets of DDR3 memory operate in a ping-pong manner to avoid image access conflicts. The detector's integral exposure imaging mode is as follows: Figure 2 As shown.

[0080] The above functions are implemented through the following modules: clock and reset control module, exposure adjustment control module, infrared array detector driver module, DDR3 access control module, image fusion processing module, and data transmission module. The flowchart is shown below. Figure 3 As shown.

[0081] 1) Clock and Reset Control Module

[0082] Asynchronous reset, synchronous release. Receives external pin input reset signal and reference clock signal, and generates local synchronous reset signal and operating clock.

[0083] 2) Exposure Adjustment Control Module

[0084] After reset and release, the system receives control commands from the bus, parses the number of exposures generated, and the integration time (exposure time) corresponding to each exposure. It also determines whether the number of exposures and the exposure time satisfy the current imaging control signal period (frame rate). If they do, it outputs a valid marker signal, integration time, and number of exposures. Simultaneously, based on the level signal of the external imaging control signal interface, it determines whether to use the signal input from the external control signal interface as the global imaging control signal, or to autonomously generate a global imaging control signal, and sends it to each module to control the generation of relevant signals. Furthermore, the period of either the external imaging control signal or the autonomously generated imaging control signal can be obtained through bus control commands (satisfying both autonomous generation and external programmable control). The system then sends the imaging control signal, the status register values ​​parsed from other control commands, and broadcast auxiliary data to the data transmission module.

[0085] 3) Infrared array detector drive module

[0086] After reset and release, under the control of the internal clock, the pixel clock of the infrared array detector is generated by frequency division, and an integration time signal that meets the requirements is generated according to the effective number of exposures and integration time, controlling the detector to complete the required exposure, as shown in the timing sequence. Figure 4 As shown. It receives image data output from the detector, processes and organizes the data, and stores it in the FPGA's on-chip memory area (BRAM resource).

[0087] 4) DDR3 access control module

[0088] Internally, a state machine and counter are set up. By detecting the imaging control signal (synchronization signal), a ping-pong marker is triggered, writing the image data stored in the BRAM into the first DDR3 group, and reading the image data from the second DDR3 group to transfer it to the image fusion processing module. During the image reading process, the address transitions controlled by the parsing instruction complete the image frame selection and output.

[0089] 5) Image fusion processing module

[0090] The first frame, nth row of data in the image to be merged is directly stored in Bram1. When the second frame, nth row of data arrives, all previously stored data in Bram1 is read out, added to the corresponding pixel data of the current row, and the superimposed row data is stored in Bram2. When the third row arrives, the image in Bram2 is read out, added, and then stored in Bram1. This process is repeated until the nth row of the Nth frame is reached. Bram1 ultimately stores the same row of image data accumulated from multiple exposures. The accumulated data is read out to calculate the image mean parameter. During the fusion process, the system automatically determines whether the process is complete or terminated. If it is complete or terminated, the processed image is output; otherwise, the fusion process continues. The specific operation process is as follows: Figure 5 As shown. Furthermore, if the fusion switch is turned on and off, the selected original image frame will be output to meet the actual needs of original image transmission.

[0091] 6) Data transmission module

[0092] The image data received from the image fusion processing module is combined with auxiliary data and then output through the data transmission interface. The process is as follows: Figure 5 As shown.

[0093] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

[0094] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. An infrared focal plane array detector image multi-modal fusion system, comprising: The application relates to a clock and reset control module, an exposure adjustment control module, an infrared area array detector driving module, an access control module, an image fusion processing module and a data transmission module. The clock and reset control module receives external pin input reset signals and reference clock signals, generates local synchronous reset signals and working clocks; The exposure adjustment control module generates exposure times, corresponding integral time code values of each exposure, imaging control signals, state register values and auxiliary data according to external bus control instructions; after validity judgment of the imaging control signals, the exposure times and the integral time code values, the exposure adjustment control module sends the imaging control signals, the exposure times and the integral time code values to the infrared area array detector driving module; the exposure adjustment control module sends the imaging control signals, the state register values and the auxiliary data to the data transmission module; The infrared area array detector driving module divides the working clock sent by the clock and reset control module, outputs pixel clock signals to the infrared area array detector, generates integral time signals according to the pixel clock signals, the exposure times and the integral time code values sent by the exposure adjustment control module, and sends the integral time signals to the infrared area array detector to control the infrared area array detector to complete exposure; the infrared area array detector driving module generates image data after complete row arrangement processing of original image data output by the infrared area array detector, and sends the generated image data and the imaging control signals sent by the exposure adjustment control module to the access control module; The access control module comprises two groups of storage units; the access control module stores the generated image data into a BRAM storage area in a ping-pong storage mode according to the imaging control signals; The image fusion processing module analyzes fusion times according to external bus control instructions, reads the generated image data from the BRAM storage area according to the fusion times, performs fusion processing, and sends the fused image data to the data transmission module; The data transmission module arranges the state register values and the auxiliary data and the fused image data sent by the image fusion processing module according to the imaging control signals sent by the exposure adjustment control module, generates arranged image data, and outputs the arranged image data to the outside through a data transmission interface; The ping-pong storage mode is specifically as follows: according to the imaging control signals, a ping-pong mark is triggered, image data in the BRAM storage area is written into one group of storage units, and image data in the other two groups of storage units is read out and transmitted to the image fusion processing module; The fusion processing is specifically as follows: S61, the storage area is divided into a first partition Bram1 and a second partition Bram2; S62, the i-th frame n-th row data of the image to be fused is stored in the first partition Bram1; wherein i<=N, n<=M, N is the maximum frame number, and M is the maximum row number of each frame image; S63, when the i+1-th frame n-th row data arrives, the image data in the first partition Bram1 is read out, added to and averaged with each pixel in the i+1-th frame n-th row data, fused into a frame image, and stored in the second partition Bram2; S64, when the i+2-th frame n-th row data arrives, the image data in the second partition Bram2 is read out, added to and averaged with the i+2-th frame n-th row data, fused into a frame image, and stored in the first partition Bram1. ​ S65, repeating steps S62-S64 until the fusion times are reached or the entire image data fusion processing is completed.

2. The infrared focal plane array detector image multi-mode fusion system according to claim 1, wherein, The image data output by the infrared area array detector is processed in row arrangement and stored in a storage area. The specific method is as follows: according to the channel order of the image output by the infrared area array detector, the received area array image data of each channel is arranged in complete rows in order and stored in the storage area.

3. The infrared focal plane array image multi-mode fusion system of claim 1, wherein, The imaging control signal, the exposure times and the integration time code value are judged for validity, and the specific method is as follows: The current imaging period is obtained according to the imaging control signal. The total length of the exposure times and the integration time is calculated. It is judged whether the total length is less than or equal to the current imaging period. If yes, it is judged as valid data, and the imaging control signal, the integration time code value and the exposure times are output. If not, it is judged as invalid data.

4. The infrared focal plane array image multi-mode fusion system of claim 1, wherein, The integration time signal is generated according to the pixel clock signal, the exposure times and the integration time code value sent by the exposure adjustment control module, and the specific method is as follows: The counting value of the counter is generated according to the period of the pixel clock signal. The pixel clock number value is obtained according to the integration time code value. When the counting value of the counter is less than or equal to the pixel clock number value, the high-low level signal meeting the working requirements of the detector is generated, which is the integration time signal.

5. The infrared focal plane array image multi-mode fusion system of claim 1, wherein, The state register value and the auxiliary data are arranged with the fused image data sent by the image fusion processing module according to the imaging control signal sent by the exposure adjustment control module, and the specific method is as follows: The auxiliary data and the state register value are collected to obtain the collection data at each rising edge of the imaging control signal. The collection data is arranged in the order of collection, and the image data is framed to form the arranged image data.

6. The infrared focal plane array image multi-mode fusion system of claim 1, wherein: The fusion times are less than or equal to the exposure times.

7. The infrared focal plane array image multi-mode fusion system of claim 1, wherein: The bus control instruction controls the image fusion mode to be on or off. If the image fusion mode is off, the data transmission module outputs the original image data sent by the infrared area array detector. If the image fusion mode is on, the data transmission module outputs the fused image data.

Images