Infrared area array detector image multi-mode fusion system

Abstract

The invention discloses an infrared area array detector image multi-mode fusion system which comprises a clock and reset control module, an exposure adjustment control module, an infrared area array detector driving module, an access control module, an image fusion processing module and a data transmission module. According to the invention, the FPGA is utilized to realize image fusion imaging software system design under fixed frame frequency and variable frame frequency of the area array infrared detector, two groups of DDR3 memories with high reliability and large capacity are utilized to realize multi-frame image caching of the high-speed large area array infrared detector, and the frame frequency, the exposure time and the exposure times are dynamically adjusted in real time through an instruction; function modes are automatically judged, and different fusion modes are switched in real time. The target image capability is improved, compatibility of different requirements of image fusion of the array infrared detector on the satellite is solved and realized, the function expansibility of a camera system is promoted and improved, and the application efficiency is improved.

Description

Technical Field

[0001] The invention relates to an infrared surface array detector image multi-mode fusion system, belonging to the technical field of on-board data comprehensive processing and application. Background Art

[0002] Image fusion technology is a technology that processes and calculates image data from a unified target source to obtain high-quality target information to the maximum extent. In the aerospace field, infrared array detectors are increasingly widely used due to their high-intensity and excellent performance.

[0003] By continuously shooting the array remote sensing detectors at a high frame rate along the on-orbit push-sweep direction and performing high frame rate continuous exposure imaging, multiple frames of highly overlapping images of the same ground object can be obtained, thereby obtaining richer raw data. Based on the images of the same scene, after image fusion, a response curve with a large dynamic range can be obtained, and the target image with a high signal-to-noise ratio can be output.

[0004] Faced with different working orbit parameters, different working modes, attitude control capabilities of different satellite platforms and other conditions, the traditional form of image fusion processing after push-frame imaging is relatively simple, that is, image data acquisition and fusion processing are completed with a fixed frame rate and a fixed number of exposures. The application conditions are relatively simple and fixed, and the exposure time and number, as well as the image data to be processed cannot be selected and adjusted more accurately, which is slightly insufficient in more complex application scenarios. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the deficiencies of the prior art and to provide an infrared array detector image multi-mode fusion system.

[0006] The technical solution of the present invention is:

[0007] The present invention discloses an infrared surface array detector image multi-mode fusion system, comprising: a clock and reset control module, an exposure adjustment control module, an infrared surface array detector drive module, an access control module, an image fusion processing module and a data transmission module; wherein,

[0008] The clock and reset control module receives the reset signal and reference clock signal input from the external pin and generates a local synchronous reset signal and a working clock;

[0009] The exposure adjustment control module generates the number of exposures, the integral time code value corresponding to each exposure, the imaging control signal, the state register value and the auxiliary data according to the external bus control instruction; after judging the validity of the imaging control signal, the number of exposures and the integral time code value, it sends them to the infrared array detector driving module; and sends the imaging control signal, the state register value and the auxiliary data to the data transmission module;

[0010] The infrared array detector driving module divides the working clock sent by the clock and reset control module, and outputs a pixel clock signal to the infrared array detector; generates an integral time signal according to the pixel clock signal, the exposure times sent by the exposure adjustment control module, and the integral time code value, and sends it to the infrared array detector to control the infrared array detector to complete the exposure; performs complete line arrangement processing on the original image data output by the infrared array detector to generate image data; and sends the generated image data and the imaging control signal sent by the exposure adjustment control module to the access control module;

[0011] The access control module includes two groups of storage units; according to the imaging control signal, the generated image data is stored in the BRAM storage area by adopting a ping-pong storage method;

[0012] The image fusion processing module analyzes the fusion times according to the external bus control instruction; reads the generated image data from the BRAM storage area according to the fusion times, performs fusion processing, and sends the fused image data to the data transmission module;

[0013] The data transmission module arranges the status register value and the auxiliary data with the fused image data sent by the image fusion processing module according to the imaging control signal sent by the exposure adjustment control module, generates the arranged image data, and outputs it to the outside through the data transmission interface.

[0014] Furthermore, in the above fusion system, the ping-pong storage method is specifically: according to the imaging control signal, the ping-pong mark is triggered, the image data in the BRAM storage area is written into one group of storage units, and the image data in the other two groups of storage units are read and transmitted to the image fusion processing module.

[0015] Furthermore, in the above fusion system, the array image data is processed by row arrangement and stored in a storage area. The specific method is: according to the channel sequence of the infrared array detector output image, the received array image data of each channel are arranged in sequence into complete rows and stored in the storage area.

[0016] Furthermore, in the above fusion system, the validity judgment of the imaging control signal, the number of exposures and the integrated time code value is specifically as follows:

[0017] According to the imaging control signal, the current imaging cycle is obtained;

[0018] Calculate the total length of exposure times and integration time;

[0019] Determine whether the total time is less than or equal to the current imaging cycle; if so, it is determined to be valid data, and the imaging control signal, integral time code value and exposure times are output; otherwise, it is determined to be invalid data.

[0020] Furthermore, in the above fusion system, the integration time signal is generated according to the pixel clock signal, the number of exposures sent by the exposure adjustment control module and the integration time code value, and the specific method is as follows:

[0021] According to the cycle of the pixel clock signal, a count value of the counter is generated;

[0022] According to the integrated time code value, the pixel clock number value is obtained;

[0023] When the count value of the counter is less than or equal to the number of pixel clocks, a high or low level signal that meets the working requirements of the detector is generated, which is the integration time signal.

[0024] Furthermore, in the above fusion system, according to the imaging control signal sent by the exposure adjustment control module, the state register value and the auxiliary data are arranged with the fused image data sent by the image fusion processing module, and the specific method is as follows:

[0025] At each rising edge of the imaging control signal, the auxiliary data and the status register value are collected to obtain the collected data;

[0026] The collected data are arranged in the order of collection, and framed with the image data to form the compiled image data.

[0027] Furthermore, in the above fusion system, the fusion processing is performed by:

[0028] S61, dividing the storage area into a first partition Bram1 and a second partition Bram2;

[0029] S62, storing the nth row data of the i-th frame of the image to be fused into a partition Bram1; wherein i≤N, n≤M, N is the maximum number of frames; M is the maximum number of rows per frame;

[0030] S63, when the n-row data of the i+1-th frame arrives, the image data in the first partition Bram1 is read out, and the corresponding pixel data in the n-row data of the i+1-th frame is added and averaged, and then merged into a frame of image, which is stored in the second partition Bram2;

[0031] S64, when the n-th row data of the i+2-th frame arrives, the image data in the second partition Bram2 is read out, added to the n-th row data of the i+2-th frame and averaged, and then merged into one frame of image, which is stored in the first partition Bram1;

[0032] S65. Repeat steps S62 to S64 until the fusion times are reached or the fusion processing of all image data is completed.

[0033] Furthermore, in the above fusion system, the number of fusions is less than or equal to the number of exposures.

[0034] Furthermore, in the above-mentioned fusion system, the external bus control instruction controls the image fusion mode to be on or off; if the image fusion mode is off, the data transmission module outputs the original image data sent from the infrared array detector; if the image fusion mode is on, the data transmission module outputs the fused image data.

[0035] The beneficial effects of the present invention and the prior art are:

[0036] (1) The present invention uses FPGA to implement the image fusion imaging software system design under fixed frame rate and variable frame rate of the array infrared detector, and uses two sets of high-reliability and large-capacity DDR3 storage ping-pong cache to realize high-speed large array infrared detector multi-frame image cache, and dynamically adjusts the frame rate, exposure time, and exposure times through instructions in real time, automatically distinguishes the function mode, and switches between different fusion modes in real time. While improving the target image capability, it solves and realizes the compatibility of different requirements for image fusion of satellite array infrared detectors, promotes the functional scalability of the camera system, and improves the application efficiency;

[0037] (2) The present invention can realize multi-frame image fusion or original image output in multi-frame rate and multi-exposure modes, and switch between modes in real time, with high controllability and more in line with actual application effects;

[0038] (3) The two groups of DDR3 storage ping-pong caches of the present invention realize massive image data caching, flexible image frame selectivity, and no data access conflict;

[0039] (4) The system of the present invention has high overall controllability, high compatibility, and is easy to expand, which is conducive to software architecture and product development.

[0040] (5) The present invention uses FPGA to implement the design of the infrared array detector image fusion imaging software system, making full use of on-chip storage resources and external memory to complete image fusion and download in real time, saving ground processing time, and dynamically adjusting the exposure and fusion modes through instructions. The functional module automatically identifies the fusion switch, fusion mode, completion mark, and image selection. The system is more in line with the actual application environment, and the functions are easy to expand and develop, which helps to improve the overall architecture and productization of the software. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the system composition and relationship diagram of the present invention;

[0042] Figure 2 It is a schematic diagram of the exposure mode of the detector of the present invention;

[0043] Figure 3 It is a schematic diagram of the fusion processing flow of the present invention;

[0044] Figure 4 It is a schematic diagram of the detector configuration of the present invention;

[0045] Figure 5 It is a schematic diagram of the fusion operation process of the present invention;

[0046] Figure 6 It is a schematic diagram of the data transmission module of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The present invention discloses an infrared surface array detector image multi-mode fusion system, comprising: a clock and reset control module, an exposure adjustment control module, an infrared surface array detector drive module, an access control module, an image fusion processing module and a data transmission module; wherein,

[0049] The clock and reset control module receives the reset signal and reference clock signal input from the external pin and generates a local synchronous reset signal and a working clock;

[0050] The exposure adjustment control module generates the number of exposures, the integral time code value corresponding to each exposure, the imaging control signal, the state register value and the auxiliary data according to the external bus control instruction; after judging the validity of the imaging control signal, the number of exposures and the integral time code value, it sends them to the infrared array detector driving module; and sends the imaging control signal, the state register value and the auxiliary data to the data transmission module;

[0051] The infrared array detector driving module divides the working clock sent by the clock and reset control module, and outputs a pixel clock signal to the infrared array detector; generates an integral time signal according to the pixel clock signal, the exposure times sent by the exposure adjustment control module, and the integral time code value, and sends it to the infrared array detector to control the infrared array detector to complete the exposure; performs complete line arrangement processing on the original image data output by the infrared array detector to generate image data; and sends the generated image data and the imaging control signal sent by the exposure adjustment control module to the access control module;

[0052] The access control module includes two groups of storage units; according to the imaging control signal, the generated image data is stored in the BRAM storage area by adopting a ping-pong storage method;

[0053] The image fusion processing module analyzes the fusion times according to the external bus control instruction; reads the generated image data from the BRAM storage area according to the fusion times, performs fusion processing, and sends the fused image data to the data transmission module;

[0054] The data transmission module, according to the imaging control signal sent by the exposure adjustment control module, arranges the status register value and the auxiliary data with the fused image data sent by the image fusion processing module, generates arranged image data, and outputs it to the outside through the data transmission interface. Figure 6 shown.

[0055] Preferably, the ping-pong storage method is specifically as follows: according to the imaging control signal, a ping-pong mark is triggered, the image data in the BRAM storage area is written into one group of storage units, and the image data in the other two groups of storage units are read and transmitted to the image fusion processing module.

[0056] Preferably, the array image data is processed by row arrangement and stored in a storage area. The specific method is: according to the channel sequence of the infrared array detector output image, the received array image data of each channel is arranged into complete rows in sequence and stored in the storage area.

[0057] Preferably, the effectiveness of the imaging control signal, the number of exposures and the integrated time code value is judged as follows:

[0058] According to the imaging control signal, the current imaging cycle is obtained;

[0059] Calculate the total length of exposure times and integration time;

[0060] Determine whether the total time is less than or equal to the current imaging cycle; if so, it is determined to be valid data, and the imaging control signal, integral time code value and exposure times are output; otherwise, it is determined to be invalid data.

[0061] Preferably, an integral time signal is generated according to the pixel clock signal, the number of exposures sent by the exposure adjustment control module and the integral time code value, and the specific method is as follows:

[0062] According to the cycle of the pixel clock signal, a count value of the counter is generated;

[0063] According to the integrated time code value, the pixel clock number value is obtained;

[0064] When the count value of the counter is less than or equal to the number of pixel clocks, a high or low level signal that meets the working requirements of the detector is generated, which is the integration time signal.

[0065] Preferably, according to the imaging control signal sent by the exposure adjustment control module, the state register value and the auxiliary data are arranged with the fused image data sent by the image fusion processing module, and the specific method is as follows:

[0066] At each rising edge of the imaging control signal, the auxiliary data and the status register value are collected to obtain the collected data;

[0067] The collected data are arranged in the order of collection, and framed with the image data to form the compiled image data.

[0068] Preferably, a fusion process is performed, and the specific method is as follows:

[0069] S61, dividing the storage area into a first partition Bram1 and a second partition Bram2;

[0070] S62, storing the nth row data of the i-th frame of the image to be fused into a partition Bram1; wherein i≤N, n≤M, N is the maximum number of frames; M is the maximum number of rows per frame;

[0071] S63, when the n-row data of the i+1-th frame arrives, the image data in the first partition Bram1 is read out, and the corresponding pixel data in the n-row data of the i+1-th frame is added and averaged, and then merged into a frame of image, which is stored in the second partition Bram2;

[0072] S64, when the n-th row data of the i+2-th frame arrives, the image data in the second partition Bram2 is read out, added to the n-th row data of the i+2-th frame and averaged, and then merged into one frame of image, which is stored in the first partition Bram1;

[0073] S65. Repeat steps S62 to S64 until the fusion times are reached or the fusion processing of all image data is completed.

[0074] Preferably, the number of fusions is less than or equal to the number of exposures.

[0075] Preferably, the external bus control instruction controls the image fusion mode to be turned on or off; if the image fusion mode is off, the data transmission module outputs the original image data sent from the infrared array detector; if the image fusion mode is on, the data transmission module outputs the fused image data.

[0076] Example

[0077] First, the detector is exposed multiple times according to the imaging control signal, and multiple frames of images are sequentially stored in DDR3. The division of the storage area and the corresponding relationship between the images under different exposure times are fixed and adjusted by instructions. When the image fusion mode is turned on, the image data frames that meet the requirements are read out row by row, and the superposition and averaging are completed pixel by pixel to form a frame of image; when the fusion mode is turned off, the original data of a certain frame of image can be selectively output and arranged and output through the data transmission module. During the data access process, two sets of DDR3 ping-pong operations are used to ensure the complete frame of the image and avoid image access conflicts. If the image fusion mode is turned off, you can choose to output a certain frame of original image data. The data processing flow is as follows: Figure 5 shown.

[0078] like Figure 1As shown, the present invention is an infrared array detector image multi-mode fusion system, which dynamically adjusts the detector drive timing of multiple frame rates and different exposure times through bus instructions, adjusts image storage selection and image fusion, and realizes real-time switching of different fusion modes. The system composition is as follows Figure 1 shown.

[0079] In the image fusion mode, firstly, the detector is exposed multiple times according to the imaging control signal, and the multiple frames of images are sequentially stored in the external DDR3 to ensure that the division of the storage area and the corresponding relationship between the images under different exposure times are fixed. Secondly, the multiple frames of images that meet the instruction requirements are read out, and the superposition and averaging are completed pixel by pixel, and then fused into one frame of image to complete the data transmission. When the mode is turned on, the two groups of DDR3 ping-pong operation avoids image access conflicts. Figure 2 shown.

[0080] The above functions are realized by the following modules: clock and reset control module, exposure adjustment control module, infrared array detector drive module, DDR3 access control module, image fusion processing module, data transmission module. The flow chart is as follows: Figure 3 shown.

[0081] 1) Clock and reset control module

[0082] Asynchronous reset, synchronous release. Receive external pin input reset signal and reference clock signal, generate local synchronous reset signal and working clock.

[0083] 2) Exposure adjustment control module

[0084] After reset is released, the control instruction of the bus is received, and the number of exposures and the integration time (exposure time) corresponding to each exposure are parsed. At the same time, it is determined whether the number of exposures and the exposure time meet the current imaging control signal cycle (frame rate). If they meet the requirements, a valid mark signal and the integration time and number of times are output; at the same time, according to the level signal of the external imaging control signal interface, it is determined whether to use the signal input from the external control signal interface as a global imaging control signal, or to generate a global imaging control signal autonomously and send it to each module to control the generation of related signals. In addition, the external imaging control signal or the autonomously generated imaging control signal cycle can be obtained through the bus control instruction (satisfying autonomous generation and external program control); the imaging control signal, the status register value parsed by other control instructions, and the broadcast auxiliary data are sent to the data transmission module;

[0085] 3) Infrared array detector driver module

[0086] After reset is released, under the control of the internal clock, the pixel clock of the infrared array detector is generated by frequency division, and the integration time signal that meets the requirements is generated according to the effective exposure times and integration time, and the detector is controlled to complete the exposure that meets the requirements. The timing is as follows: Figure 4 The image data output by the detector is received, and the data is processed and stored in the FPGA on-chip storage area (BRAM resource).

[0087] 4) DDR3 access control module

[0088] The state machine and counter are set up internally. By detecting the imaging control signal (synchronization signal), the ping-pong mark is triggered, the image data stored in the BRAM is written into the first group of DDR3, and the image data in the second group of DDR3 is read and transmitted to the image fusion processing module. In the process of reading the image, the address jump is controlled by the parsing instruction to complete the image frame selection and output.

[0089] 5) Image fusion processing module

[0090] The nth row data of the first frame of the image to be fused is directly stored in Bram1. When the nth row data of the second frame arrives, all the data previously stored in Bram are read out, and after adding to each pixel of the row data, the superimposed row data is stored in Bram2. When the third row arrives, the image of Bram2 is read out and added and stored in Bram1. Repeat the above process until the nth row of the Nth frame image is reached. What is finally stored in Bram is the image data of the same row after the accumulation of multiple exposure images. The accumulated data is read out to calculate the image mean parameters. During the fusion process, it automatically determines whether it is completed and the cut-off state. If it is completed or cut-off, the processed image is output, otherwise the fusion process continues. The specific operation process is as follows: Figure 5 In addition, if the fusion switch is turned off, the selected original image frame is output to meet the actual needs of original image transmission.

[0091] 6) Data transmission module

[0092] Receive the image data output by the image fusion processing module, arrange it with the auxiliary data, and output the image data through the data transmission interface. The process is as follows: Figure 5 shown.

[0093] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

[0094] The contents not described in detail in the specification of the present invention belong to the common knowledge of the professionals in this field.

Claims

1. An infrared array detector image multi-mode fusion system, characterized in that: include: Clock and reset control module, exposure adjustment control module, infrared array detector drive module, access control module, image fusion processing module and data transmission module; wherein, The clock and reset control module receives the reset signal and reference clock signal input from the external pin and generates a local synchronous reset signal and a working clock; The exposure adjustment control module generates the number of exposures, the integral time code value corresponding to each exposure, the imaging control signal, the state register value and the auxiliary data according to the external bus control instruction; after judging the validity of the imaging control signal, the number of exposures and the integral time code value, it sends them to the infrared array detector driving module; and sends the imaging control signal, the state register value and the auxiliary data to the data transmission module; The infrared array detector driving module divides the working clock sent by the clock and reset control module, and outputs a pixel clock signal to the infrared array detector; generates an integral time signal according to the pixel clock signal, the exposure times sent by the exposure adjustment control module, and the integral time code value, and sends it to the infrared array detector to control the infrared array detector to complete the exposure; performs complete line arrangement processing on the original image data output by the infrared array detector to generate image data; and sends the generated image data and the imaging control signal sent by the exposure adjustment control module to the access control module; The access control module includes two groups of storage units; according to the imaging control signal, the generated image data is stored in the BRAM storage area by adopting a ping-pong storage method; The image fusion processing module analyzes the fusion times according to the external bus control instruction; reads the generated image data from the BRAM storage area according to the fusion times, performs fusion processing, and sends the fused image data to the data transmission module; The data transmission module arranges the status register value and the auxiliary data with the fused image data sent by the image fusion processing module according to the imaging control signal sent by the exposure adjustment control module, generates the arranged image data, and outputs it to the outside through the data transmission interface.

2. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The ping-pong storage method is specifically as follows: according to the imaging control signal, the ping-pong mark is triggered, the image data in the BRAM storage area is written into one group of storage units, and the image data in the other two groups of storage units are read and transmitted to the image fusion processing module.

3. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The array image data is processed by row arrangement and stored in the storage area. The specific method is: according to the channel sequence of the infrared array detector output image, the received array image data of each channel is arranged into complete rows in sequence and stored in the storage area.

4. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The validity judgment of the imaging control signal, the number of exposures and the integrated time code value is specifically as follows: According to the imaging control signal, the current imaging cycle is obtained; Calculate the total length of exposure times and integration time; Determine whether the total time is less than or equal to the current imaging cycle; if so, it is determined to be valid data, and the imaging control signal, integral time code value and exposure times are output; otherwise, it is determined to be invalid data.

5. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The method of generating an integral time signal according to the pixel clock signal, the exposure times sent by the exposure adjustment control module and the integral time code value is as follows: According to the cycle of the pixel clock signal, a count value of the counter is generated; According to the integrated time code value, the pixel clock number value is obtained; When the count value of the counter is less than or equal to the number of pixel clocks, a high or low level signal that meets the working requirements of the detector is generated, which is the integration time signal.

6. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: According to the imaging control signal sent by the exposure adjustment control module, the status register value and the auxiliary data are arranged with the fused image data sent by the image fusion processing module. The specific method is as follows: At each rising edge of the imaging control signal, the auxiliary data and the status register value are collected to obtain the collected data; The collected data are arranged in the order of collection, and framed with the image data to form the compiled image data.

7. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The fusion process is specifically performed as follows: S61, dividing the storage area into a first partition Bram1 and a second partition Bram2; S62, storing the nth row data of the i-th frame of the image to be fused into a partition Bram1; wherein i≤N, n≤M, N is the maximum number of frames; M is the maximum number of rows per frame; S63, when the n-row data of the i+1-th frame arrives, the image data in the first partition Bram1 is read out, and the corresponding pixel data in the n-row data of the i+1-th frame is added and averaged, and then merged into a frame of image, which is stored in the second partition Bram2; S64, when the n-th row data of the i+2-th frame arrives, the image data in the second partition Bram2 is read out, added to the n-th row data of the i+2-th frame and averaged, and then merged into one frame of image, which is stored in the first partition Bram1; S65. Repeat steps S62 to S64 until the fusion times are reached or the fusion processing of all image data is completed.

8. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The number of fusion times is less than or equal to the number of exposure times.

9. The infrared array detector image multi-mode fusion system according to claim 1, characterized in that: The external bus control instruction controls the image fusion mode to be turned on or off; if the image fusion mode is off, the data transmission module outputs the original image data sent from the infrared array detector; if the image fusion mode is on, the data transmission module outputs the fused image data.

Images