A pixel-based merging space-borne high-frame-rate visible camera imaging system and method

Abstract

The application discloses a kind of based on image element merging spaceborne high frame frequency visible camera imaging system and method, it is related to spaceborne hyperspectral imaging technical field, main purpose is to solve the image data merging efficiency low of existing visible hyperspectral imaging system and the problem of image element size matching under different needs.This system mainly includes main control, FPGA, visible light detector and on-board real-time processing unit;Control command is sent to FPGA by main control, and system state information is received back by FPGA;Visible light detector is driven by FPGA, image data returned by visible light detector is cached, arranged and merged based on multilevel first-in-first-out array structure, and image data after caching, arranging and merging is sent to on-board real-time processing unit;Image data after caching, arranging and merging is processed by on-board real-time processing unit, and image feature data for subsequent image processing is obtained.Processing high-speed LVDS image data is mainly used.

Description

Technical Field

[0001] This invention relates to the field of spaceborne hyperspectral imaging technology, and in particular to an imaging system and method for a spaceborne high frame rate visible camera based on pixel merging. Background Technology

[0002] Visible hyperspectral imaging systems are a novel type of remote sensing system that integrates multiple disciplines, and have become one of the most important space-based Earth observation technologies in the world today. Based on its sophisticated spectroscopic optical system and unique imaging method, it can passively sense the high-resolution spectral and spatial geometric features of targets in a scene, capture subtle differences in objects in the visible and near-infrared bands, and provide a wealth of spectral information, which is helpful for the analysis of the spectral characteristics of materials. It has wide applications in agriculture, geology, environmental monitoring and other fields.

[0003] Pixel combining is a widely used technique in image sensors, primarily used to improve the signal-to-noise ratio and enhance imaging performance in low-light conditions. In addition, the combined "superpixel" has a larger effective photosensitive area, capturing more photon signals and significantly improving the detector's optical sensitivity. Simultaneously, it reduces the amount of data generated for the image, decreasing data transmission and processing time, which is particularly important in high frame rate applications.

[0004] Currently, visible hyperspectral imaging system architectures based on CMOS image sensors that employ voltage-domain merging require multiple readouts or parallel readouts, followed by merging in the analog domain or, after analog-to-digital conversion, in the digital domain. This architecture primarily relies on the sensor's hardware readout circuitry; each pixel still needs to be read out, and the multiple steps of analog-to-digital conversion and averaging require time. Often, when merging pixels, the increase in frame rate is far less than the resolution decrease caused by merging pixels, resulting in image data merging efficiency that cannot meet the data reception rate requirements of high-speed LVDS image data. Summary of the Invention

[0005] In view of this, the present invention provides an imaging system and method for a spaceborne high frame rate visible camera based on pixel merging, the main purpose of which is to solve the problems of low image data merging efficiency and pixel size matching under different requirements in existing visible hyperspectral imaging systems.

[0006] According to one aspect of the present invention, a high frame rate visible camera imaging system based on pixel merging is provided, comprising: a main controller, an FPGA, a visible light detector and an on-board real-time processing unit, wherein the FPGA includes an RS422 communication module, a detector driver module, a data buffering and merging module and a GTX image transmission module;

[0007] The main controller is communicatively connected to the FPGA and is used to send control commands to the FPGA and receive system status information returned by the FPGA.

[0008] The FPGA is communicatively connected to the visible light detector, and is used to receive control commands sent by the master controller through the RS422 communication module, and to transmit system status information back to the master controller; it configures detector driving parameters through the detector driving module, and drives the visible light detector through the SPI interface according to the detector driving parameters; it receives the image transmitted back by the visible light detector through the LVDS interface of the data buffering, orchestration and merging module, performs pixel data buffering, orchestration and merging operations on the image, and sends the buffered, orchestrated and merged image data to the on-board real-time processing unit;

[0009] The on-board real-time processing unit is used to perform feature processing on the cached, arranged and merged image data to obtain image feature data for subsequent image processing.

[0010] The data cache orchestration and merging module includes a four-layer first-in-first-out array and a divider. The four-layer first-in-first-out array includes a first-level first-in-first-out array layer, a second-level first-in-first-out array layer, a third-level first-in-first-out array layer and a fourth-level first-in-first-out array layer.

[0011] The first-level first-in-first-out array layer is used to receive two rows of image data written simultaneously at a preset time interval, and to work with the second-level first-in-first-out array layer to complete the merging of pixels in the row direction.

[0012] The second-level first-in-first-out array layer is used in conjunction with the third-level first-in-first-out array layer to complete the pixel merging of the row-merged image pixel data in the column direction, and to cache the image data that has completed pixel merging in both the row and column directions;

[0013] The divider is used to perform an averaging operation on the image data that has been merged in the row and column directions to obtain the image data after buffering, arrangement and merging.

[0014] The fourth-level first-in-first-out array layer is used to read and cache the cached, arranged and merged image data in the third-level first-in-first-out array layer when the cache space of the third-level first-in-first-out array layer is full.

[0015] Wherein, both the cell merging in the row direction and the cell merging in the column direction are odd-numbered cell merging.

[0016] Furthermore, the first-level first-in-first-out array layer includes a top-level array and a bottom-level array that equally distribute the number of first-level first-in-first-out arrays; the second-level first-in-first-out array layer includes at least 6 second-level first-in-first-out arrays; and the third-level first-in-first-out array layer includes at least 2 third-level first-in-first-out arrays.

[0017] The top-level array is used to receive and cache the odd-numbered rows of image data that are written, and the bottom-level array is used to receive and cache the even-numbered rows of image data that are written.

[0018] The second-level first-in-first-out array layer is used to read cached image data from the top-level array and the bottom-level array in groups of two second-level first-in-first-out arrays respectively, and to merge the read pixel data in the row direction when the number of cached pixels in the top-level array or the bottom-level array meets the preset merging number, and to cache the row merged image data after the row direction pixel merging is completed.

[0019] The three-level first-in-first-out array layer is used to read the row-merged image data in each of the two-level first-in-first-out arrays when the number of the two-level first-in-first-out arrays that cache the row-merged image data meets the preset merging number, and to merge the read pixel data in the column direction, and to cache the image data that has completed the pixel merging in the row and column directions.

[0020] The first-level FIFO array layer includes 50 first-level FIFO arrays with a depth of 248, the second-level FIFO array layer includes 6 second-level FIFO arrays with a depth of 2048, the third-level FIFO array layer includes 2 third-level FIFO arrays with a depth of 2048, and the fourth-level FIFO array layer includes 1 fourth-level FIFO array with a depth of 4096.

[0021] According to another aspect of the present invention, an imaging method based on a pixel-merging spaceborne high frame rate visible camera is provided, comprising:

[0022] The main controller sends control commands to the FPGA and receives system status information returned by the FPGA. The FPGA includes an RS422 communication module, a detector driver module, a data buffer, orchestration and merging module, and a GTX image transmission module.

[0023] The system receives control commands sent by the master controller via the RS422 communication module and transmits system status information back to the master controller; it configures detector driving parameters via the detector driving module and drives the visible light detector via the SPI interface based on the detector driving parameters; it receives images transmitted back from the visible light detector via the LVDS interface of the data buffering, orchestration, and merging module, performs pixel data buffering, orchestration, and merging operations on the images, and transmits the buffered, orchestrated, and merged image data to the on-board real-time processing unit via the GTX image transmission module.

[0024] The data caching, orchestration, and merging module in the FPGA performs pixel data caching, orchestration, and merging operations on the image, including:

[0025] Based on the first-level first-in-first-out array layer in the data cache orchestration and merging module, two rows of image data written simultaneously are received at a preset time interval, and pixel merging in the row direction is completed in conjunction with the second-level first-in-first-out array layer.

[0026] Based on the second-level first-in-first-out array layer in the data caching and orchestration merging module, the third-level first-in-first-out array layer jointly completes the pixel merging of the row-merged image pixel data in the column direction, and caches the image data that has completed pixel merging in both the row and column directions.

[0027] Based on the divider, the average operation is performed on the image data after pixel merging in the row and column directions to obtain the image data after buffering, arrangement and merging.

[0028] Based on the four-level first-in-first-out array layer in the data cache orchestration and merging module, when the cache space of the three-level first-in-first-out array layer is full, the cached, orchestrated and merged image data in the three-level first-in-first-out array layer is read and cached.

[0029] Both the cell merging in the row direction and the cell merging in the column direction are odd-number cell merging;

[0030] The on-board real-time processing unit performs feature processing on the cached, orchestrated, and merged image data to obtain image feature data for subsequent image processing.

[0031] Furthermore, the data caching, orchestration, and merging module based on the FPGA performs pixel data caching, orchestration, and merging operations on the image, specifically including:

[0032] The top-level array receives and caches the written odd-numbered rows of image data, and the bottom-level array receives and caches the written even-numbered rows of image data.

[0033] Through the two-level first-in-first-out array layer, the cached image data is read from the top-level array and the bottom-level array in groups of two two-level first-in-first-out arrays respectively. When the number of cached pixels in the top-level array or the bottom-level array meets the preset merging number, the read pixel data is merged in the row direction, and the row merged image data after the row direction pixel merging is completed is cached.

[0034] Through the three-level first-in-first-out array layer, when the number of the second-level first-in-first-out arrays that cache the row-merged image data meets the preset merging quantity, the row-merged image data in each second-level first-in-first-out array is read, and the read pixel data is merged in the column direction, and the image data that has completed the pixel merging in the row and column directions is cached.

[0035] The first-level FIFO array layer includes 50 first-level FIFO arrays with a depth of 248, the second-level FIFO array layer includes 6 second-level FIFO arrays with a depth of 2048, the third-level FIFO array layer includes 2 third-level FIFO arrays with a depth of 2048, and the fourth-level FIFO array layer includes 1 fourth-level FIFO array with a depth of 4096.

[0036] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0037] This invention provides an imaging system and method for a high-frame-rate visible light camera onboard via pixel merging. In this embodiment, the main controller sends control commands to an FPGA and receives system status information from the FPGA. The FPGA drives the visible light detector, performing buffering, arrangement, and merging processing on the image data transmitted from the detector based on a multi-level first-in-first-out array structure. The buffered, arranged, and merged image data is then sent to the onboard real-time processing unit. The onboard real-time processing unit performs feature processing on the buffered, arranged, and merged image data to obtain image feature data for subsequent image processing. This transforms the existing hardware-based pixel merging process into a hardware-software combined pixel merging process. This allows the merging level and the number of image data rows processed simultaneously to match the format and resolution of the high-speed LVDS image data transmitted from the detector, significantly improving the flexibility and efficiency of the image data and thus better meeting the data reception rate requirements of high-speed LVDS image data.

[0038] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. Furthermore, in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 This diagram illustrates a block diagram of a satellite-borne high frame rate visible camera imaging system based on pixel merging, according to an embodiment of the present invention.

[0041] Figure 2 This invention provides a block diagram of another satellite-borne high frame rate visible camera imaging system based on pixel merging, according to an embodiment of the present invention.

[0042] Figure 3 This illustration shows a flowchart of a pixel merging process based on a four-level structure provided by an embodiment of the present invention;

[0043] Figure 4 The diagram illustrates a flowchart of an imaging method for a high-frame-rate visible camera on a satellite based on pixel merging, according to an embodiment of the present invention. Detailed Implementation

[0044] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0045] To address the issues of low image data merging efficiency and pixel size matching under varying requirements in existing visible hyperspectral imaging systems, this invention provides a pixel-merging-based spaceborne high-frame-rate visible camera imaging system, such as... Figure 1 As shown, the system includes: a main controller 110, an FPGA 120, a visible light detector 130, and an on-board real-time processing unit 140;

[0046] The main controller 110 is communicatively connected to the FPGA 120, used to send control commands to the FPGA 120 and receive system status information returned by the FPGA 120; the FPGA 120 is communicatively connected to the visible light detector 130, used to drive the visible light detector 130, and to perform buffering, arrangement and merging processing on the image data returned by the visible light detector 130 based on a multi-level first-in-first-out array structure, and to send the buffered, arranged and merged image data to the on-board real-time processing unit 140; the on-board real-time processing unit 140 is used to perform feature processing on the buffered, arranged and merged image data to obtain image feature data for subsequent image processing.

[0047] In this embodiment of the invention, a hardware and software architecture for receiving LVDS image data in a high-frame-rate visible light camera imaging system with pixel merging is built based on a Field-Programmable Gate Array (FPGA 120). This allows the FPGA 120 to drive the visible light detector 130 under the control of the main controller, performing pixel data buffering, arrangement, and merging processing on the image data transmitted back from the visible light detector 130. The pixel data buffering, arrangement, and merging processing includes pixel data buffering, arrangement, and merging processing. The buffered, arranged, and merged image data is then transmitted to the onboard real-time processing unit 140, whereby the real-time processing unit performs in-depth processing on the buffered, arranged, and merged image data, specifically including feature extraction and compression encoding, to obtain image feature data for subsequent image processing. This image feature data can be used in various application scenarios such as ground feature recognition, environmental monitoring, and disaster assessment, providing scientific support for resource management and decision-making.

[0048] It should be noted that existing image data arrangement and merging processing is mainly based on integrated circuits specifically built for the data transmission channels and parameters of the current visible light detector 130. This is only compatible with visible light detectors of the corresponding specifications, and the number of channels for image data arrangement and merging operations is limited by the hardware composition of the integrated circuit, resulting in low data processing efficiency and low compatibility with the visible light detector 130. This solution utilizes a multi-level First-In-First-Out (FIFO) array structure in the FPGA 120 to perform buffering, arrangement, and merging processing on images transmitted from the visible light detector 130. Regardless of the detector's speed, resolution, or data format, the adaptation requirements can be addressed by adjusting the software logic within the FPGA 120, providing stable and efficient image data reception and processing capabilities for various visible light detectors 130 in various application scenarios. Furthermore, it ensures the compatibility and scalability of the current system architecture.

[0049] Among them, such as Figure 2 As shown, the FPGA includes an RS422 communication module, a detector driver module, a data buffering, orchestration, and merging module, and a GTX image transmission module. The FPGA receives control commands sent by the master controller through the RS422 communication module and transmits system status information back to the master controller. The detector driver module is used to configure the detector driving parameters and drive the visible light detector through the SPI interface according to the detector driving parameters. The data buffering, orchestration, and merging module is used to receive images transmitted back from the visible light detector through the LVDS interface, perform pixel data buffering, orchestration, and merging operations on the images, and send the buffered, orchestrated, and merged image data to the on-board real-time processing unit through the GTX image transmission module.

[0050] In this embodiment of the invention, the main controller interacts with the FPGA via an RS422 communication module, responsible for issuing commands and receiving status feedback to achieve precise control and status monitoring of the system. The FPGA, as the core of the system, is directly connected to the detector and undertakes the configuration and control tasks of the detector, completing the detector's initialization configuration and real-time drive through dedicated signals. The initialization configuration includes setting parameters such as the working mode, exposure time, and data format. The detector transmits the acquired image data back to the FPGA through multiple high-speed LVDS (Low-Voltage Differential Signaling) interfaces. The high-speed and low-power characteristics of LVDS ensure a data transmission rate of 10Gbps while maintaining signal integrity and reducing the bit error rate. After receiving the detector data, the FPGA internally caches the data through a high-speed caching mechanism and uses orchestration and merging algorithms to perform timing reordering and format merging of the multi-channel pixel data, i.e., caching, orchestration, and merging processing, to reduce the complexity of subsequent processing. The processed image data is transmitted to the on-board real-time processing unit through the FPGA's GTX (GeForce Technology) high-speed serial interface of the image transmission module. The above system architecture adopts a modular and standardized design. The interface between the FPGA and the detector supports rapid adaptation to different detector models, enabling the system to meet various detector application scenarios based on LVDS, and has extremely high scalability and flexibility.

[0051] The data caching, orchestration, and merging module includes a four-layer FIFO array and a divider. The four-layer FIFO array comprises a first-level FIFO array layer, a second-level FIFO array layer, a third-level FIFO array layer, and a fourth-level FIFO array layer. The first-level FIFO array layer receives two rows of image data written simultaneously at preset time intervals and, together with the second-level FIFO array layer, completes pixel merging in the row direction. The second-level FIFO array layer, together with the third-level FIFO array layer, completes pixel merging of the merged image data in the row direction and caches the image data after pixel merging in both the row and column directions. The divider performs an averaging operation on the image data after pixel merging in both the row and column directions to obtain the cached, orchestrated, and merged image data. The fourth-level FIFO array layer, when the cache space of the third-level FIFO array layer is full, reads and caches the cached, orchestrated, and merged image data from the third-level FIFO array layer.

[0052] In this embodiment of the invention, during the image data caching and arrangement process, cell merging in the row direction is completed between the first-level FIFO layer and the second-level FIFO layer, and cell merging in the column direction is completed between the second-level FIFO layer and the third-level FIFO layer. After completing the cell merging in the row direction, the average operation is performed by dividing by the product of the number of rows and columns of the cell matrix using the divider IP core. Both the row-direction and column-direction cell merging are odd-numbered cell merging. In existing cell merging processes based on hardware integrated circuits, merging odd-numbered cells is more difficult, so even-numbered cells are usually merged. By using a four-layer FIFO array, odd-numbered cell merging can be achieved during the cell merging process, meeting the needs of more image data merging application scenarios. When merging 3×3 odd-numbered cells, the average operation is performed by dividing by 9 using the divider IP core. For merging cells of any other size, it can be easily achieved by simply adjusting the number of different levels of FIFO in the FIFO structure.

[0053] In one embodiment of the present invention, for further explanation and limitation, the first-level first-in-first-out array layer includes a top-level array and a bottom-level array that equally distribute the number of first-level first-in-first-out arrays, the second-level first-in-first-out array layer includes at least 6 second-level first-in-first-out arrays, and the third-level first-in-first-out array layer includes at least 2 third-level first-in-first-out arrays.

[0054] The system consists of three layers: a top-level array for receiving and buffering odd-numbered rows of image data, a bottom-level array for receiving and buffering even-numbered rows of image data, and a second-level FIFO array layer for reading buffered image data from the top-level and bottom-level arrays in pairs. If the number of buffered pixels in either the top-level or bottom-level array meets a preset merging requirement, the layer merges the read pixel data in the row direction and buffers the merged image data. A third-level FIFO array layer, provided the number of second-level FIFO arrays containing merged row image data meets a preset merging requirement, reads merged row image data from each second-level FIFO array, merges the read pixel data in the column direction, and buffers the buffered, arranged, and merged image data after column-direction pixel merging.

[0055] In this embodiment of the invention, since the visible light detector outputs two lines of image data simultaneously within one row time, the first-level FIFO array layer receives the two lines of image data simultaneously input through the top-level array and the bottom-level array, respectively. For example, the top-level array receives image data from odd-numbered rows, and the bottom-level array receives image data from even-numbered rows. After receiving the image data, the image data written in each of the first-level FIFO arrays in the top-level array is spliced ​​in the row direction according to a preset merging number, and the spliced ​​image data is cached in one of the second-level FIFO arrays in the second-level FIFO array layer. The image data in the bottom-level array is spliced ​​in the same way as the top-level array, and the spliced ​​image data is cached in another second-level FIFO array in the second-level FIFO array layer. At the same time, the first-level FIFO array writes the next two lines of image data. The preset merging number corresponds to the size of the merging matrix. For example, if the preset merging number is 3, then every three pixels are merged into one in the row direction and every three pixels are merged into one in the column direction. Therefore, after receiving the first two rows of input image data and completing row merging, the number of image data columns cached in the second-level FIFO array layer is still insufficient to meet the merging requirements. Therefore, it is necessary to wait for subsequent input image data to be merged in rows and cached in the third and fourth second-level FIFO arrays so that the number of second-level FIFO arrays with cached image data meets the preset merging quantity. Then, pixel merging in the column direction is performed, and the image data after completing the cache, arrangement, and merging of the column direction pixel merging is cached in one of the third-level FIFO array layers.

[0056] The first-level FIFO array layer includes 50 first-level FIFO arrays with a depth of 248, the second-level FIFO array layer includes 6 second-level FIFO arrays with a depth of 2048, the third-level FIFO array layer includes 2 third-level FIFO arrays with a depth of 2048, and the fourth-level FIFO array layer includes 1 fourth-level FIFO array with a depth of 4096.

[0057] In this embodiment of the invention, a visible light detector with a data output format of 6144×126 16-bit pixel data and a maximum operating frame rate of 900Hz is used as an example to illustrate the pixel buffer arrangement and merging process. This visible light detector has an image width of 6144×6144 pixels and outputs two rows of pixel data simultaneously within one line time, i.e., 2×6144 pixels. It utilizes 50 channels (25 top channels and 25 bottom channels) for cross-output data, including some useless pixels. The two rows of pixel data are output through 25 channels each, with odd-numbered rows (rows 1, 3, 5, etc.) output through 25 top channels and even-numbered rows (rows 2, 4, 6, etc.) output through 25 bottom channels. The pixel merging process is based on a four-level structure, as follows: Figure 3 As shown. At the start of the first row time, two rows of image data are simultaneously written to 50 level-one FIFOs. When the first row time ends and the second row time begins, i.e., when the position of the first vertical line in the image is reached, the following two steps will be performed simultaneously:

[0058] Step a: The first-level FIFO layer is filled with 248 valid data points. Taking the top 25 channels, i.e., the first row of image data, as an example, the read enable signals of FIFO1-1 (Note: FIFOX-Y represents the Yth FIFO of the Xth level FIFO), FIFO1-2, FIFO1-3, etc., are sequentially pulled high to form a complete 6200-pixel image (containing 56 invalid data points, which can be discarded by the transmitting module). During this process, the first-level FIFO layer reads 3 pixels and places them into 3 registers for buffering. When each register has buffered 3 pixels, the write enable of the second-level FIFO layer is pulled high, and the merged result of every three pixels in the same row is written to FIFO2-1, completing the pixel merging in the row direction. At the same time, because the detector outputs two rows of image data simultaneously in one row time, the second row of image data also completes the pixel merging in the row direction, and the result is also stored in FIFO2-2.

[0059] Step b: Simultaneously, two new rows of image data are output, and 50 primary FIFOs are writing image data to the 3rd and 4th rows. After writing 248 valid data entries, the write operations for FIFO2-3 and FIFO2-4 are completed as in step a. After the pixel merging in the row direction of the 3rd row of image data is completed and cached in 2048 positions of a secondary FIFO, the first three secondary FIFOs have met the preset merging quantity in the column direction. Therefore, the image data of the 1st, 2nd, and 3rd rows are merged in the column direction. That is, the read enable of FIFO2-1, FIFO2-2, and FIFO2-3 are simultaneously raised, and the sum of the three numbers is read and sent to the divider for division by 9 and average operation to obtain the cached, arranged, and merged image data. The cached, arranged, and merged image data is then cached in the first tertiary FIFO. The same process applies to the pixel data in rows 4, 5, and 6. The difference is that rows 1, 2, and 3 cache the data from rows 1 and 2, waiting for row 3 to arrive before merging them and storing the result in the first three-level FIFO. Rows 4, 5, and 6 cache the data from row 4, waiting for row 5 and 6 to arrive before merging them and storing the result in the second three-level FIFO.

[0060] Steps a and b above demonstrate the merging process of 6 rows of image data. The entire pixel merging process is carried out in a cycle, which is to merge the row and column pixels of the 6 rows of image data into two buffers, arrange and merge the image data. When the third-level FIFO is full of 2048 image data, the image data is buffered into the fourth-level FIFO.

[0061] This invention provides a pixel-merging-based onboard high-frame-rate visible camera imaging system. In this embodiment, the main controller sends control commands to an FPGA and receives system status information from the FPGA. The FPGA drives the visible light detector, performing buffering, arrangement, and merging processing on the image data transmitted from the detector using a multi-level first-in-first-out array structure. The buffered, arranged, and merged image data is then sent to the onboard real-time processing unit. The onboard real-time processing unit performs feature processing on the buffered, arranged, and merged image data to obtain image feature data for subsequent image processing. This transforms the existing hardware-based pixel merging process into a hardware-software combined pixel merging process. This allows the merging level and the number of image data rows processed simultaneously to match the format and resolution of the high-speed LVDS image data transmitted from the detector, significantly improving the flexibility and efficiency of the image data and thus better meeting the data reception rate requirements of high-speed LVDS image data.

[0062] Furthermore, as a response to the above Figure 1The implementation of the method shown in this embodiment of the invention provides an imaging method based on pixel merging for a high-frame-rate visible camera on a satellite, such as... Figure 4 As shown, the method includes:

[0063] 201. The main controller sends control commands to the FPGA and receives system status information returned by the FPGA.

[0064] 202. The visible light detector is driven by the FPGA, and the image data returned by the visible light detector is cached, arranged and merged based on the multi-level first-in-first-out array structure. The cached, arranged and merged image data is then sent to the on-board real-time processing unit.

[0065] 203. The on-board real-time processing unit performs feature processing on the cached, arranged and merged image data to obtain image feature data for subsequent image processing.

[0066] The step of buffering, arranging, and merging the image data transmitted back from the visible light detector based on a multi-level first-in-first-out array structure includes:

[0067] The system receives control commands sent by the master controller through the RS422 communication module in the FPGA and transmits system status information back to the master controller.

[0068] Configure detector driving parameters based on the detector driving module in the FPGA, and drive the visible light detector through the SPI interface according to the detector driving parameters;

[0069] Based on the data caching, orchestration, and merging module in the FPGA, the image transmitted back by the visible light detector is received through the LVDS interface. The image is then cached, orchestrated, and merged with pixel data. The cached, orchestrated, and merged image data is then transmitted to the on-board real-time processing unit through the GTX image transmission module.

[0070] The data caching, orchestration, and merging module in the FPGA performs pixel data caching, orchestration, and merging operations on the image, including:

[0071] Based on the first-level first-in-first-out array layer in the data cache orchestration and merging module, two rows of image data written simultaneously are received at a preset time interval, and pixel merging in the row direction is completed in conjunction with the second-level first-in-first-out array layer.

[0072] Based on the second-level first-in-first-out array layer in the data caching and orchestration merging module, the third-level first-in-first-out array layer jointly completes the pixel merging of the row-merged image pixel data in the column direction, and caches the image data that has completed pixel merging in both the row and column directions.

[0073] Based on the divider, the average operation is performed on the image data after pixel merging in the row and column directions to obtain the image data after buffering, arrangement and merging.

[0074] Based on the four-level first-in-first-out array layer in the data cache orchestration and merging module, when the cache space of the three-level first-in-first-out array layer is full, the cached, orchestrated and merged image data in the three-level first-in-first-out array layer is read and cached.

[0075] Wherein, both the cell merging in the row direction and the cell merging in the column direction are odd-numbered cell merging.

[0076] Furthermore, the data caching, orchestration, and merging module based on the FPGA performs pixel data caching, orchestration, and merging operations on the image, specifically including:

[0077] The top-level array receives and caches the written odd-numbered rows of image data, and the bottom-level array receives and caches the written even-numbered rows of image data.

[0078] Through the two-level first-in-first-out array layer, the cached image data is read from the top-level array and the bottom-level array in groups of two two-level first-in-first-out arrays respectively. When the number of cached pixels in the top-level array or the bottom-level array meets the preset merging number, the read pixel data is merged in the row direction, and the row merged image data after the row direction pixel merging is completed is cached.

[0079] Through the three-level first-in-first-out array layer, when the number of the second-level first-in-first-out arrays that cache the row-merged image data meets the preset merging quantity, the row-merged image data in each second-level first-in-first-out array is read, and the read pixel data is merged in the column direction, and the image data that has completed the pixel merging in the row and column directions is cached.

[0080] The first-level FIFO array layer includes 50 first-level FIFO arrays with a depth of 248, the second-level FIFO array layer includes 6 second-level FIFO arrays with a depth of 2048, the third-level FIFO array layer includes 2 third-level FIFO arrays with a depth of 2048, and the fourth-level FIFO array layer includes 1 fourth-level FIFO array with a depth of 4096.

[0081] This invention provides an imaging method for a high-frame-rate visible light camera on a satellite based on pixel merging. In this embodiment, the main controller sends control commands to an FPGA and receives system status information returned by the FPGA. The FPGA drives the visible light detector, performing buffering, arrangement, and merging processing on the image data returned by the detector based on a multi-level first-in-first-out array structure. The buffered, arranged, and merged image data is then sent to the on-board real-time processing unit. The on-board real-time processing unit performs feature processing on the buffered, arranged, and merged image data to obtain image feature data for subsequent image processing. This transforms the existing hardware-based pixel merging process into a hardware-software combined pixel merging process. This allows the merging level and the number of image data rows processed simultaneously to match the format and resolution of the high-speed LVDS image data returned by the detector, greatly improving the flexibility and efficiency of the image data and thus better meeting the data reception rate requirements of high-speed LVDS image data.

[0082] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An imaging system based on a pixel-merging spaceborne high frame rate visible camera, characterized in that, include: The system includes a main control unit, an FPGA, a visible light detector, and an on-board real-time processing unit. The FPGA includes an RS422 communication module, a detector driver module, a data buffering and merging module, and a GTX image transmission module. The main controller is communicatively connected to the FPGA and is used to send control commands to the FPGA and receive system status information returned by the FPGA. The FPGA is communicatively connected to the visible light detector, and is used to receive control commands sent by the master controller through the RS422 communication module, and to transmit system status information back to the master controller; it configures detector driving parameters through the detector driving module, and drives the visible light detector through the SPI interface according to the detector driving parameters; it receives the image transmitted back by the visible light detector through the LVDS interface of the data buffering, orchestration and merging module, performs pixel data buffering, orchestration and merging operations on the image, and sends the buffered, orchestrated and merged image data to the on-board real-time processing unit; The on-board real-time processing unit is used to perform feature processing on the cached, arranged and merged image data to obtain image feature data for subsequent image processing. The data cache orchestration and merging module includes a four-layer first-in-first-out array and a divider. The four-layer first-in-first-out array includes a first-level first-in-first-out array layer, a second-level first-in-first-out array layer, a third-level first-in-first-out array layer and a fourth-level first-in-first-out array layer. The first-level first-in-first-out array layer is used to receive two rows of image data written simultaneously at a preset time interval, and to work with the second-level first-in-first-out array layer to complete the merging of pixels in the row direction. The second-level first-in-first-out array layer is used in conjunction with the third-level first-in-first-out array layer to complete the pixel merging of row-merged image data in the column direction, and to cache the image data that has completed pixel merging in both the row and column directions. The divider is used to perform an averaging operation on the image data that has been merged in the row and column directions to obtain the image data after buffering, arrangement and merging. The fourth-level first-in-first-out array layer is used to read and cache the image data after buffering, arrangement and merging in the third-level first-in-first-out array layer when the cache space of the third-level first-in-first-out array layer is full. Wherein, both the cell merging in the row direction and the cell merging in the column direction are odd-number cell merging; The first-level first-in-first-out array layer includes a top-level array and a bottom-level array with an evenly distributed number of first-level first-in-first-out arrays; the second-level first-in-first-out array layer includes at least 6 second-level first-in-first-out arrays; and the third-level first-in-first-out array layer includes at least 2 third-level first-in-first-out arrays. The top-level array is used to receive and cache the odd-numbered rows of image data that are written, and the bottom-level array is used to receive and cache the even-numbered rows of image data that are written. The second-level first-in-first-out array layer is used to read cached image data from the top-level array and the bottom-level array in groups of two second-level first-in-first-out arrays respectively, and to merge the read pixel data in the row direction when the number of cached pixels in the top-level array or the bottom-level array meets the preset merging number, and to cache the row merged image data after the row direction pixel merging is completed. The three-level first-in-first-out array layer is used to read the row-merged image data in each of the two-level first-in-first-out arrays when the number of the two-level first-in-first-out arrays that cache the row-merged image data meets the preset merging number, and to merge the read pixel data in the column direction, and to cache the image data that has completed the pixel merging in the row and column directions. The first-level FIFO array layer includes 50 first-level FIFO arrays with a depth of 248, the second-level FIFO array layer includes 6 second-level FIFO arrays with a depth of 2048, the third-level FIFO array layer includes 2 third-level FIFO arrays with a depth of 2048, and the fourth-level FIFO array layer includes 1 fourth-level FIFO array with a depth of 4096.

2. An imaging method based on pixel merging for a satellite-borne high frame rate visible camera, wherein the method is applied to the satellite-borne high frame rate visible camera imaging system based on pixel merging as described in claim 1, characterized in that, include: The main controller sends control commands to the FPGA and receives system status information returned by the FPGA. The FPGA includes an RS422 communication module, a detector driver module, a data buffer, orchestration and merging module, and a GTX image transmission module. The system receives control commands sent by the master controller via the RS422 communication module and transmits system status information back to the master controller; it configures detector driving parameters via the detector driving module and drives the visible light detector via the SPI interface based on the detector driving parameters; it receives images transmitted back from the visible light detector via the LVDS interface of the data buffering, orchestration, and merging module, performs pixel data buffering, orchestration, and merging operations on the images, and transmits the buffered, orchestrated, and merged image data to the on-board real-time processing unit via the GTX image transmission module. The data caching, orchestration, and merging module in the FPGA performs pixel data caching, orchestration, and merging operations on the image, including: Based on the first-level first-in-first-out array layer in the data cache orchestration and merging module, two rows of image data written simultaneously are received at a preset time interval, and pixel merging in the row direction is completed in conjunction with the second-level first-in-first-out array layer. Based on the second-level first-in-first-out array layer in the data caching orchestration and merging module, the third-level first-in-first-out array layer is used to complete the pixel merging of row-merged image data in the column direction, and to cache the image data that has completed pixel merging in both the row and column directions. Based on the divider, the average operation is performed on the image data after pixel merging in the row and column directions to obtain the image data after buffering, arrangement and merging. Based on the four-level first-in-first-out array layer in the data cache orchestration and merging module, when the cache space of the three-level first-in-first-out array layer is full, the cached, orchestrated and merged image data in the three-level first-in-first-out array layer is read and cached. Both the cell merging in the row direction and the cell merging in the column direction are odd-number cell merging; The on-board real-time processing unit performs feature processing on the cached, orchestrated, and merged image data to obtain image feature data for subsequent image processing.

Images