Multispectral image data stream processing system based on FPGA

Abstract

The invention discloses a multispectral image data stream processing system based on an FPGA (Field Programmable Gate Array), which is provided with an FPGA data processing module comprising a plurality of spectral image information receiving sub-modules and a DVP (Digital Video Protocol) parsing sub-modules, and each DVP parsing sub-module receives and filters parallel spectral image information transmitted by one spectral image information receiving sub-module; the arbitration channel control sub-module is used for receiving all valid data filtered by the DVP analysis sub-modules, storing the valid data transmitted by each DVP analysis sub-module in blocks, and when the data size of each block storage area is larger than the maximum cache amount, sending the data to the arbitration channel control sub-module; and according to the polling priorities from high to low, outputting the effective data of each block storage area in sequence, and finally storing the effective data to a cache particle module. According to the multispectral image data stream processing system based on the FPGA, real-time parallel processing can be carried out on high-parallelism and high-throughput image stream data, and finally obtained spectral images are high in quality and obvious in feature.

Description

Technical Field

[0001] The present invention relates to the technical field of image acquisition, and in particular to a multi-spectral image data stream processing system based on FPGA. Background Art

[0002] With the development of science and technology, in order to obtain image information more comprehensively, special photographic equipment has been rapidly developing towards imaging with a wider field of view, higher resolution and more spatial information, providing an effective path for the realization of informatization. Among the many image acquisition devices, multispectral image acquisition devices use multiple spectral bands to simultaneously image the same object during operation, which can obtain both the image information and the spectral information of the target. It has the characteristics of stronger spectral resolution and higher spectral identification ability, and has gradually become a hot research direction in today's scientific research field.

[0003] The multi-spectral image acquisition device usually has multiple spectral image information acquisition units. When processing the acquired spectral image information, it is necessary to process the multiple spectral image information acquired by the spectral image information acquisition units in parallel. However, the existing multi-spectral image acquisition devices usually use embedded platforms such as STM32 to process the acquired information data. Although such embedded platforms are easy to develop and design, they are limited by the internal processor architecture and have insufficient processing capabilities for the high-parallelism and high-throughput image stream data formed when there are a large number of spectral image information acquisition units. This will result in the inability to process the data in real time and in parallel, which in turn leads to low quality and unclear features of the spectral image finally obtained. Summary of the invention

[0004] Based on this, it is necessary to provide a multispectral image data stream processing system based on FPGA to address the above technical problems, which can perform real-time parallel processing of high-parallel, high-throughput image stream data, and the final spectral image obtained has high quality and obvious features.

[0005] The present invention provides a multi-spectral image data stream processing system based on FPGA, which is provided with an FPGA data processing module, a downstream receiving module and a high-speed cache particle module;

[0006] FPGA data processing modules include:

[0007] A plurality of spectral image information receiving submodules, each of which receives parallel spectral image information collected by a spectral image collecting unit;

[0008] A plurality of DVP protocol parsing submodules equal in number to the spectral image information receiving submodules, each DVP protocol parsing submodule receiving and filtering parallel spectral image information transmitted by a spectral image information receiving submodule;

[0009] The arbitration channel control submodule is used to receive the valid data filtered by all DVP protocol parsing submodules, and store the valid data transmitted by each DVP protocol parsing submodule in blocks. When the amount of data in each block storage area is greater than the maximum cache amount, the valid data of each block storage area is output in sequence from high to low according to the polling priority;

[0010] The data cache submodule is used to receive and store the valid data of each block storage area output by the arbitration channel control submodule to the cache particle module;

[0011] The downstream communication submodule is used to extract and transmit the valid data stored in the data cache submodule to the downstream receiving module.

[0012] In one embodiment, the data cache submodule includes a ping-pong unit, the cache particle module includes multiple cache partitions, and the ping-pong unit is used to monitor each frame of valid data stored in the cache particle module by the data cache submodule and assist in selecting a cache partition for each frame of valid data.

[0013] In one embodiment, the arbitration channel control submodule includes a plurality of arbitration channel units equal in number to the DVP protocol parsing units, each arbitration channel unit is used to receive and store valid data transmitted by a DVP protocol parsing submodule, and store the stored valid data to the data cache submodule;

[0014] Each group of arbitration channel units includes an asynchronous FIFO subunit, an application arbitration subunit and a transmission channel switch subunit;

[0015] The asynchronous FIFO subunit is a block storage area used to temporarily store valid data transmitted by a DVP protocol parsing unit;

[0016] The arbitration application subunit is used to monitor the amount of cached data in the asynchronous FIFO subunits of the same group in real time, and when the amount of cached data is greater than the maximum cache amount of the asynchronous FIFO subunit, an opening signal is sent to the transmission channel switch subunit;

[0017] The transmission channel switch subunit is used to transmit the valid data temporarily stored in the same group of asynchronous FIFO subunits to the data cache submodule after receiving the start signal sent by the application arbitration subunit.

[0018] In one embodiment, the downstream communication submodule includes a high-speed Ethernet interface unit, an external storage medium control unit, and a display unit;

[0019] Downstream receiving modules include Ethernet, SD card and display screen;

[0020] The high-speed Ethernet interface unit is used to obtain all valid data in the high-speed cache particle module and send all valid data to the Ethernet in the form of a general protocol;

[0021] The external storage medium control subunit is used to obtain all valid data in the cache particle module and send all valid data to the SD card;

[0022] The display subunit is used to obtain all valid data in the cache particle module and send all valid data to the display screen.

[0023] In one embodiment, the FPGA data processing module further includes an acquisition configuration submodule, and the acquisition configuration submodule is used to sequentially configure parameters of registers inside the spectral image acquisition unit through the serial interface of the spectral image acquisition unit.

[0024] In one of the embodiments, the spectral image information receiving submodule is connected to the spectral image acquisition unit via an FFC flexible flat cable.

[0025] In one embodiment, the FPGA data processing module is an ALTERA-CYCLONE IV series FPGA processing chip with model number EP4CE30F23C7.

[0026] In one embodiment, the number of spectral image information receiving submodules is 2 to 15.

[0027] The beneficial effects of the present invention are as follows: the FPGA-based multi-spectral image data stream processing system of the present invention uses an FPGA data processing module to process the received parallel spectral image information collected by multiple spectral image acquisition units. During the processing, the arbitration channel control submodule can sequentially transmit the valid data of each block storage area in order from high to low according to the polling priority and finally store it in the cache particle module. The arbitration channel control submodule can control the orderly transmission of parallel data streams, so that the FPGA data processing module has a higher parallel data processing capability. In addition, in the present invention, the valid data is stored in the cache particle module, and the cache particles have a higher cache rate, which can improve the real-time performance of the FPGA data processing module in processing data. In summary, the system of the present invention can perform real-time parallel processing of high-parallel and high-throughput data, so that the quality of the spectral image finally obtained can be higher and the features are more obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the connection relationship between the FPGA-based multispectral image data stream processing system and multiple spectral image acquisition units provided in an embodiment of the present invention;

[0029] Figure 2A timing diagram of the DVP protocol provided in an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the connection relationship between the internal structure of the arbitration channel control submodule and the DVP protocol parsing submodule and the data cache submodule provided in an embodiment of the present invention.

[0031] Figure 4 It is a schematic diagram of the connection relationship between the downstream communication submodule and the downstream receiving module in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] For multispectral image acquisition, the most important metric is the number of spectral bands, but the number of spectral bands and real-time performance are mutually restrictive. The more spectral bands there are, the lower the real-time performance will be. This means that for an acquisition device with poor parallel capability and low throughput, the number of spectral bands it acquires is greatly restricted by the parallel capability and throughput, and it is difficult to process data from more spectral bands, and ultimately it is difficult to obtain spectral images with higher quality and more obvious features.

[0034] The FPGA-based multispectral image data stream processing system of the present invention is capable of processing highly parallel and high-throughput data streams. Under the premise of meeting real-time requirements, it can process a larger number of spectral segments, thereby obtaining spectral images with higher quality and more obvious features.

[0035] In one embodiment, Figure 1 As shown, Figure 1 It is a schematic diagram of the connection relationship between the FPGA-based multispectral image data stream processing system and multiple spectral image acquisition units provided in an embodiment of the present invention. The FPGA-based multispectral image data stream processing system of this embodiment is provided with an FPGA data processing module, a downstream receiving module and a cache particle module.

[0036] In this embodiment, the cache particle module uses DDR2 as a cache unit. The DDR2 memory supports the characteristics of 4-bit pre-reading and double-edge sampling and has a high data transmission rate.

[0037] Among them, the FPGA data processing module includes:

[0038] A plurality of spectral image information receiving submodules, each of which receives parallel spectral image information collected by a spectral image collecting unit.

[0039] The spectral image information collected by each spectral image acquisition unit is 8-line parallel data. In this embodiment, the number of spectral image information receiving submodules is 2 to 15, that is, the number of spectral bands that can be processed by the FPGA data processing module in this embodiment is 2 to 15, which can process not only low-quantity spectral band data but also high-quantity spectral band data.

[0040] Specifically, the present invention takes the number of spectral image information receiving submodules as 9 as an example to illustrate the system of the present invention.

[0041] A plurality of DVP protocol parsing submodules equal in number to the spectral image information receiving submodules, each of which receives and filters the parallel spectral image information transmitted by a spectral image information receiving submodule.

[0042] The DVP protocol parsing submodule can sample and collect the parallel spectral image information transmitted by the spectral image information receiving submodule to implement data filtering. Figure 2 As shown, Figure 2 The DVP protocol timing diagram provided in the embodiment of the present invention, the parallel spectral image information data stream is controlled by VSync and HRef. When the VSync signal is at a low level and the HRef is at a high level, the parallel spectral image information data is sampled at the moment when the reference clock trigger edge output by the spectral image acquisition unit is triggered to obtain the valid data of the parallel spectral image information at the current pixel point. Figure 2 The HRef signal in the image is used to indicate the start or end of each row of data in the image. The DVP protocol parsing submodule receives and filters the parallel spectral image information transmitted by the spectral image information receiving submodule, which can effectively prevent data leakage.

[0043] The arbitration channel control submodule, the arbitration channel control unit is used to receive the valid data filtered by all DVP protocol parsing submodules, and store the valid data transmitted by each DVP protocol parsing submodule in blocks. When the amount of data in each block storage area is greater than the maximum cache amount, the valid data of each block storage area is output in sequence from high to low according to the polling priority.

[0044] The arbitration channel control submodule outputs the valid data of each block storage area in sequence from high to low according to the polling priority, which can control the orderly transmission of parallel data streams, so that the FPGA data processing module has a higher parallel data processing capability.

[0045] The data cache submodule is used to receive and store the valid data of each block storage area output by the arbitration channel control submodule to the cache particle module.

[0046] The high cache rate of the cache particle module can improve the real-time performance of data processing of the entire acquisition system, thereby enabling the system to ultimately obtain high-resolution image data.

[0047] The downstream communication submodule is used to extract and transmit the valid data stored in the data cache submodule to the downstream receiving module. The downstream receiving module is used to display or transfer the valid data.

[0048] Specifically, the spectral image information receiving submodule is connected to the spectral image acquisition unit through an FFC flexible flat cable. The FFC flexible flat cable can be bent and wound more arbitrarily, and can be bent and folded freely in some narrow spaces without being restricted by the narrow space, which can reduce the volume of the entire system hardware.

[0049] In addition, the FPGA data processing module of this embodiment is an FPGA processing chip of the ALTERA-CYCLONE IV series, model EP4CE30F23C7, which has high performance and low power consumption.

[0050] The FPGA-based multispectral image data stream processing system of this embodiment has high real-time performance and high parallel data processing capabilities, and can process data of more spectral bands, so that the final spectral image has higher quality and more obvious features.

[0051] In one embodiment, the data cache submodule includes a ping-pong unit, the cache particle module includes multiple cache partitions, and the ping-pong unit is used to monitor each frame of valid data stored in the cache particle module by the data cache submodule and assist in selecting a cache partition for each frame of valid data.

[0052] The ping-pong unit can dynamically allocate different cache spaces for each frame of valid data, ensuring that when new data is written to the DDR, old data that has not been read will not be overwritten, thereby improving the system's cache efficiency and cache bandwidth utilization.

[0053] In one embodiment, if Figure 3 As shown, Figure 3 A schematic diagram of the connection relationship between the internal structure of the arbitration channel control submodule and the DVP protocol parsing submodule and the data cache submodule provided in an embodiment of the present invention. The arbitration channel control submodule includes a plurality of groups of arbitration channel units equal in number to the DVP protocol parsing units. Each group of arbitration channel units is used to receive and store valid data transmitted by a DVP protocol parsing submodule, and store the stored valid data in the data cache submodule.

[0054] Each group of arbitration channel units includes an asynchronous FIFO sub-unit, an application arbitration sub-unit and a transmission channel switch sub-unit.

[0055] The asynchronous FIFO subunit is a block storage area used to temporarily store valid data transmitted by a DVP protocol parsing unit. The asynchronous FIFO subunit acts as a buffer to cope with the mismatch of data processing rate, ensure smooth and orderly data flow, and prevent data loss or congestion during high-speed data transmission.

[0056] The arbitration application subunit is used to monitor the amount of cached data in the same group of asynchronous FIFO subunits in real time, and when the amount of cached data is greater than the maximum cache amount of the asynchronous FIFO subunit, an opening signal is sent to the transmission channel switch subunit.

[0057] The transmission channel switch subunit is used to transmit the valid data temporarily stored in the same group of asynchronous FIFO subunits to the data cache submodule after receiving the start signal sent by the application arbitration subunit.

[0058] In order to ensure the orderly transmission of data streams, only one transmission channel switch can be turned on at the same time. Therefore, the transmission channel switch subunit is used as the switch for each data stream channel. The opening of the transmission channel switch subunit is managed by the application arbitration subunit. When the valid data is transmitted, the transmission channel switch subunit is closed.

[0059] In one embodiment, Figure 4 As shown, Figure 4 It is a schematic diagram of the connection relationship between the downstream communication submodule and the downstream receiving module of an embodiment of the present invention. The downstream communication submodule includes a high-speed Ethernet interface unit, an external storage medium control unit and a display unit; the downstream receiving module includes Ethernet, an SD card and a display screen.

[0060] The high-speed Ethernet interface unit is used to obtain all valid data in the high-speed cache particle module and send all valid data to the Ethernet in the form of a general protocol. The external storage medium control subunit is used to obtain all valid data in the high-speed cache particle module and send all valid data to the SD card. The display subunit is used to obtain all valid data in the high-speed cache particle module and send all valid data to the display screen.

[0061] The FPGA-based multispectral image data stream processing system of this embodiment can apply the spectral data processed by the FPGA data processing module in various forms.

[0062] In one of the embodiments, the FPGA data processing module also includes an acquisition configuration submodule, which is used to configure parameters of registers inside the spectral image acquisition unit in sequence through the serial interface of the spectral image acquisition unit to achieve the setting of parameters such as resolution, frame number and white balance of the spectral image data collected by the spectral image acquisition unit.

[0063] In this embodiment, one acquisition configuration submodule is used to configure register parameters of all spectral image acquisition units, which can reduce system design redundancy and resource consumption.

[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A multispectral image data stream processing system based on FPGA, characterized in that: An FPGA data processing module, a downstream receiving module and a cache particle module are provided; The FPGA data processing module includes: A plurality of spectral image information receiving submodules, each of which receives parallel spectral image information collected by a spectral image collecting unit; A plurality of DVP protocol parsing submodules equal in number to the spectral image information receiving submodules, each of which receives and filters parallel spectral image information transmitted by a spectral image information receiving submodule; The arbitration channel control submodule is used to receive the valid data filtered by all DVP protocol parsing submodules, and store the valid data transmitted by each DVP protocol parsing submodule in blocks. When the amount of data in each block storage area is greater than the maximum cache amount, the valid data of each block storage area is output in sequence from high to low according to the polling priority; A data cache submodule, used for receiving and storing valid data of each block storage area output by the arbitration channel control submodule to the cache particle module; The downstream communication submodule is used to extract and transmit the valid data stored in the data cache submodule to the downstream receiving module.

2. The FPGA-based multispectral image data stream processing system according to claim 1, characterized in that: The data cache submodule includes a ping-pong unit, the cache particle module includes a plurality of cache partitions, and the ping-pong unit is used to monitor each frame of valid data stored in the cache particle module by the data cache submodule and assist in selecting a cache partition for each frame of valid data.

3. The FPGA-based multispectral image data stream processing system according to claim 2, characterized in that: The arbitration channel control submodule includes a plurality of arbitration channel units whose number is equal to that of the DVP protocol parsing units, and each arbitration channel unit is used to receive and store valid data transmitted by a DVP protocol parsing submodule, and store the stored valid data into the data cache submodule; Each group of arbitration channel units includes an asynchronous FIFO subunit, an application arbitration subunit and a transmission channel switch subunit; The asynchronous FIFO subunit is a block storage area for temporarily storing valid data transmitted by the DVP protocol parsing unit; The arbitration application subunit is used to monitor the amount of cached data in the same group of asynchronous FIFO subunits in real time, and when the amount of cached data is greater than the maximum cache amount of the asynchronous FIFO subunit, send an opening signal to the transmission channel switch subunit; The transmission channel switch subunit is used to transmit the valid data temporarily stored in the same group of asynchronous FIFO subunits to the data cache submodule after receiving the start signal sent by the application arbitration subunit.

4. The FPGA-based multispectral image data stream processing system according to claim 3, characterized in that: The downstream communication submodule includes a high-speed Ethernet interface unit, an external storage medium control unit and a display unit; The downstream receiving module includes Ethernet, SD card and display screen; The high-speed Ethernet interface unit is used to obtain all valid data in the high-speed cache particle module and send all valid data to the Ethernet in the form of a general protocol; The external storage medium control subunit is used to obtain all valid data in the cache particle module and send all valid data to the SD card; The display subunit is used to obtain all valid data in the cache particle module and send all valid data to the display screen.

5. The FPGA-based multispectral image data stream processing system according to claim 4, characterized in that: The FPGA data processing module also includes an acquisition configuration submodule, which is used to sequentially configure parameters of registers inside the spectral image acquisition unit through the serial interface of the spectral image acquisition unit.

6. The FPGA-based multispectral image data stream processing system according to claim 1, characterized in that: The spectral image information receiving submodule is connected to the spectral image acquisition unit via an FFC flexible flat cable.

7. The FPGA-based multispectral image data stream processing system according to claim 1, characterized in that: The FPGA data processing module is an FPGA processing chip of the ALTERA-CYCLONE IV series, model EP4CE30F23C7.

8. The FPGA-based multispectral image data stream processing system according to claim 1, characterized in that: The number of the spectral image information receiving submodules is 2 to 15.

Images