Remote sensing image slicing system and method based on FPGA

Abstract

The invention provides a remote sensing image slicing system and method based on an FPGA (Field Programmable Gate Array). The system comprises a CPU (Central Processing Unit) module and an FPGA module, the CPU module is used for decoding an input original image and transmitting decoded image data and a corresponding first configuration parameter to the FPGA module; the FPGA module comprises an image slicing module and a memory module; the image slicing module is used for segmenting the original image into a plurality of slice images based on the first configuration parameter, and the memory module is used for storing the original image and the generated slice images. A CPU module and an FPGA module are combined, a plurality of image slices can be processed at the same time by means of the parallel computing capacity of an FPGA, and the processing speed and efficiency are improved; and the method has relatively low processing delay and flexibility.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image slicing system and method based on FPGA. Background Art

[0002] Remote sensing images usually have high resolution and large data volumes. A single image file may be very large. Processing images may result in excessive memory usage and low processing efficiency. Traditional computing resources may not be able to meet the needs of large-scale remote sensing image processing. It is usually necessary to use parallel computing and distributed processing technologies to improve processing efficiency.

[0003] Based on the needs and challenges of large-scale remote sensing image processing, remote sensing image slicing processing technology has emerged. Remote sensing images usually require preprocessing operations such as denoising, enhancement, correction, and feature extraction. Direct preprocessing of the entire image requires a lot of computing resources and time, and different areas or targets of remote sensing images may require different processing methods. After slicing the image, personalized preprocessing operations can be performed on each small block to improve processing accuracy and efficiency; in addition, some remote sensing applications require real-time processing of remote sensing image data. After dividing the image into small blocks, parallel processing can be achieved to improve real-time performance and response speed; when transmitting remote sensing image data over the network, dividing the image into small blocks can reduce the size of each data packet and improve the efficiency and stability of data transmission; for deep learning model training, slicing large-scale remote sensing images into small blocks can help improve training efficiency and reduce memory usage, and can also use distributed computing resources for training.

[0004] Remote sensing image slicing can combine parallel computing, personalized processing, and real-time requirements to effectively process large-scale remote sensing image data, improve processing efficiency and accuracy, and provide support for remote sensing image analysis and applications. However, existing remote sensing image slicing systems usually use CPUs, which have low processing speeds, large full-link delays, and high memory pressure and cost investment.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0006] The object of the present invention is to provide a remote sensing image slicing system and method based on FPGA to obtain higher processing efficiency, lower latency and better flexibility.

[0007] In order to solve the above problems, in the first aspect, a remote sensing image slicing system based on FPGA is provided below, including a CPU module and an FPGA module; The CPU module is used to decode the input original image and transmit the decoded image data and the corresponding first configuration parameters to the FPGA module; The FPGA module includes an image slicing module and a memory module; The image slicing module is used to slice the original image into a plurality of slice images based on the first configuration parameter, and the memory module is used to store the original image and the generated slice images.

[0008] By combining the CPU module and the FPGA module and taking advantage of the parallel computing capability of the FPGA, multiple image slices can be processed simultaneously, thus improving processing speed and efficiency. Moreover, due to the hardware implementation of the FPGA, it has a lower processing delay and is more suitable for the remote sensing image processing field with high real-time requirements. At the same time, the use of FPGA can reduce the CPU load pressure and make the CPU embedded as an independent device to reduce costs. The FPGA can also be customized according to actual needs to avoid wasting computing resources, improve resource utilization and be more flexible.

[0009] The image slicing module includes a coordinate calculation module, a burst reading information generation module, a slice reading module and a slice writing module connected in sequence; the coordinate calculation module is used to receive the first configuration parameter and calculate the address information corresponding to the slice image to be formed; the burst reading information generation module is used to receive the address information and generate corresponding burst information; the slice reading module is used to receive the burst information and read the corresponding image data from the memory module; the slice writing module is used to receive the image data and write it into the memory module to generate the slice image.

[0010] The memory module is provided with an original image frame buffer queue and a slice buffer queue; the original image is stored in the original image frame buffer queue, and the slice image is stored in the slice buffer queue.

[0011] The FPGA module also includes a first PCIe core and a first register; the CPU module transmits the decoded image data to the original image frame cache queue through the first PCIe core; the CPU module transmits the first configuration parameter to the first register through the first PCIe core.

[0012] The FPGA module includes a second PCIe core and a second register; the slice image is transmitted to a subsequent module through the second PCIe core, and the subsequent module includes a GPU module; the second register is used to store second configuration parameters related to the GPU module, and the second register is connected to the second PCIe core and the image slice module.

[0013] On the other hand, the present application further provides a remote sensing image slicing method based on FPGA, which is performed based on the remote sensing image slicing system according to any one of the first aspects, comprising: Input the original image into the CPU module for decoding to obtain multi-band image cube data, and set the corresponding first configuration parameters in the CPU module; sequentially transmitting the image data of a spectral band in the image cube data and the corresponding first configuration parameter to the FPGA module; Based on the first configuration parameter, the original image is segmented into a plurality of the slice images by the image slicing module.

[0014] The FPGA module includes an image slicing module and a memory module, and the image slicing module includes a coordinate calculation module, a burst read information generation module, a slice read module and a slice write module connected in sequence; the step of slicing the original image into a plurality of slice images through the image slicing module includes: generating a corresponding two-dimensional coordinate system according to the size of the original image, calculating the coordinate information of each slice image to be formed through the coordinate calculation module based on the configuration parameters, and converting the coordinate information into the address information corresponding to the slice image in the memory module; transmitting the address information to the burst read information generation module, generating corresponding burst information through the burst read information generation module, and the burst information includes a burst head address and a burst length; transmitting the burst information to the slice read module, reading the corresponding image data from the memory module through the slice read module; writing the read image data into the memory module through the slice write module to generate the slice image.

[0015] The configuration parameters include the size of the original image, the size of the slice image to be formed, and the overlapping area of ​​adjacent slice images; the overlapping area is determined based on the size of the target to be identified, and the overlapping area at the boundary is adaptively adjusted according to the remaining pixels.

[0016] The burst first address is the address corresponding to the first pixel point of each row of the slice image to be formed in the memory module, and the burst length is the length of each row of the slice image to be formed.

[0017] The slicing method is a flow-type process.

[0018] Compared with the prior art, the beneficial effects of the present invention mainly include the following: 1) Improve processing efficiency: Utilize the parallel computing capability of FPGA to process multiple image slices simultaneously, thus improving processing speed and efficiency; 2) Reduce latency: Due to the hardware implementation of FPGA, it has lower processing latency and is suitable for remote sensing image processing with high real-time requirements; 3) Reduce CPU computing pressure: Using FPGA to implement can reduce CPU load pressure, and can make the CPU embedded and made into an independent device, thus reducing costs; 4) Save resources: FPGA can be customized according to actual needs, thus avoiding wasting computing resources and improving resource utilization; 5) Flexibility: FPGA can be customized according to different processing requirements and is suitable for various remote sensing image processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the specific embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a working principle diagram of the remote sensing image slicing system provided by the present invention.

[0021] Figure 2 This is a principle block diagram of the remote sensing image slicing system provided by the present invention.

[0022] Figure 3 This is a flow chart of sending original images provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the slice position and coordinates provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the slice image data reading process provided by the present invention.

[0025] Figure 6 This is a flow chart of slice image data reading provided by the present invention.

[0026] Figure 7 This is a schematic diagram of the slice image data writing process provided by the present invention. DETAILED DESCRIPTION

[0027] The above and other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of a preferred embodiment with reference to the drawings. The directional terms mentioned in the following embodiments, such as up, down, left, right, front or back, etc., are only referenced to the directions of the attached drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0028] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0029] The steps in the following embodiments do not correspond one to one with the summary of the invention.

[0030] Embodiment 1 Remote sensing images have high resolution and large data volumes. A single image file is very large, and direct image processing is inconvenient. In order to achieve various requirements such as parallel computing, personalized processing, and real-time performance, remote sensing images can be sliced ​​first to divide a large-scale remote sensing image into multiple slice images that are easy to process later (for example, denoising, enhancement, correction, feature extraction, and other preprocessing operations).

[0031] This embodiment provides a remote sensing image slicing system and method based on FPGA. Figure 1 and Figure 2 The remote sensing image slicing system in this embodiment includes a CPU module and an FPGA module; wherein the CPU module is used to receive the original image input from the outside and then transmit it to the FPGA module, and the FPGA module performs slicing processing on the received image data to obtain the sliced ​​image and transmit it to the subsequent module (generally a GPU module) for subsequent processing.

[0032] Specifically, the remote sensing image slicing system in this embodiment includes an image decoding module, a memory module and an image slicing module.

[0033] The image decoding module (i.e., the CPU module) is implemented in the main control CPU, and is mainly used to implement image decoding of mainstream image file formats such as Tiff, Img, JPG, BMP, etc. The image decoding module will parse the input original image into multi-band image cube data and then transmit it to the FPGA module for cache. Specifically, a first PCIe core and a memory module are provided in the FPGA module. The memory module (in this embodiment, the memory module is DDR4) is provided with an original image frame cache queue for storing original image data. The image decoding module transfers the parsed image data to the original image frame cache queue through the first PCIe core. The image decoding module, the first PCIe core, and the memory module together constitute an original image sending module. The specific implementation process of the original image sending module is as follows. Figure 3 Reference Figure 3 First, the image decoding module sends an instruction (i.e., enabling the slicing function) to the FPGA module to confirm whether the image data can be transmitted at this time. When it is determined that the transmission can be carried out, the data transmission starts until the frame image is sent down.

[0034] The image slicing module specifically includes a coordinate calculation module, a burst read information generation module, a slice read module, and a slice write module. Among them, the coordinate calculation module is used to calculate the address of each slice in the memory module and transmit the address information to the burst read information generation module; the burst read information generation module confirms the burst first address and burst length according to the received address information and transmits them to the slice read module; the slice read module reads the corresponding image data from the memory module according to the received burst information and transmits it to the slice write module; the slice write module writes the received image data into the slice cache queue in the memory module for storage, and waits for subsequent transmission to the GPU module for processing.

[0035] Specifically, for the segmentation process of a frame of image, for example, an input image with a size (resolution) of 4096*4096 (in pixels, omitted below) (i.e., image data of a spectral band in the image cube data or a part of it) is taken as an example. The image is a two-dimensional structure (i.e., composed of a pixel array of 4096 rows*4096 columns). When it is stored in the original image frame buffer queue, it exists in the form of a one-dimensional structure, that is, the data corresponding to the 4096 rows of the original image (each row has 4096 pixels) are connected end to end and arranged in sequence in the same row; in the process of transmitting the original image data, the image decoding module will also transmit the relevant configuration parameters of the frame image data (i.e., the first configuration parameters), and the first configuration parameters include preset information such as the size of the image data, the size of the desired slice image, and the overlap size of adjacent slice images. In general, the first configuration parameters will be transmitted to the first register through the first PCIe core.

[0036] The coordinate calculation module interacts with the first register to calculate the coordinate information and address information of the slice image according to the received first configuration parameter. Figure 4 As shown, the original image is a two-dimensional structure (to generate a two-dimensional coordinate system), and it is expected that the original image will be divided into m*n small slice images, and the size of each slice image is the same; in order to prevent information loss, an overlapping area (overlapping area x) is set between adjacent slices. When reaching the right boundary and the lower boundary, the overlapping area is adaptively adjusted according to the remaining pixels (overlapping area x' and y). The values ​​of the overlapping area (including overlapping areas x, x' and y) are configured by the CPU module and recorded in the first configuration parameter. The setting of the overlapping area is mainly to avoid the problem when the target to be identified is located at the boundary of different slices. When the target to be identified is located on the boundary, the non-overlapping segmentation method may divide the target into two different slices, resulting in unsuccessful identification. By setting the overlapping method, this problem can be avoided. The size of the overlapping area can be set according to different application situations, usually related to the size of the target to be identified. For example, if the size of the subsequent target to be identified is 32*32, an overlapping area with a width of at least 32 pixels can be set to avoid target loss.

[0037] For ease of understanding, an example is given below. When the size of the input original image is 4096*4096, the size of the required slice image is 512*512, and the overlap area x is pre-set to 10, the original image will be divided into 9*9 small slice images, and the corresponding overlap areas x' and y are 442. If each pixel is regarded as a unit length, the coordinate value of each slice image in the two-dimensional coordinate system can be easily obtained. For example, the coordinates of the four vertices of the first slice (starting from the upper left corner and calculated in a clockwise direction) are (0,0), (0,512), (512,0) and (512,512), and the coordinates of the four vertices of the second slice are (0,510), (510,1022), (512,510) and (512,1022), and so on. The coordinate information of each slice image can be easily obtained. This method can be used to represent the corresponding relationship between any slice image to be formed and the original image.

[0038] After the coordinate calculation module calculates the coordinate information of the slice image, since it still needs to be read from the memory module when it is subsequently read, and the original image exists in the memory module in the form of a one-dimensional structure, the coordinate information of the slice image must be converted into its actual address information in the memory module. Specifically, according to the calculated coordinate information, for example, the coordinates of the four vertices of the first slice are (0,0), (0,512), (512,0) and (512,512), that is, it is a 512-row * 512-column pixel array. The image data corresponding to the pixel array is stored in the memory module as 512 data segments spaced apart from each other, and the length of each data segment is 512, that is, 512 pixels in each row are a segment, and the data between two adjacent rows is spaced 4096 lengths (that is, the length of the original image). Then, the coordinate information can be converted into the actual storage address based on this relationship.

[0039] In order to increase the speed of reading data, this application adopts a burst transmission method, so the burst reading information generation module is required to confirm the burst first address and burst length according to the address information. Figure 5 For each slice, the burst first address is the address corresponding to the first pixel point of each row of the slice image, and the burst length is the length of each row of the slice image. For this embodiment, 512 burst first addresses need to be provided for each slice, and each burst length is 512, and 512 data are read in sequence.

[0040] The slice reading module reads the corresponding image data from the memory module according to the received burst information (including the burst first address and burst length). Figure 5 and Figure 6 As shown, whenever the slice reading module receives a burst head address and burst length, it reads the data of the corresponding address from the memory module according to the axi bus protocol, and then converts it into an AXI-Stream stream through the AXI2AXIS interface and transmits it to the slice writing module. A complete slice is obtained by reading 512 data. It can be understood that this system adopts pipeline operation. The slice reading module does not need to wait for the original image to be completely cached in the memory module before reading different slices. It only needs to wait until all the data of the current slice is cached in the memory module to start reading. For example, corresponding to the first slice, when the first 512 rows of the original image are cached, the data corresponding to the first slice is sufficient, and the reading operation can be started at this time to form the first slice image, and the same is true for subsequent slice images.

[0041] The slice data writing module writes the received data stream (AXI-Stream stream) into the slice cache queue in address order, and the generation of a slice image has been completed. The generated slice image can be directly transmitted to the subsequent module (such as the GPU module), or it can be transmitted together after a certain number of slice images are generated. It can be understood that this transmission step is also a pipeline operation, and the GPU module does not need to wait until all slices of the original image are cached in the memory module before transmission. It can be reasonably matched according to the generation speed of the slice image and the processing speed of the GPU module.

[0042] The transmission of slice images from the memory module to the subsequent modules can be constructed by referring to the original image sending module. Specifically, a second PCIe core is provided in the FPGA module, and a slice cache queue for storing slice images is provided in the memory module. The memory module, the second PCIe core and the GPU module together constitute a slice image upload module. The specific implementation process of this module is as follows: Figure 7 As shown. First, the GPU module sends an idle flag to the FPGA module to indicate that data transmission is possible. When the FPGA module has generated a specified number of slices, the image slice module in the FPGA module generates an interrupt signal and sends it to the GPU module through the second PCIe core. At this time, the GPU module is set to a busy state and reads a specified number of slice images from the slice cache queue through the second PCIe core and performs corresponding processing. After processing the slice images read this time, the above process is repeated again until all slice images are processed.

[0043] Likewise, a second register is provided in the FPGA module, and the second register stores configuration parameters related to the GPU module. The second register can interact with the second PCIe core and the image slicing module.

[0044] In order to facilitate the description, some common English nouns or letters used in the present invention are only used for exemplary reference rather than restrictive interpretation or specific usage, and the protection scope of the present invention should not be limited by their possible Chinese translations or specific letters.

[0045] It should also be noted that, in this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

Claims

1. A remote sensing image slicing system based on FPGA, characterized in that: Including CPU module and FPGA module; The CPU module is used to decode the input original image and transmit the decoded image data and the corresponding first configuration parameters to the FPGA module; The FPGA module includes an image slicing module and a memory module; The image slicing module is used to slice the original image into a plurality of slice images based on the first configuration parameter, and the memory module is used to store the original image and the generated slice images.

2. The FPGA-based remote sensing image slicing system according to claim 1, characterized in that: The image slicing module includes a coordinate calculation module, a burst reading information generation module, a slice reading module and a slice writing module connected in sequence; The coordinate calculation module is used to receive the first configuration parameter and calculate the address information corresponding to the slice image to be formed; The burst read information generation module is used to receive the address information and generate corresponding burst information; The slice reading module is used to receive the burst information and read the corresponding image data from the memory module; The slice writing module is used to receive the image data and write it into the memory module to generate the slice image.

3. The FPGA-based remote sensing image slicing system according to claim 2, characterized in that: The memory module is provided with an original image frame buffer queue and a slice buffer queue; The original image is stored in the original image frame buffer queue, and the slice image is stored in the slice buffer queue.

4. The FPGA-based remote sensing image slicing system according to claim 3, characterized in that: The FPGA module also includes a first PCIe core and a first register; The CPU module transmits the decoded image data to the original image frame buffer queue through the first PCIe core; The CPU module transmits the first configuration parameter to the first register through the first PCIe core.

5. The FPGA-based remote sensing image slicing system according to claim 3, characterized in that: The FPGA module includes a second PCIe core and a second register; The slice image is transmitted to a subsequent module through the second PCIe core, and the subsequent module includes a GPU module; The second register is used to store a second configuration parameter related to the GPU module, and the second register is connected to the second PCIe core and the image slicing module.

6. A remote sensing image slicing method based on FPGA, characterized in that: Based on the remote sensing image slicing system according to any one of claims 1 to 5, the method comprises: Input the original image into the CPU module for decoding to obtain multi-band image cube data, and set the corresponding first configuration parameters in the CPU module; sequentially transmitting the image data of a spectral band in the image cube data and the corresponding first configuration parameter to the FPGA module; Based on the first configuration parameter, the original image is segmented into a plurality of the slice images by the image slicing module.

7. The remote sensing image slicing method based on FPGA according to claim 6, characterized in that: The FPGA module includes an image slicing module and a memory module, wherein the image slicing module includes a coordinate calculation module, a burst reading information generation module, a slice reading module and a slice writing module connected in sequence; The step of dividing the original image into a plurality of slice images by the image slicing module comprises: Generate a corresponding two-dimensional coordinate system according to the size of the original image, calculate the coordinate information of each slice image to be formed through the coordinate calculation module based on the configuration parameters, and convert the coordinate information into the address information corresponding to the slice image in the memory module; The address information is transmitted to the burst read information generation module, and the burst read information generation module generates corresponding burst information, wherein the burst information includes a burst first address and a burst length; Transmitting the burst information to the slice reading module, and reading the corresponding image data from the memory module through the slice reading module; The slice writing module writes the read image data into the memory module to generate the slice image.

8. The remote sensing image slicing method based on FPGA according to claim 7, characterized in that: The configuration parameters include the size of the original image, the size of the slice image to be formed, and the overlapping area of ​​adjacent slice images; the overlapping area is determined based on the size of the target to be identified, and the overlapping area at the boundary is adaptively adjusted according to the remaining pixels.

9. The remote sensing image slicing method based on FPGA according to claim 7, characterized in that: The burst first address is the address corresponding to the first pixel point of each row of the slice image to be formed in the memory module, and the burst length is the length of each row of the slice image to be formed.

10. The remote sensing image slicing method based on FPGA according to claim 7, characterized in that: The slicing method is a flow-type process.

Images