Rapid display method of oversized image

Abstract

The invention belongs to the technical field of remote sensing satellite images, and particularly relates to a rapid display method of an oversized image. The method comprises the following steps: S1, acquiring a super-large image, and performing parallel reading on the super-large image by an FPGA; s2, a line buffer area and a window buffer area are arranged on the FPGA, and data flow buffering is carried out on the image at the position selected by the user through the line buffer area and the window buffer area; bilinear interpolation and a Lanczos filtering algorithm are embedded into the FPGA, and image rendering is carried out on the image at the position selected by the user; and caching and displaying the image at the position selected by the user in real time. On the basis of the memory mapping and FPGA hardware acceleration technology, an efficient transmission communication mechanism of the GPU, the memory and the FPGA is designed, and rapid browsing and displaying of oversized images are achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing satellite images, and in particular relates to a method for quickly displaying a super-large image. Background Art

[0002] With the continuous advancement of space technology, remote sensing satellite and UAV images have achieved significant improvements in spatial resolution, spectral resolution and temporal resolution, and the amount of remote sensing data generated has also grown exponentially. These high-resolution remote sensing data have shown important application value in many fields such as agriculture, forestry, environment, natural resources, and disaster prevention and mitigation. However, how to efficiently and quickly present these massive and high-resolution remote sensing data to users has become a key issue that needs to be solved urgently. Mosaiced remote sensing images often reach hundreds of GB or even tens of TB, which makes traditional geographic information or remote sensing software slow in opening these images, thus restricting the deep mining and widespread use of remote sensing images in practical applications.

[0003] For the real-time display of the above-mentioned large-size and large-capacity remote sensing images, the traditional method is to perform multiple Gaussian filtering and downsampling processing on the original image based on the Gaussian pyramid principle. The size of each layer of remote sensing images gradually decreases, and the resolution also decreases accordingly. The number of levels depends on the original resolution of the remote sensing image and the application requirements. For example, if the resolution of the original remote sensing image is 1 meter, 4-5 levels may be required to build a pyramid with a resolution of 1 meter to 16 meters. The resolution ratio is downsampled according to a multiple of 2, that is, the original resolution to the next layer will be halved. In the process of building the pyramid, an index file is established to record the resolution, size, geographic coordinate range and other information of each layer of remote sensing images. In this way, when displaying or analyzing remote sensing images, remote sensing image data of the corresponding level can be quickly located and read as needed. However, this usually requires a separate pyramid file to be built before display. On the one hand, this process takes a long time; on the other hand, the generated pyramid file is also large in size. That is, the current method of constructing a Gaussian pyramid in layers and blocks is used to perform fast roaming and zooming of super-large images. However, the process of building an image pyramid is time-consuming, especially for high-resolution remote sensing images or drone remote sensing images. Summary of the invention

[0004] In view of this, the present invention aims to provide a method for quickly displaying super-large images, so as to solve the problem in the prior art that, during the display process of super-large images, the super-large images appear stuck and load too slowly due to the image dragging speed being too fast. The present invention is based on memory mapping and FPGA hardware acceleration technology, and designs an efficient transmission and communication mechanism of GPU, memory and FPGA, so as to realize the fast browsing and display of super-large images.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows: A method for quickly displaying a super large image comprises the following steps: S1: Acquire a super large image, and FPGA reads the super large image in parallel; S2: Set a line buffer and a window buffer on the FPGA, use the line buffer and the window buffer to buffer the data stream of the image at the user selected position, embed the bilinear interpolation and Lanczos filtering algorithms into the FPGA, render the image at the user selected position, and cache and display the image at the user selected position in real time.

[0006] Furthermore, step S1 also includes: obtaining relevant parameters of the display, the relevant parameters of the display including real-time resolution, real-time refresh rate and size of the display window.

[0007] Furthermore, step S1 specifically includes the following steps: S11: determining the resolution of each tile and the number and size of the tiles to be rendered based on relevant parameters of the display and in combination with the scaling of the super large image by the user; S12: according to the number and size of tiles determined in step S11, the super large image is divided to obtain m tiles; the header file of the super large image is read by the CPU, and the header file is divided to obtain a two-dimensional coordinate position list corresponding to the m tiles one by one; S13: CPU based on the width of the oversized image W ,high H And the pixel spatial resolution parameters of each tile M , transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally to each processor of the PPGA in real time, so that each processor reads each tile in parallel; S14: Each processor of the PPGA stores each tile in the form of a file pointer to the FPGA and numbers each tile.

[0008] Furthermore, the FPGA stores tiles in order from left to right and from top to bottom.

[0009] Furthermore, buffering the data stream of the image at the position selected by the user specifically includes the following steps: S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer start address of the solid state drive until all pixel data of the current tile is completely written into the row buffer; Set the size of the window buffer to the same size as the current display; S22: repeating step S21 until all tiles of the current display window are written into the row buffer, and the image data corresponding to all tiles of the current row buffer are written into the window buffer; S23: Move the display window from left to right, and repeat steps S21 to S22 until all tiles at the position selected by the user are buffered.

[0010] Furthermore, step S2 also includes: based on the tiles corresponding to the position selected by the user and the resolution of rendering each tile, using the FPGA to parallelly calculate the scaling value of the pixels contained in each tile to accelerate the rendering process; the FPGA generates an RGB array of each tile, and transmits each RGB array to the video memory chip of the graphics card through the system bus for real-time rendering, so that the FPGA can realize the splicing and display of each tile based on the user's scaling size of the super-large image and the relative position of each tile to the super-large image.

[0011] Furthermore, the resolution of the super large image is greater than 20000×20000, and the super large image is stored in a solid state drive.

[0012] Compared with the prior art, the invention can achieve the following beneficial effects: The invention creates a method for fast display of super-large images. On the one hand, the image position is calculated in blocks based on the parallel computing advantage of FPGA, and the interpolation algorithm and the filtering algorithm are embedded. When the interpolation algorithm and the filtering algorithm are deployed in the FPGA, when the computer is replaced, the original FPGA card can be directly pulled out and inserted into the PCIe card slot in the new device, which is ready for use, eliminating the setting link of the traditional software environment. On the other hand, a high-speed memory and video memory transmission mechanism is constructed to cache the current view and its surrounding data into the video memory of the high-speed graphics card to achieve real-time zoom display of super-large images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of a process flow of a method for quickly displaying a super-large image according to an embodiment of the present invention; Figure 2 A schematic diagram of a sub-meter high-resolution remote sensing image according to an embodiment of the present invention; Figure 3 This is a graph of response time simulation results described in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0015] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0016] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0017] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0019] like Figure 1 As shown, the present invention proposes a method for quickly displaying a super-large image, which specifically includes the following steps: S1: acquiring a super-large image, and FPGA parallelizing the reading of the super-large image; S2: setting a line buffer and a window buffer on the FPGA, using the line buffer and the window buffer to buffer the data stream of the image at the position selected by the user, embedding bilinear interpolation and Lanczos filtering algorithms into the FPGA, rendering the image at the position selected by the user, and caching and real-time displaying the image at the position selected by the user.

[0020] It should be noted that the user can use the mouse wheel to zoom in or out of the super large image as needed, the area selected by the mouse is the user selected position, and then the image at the user selected position is processed in real time to cache and display the image at the user selected position in real time.

[0021] In some embodiments, step S1 further includes: acquiring relevant parameters of the display, the relevant parameters of the display including real-time resolution, real-time refresh rate and size of the display window.

[0022] It should be noted that the FPGA accelerator card runs on an x86-based Windows platform, the host CPU architecture is x86, the memory is greater than or equal to 16GB, the graphics card is NVIDIA's Quadro or Tesla professional card, and the hard disk is an NVME-based SSD hard disk. The FPGA accelerator card is located in the PCIe x16 slot.

[0023] Furthermore, the advantage of FPGA is that it can parallelize the processing logic. FPGA is used to read and decode very large images (image data) in parallel. The very large image is divided into tiles, and then the tiles are read and decoded in parallel by multiple processing modules of FPGA (hard processor system (HPS) directly integrated inside the FPGA chip, with performance close to that of an independent CPU, but can interact with FPGA logic resources at high speed). The data of the very large image is stored in a solid-state drive.

[0024] In some embodiments, step S1 specifically includes the following steps: S11: based on the relevant parameters of the display, combined with the user's scaling of the super-large image, determine the resolution of rendering each tile and the number and size of the tiles; S12: according to the number and size of the tiles determined in step S11, divide the super-large image to obtain m tiles; use the CPU to read the header file of the super-large image, and divide the header file to obtain a two-dimensional coordinate position list corresponding to the m tiles; S13: the CPU calculates the two-dimensional coordinate position list according to the width of the super-large image. W ,high H And the pixel spatial resolution parameters of each tile M , transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally to each processor of the PPGA in real time, so that each processor reads each tile in parallel; S14: each processor of the PPGA stores each tile in the form of a file pointer to the FPGA, and numbers each tile.

[0025] It should be noted that a DMA controller is provided in the FPGA to achieve high-speed data transmission from the storage device to the FPGA and reduce the delay of image data in transmission. The image data of the super-large image is extracted according to the user's needs. The image data is directly transferred through the DMA controller. Through the communication between the FPGA and the SSD (Solid State Disk) master control, the image data of the super-large image is first read in blocks (divided into tiles), and then directly transmitted to the flash memory signal of the FPGA through the PCIe channel for subsequent operations.

[0026] Furthermore, the super-large image is a complete raster image, and the FPGA uniquely numbers each tile, which enables the FPGA to quickly process the tiles in parallel and form an RGB array for each tile. The advantage of numbering is that after processing, the image required by the user can be quickly spliced ​​in the actual image order.

[0027] In some embodiments, the FPGA stores tiles in order from left to right and from top to bottom.

[0028] It should be noted that when the view is adjusted (zoomed), the FPGA extracts the corresponding image data in blocks according to the position of the file pointer for processing and transmission, without the need to pre-load the entire image into the memory, reducing I / O operations and improving the reading speed of image data.

[0029] In some embodiments, data stream buffering for an image at a position selected by a user specifically includes the following steps: S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer starting address of the solid-state drive until all pixel data of the current tile is completely written into the row buffer; setting the size of the window buffer to be the same as the size of the current display; S22: repeating step S21 until all tiles of the current display window are written into the row buffer, and writing image data corresponding to all tiles of the current row buffer into the window buffer; S23: moving the display window from left to right, and repeating steps S21-S22 until all tiles at the position selected by the user are buffered.

[0030] It should be noted that the AXI-Stream (Advanced eXtensible Interface Stream, efficient data stream transmission protocol) method is used to access image data. When storing tile data on FPGA, it is stored from left to right and from bottom to top. Processing data streams on FPGA requires setting up a virtual memory architecture to access data in FPGA flash memory multiple times. The row buffer and window buffer are used to read and write image data stored in the solid-state drive and perform image processing operations respectively. The row buffer is mainly used to store complete image rows, while the window buffer is mainly used to read tiles that need to be operated for algorithm operation.

[0031] The row buffer continuously receives new image data. As the window moves to the right, the image pixels in the window gradually move to the left, thereby updating the data in the window. At the beginning, the row buffer is empty, and data is gradually loaded from the starting address of the row buffer until all the image data of the first row is completely written into the row buffer. When the first row of data is written, the image data of the first row and the first column begins to move upward, and the buffering process is performed in sequence until the last row of image data is completely written into the row buffer. In this process, all the pixel values ​​of the tile 512×512 are expanded into a row of 1×262144 and stored in the row buffer. Through the above loop operation, the row buffer continues to retain 1×262144 image data until the window buffer operation can be performed.

[0032] When the row buffer is filled, the image data in the row buffer will be written into the window buffer. The image data read by the window buffer is preprocessed for the next step of algorithm processing. As the display window moves to the right, the new image data will fill the window buffer. The window buffer moves from left to right and from top to bottom until the entire image data of the super large image is buffered in blocks according to the scaling size.

[0033] Furthermore, step S2 also includes: based on the tiles corresponding to the position selected by the user and the resolution of rendering each tile, using the FPGA to parallelly calculate the scaling value of the pixels contained in each tile to accelerate the rendering process; the FPGA generates an RGB array of each tile, and transmits each RGB array to the video memory chip of the graphics card through the system bus for real-time rendering, so that the FPGA can realize the splicing and display of each tile based on the user's scaling size of the super-large image and the relative position of each tile to the super-large image.

[0034] It should be noted that the present invention embeds bilinear interpolation and Lanczos filtering algorithms into FPGA, replacing the traditional process of tile generation of super large images.

[0035] Cache the image data of the current view and its surroundings into high-speed video memory to achieve real-time zoom display of ultra-large images. The traditional way is that the FPGA transfers the image data to the host RAM through DMA (Direct Memory Access), and then the GPU reads the data from the host RAM through DMA. After running a process, the overall throughput will be reduced and the delay will increase. The present invention is based on the GPU Direct RDMA (Remote Direct Memory Access) GPU and FPGA communication method. Through this interface (NVIDIA's own GPUDirect interface), the bus address of the GPU video memory can be obtained, and the bus address of the GPU video memory is used as the source / destination address of the FPGA DMA read / write.

[0036] Figure 2 It is a sub-meter high-resolution remote sensing image with an image size of 2.1TB. Figure 3 As shown in the figure, for the rapid display of sub-meter high-resolution remote sensing images, when the parallel number of FPGA kernel execution is 32 and the image block size is 512×512, the real-time display response is the fastest, with a response time of 0.145ms.

[0037] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0038] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for quickly displaying a super large image, characterized in that: The specific steps include: S1: Acquire a super large image, and FPGA reads the super large image in parallel; S2: Set a line buffer and a window buffer on the FPGA, use the line buffer and the window buffer to buffer the data stream of the image at the user selected position, embed the bilinear interpolation and Lanczos filtering algorithms into the FPGA, render the image at the user selected position, and cache and display the image at the user selected position in real time.

2. The method for quickly displaying a super large image according to claim 1, characterized in that: Step S1 also includes: obtaining relevant parameters of the display, which include real-time resolution, real-time refresh rate and size of the display window.

3. The method for quickly displaying a super large image according to claim 2, characterized in that: Step S1 specifically includes the following steps: S11: determining the resolution of each tile and the number and size of the tiles to be rendered based on relevant parameters of the display and in combination with the scaling of the super large image by the user; S12: Divide the super large image according to the number and size of tiles determined in step S11 to obtain m tiles; Use the CPU to read the header file of the super large image and divide the header file to obtain a two-dimensional coordinate position list corresponding to m tiles; S13: CPU based on the width of the oversized image W ,high H And the pixel spatial resolution parameters of each tile M , transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally to each processor of the PPGA in real time, so that each processor reads each tile in parallel; S14: Each processor of the PPGA stores each tile in the form of a file pointer to the FPGA and numbers each tile.

4. The method for quickly displaying a super large image according to claim 1, characterized in that: FPGA stores tiles in order from left to right and from top to bottom.

5. The method for quickly displaying a super large image according to claim 3, characterized in that: The data stream buffering of the image at the position selected by the user specifically includes the following steps: S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer start address of the solid state drive until all pixel data of the current tile is completely written into the row buffer; Set the size of the window buffer to the same size as the current display; S22: repeating step S21 until all tiles of the current display window are written into the row buffer, and the image data corresponding to all tiles of the current row buffer are written into the window buffer; S23: Move the display window from left to right, and repeat steps S21 to S22 until all tiles at the position selected by the user are buffered.

6. The method for quickly displaying a super large image according to claim 5, characterized in that: The step S2 further includes: based on the tiles corresponding to the positions selected by the user and the resolution of rendering each tile, using the FPGA to calculate in parallel the scaling value of the pixels contained in each tile to accelerate the rendering process; the FPGA generates an RGB array of each tile, and transmits each RGB array to the video memory chip of the graphics card through the system bus for real-time rendering, so that the FPGA realizes the splicing and display of each tile based on the scaling size of the super-large image selected by the user and the relative position of each tile to the super-large image.

7. The method for quickly displaying a super large image according to claim 1, characterized in that: The resolution of the super large image is greater than 20000×20000, and the super large image is stored in a solid state drive.

Images