Fast Display Method for Ultra-Large Images

Abstract

The present invention belongs to the field of remote sensing satellite imagery technology, and in particular relates to a method for rapidly displaying very large images. The method comprises: S1: acquiring a very large image, and parallelizing the reading of the very large image using an FPGA; S2: setting a row buffer and a window buffer on the FPGA, and utilizing the row buffer and the window buffer to buffer the data stream of an image at a user-selected location; embedding bilinear interpolation and Lanczos filtering algorithms into the FPGA to render the image at the user-selected location; and caching and displaying the image at the user-selected location in real time. Based on memory mapping and FPGA hardware acceleration technology, the present invention designs an efficient transmission and communication mechanism between the GPU, memory, and FPGA, enabling rapid browsing and display of very large images.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing satellite images, and in particular relates to a method for quickly displaying an ultra-large image. Background Art

[0002] With the continuous advancement of space technology, remote sensing satellite and drone imagery has achieved significant improvements in spatial, spectral, and temporal resolution, and the amount of remote sensing data generated has also increased exponentially. This high-resolution remote sensing data has demonstrated significant application value in a variety of fields, including agriculture, forestry, environment, natural resources, and disaster prevention and mitigation. However, how to efficiently and quickly present this massive amount of high-resolution remote sensing data to users has become a critical issue that urgently needs to be addressed. Mosaiced remote sensing images often reach hundreds of gigabytes or even tens of terabytes, resulting in slow opening of these images by traditional geographic information or remote sensing software, thus restricting the in-depth exploration and widespread use of remote sensing images in practical applications.

[0003] For the real-time display of large-scale, high-volume remote sensing images, the traditional method involves applying multiple Gaussian filters and downsampling to the original image based on the Gaussian pyramid principle. Each layer of the remote sensing image gradually reduces in size and resolution. The number of layers depends on the original resolution of the remote sensing image and the application requirements. For example, if the original remote sensing image has a resolution of 1 meter, constructing a pyramid covering resolutions from 1 meter to 16 meters may require four to five layers. The resolution is downsampled by multiples of two, meaning that the original resolution is halved at the next layer. During the pyramid construction process, an index file is created, recording information such as the resolution, size, and geographic coordinate range of each layer. This allows for rapid location and access of the corresponding layer of remote sensing image data when displaying or analyzing the remote sensing image. However, this typically requires separate pyramid file construction prior to display, which is time-consuming and results in a large file size. Currently, the method of constructing a layered and tiled Gaussian pyramid allows for rapid navigation and zooming of very large images. However, the process of building an image pyramid is time-consuming, especially for high-resolution remote sensing images or drone remote sensing images. Summary of the Invention

[0004] In view of this, the present invention aims to provide a method for quickly displaying ultra-large images to solve the problem in the prior art that, during the display process of ultra-large images, the ultra-large images may be stuck and loaded too slowly due to the image dragging speed being too fast. Based on memory mapping and FPGA hardware acceleration technology, the present invention designs an efficient transmission and communication mechanism among GPU, memory and FPGA, thereby realizing the rapid browsing and display of ultra-large images.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0006] A method for quickly displaying an extremely large image comprises the following steps:

[0007] S1: Acquire a large image, and FPGA reads the large image in parallel;

[0008] S2: Set up a line buffer and a window buffer on the FPGA, use the line buffer and the window buffer to buffer the data stream of the image at the user-selected position, embed the bilinear interpolation and Lanczos filtering algorithms into the FPGA, render the image at the user-selected position, and cache and display the image at the user-selected position in real time.

[0009] Furthermore, step S1 also includes: obtaining relevant parameters of the display, which include real-time resolution, real-time refresh rate and size of the display window.

[0010] Furthermore, step S1 specifically includes the following steps:

[0011] S11: Determine the resolution, number, and size of each tile to be rendered based on relevant parameters of the display and the user's scaling of the super-large image;

[0012] S12: Divide the super-large image into m tiles according to the number and size of tiles determined in step S11; use the CPU to read the header file of the super-large image and divide the header file to obtain a two-dimensional coordinate position list corresponding to the m tiles;

[0013] S13: CPU based on the width of the large image W ,high H And the pixel spatial resolution parameters of each tile M ,transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally into each processor of the FPGA in real time, so that each processor reads each tile in parallel;

[0014] S14: Each processor of the FPGA stores each tile in the form of a file pointer to the FPGA and numbers each tile.

[0015] Furthermore, the FPGA stores tiles in order from left to right and from top to bottom.

[0016] Furthermore, buffering the data stream of the image at the position selected by the user specifically includes the following steps:

[0017] S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer starting address of the solid-state drive until all pixel data of the current tile is completely written into the row buffer;

[0018] Set the size of the window buffer to the same as the current display size;

[0019] S22: Repeat step S21 until all tiles of the current display window are written into the row buffer, and the image data corresponding to all tiles of the current row buffer are written into the window buffer;

[0020] S23: Move the display window from left to right and repeat steps S21 to S22 until all tiles at the position selected by the user are buffered.

[0021] Furthermore, step S2 also includes: based on the tiles corresponding to the user-selected position and the resolution of rendering each tile, using the FPGA to parallelly calculate the scaling value of the pixels contained in each tile to accelerate the rendering process; the FPGA generates an RGB array for each tile, and transmits each RGB array to the graphics card's memory chip through the system bus for real-time rendering, so that the FPGA can realize the splicing and display of each tile based on the user's scaling of the super-large image and the relative position of each tile to the super-large image.

[0022] Furthermore, the resolution of the super-large image is greater than 20000×20000, and the super-large image is stored in a solid state drive.

[0023] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0024] The present invention creates a method for rapidly displaying very large images. Firstly, it leverages the parallel computing advantages of FPGAs to calculate image positions in blocks and embed interpolation and filtering algorithms within the FPGA. By deploying these algorithms within the FPGA, when replacing a computer, the original FPGA card can be simply removed and inserted into the PCIe slot of the new device, eliminating the traditional software environment setup process. Secondly, a high-speed memory-to-video memory transfer mechanism is established to cache the current view and its surrounding data in the high-speed graphics card's video memory, enabling real-time zooming and display of very large images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0026] Figure 1A schematic diagram of a flow chart of a method for quickly displaying an ultra-large image according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a sub-meter high-resolution remote sensing image according to an embodiment of the present invention;

[0028] Figure 3 This is a diagram of the response time simulation results described in the embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0030] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0031] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0032] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0033] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0034] like Figure 1As shown, the present invention proposes a method for quickly displaying an ultra-large image, which specifically includes the following steps: S1: obtaining an ultra-large image, and parallelizing reading the ultra-large image using an FPGA; S2: setting a line buffer and a window buffer on the FPGA, using the line buffer and the window buffer to buffer the data stream of the image at the user-selected position, embedding bilinear interpolation and Lanczos filtering algorithms into the FPGA, rendering the image at the user-selected position, and caching and real-time displaying the image at the user-selected position.

[0035] It should be noted that the user uses the mouse wheel to zoom in or out of the oversized image as needed, and the area selected by the mouse is the user-selected position, and then the image at the user-selected position is processed in real time to cache and display the image at the user-selected position in real time.

[0036] In some embodiments, step S1 further includes: acquiring relevant parameters of the display, where the relevant parameters of the display include real-time resolution, real-time refresh rate, and size of the display window.

[0037] It should be noted that the FPGA accelerator card runs on an x86-based Windows platform, with an x86 CPU architecture, 16GB or greater of memory, an NVIDIA Quadro or Tesla graphics card, and an NVMe-based SSD. The FPGA accelerator card is located in a PCIe x16 slot.

[0038] Furthermore, FPGAs offer the advantage of parallelizing processing logic. FPGAs can be used to read and decode very large images (image data) in parallel. The large image is divided into tiles, and then the FPGA's multiple processing modules (hard processor systems (HPSs) directly integrated within the FPGA chip, with performance similar to that of independent CPUs but capable of high-speed interaction with FPGA logic resources) read and decode these tiles in parallel. The large image data is stored on solid-state drives.

[0039] In some embodiments, step S1 specifically includes the following steps: S11: based on the relevant parameters of the display, combined with the user's scaling of the super-large image, determining the resolution of each tile and the number and size of the tiles; S12: dividing the super-large image according to the number and size of the tiles determined in step S11 to obtain m tiles; using the CPU to read the header file of the super-large image and divide the header file to obtain a two-dimensional coordinate position list corresponding to the m tiles; S13: the CPU calculates the number and size of the tiles according to the width of the super-large image. W ,high H And the pixel spatial resolution parameters of each tile M, transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally into each processor of the FPGA in real time, so that each processor reads each tile in parallel; S14: each processor of the FPGA stores each tile in the form of a file pointer to the FPGA and numbers each tile.

[0040] It's important to note that the FPGA includes a DMA controller to enable high-speed data transfer from the storage device to the FPGA, reducing image data transmission latency. To extract image data for very large images as required, the DMA controller directly handles this data. Through communication between the FPGA and the SSD (Solid State Disk) controller, the image data is first read in blocks (tiles) and then transferred directly to the FPGA's flash memory via the PCIe channel for subsequent operations.

[0041] Furthermore, for very large images that are complete raster images, the FPGA uniquely numbers each tile, enabling rapid parallel processing of the tiles and forming an RGB array for each tile. The advantage of this numbering is that once processing is complete, the user's desired image can be quickly assembled in the order of the actual images.

[0042] In some embodiments, the FPGA stores tiles in order from left to right and from top to bottom.

[0043] It should be noted that when the view is adjusted (zoomed), the FPGA extracts the corresponding image data in blocks according to the position of the file pointer for processing and transmission, without the need to pre-load the entire image into the memory, reducing I / O operations and improving the reading speed of image data.

[0044] In some embodiments, data stream buffering for an image at a position selected by a user specifically includes the following steps: S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer starting address of the solid-state drive until all pixel data of the current tile is completely written into the row buffer; setting the size of the window buffer to be the same as the size of the current display; S22: repeating step S21 until all tiles of the current display window are written into the row buffer, and writing the image data corresponding to all tiles of the current row buffer into the window buffer; S23: moving the display window from left to right, and repeating steps S21~S22 until all tiles at the position selected by the user are buffered.

[0045] It should be noted that image data is accessed using AXI-Stream (Advanced eXtensible Interface Stream), and the FPGA stores tile data in a left-to-right, bottom-to-top order. Processing data streams on the FPGA requires a virtual memory architecture to multiplex data accesses from the FPGA flash memory. Row buffers and window buffers are used to read and write image data stored on the SSD and process the image data, respectively. The row buffer is primarily used to store complete image rows, while the window buffer is primarily used to read tiles requiring computation, facilitating algorithm operations.

[0046] The row buffer continuously receives new image data. As the window moves to the right, the image pixels in the window gradually move to the left, thereby updating the data within the window. Initially, the row buffer is empty, and data is loaded gradually from the starting address of the row buffer until all the image data of the first row is completely written to the row buffer. After the first row of data is written, the image data of the first row and first column begins to move upward, and the buffering process continues in sequence until the last row of image data is completely written to the row buffer. During this process, all the pixel values of the tile (512×512) are expanded into a row of 1×262144 and stored in the row buffer. Through the above loop operation, the row buffer continuously retains 1×262144 image data until the window buffer operation can be performed.

[0047] When the row buffer is full, the image data in the row buffer is written to the window buffer. The image data read from the window buffer is preprocessed for the next step of the algorithm. As the display window moves rightward, new image data fills the window buffer. The window buffer moves from left to right and from top to bottom until the entire image data of the oversized image is buffered in blocks according to the scaled size.

[0048] Furthermore, step S2 also includes: based on the tiles corresponding to the user-selected position and the resolution of rendering each tile, using the FPGA to parallelly calculate the scaling value of the pixels contained in each tile to accelerate the rendering process; the FPGA generates an RGB array for each tile, and transmits each RGB array to the graphics card's memory chip through the system bus for real-time rendering, so that the FPGA can realize the splicing and display of each tile based on the user's scaling of the super-large image and the relative position of each tile to the super-large image.

[0049] It should be noted that the present invention embeds bilinear interpolation and Lanczos filtering algorithms into FPGA, replacing the traditional process of tile generation of super-large images.

[0050] Image data for the current view and its surroundings is cached in high-speed video memory to enable real-time zooming of very large images. Traditionally, the FPGA transfers image data to host RAM via DMA (Direct Memory Access), and the GPU then reads the data from the host RAM via DMA. This reduces overall throughput and increases latency after a single process. This invention utilizes GPU Direct RDMA (Remote Direct Memory Access) for GPU-FPGA communication. This interface (NVIDIA's native GPUDirect interface) allows the bus address of the GPU video memory to be obtained and used as the source / destination address for FPGA DMA reads and writes.

[0051] Figure 2 It is a sub-meter high-resolution remote sensing image with an image size of 2.1TB. Figure 3 As shown in the figure, for the rapid display of sub-meter high-resolution remote sensing images, when the parallel number of FPGA kernel execution is 32 and the image block size is 512×512, the real-time display response is the fastest, with a response time of 0.145ms.

[0052] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0053] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for quickly displaying an ultra-large image, characterized in that: The fast display method for large images uses GPU-FPGA communication based on GPUDirect RDMA. The bus address of the GPU memory is obtained through the GPUDirect interface of GPUDirect RDMA, and the bus address of the GPU memory is used as the source or destination address for FPGA DMA reading or writing. The method for quickly displaying an extremely large image specifically includes the following steps: S1: Acquire a large image, and FPGA reads the large image in parallel; S11: Determine the resolution, number, and size of each tile to be rendered based on relevant parameters of the display and the user's scaling of the super-large image; S12: Divide the super large image into m tiles according to the number and size of the tiles determined in step S11; Use the CPU to read the header file of the super large image and divide the header file to obtain a two-dimensional coordinate position list corresponding to m tiles; S13: CPU based on the width of the large image W ,high H And the pixel spatial resolution parameters of each tile M ,transmit the two-dimensional coordinate position list of each tile to the FPGA through the PCIe bus, and divide each tile equally into each processor of the FPGA in real time, so that each processor reads each tile in parallel; S14: Each processor of the FPGA stores each tile in the form of a file pointer to the FPGA and numbers each tile; S2: Set up a line buffer and a window buffer on the FPGA, use the line buffer and the window buffer to buffer the data stream of the image at the user-selected position, embed the bilinear interpolation and Lanczos filtering algorithms into the FPGA, render the image at the user-selected position, and cache and display the image at the user-selected position in real time.

2. The method for quickly displaying a large image according to claim 1, wherein: Step S1 also includes: obtaining relevant parameters of the display, which include real-time resolution, real-time refresh rate and size of the display window.

3. The method for quickly displaying a large image according to claim 1, wherein: FPGA stores tiles in order from left to right and from top to bottom.

4. The method for quickly displaying a large image according to claim 1, wherein: The data stream buffering of the image at the user-selected position specifically includes the following steps: S21: placing the display window at the starting position of the position selected by the user, and making the current row buffer empty, the row buffer loads a tile from the buffer starting address of the solid-state drive until all pixel data of the current tile is completely written into the row buffer; Set the size of the window buffer to the same as the current display size; S22: Repeat step S21 until all tiles of the current display window are written into the row buffer, and the image data corresponding to all tiles of the current row buffer are written into the window buffer; S23: Move the display window from left to right and repeat steps S21 to S22 until all tiles at the position selected by the user are buffered.

5. The method for quickly displaying a large image according to claim 4, wherein: Step S2 further includes: using the FPGA to parallelly calculate the scaling values of the pixels contained in each tile based on the tiles corresponding to the user-selected position and the resolution of rendering each tile to accelerate the rendering process; the FPGA to generate an RGB array for each tile, and transmit each RGB array to the graphics card's memory chip via the system bus for real-time rendering, so that the FPGA can achieve splicing and display of each tile based on the user's scaling of the super-large image and the relative position of each tile with respect to the super-large image.

6. The method for quickly displaying a large image according to claim 1, wherein: The resolution of the super large image is greater than 20000×20000, and the super large image is stored in a solid state drive.

Images