Interactive display method of super large-scale remote sensing images based on GPU and GDAL

Through the technology based on GPU and GDAL, the rapid loading and real-time interaction of ultra-large-sized remote sensing images are achieved using matrix transformation, which solves the problem of slow interaction response in the existing technology, improves loading speed and real-time interaction, and is suitable for project maintenance and secondary Development.

CN115018691BActive Publication Date: 2024-11-05XIAMEN UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111507545.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-11-05
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

When existing remote sensing image processing software processes ultra-large-sized images, the interaction response is slow, making it difficult to achieve fast loading and real-time display. Especially in multi-image superposition scenarios, the CPU is overloaded, resulting in increased latency.

Method used

Using the interactive display method of ultra-large-size remote sensing images based on GPU and GDAL, through the steps of preprocessing, viewport cropping, block loading and rendering, matrix transformation is used to achieve concise cutting and transformation, reducing loading burden, improving loading speed and real-time interaction sex.

Benefits of technology

It realizes fast loading and real-time interaction of ultra-large-sized remote sensing images, which improves loading speed by about 50% compared to Monteverdi-class technology, has a simpler architecture and higher execution efficiency, and is suitable for project maintenance and secondary development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115018691B_ABST
    Figure CN115018691B_ABST
Patent Text Reader

Abstract

The present invention provides an interactive display method of super-large-size remote sensing images based on GPU and GDAL, including: preprocessing step, viewport clipping step, block loading step and rendering display step. The present invention is an implementation of the block loading technology using GPU, which has the advantages of fast speed and good presentation effect. At the same time, compared with heavy technologies such as Monteverdi that require a large number of references to external frameworks, the architecture of the present invention is more streamlined, all transformations are implemented using matrices, various types of clipping and transformation are more concise, and it is more friendly to project maintenance and secondary development, and the execution efficiency is also higher. Compared with Monteverdi, the present invention can increase the loading speed by about 50%.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing technology, and particularly designs a method for interactively displaying super-large-size remote sensing images based on GPU and GDAL. Background Art

[0002] With the rapid development of remote sensing technology, the size of remote sensing images is becoming larger and larger, but the requirements for interactive feedback and loading speed are getting higher and higher. Users of remote sensing image processing software often involve geometric operations such as image scaling, rotation, and translation; pixel-level processing such as channel pixel range stretching and algorithm real-time preview; and multi-layer sampling operations such as multi-image overlay, rolling shutter, and multi-image overlay operations. Users usually hope that the feedback of these operations is as fast as possible, or even in real time. However, super-large images will inevitably require more processing time, resulting in slower interactive response.

[0003] How to quickly interact with user operations and quickly present processing results under limited space overhead is a problem that all major remote sensing image processing software must face. The current processing methods can be mainly divided into the following categories:

[0004] (1) One-time loading technology [1-6]: This type of technology usually loads the image into the memory at one time. The advantage is that the subsequent operations will have a higher processing speed and are more friendly to geometric operations. However, the disadvantages are also obvious. First, it is limited by the memory size and has poor compatibility with ultra-large-scale images. At the same time, since the image data has been solidified when loaded, pixel-level operations such as pixel stretching and multi-layer sampling cannot be performed. This type of technology is very cost-effective and mainstream processing software basically does not use this type of technology.

[0005] (2) Block loading and clipping technology based on CPU processing [7-9]. This type of technology uses the underlying layer with regional loading function such as GDAL to only load images of specific areas and specific resolutions in the viewport. Since the pixels are reduced to the viewport range, it is possible for the CPU to perform real-time processing. This type of technology usually has good interactivity and strong applicability. The advantage of this type of technology is that it has a wide range of applications and a decent interactive experience. Some commercial software also uses this technology to implement [10-11]. However, with the advancement of user interaction interfaces, the number of pixels that the viewport needs to carry is increasing. This type of technology usually has a large delay. Especially for scenes with multiple images superimposed, since the CPU is responsible for both geometric and pixel operations, and also needs to perform sampling operations on multiple images, the operation delay will be greatly increased.

[0006] (3) GPU-based block loading technology.

[12] This type of technology uses GPU for rendering, which has better rendering speed and effect. This type of technology usually requires a more complex implementation. For example, Monteverdi itself contains a very complex implementation and introduces many third-party image cutting and segmentation libraries for implementation. In addition, this type of technology has certain requirements for the graphics card of the machine, and its compatibility is slightly worse than the second technology. However, with the development of hardware, this disadvantage is usually no longer a major problem.

[0007] [1]. Bian Litao, Liu Shaotang, Jiang Bin. Research on fast display method of multi-temporal and large-scale remote sensing images[J]. Surveying and Mapping and Spatial Geographic Information, 2020, 43(7):4.

[0008] [2]. Wang Yanan, Lai Jibao, Zhou Ke, et al. Design and implementation of standard interface for multi-type remote sensing image files based on GDAL[J]. Journal of Henan University: Natural Science Edition, 2012, 42(6):5.

[0009] [3]. Zhao Yan, Wang Siyuan, Bi Haiyun, et al. Research on key technologies of remote sensing image browsing based on GDAL[J]. Computer Engineering, 2012, 38(23):5.

[0010] [4]. Zhang Lina, Zhang Dongfang, Bai Yabin. Research on fast reading and display method of remote sensing images based on GDAL[J]. Western Resources, 2014(1):3.

[0011] [5]. Yang Xuebo. Remote sensing image processing based on GDAL library[J]. Urban Geography, 2015(4X):2.

[0012] [6]. Wang Yun, Chen Hao, Wan Hongxia. Application of processing remote sensing image GeoTIFF data based on QT and GDAL[J]. Network New Media Technology, 2012, 1(5):3.

[0013] [7]. Zhang Hongwei, Tong Hengjian, Zuo Boxin, et al. Fast browsing of remote sensing images larger than 2G based on GDAL[J]. Computer Engineering and Applications, 2012, 48(13): 159-162.

[0014] [8]. Chen Xingyu. Research on satellite remote sensing image processing system based on OPENGL and GDAL[D]. South China University of Technology.

[0015] [9]. Yuan Ziqi. Research and development of remote sensing image change detection and comparison viewing tool based on GDAL[D]. Beijing Forestry University.

[0016]

[10] .L3HARRIS.Envi. https: / / www.l3harrisgeospatial.com / Software- Technology / ENVI [EB / OL]

[0017]

[11] . Aerospace Vision.PIE. https: / / www.piesat.cn / product / PIE-Basic / index.html [EB / OL]

[0018]

[12] .Grizonnet,M.,Michel,J.,Poughon,V.et al.Orfeo ToolBox:open sourceprocessing of remote sensing images.Open geospatial data,softw.stand.2,15(2017). https: / / doi.org / 10.1186 / s40965-017-0031-6 . Summary of the invention

[0019] The purpose of the present invention is to overcome the shortcomings of the existing methods and provide a method for interactive display of ultra-large-scale remote sensing images based on GPU and GDAL, which has fast loading speed and good interactive real-time performance.

[0020] In order to achieve the above-mentioned invention object, the technical solution adopted is as follows:

[0021] A method for interactive display of super-large remote sensing images based on GPU and GDAL. The main steps can be divided into five parts: preprocessing, viewport clipping, block loading, rendering display and multi-image overlay display, as follows:

[0022] 1. Preprocessing steps

[0023] 1.1. The user loads the image and pre-processes the loaded image. At this time, the image pixel details are not read, only the basic information of the image, such as length, width, number of channels, geographic information, etc. The image is logically divided into blocks, and the block size is determined by the user's operating window size and the texture format supported by the graphics card. According to different resolution levels, the logical level of the pyramid is established.

[0024] 1.2. Load the image geometry information and generate the transformation matrix M from the pixel coordinate system to the interactive coordinate system i The transformation matrix of the 2D space is a 3×3 homogeneous transformation matrix, which has the following form:

[0025]

[0026] The homogeneous transformation matrix is ​​used to perform a linear transformation from a two-dimensional homogeneous vector to another homogeneous space. 11 、M 12 、M 21 、M 22 The four components describe transformations such as rotation, shearing, symmetric scaling, etc. 13 、M 23 Describes the translation transformation, M 31 、M 32Describe the projection transformation, M 33 is the overall scaling factor.

[0027] Since the result needs to be projected onto the three-dimensional plane (z=0) corresponding to the viewport plane, it is necessary to expand the matrix to the homogeneous transformation matrix of the three-dimensional space and discard the Z-axis component. The resulting transformation matrix is:

[0028]

[0029] All transformation matrices (such as M v 、M b ) are obtained in this way.

[0030] 1.3. Call GDAL to calculate the pixel range of the image channel as a parameter for subsequent operations such as pixel stretching.

[0031] 2. Viewport clipping steps

[0032] 2.1. Maintain a transformation matrix M from the viewport coordinate system to the interactive coordinate system (the geographic coordinate system of the image) v The transformation matrix is ​​composed of various linear changes such as scaling, offset, shearing, rotation, etc.

[0033] 2.2. When an interaction occurs that causes the viewport to change or the user loads a new image, an update is triggered.

[0034] 2.3. Viewport positioning: The method is to transform the four-point boundary of the viewport into the image pixel coordinate system by the viewport matrix to determine the viewport range:

[0035] P i =P v ×M v ×M i -1

[0036] Among them, P i Represents the image pixel coordinates; P v Represents pixel coordinates.

[0037] In this step, you can preload some blocks by appropriately expanding the viewport range to avoid frequent loading caused by slight movements of the user.

[0038] 2.4. Determine the clipping range: Intersect the viewport range with the logical block, and put the blocks within the viewport range into the update sequence. Select the appropriate pyramid level according to the clipping size. For example, in a practical application, use a pyramid image with the same pixel size as the viewport to ensure the accuracy of the image. Intersect the logical block with the viewport to determine the range of blocks that need to be loaded.

[0039] 2.5. Use the incremental update method to handle the update sequence, keep the blocks with the same attributes and mark them as loaded. Unload the blocks that do not exist in the new sequence. Add the newly added blocks and mark them as unloaded. The purpose of this step is to further reduce the loading burden.

[0040] 3. Block loading steps

[0041] 3.1. Use GDAL to load the blocks that are not loaded in the update sequence. For each block, calculate the transformation matrix M from the block pixel to the overall coordinate system according to its position in the overall block. b .

[0042] 3.2. Use GDAL to load data into OpenGL texture according to the channel selected by the user. In this step, you can select the appropriate texture format according to the image format to speed up the loading process, or you can normalize it to a certain texture format.

[0043] 4. Rendering display steps

[0044] 4.1. Synthetic transformation matrix: For each block in the update sequence, the following matrix transformation is required to map it to the pixels of the viewport:

[0045] P v =P b ×M b ×M i ×M v -1

[0046] Among them, P v Represents the pixel coordinates mapped to the viewport; P b Represents the pixel coordinates on the tiled image.

[0047] When binding rendering, you also need to b Perform the inverse transformation M of the texture coordinates t Convert to the corresponding pixel, and v Perform viewport normalization M n , so the complete transformation is:

[0048] P v =P b ×M t ×M b ×M i ×M v -1 ×M n

[0049] The matrix is ​​set as a parameter of the vertex shader, and the GPU performs the geometric transformation.

[0050] 4.2. Bind fragment shader: Bind the corresponding fragment shader according to user interaction. Taking channel pixel stretching as an example, use the following formula to calculate the pixel:

[0051]

[0052] Among them, TM is the sampling value of the channel band, U and L are the upper and lower limits of the band value, which are determined by the user, and C is the final color. The user's adjustments to U and L are passed to the fragment shader as shader parameters in real time, calling the GPU to complete various pixel stretching.

[0053] 4.3. Bind the predefined face (standard square), and then perform rendering by other post-processing such as Alpha Test.

[0054] 5. Steps for displaying multiple images

[0055] Specifically, the transformation of the image and viewport coordinate systems is performed through matrices and finally normalized to the interactive coordinate system and then unified by the viewport transformation matrix M v Convert to viewport. Therefore, it is easy to overlay multiple images. Just replace the M i That's it.

[0056] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention discloses a method for interactively displaying super-large-sized remote sensing images based on GPU and GDAL, including a preprocessing step, a viewport clipping step, a block loading step and a rendering display step. Through the processing of these steps, the present invention has fast loading speed and good interactive real-time performance when interactively displaying super-large-sized remote sensing images. Compared with heavy technologies such as Monteverdi that require a large number of references to external frameworks, the architecture of the present invention is more streamlined, all transformations are implemented using matrices, various types of clipping and transformation are more concise, and are more friendly to project maintenance and secondary development, and have higher execution efficiency. Compared with Monteverdi, the present invention can increase the loading speed by about 50%. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of the method for interactively displaying super-large-size remote sensing images based on GPU and GDAL in this embodiment;

[0059] Figure 2 It is a complete process flow diagram of this embodiment;

[0060] Figure 3 A schematic diagram of establishing a multi-resolution image pyramid in this embodiment;

[0061] Figure 4 Schematic diagram of viewport coordinate system transformation in this embodiment;

[0062] Figure 5 A schematic diagram of image segmentation and viewport clipping in this embodiment;

[0063] Figure 6 Schematic diagram of incremental loading of blocks in this embodiment; (a) represents the original update sequence; (b) represents incremental loading;

[0064] Figure 7 FIG. 4 is a schematic diagram of multi-image overlay in this embodiment. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with the accompanying drawings.

[0066] See also Figure 1 and Figure 2 As shown, this embodiment provides a method for interactively displaying super-large-size remote sensing images based on GPU and GDAL, including the following steps:

[0067] S101, preprocessing step: receiving the user's image loading instruction, logically dividing the image into blocks; establishing a multi-resolution logical pyramid according to different resolution levels; loading the image geometric information, and generating a transformation matrix M from the pixel coordinate system to the interactive coordinate system i ;Call GDAL to calculate the pixel range of the image channel;

[0068] S102, viewport clipping step: generate a transformation matrix M from the viewport coordinate system to the interactive coordinate system v , based on the user's operation to change the viewport attributes v Perform update; transform the four-point boundary of the viewport from the viewport matrix to the image pixel coordinate system to calculate the viewport range; select the appropriate pyramid level according to the size of the viewport pixel, and add the logical blocks within the viewport range to the update sequence; use the incremental update method to handle the update of the update sequence;

[0069] S103, block loading step: calling GDAL to load the unloaded blocks in the update sequence, and using GDAL to load data into the OpenGL texture according to the channel selected by the user;

[0070] S104, rendering display step: for the blocks in the update sequence, synthesize the geometric transformation matrix of each matrix sequence and bind it to the vertex shader; bind the fragment shader of channel stretching according to user interaction; bind predefined vertices and faces, and perform rendering.

[0071] In this embodiment, a 3-channel, 8-bit high-resolution color image of a certain area is used as an example for explanation. The color image is spliced ​​from Google Earth satellite images. The interactive display method is as follows:

[0072] Step 1: See Figure 3 As shown, the user clicks Load Image, uses GDAL to load the image and initialize it, with 2 n The multi-resolution logical pyramid is established (not actually loaded, but only the pyramid relationship is established logically), and the geometric information of the image is loaded to calculate the transformation matrix M i , initialize other parameters.

[0073] Step 2: See Figure 4 As shown, the viewport transformation matrix M is updated when the viewport attributes (zoom, translation, rotation, etc.) are initialized or the user changes them. v , and trigger an update.

[0074] Step 3: See Figure 5 As shown, through P i =P v ×M v ×M i -1 Calculate the viewport range (a polygonal area) and select the appropriate pyramid level based on the viewport pixel size. Logical blocks within the viewport range are added to the update sequence.

[0075] Step 4: See Figure 6 As shown in Figure 1, by comparing with the original update sequence, new blocks are incrementally loaded, old blocks that do not exist are unloaded, and the same blocks are retained. Each new block records the local to global transformation matrix M b .

[0076] Step 5: Call GDAL to load new blocks of appropriate resolution according to the level of the image pyramid. For No Data data, assign the Alpha channel to 0. At this point, all images have been loaded and rendering begins.

[0077] Step 6: For the blocks in the update sequence, transform each matrix sequence: M t ×M b ×M i ×M v -1 ×M n Synthesize the geometry transformation matrix and bind it to the vertex shader.

[0078] Step 7: Bind the fragment shader for channel stretching based on user interaction.

[0079] Step 8: Set Alpha Test to remove No Data, bind the pre-set vertex buffer and face buffer, and perform rendering.

[0080] In this embodiment, a method for interactively displaying ultra-large-size remote sensing images based on GPU and GDAL also includes a multi-image overlay display step.

[0081] For details, see Figure 7 As shown, different images, their M t 、M b 、M i These three items are different, but after these three transformations M t ×M b ×M i After that, their coordinate systems are transformed to the same geographic coordinate system. Then, just use the same viewport coordinate system transformation and normalization change M v -1 ×M n , it is easy to achieve the superposition of multiple images.

[0082] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A method for interactive display of super-large-scale remote sensing images based on GPU and GDAL, characterized in that: The steps include: Preprocessing steps: receiving the user's image loading instruction, logically dividing the image into blocks; establishing a multi-resolution logical pyramid according to different resolution levels; loading the image geometric information, and generating the transformation matrix M from the pixel coordinate system to the interactive coordinate system i ; Call GDAL to calculate the pixel range of the image channel; Viewport clipping step: Generate the transformation matrix M from the viewport coordinate system to the interactive coordinate system v , based on the user's operation to change the viewport attributes v Update; transform the four-point boundary of the viewport from the viewport matrix to the image pixel coordinate system to calculate the viewport range; select the appropriate pyramid level according to the size of the viewport pixel, and add the logical blocks within the viewport range to the update sequence; Use incremental update method to handle update sequence updates; Block loading step: call GDAL to load the blocks that are not loaded in the update sequence, and use GDAL to load data into OpenGL texture according to the channel selected by the user; Rendering display step: for the blocks in the update sequence, synthesize each matrix sequence into a geometric transformation matrix and bind it to the vertex shader; Based on user interaction, bind the fragment shader for channel stretching; bind predefined vertices and faces and perform rendering.

2. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: When the image is logically divided into blocks, the block size is based on the user's operating window size and the texture format supported by the graphics card.

3. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The establishing of a multi-resolution logical pyramid specifically includes: A multi-resolution logical pyramid is established with a factor of 2n, where n is an integer.

4. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The viewport range is calculated as follows: P i =P v ×M v ×M i -1 Among them, P i Represents the image pixel coordinates; P v Represents viewport pixel coordinates.

5. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The transformation matrix M v The synthesis is based on various types of linear changes; the linear changes include scaling, translation, rotation, and shearing.

6. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The updating of the update sequence using the incremental update method specifically includes: Compared with the original update sequence, blocks with the same attributes are retained and marked as loaded; non-existent blocks are unloaded; newly added blocks are added and marked as unloaded.

7. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The calling of GDAL to load the blocks not loaded in the update sequence specifically includes: Use GDAL to load the blocks that are not loaded in the update sequence. For each block, calculate the transformation matrix M from the block pixel to the global coordinate system according to its position in the overall block. b .

8. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 7, characterized in that: The blocks in the update sequence are synthesized into a geometric transformation matrix by combining the matrix sequences and binding it to the vertex shader, specifically including: For each block in the update sequence, when binding rendering, P b Perform the inverse transformation M of the texture coordinates t Convert to the corresponding viewport pixels and normalize them as follows: P v =P b ×M t ×M b ×M i ×M v -1 ×M n Among them, P v Represents the pixel coordinates mapped to the viewport; P b Represents the pixel coordinates on the block image; M n A matrix representing the normalized transformation of the viewport; Let the matrix P v Bind to the vertex shader.

9. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: The fragment shader of binding channel stretching according to user interaction specifically includes: Based on user interaction, the corresponding fragment shader is bound and the following formula is used to calculate the pixel: Among them, C is the final color; TM is the sampling value of the channel band; U is the upper limit of the band value; L is the lower limit of the band value; The user's adjustments to U and L are passed to the fragment shader as shader parameters, and the GPU is called to complete various pixel stretching.

10. The method for interactively displaying super-large-size remote sensing images based on GPU and GDAL according to claim 1, characterized in that: After the rendering display step, the method further includes: a multi-image overlay display step; specifically, replacing the transformation matrix M i .

Citation Information

Patent Citations

  • Extra-large remote sensing image display method based on pyramids and image rendering engine

    CN103678501A

  • Remote sensing image-oriented GPU dynamic adaptive acceleration method

    CN111612685A

  • System and method for real-time overlay of map features onto a video feed

    US20170039765A1

  • Viewability metrics of a multidimensional object in a multidimensional digital environment

    WO2020157738A2