A neural network-based comprehensive visual scene super-resolution method

Abstract

This invention discloses a comprehensive visual super-resolution method based on neural networks. Through a two-stage optimization process of offline training and online deployment, it achieves real-time super-resolution reconstruction of remote sensing images on an edge computing platform. In the offline stage, a physics-data hybrid-driven training dataset is constructed, including optical MTF analysis, atmospheric turbulence simulation, and sinc artifact enhancement. A dynamic network, RAG-SRNet, is designed to adaptively switch the RRDB computation path based on the screen projection size, and training is performed using remote sensing-specific perception loss, RaGAN adversarial loss, and spectral angle loss. In the online stage, TensorRT INT8 quantization and mixed-precision optimization are employed. CUDA-OpenGL zero-copy memory mapping enables the reuse of memory addresses for the inference output buffer and texture objects. Asynchronous inference thread pools and hardware semaphore synchronization are used to achieve pipelined parallelism for tile requests, super-resolution computation, and texture rendering. Compared to existing methods, this invention significantly improves PSNR, increases frame rate by over 25%, and reduces CPU usage by 77%, providing an industrial-grade visual enhancement solution for aerospace simulation and autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of airborne visual navigation and relates to a comprehensive visual super-resolution method based on neural networks. Background Technology

[0002] Integrated visual systems are technologies that integrate multi-source information with real-time 3D rendering, and are widely used in airborne visual navigation, aviation simulation, and autonomous driving. By fusing sensor data, geographic information, and real-time generated virtual images, this system provides operators with enhanced environmental perception capabilities beyond natural vision, especially in low visibility, complex weather, or nighttime conditions, significantly improving situational awareness and decision support.

[0003] Currently, the visual presentation quality of integrated visual systems largely depends on the underlying remote sensing map data. However, limited by the physical resolution of satellite or airborne remote sensing sensors, data acquisition costs, and processing complexity, the resolution of existing available remote sensing data often falls short of the high detail requirements of systems at near-ground distances (such as low-altitude flight, ground taxiing, and takeoff and landing). When the viewpoint approaches the ground, the terrain, buildings, roads, and other elements rendered by the system exhibit noticeable texture blurring, loss of detail, and jagged edges, severely impacting visual realism, target recognition accuracy, and the effectiveness of operational guidance.

[0004] To alleviate these problems, deep learning-based super-resolution reconstruction techniques have been introduced into the field of visual enhancement in recent years. These techniques aim to recover high-frequency details from low-resolution images through algorithms, thereby improving the spatial resolution and visual quality of images. However, existing super-resolution methods often employ simple downsampling (such as bicubic interpolation) or statistical degradation models, failing to accurately model the multiple physical degradations experienced by remote sensing images in the actual imaging process. These degradations include modulation transfer function (MTF) attenuation in optical systems, blurring caused by atmospheric turbulence, and sensor quantization noise. This results in significant differences between training data and real-world application scenarios, leading to low physical reliability of the restored details.

[0005] Furthermore, high-precision super-resolution networks are typically complex in structure and computationally intensive, making it difficult to meet the frame rate requirements of real-time rendering on edge computing platforms (such as airborne embedded devices). Existing static networks cannot dynamically adjust the amount of computation based on the visual importance of different regions in the scene, resulting in over-computation in distant areas, wasting computing power, while under-computation in near areas leads to quality degradation and creates a performance bottleneck.

[0006] On the other hand, when existing super-resolution solutions are integrated with graphics engines, they often require deep modifications to the engine source code or the introduction of additional data transfer and synchronization overhead. This makes integration difficult, highly invasive, and prone to increased latency due to memory copying, making it difficult to achieve high-quality rendering while ensuring real-time performance.

[0007] Therefore, current technology has not yet been able to provide a physically reliable, real-time efficient, and easily integrated super-resolution enhancement solution for integrated visual systems without changing the existing data sources and hardware architecture. This has become a key technical bottleneck restricting the further improvement of the system's visual performance in near-ground scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide a comprehensive visual super-resolution method and system based on neural networks to solve the problems of blurred near-ground distance images, lack of detail, and unrealistic textures caused by the reliance on low-resolution remote sensing map data in existing comprehensive visual systems. At the same time, it overcomes the technical bottlenecks of existing super-resolution methods, such as degradation modeling mismatch, contradiction between computational efficiency and real-time performance, and complex system integration.

[0009] To achieve the above objectives, this invention provides a comprehensive visual super-resolution method based on neural networks. The method includes an offline training phase and an online inference deployment phase. The specific technical solution is as follows:

[0010] The offline training phase includes:

[0011] Step S1: Based on remote sensing data with a multi-level LOD pyramid structure, construct a physical-data hybrid driven training dataset. The construction includes:

[0012] A 256×256 pixel block is extracted from Level 17 (0.5-meter resolution) of the 18-level LOD pyramid as the high-resolution ground truth (HR), and a 128×128 pixel block is extracted from Level 16 (1-meter resolution) as the low-resolution input (LR), forming a low-resolution-high-resolution image pair with a 2:1 resolution mapping relationship. The high-resolution image tiles are then subjected to optical analytical degradation based on modulation transfer function, simulated degradation based on atmospheric turbulence model, quantization noise injection, higher-order degradation processing, and sinc artifact enhancement processing in sequence.

[0013] Step S2: Construct and train a dynamic super-resolution network for frustum perception. The network includes a frustum classifier, a dynamic generator, and a discriminator. The frustum classifier outputs discrete gating signals based on the input image. The dynamic generator contains three inference paths, corresponding to the far, medium, and near scenes, respectively. The three paths have increasing numbers of residual dense blocks, channels, and computational complexity. The generator adaptively selects and activates one of the corresponding inference paths for super-resolution reconstruction based on the discrete value of the gating signal.

[0014] The online inference deployment phase includes:

[0015] Step S3: Perform INT8 quantization and mixed precision optimization based on TensorRT on the trained network, and register the inference output buffer as an OpenGL pixel buffer object through CUDA-OpenGL interoperation to realize the reuse of the video memory address of the inference output buffer and the graphics texture object.

[0016] Step S4: Create a plugin class by inheriting the tile source interface of the graphics engine, overload the initialization method and tile creation method, integrate the optimized model into the graphics engine, and execute the tile request, super-resolution calculation and texture rendering process in parallel through the asynchronous inference thread pool and hardware semaphore synchronization mechanism.

[0017] Preferably, in step S1, the optical analytical degradation based on modulation transfer function includes: obtaining the MTF curve from the satellite technical manual and constructing an optical blur kernel through inverse Fourier transform; the simulated degradation based on the atmospheric turbulence model includes: dynamically calculating the atmospheric coherence length based on the Kolmogorov turbulence model according to meteorological visibility and shooting altitude, and generating a long exposure point spread function kernel; the higher-order degradation processing includes adaptively adjusting the secondary blur intensity according to the local variance; and the sinc artifact enhancement processing includes performing sinc filtering with dynamically changing cutoff frequency.

[0018] Preferably, the view frustum classifier employs a convolutional neural network, extracts features from the input image through multiple convolutional layers, outputs three-dimensional logits through global average pooling and fully connected layers, and generates the discrete gating signal through argmax operation; the gating signal includes three levels corresponding to far-field, mid-field, and near-field, respectively corresponding to a screen projection area of ​​less than 4096 pixels², between 4096 and 65536 pixels², and greater than 65536 pixels².

[0019] Preferably, the three inference paths of the dynamic generator include: a far-field path, containing 8 residual dense blocks with 64 channels; a mid-field path, containing 16 residual dense blocks with 96 channels and inserting a window self-attention mechanism; and a near-field path, containing 23 residual dense blocks with 128 channels and adding a sinc residual branch; wherein, the first 8 residual dense blocks of the three paths share weights and the number of channels is uniformly set to 128, and the far-field path and the mid-field path adjust the number of channels to their respective needs through 1×1 convolution after sharing residual dense blocks.

[0020] Preferably, the cost function used to train the network includes: pixel loss, which employs Charbonnier loss and assigns a weight factor greater than 1 to pixels with brightness below a preset threshold; perceptual loss, which uses a ResNet-34 encoder pre-trained on the Sentinel-2 dataset to extract features and calculate L1 distance, with its weights dynamically adjusted according to the gating signal; adversarial loss, which employs a relative average generative adversarial network, with its weights dynamically adjusted according to the gating signal; and spectral angle fidelity loss, used to constrain the spectral characteristics of the reconstructed image.

[0021] Preferably, the discriminator adopts a U-Net structure, which includes a 4-layer encoder and a 4-layer decoder, with skip connections to pass features, and spectral normalization after each convolution layer.

[0022] Preferably, in step S3, the INT8 quantization and mixed precision optimization based on TensorRT includes: using INT8 quantization for computationally intensive layers and maintaining FP16 precision for quantization-sensitive layers; using entropy calibration and a calibration dataset synthesized from physical degradation, and setting a sensitivity threshold for automatic fallback.

[0023] Preferably, step S4 includes: overloading the tile creation method to implement view frustum distance judgment; for tiles that need super-resolution processing based on view frustum distance judgment, pushing the task into a lock-free queue and immediately returning a null pointer; using CUDA external semaphores to automatically trigger OpenGL texture updates upon completion of GPU-side inference; and combining double-buffered ping-pong switching to make rendering and inference completely parallel.

[0024] Based on the same inventive concept, this invention also provides a comprehensive visual super-resolution system based on neural networks, comprising: a training data construction module for performing physical-data hybrid degradation processing to construct a training dataset; a dynamic super-resolution network module including a view frustum classifier, a multi-path dynamic generator, and a discriminator for performing dynamic super-resolution inference; an inference acceleration module integrating a TensorRT engine for loading a quantized model and performing zero-copy asynchronous inference; and a graphics engine integration module integrated into the graphics engine as a plug-in for performing plug-in integration and asynchronous scheduling.

[0025] Preferably, the graphics engine integration module includes a plugin class that inherits from the graphics engine tile source interface. The plugin class overloads the initialization method and the tile creation method. In the initialization method, TensorRT engine deserialization, CUDA-OpenGL interoperability resource registration and hardware semaphore import are completed. In the tile creation method, view frustum distance judgment, asynchronous task submission and zero-copy texture upload are implemented.

[0026] Beneficial effects:

[0027] Compared with the prior art, the present invention has the following significant advantages:

[0028] 1. High image quality and strong physical fidelity: By embedding physical priors such as satellite MTF curves and the Kolmogorov atmospheric turbulence model into the training data generation process, the reconstructed high-frequency details better conform to the physical laws of real imaging. The PSNR is significantly improved compared to pure data-driven methods, and the ringing artifact suppression rate is greater than 90%, effectively avoiding the problems of excessive smoothing or the generation of unrealistic textures in traditional methods. The introduction of spectral angle fidelity loss and remote sensing-specific sensing loss further ensures that the spectral characteristics of targets such as vegetation and water bodies do not drift.

[0029] 2. Superior Real-Time Performance and Low Resource Consumption: The frustum-aware dynamic network RAG-SRNet adaptively switches between 8 / 16 / 23 RRDB computation paths based on the tile screen projection area (fast for distant scenes, high-quality for close scenes). Combined with TensorRT INT8 quantization (2-4x acceleration), GPU-OpenGL zero-copy memory mapping (texture upload time reduced from 6-8ms to <0.3ms), and asynchronous inference thread pools and double-buffered pipelines, it achieves an average inference latency of 13ms on the Jetson Orin NX edge computing platform, with a frame rate increase of over 25%, a CPU usage reduction of 77%, and VRAM usage controlled within 1.05GB, adapting to 16GB edge devices. Dynamic batch merging (1-4 tiles) further increases throughput by 40%.

[0030] 3. Excellent practicality and convenient deployment: The plug-in architecture integrates with osgEarth as a plug-in, requiring no modification to the graphics engine source code. "Plug and play" is achieved by inheriting from TileSource and overriding the createImage() method, reducing the deployment cycle from weeks to one day. Supports physical parameters (MTF curves, atmospheric...). The view frustum activation distance is dynamically configured via a .earth file, and a runtime monitoring API (average latency, frame rate, cache hit rate, view frustum signal distribution, etc.) is provided to facilitate system integration and on-site optimization. This solution provides the first end-to-end industrial-grade solution for the high-definition upgrade of integrated visual systems, encompassing data generation, model design, edge acceleration, and graphics engine integration. It can be widely applied in fields such as aerospace simulation, intelligent driving, and virtual reality.

[0031] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0033] Figure 1 This is a schematic diagram of the overall technical framework.

[0034] Figure 2 A schematic diagram of the process for modeling physical-data hybrid degradation.

[0035] Figure 3 This is a schematic diagram of the dynamic network architecture for view cone perception, where 21 is the view cone classifier, 22 is the dynamic generator, and 23 is the U-Net discriminator.

[0036] Figure 4 This is a schematic diagram of GPU-OpenGL zero-copy memory mapping. Detailed Implementation

[0037] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0038] This invention provides a comprehensive visual super-resolution method based on neural networks, comprising an offline training phase (S1, S2) and an online inference and deployment phase (S3, S4). The offline training phase utilizes physical-data hybrid-driven training data construction and dynamic super-resolution network training with frustum perception to obtain a deep neural network model capable of reconstructing low-resolution remote sensing tiles into high-resolution tiles with high quality. The online inference and deployment phase employs GPU-rendering collaborative acceleration optimization and pluggable graphics engine integration to deploy the trained model to an edge computing platform, achieving real-time, low-latency, and low-resource-consumption super-resolution rendering.

[0039] The goal of the offline training phase is to generate a deep neural network model (RAG-SRNet) capable of reconstructing low-resolution remote sensing tiles into high-resolution tiles with high quality. S1 is responsible for creating LR-HR training pairs with physical degradation characteristics; S2 is responsible for designing the dynamic network structure and cost function of frustum perception, and training usable model weights using the data from S1.

[0040] The goal of the online inference deployment phase is to deploy the trained model, after engineering optimization, to a real-world edge computing platform (such as Jetson Orin NX) and seamlessly integrate it with the graphics engine (osgEarth) to achieve real-time, low-latency, and low-resource-consumption super-resolution rendering. S3 is responsible for quantization compression (INT8), zero-copy memory mapping, asynchronous inference thread pools, and GPU double buffering to eliminate computational bottlenecks and data transfer overhead. S4 encapsulates these optimizations into an osgEarth plugin, achieving transparent "plug-and-play" integration through an overloaded tile source interface, and providing runtime configuration and monitoring.

[0041] The following details each step:

[0042] S1: Construction of training data based on integrated visual remote sensing data:

[0043] The training data construction based on integrated visual remote sensing data employs a physical-data hybrid degradation modeling method, such as... Figure 2 As shown, the specific steps include the following.

[0044] S1.1: LOD layered tile mapping and view frustum sampling:

[0045] The data construction employs a physical-data hybrid training data construction method based on an 18-level LOD pyramid. The data construction module dynamically reads tiles from the osgEarth image database (which stores 18-level LOD pyramids). Specifically, during training, 256×256 pixel blocks are extracted from Level 17 (0.5-meter resolution) as the HR (High Resolution) ground truth, while LR (Low Resolution) inputs are extracted from Level 16 (1-meter resolution) as 128×128 pixel blocks, forming a strict 2:1 resolution mapping. Tile coordinates are encoded using osgEarth's TileKey::getTileId() method, generating a list of meta_info.txt files, with an example format as follows:

[0046] / data / rs_l17 / 032110_2305.tif # HR tile path

[0047] / data / rs_l16 / 032110_2305.tif # LR tile path (automatically generated)

[0048] S1.2 Physical Imaging Link Resolution Degradation:

[0049] The degradation module includes a physical degradation unit and a data augmentation unit.

[0050] Physical degradation unit:

[0051] (1) Optical MTF Resolution Degradation: Obtain the MTF (Modulation Transfer Function) curve of the optical system from the satellite technical manual. For example, the MTF of the WorldView-3 satellite is 0.12 at the Nyquist frequency. Construct the optical blur kernel k_optical through a two-dimensional inverse Fourier transform. Its mathematical expression is:

[0052]

[0053] in, This represents the two-dimensional inverse Fourier transform, where f_x and f_y represent the spatial frequencies of the image in the horizontal and vertical directions, respectively (unit: period per pixel or period per millimeter). This kernel directly truncates high-frequency components in the frequency domain, and compared with traditional Gaussian blur kernels, it can accurately simulate the diffraction-limited blur of optical systems, avoiding the over-smoothing of statistical kernels.

[0054] (2) Atmospheric turbulence degradation: Calculate the Kolmogorov atmospheric coherence length based on meteorological metadata (e.g., visibility 10 km, shooting altitude 500 km). Its approximate expression is:

[0055]

[0056] In the formula: λ is the wavelength of light (unit: meters), usually taken as approximately the center wavelength of visible light. θ is the observed zenith angle (in radians), secθ is its secant value; h is the altitude at which the satellite or aircraft takes the picture (in meters). Reference atmospheric pressure (unit: hectopascal, usually taken as 1013.25 hPa).

[0057] based on Generate a long exposure point spread function (PSF) kernel k_atmosphere to simulate long exposure blur caused by atmospheric turbulence.

[0058] (3) Composite blur: The optical blur kernel is convolved with the atmospheric turbulence blur kernel to obtain the total blur kernel:

[0059]

[0060] in, This represents a two-dimensional discrete convolution operation. Then, k_total is convolved with the HR image patch to obtain the physically degraded image.

[0061] Data Augmentation Unit:

[0062] After physical blurring is completed, random Gaussian noise (noise standard deviation σ∈[0,15], pixel value range 0~255) and 10-bit quantization noise are added. The quantization noise simulates the sensor's analog-to-digital conversion process, and its calculation method is as follows:

[0063]

[0064] Where: hr is the pixel value of the high-resolution image; round() is the rounding function; n_quant is the quantization noise value. This operation simulates the readout noise and analog-to-digital conversion error of a 10-bit sensor.

[0065] S1.3 Higher-order Degradation and sinc Artifact Enhancement

[0066] A second-order degradation process is then performed on the first-order degradation result (i.e., the image after the physical degradation and data augmentation described above):

[0067] Adaptive quadratic blur: Calculating the local variance map of the first-order degraded image ,in This represents the pixel value of the image after first-order degradation. In local variance... The region (corresponding to smooth ground features) is blurred using a Gaussian blur kernel with a standard deviation σ = 1.5; The region (corresponding to the strong edge region) is treated with σ=0.5. This strategy is used to simulate adaptive sharpening and smoothing strategies in ground data processing.

[0068] sinc artifact enhancement: Sinc filtering is performed with a probability of 0.8, and the filter cutoff frequency (normalized angular frequency) ω_c is dynamically adjusted according to the number of training iterations t.

[0069]

[0070] In the formula: t is the number of global iterations during the training process; π is the constant of pi; sin() is the sine function.

[0071] This dynamic adjustment strategy forces the super-resolution model to learn the ability to suppress ringing artifacts at different bandwidths, achieving a suppression rate of over 90% for the common "white edge ghosting" problem in satellite imagery.

[0072] To improve near-field quality, a viewpoint access frequency statistician is introduced: Ten typical flight paths are pre-flew in a simulation environment, and the rendered pixel area (in screen pixels, denoted as screen_pixels) of each Level 17 tile is recorded. During training, weighted sampling is performed using weight = sqrt(screen_pixels / 256²), where 256 is the side length (in pixels) of the HR tile. Near-field tiles (with large screen area) have a weight of up to 3.0, while far-field tiles have a weight as low as 0.5, thus ensuring that the model's convergence speed in the near-field region is improved by approximately 2.5 times.

[0073] The final output of step S1 is a pair of training data (LR, HR), where LR is a low-resolution image patch of 128×128×3 and HR is a high-resolution ground truth image patch of 256×256×3. This data pair will serve as both the input and supervision signal for the network in step S2.

[0074] S2: Dynamic network architecture and cost function design for view frustum perception:

[0075] The architecture diagram of the View Cone Perceptual Dynamic Network (RAG-SRNet) is as follows: Figure 3As shown, the network consists of three sub-modules: a view cone classifier, a dynamic generator, and a U-Net discriminator. The LR-HR data pairs output by S1 are used to train the network.

[0076] S2.1 Network Architecture Design

[0077] like Figure 3 As shown, RAG-SRNet 20 consists of a view cone classifier 21, a dynamic generator 22, and a U-Net discriminator 23. The input-output relationships of each module are as follows:

[0078] The frustum classifier receives an LR image (128×128×3) as input and outputs a discrete gated signal signal ∈{0,1,2}.

[0079] Dynamic generator: Receives the same LR image (preprocessed by pixel-unshuffle) and a gated signal signal, and outputs a super-resolution image SR (256×256×3).

[0080] U-Net discriminator: During the training phase, it receives the ground truth (HR) value and the SR image, and outputs a pixel-wise confidence map (256×256×1) for calculating the adversarial loss.

[0081] S2.1.1 Hardware Implementation of the View Frustum Classifier (RAG-Classifier)

[0082] The frustum classifier 21 serves as the control center of this invention. Its input is the screen projection parameters calculated in real time by the graphics engine. However, in the network implementation, the LR image is directly used as the input (because there is a statistical correspondence between the screen projection area of ​​the tile and the scale of the LR image; a more accurate implementation could also use the projection area as an additional feature for stitching, but this embodiment uses the LR image directly for simplification). The specific implementation method is as follows:

[0083] Classifier Network Structure: The classifier employs a lightweight convolutional neural network structure. The input is a 128×128 pixel LR tile (3-channel RGB). The network consists of three 3×3 convolutional layers:

[0084] First layer: 3 input channels, 16 output channels, stride = 2, output resolution 64×64;

[0085] Second layer: 16 input channels, 32 output channels, step size = 2, output resolution 32×32;

[0086] The third layer has 32 input channels and 1 output channel (i.e., a single-channel feature map), with a stride of 2 and an output resolution of 16×16. Global Average Pooling (GAP) then compresses the 16×16×1 feature map into a 1-dimensional feature vector (length 1). Finally, a fully connected layer projects the 1-dimensional feature onto a 3-dimensional logits space (i.e., the unnormalized scores of the three classes).

[0087] Gated signal generation: Classifier output logits (3D vector) After that, instead of using softmax normalization, the index of the maximum value is obtained directly through the argmax operation to generate a discrete integer gated signal signal:

[0088]

[0089] This signal takes values ​​of 0, 1, or 2, corresponding to:

[0090] signal = 0: distant view, screen projection area A_screen < 4096 pixels²;

[0091] signal = 1: Medium shot, 4096 ≤ A_screen ≤ 65536 pixels²;

[0092] signal = 2: Close-up, A_screen > 65536 pixels².

[0093] A_screen is obtained through the following process: In the osgEarth plugin, the screen projection area is calculated through real-time transformation of the camera view frustum matrix and the tile geographic bounding box. Specifically, the product matrix M of the camera view matrix and the projection matrix is ​​obtained, the coordinates of the 8 geographic corner points (3D world coordinates) of the tile are transformed to the normalized device coordinate system (NDC), and then mapped to the screen pixel coordinates. The area of ​​its bounding rectangle (in pixels squared) is calculated as the screen projection size A_screen.

[0094] In this embodiment, during the super-resolution network training phase, the classifier parameters are fixed (requires_grad=False) and do not participate in backpropagation to ensure the stability of the view frustum calculation. The gating signal is updated every 100 milliseconds, and the same tile reuses the signal between adjacent frames, thereby reducing the classifier call frequency from 60 times per second (60 Hz rendering) to about 6 times per second.

[0095] S2.1.2 Three-path architecture and routing mechanism of dynamic generator 22:

[0096] The dynamic generator consists of three independent subnetworks that are selectively activated during inference based on gating signals. During training, the three paths share the weights of the first eight RRDBs.

[0097] The input for all paths is the same: the LR image (128×128×3) first passes through a shared preprocessing layer, the Pixel-Unshuffle operation (downsampling factor r_down = 2), which reduces the spatial resolution to 64×64 while expanding the number of channels to 12 (3×2²=12). This 64×64×12 feature map serves as the common input for the three paths.

[0098] FastPath (Signal 0, Vision Path): Employs a lightweight architecture. It contains 8 Residual-in-Residual Dense Blocks (RRDBs). Each RRDB consists of 3 dense convolutional layers. The output of each layer is concatenated with the outputs of all preceding layers along the channel dimension, and finally fused using a 1×1 convolution with a residual scaling factor of 0.2. To support weight sharing, the number of channels in the first 8 RRDBs of all three paths is uniformly set to 128. FastPath contains a total of 8 RRDBs (all shared). After these 8 shared RRDBs, a 1×1 convolutional layer (denoted as conv_down_fast) reduces the number of channels from 128 to 64 to match the computational requirements of the lightweight path. The network front-end has performed Pixel-Unshuffle. The backbone network extracts features in a low-resolution 64×64 space. After processing by 8 shared RRDB layers, it undergoes dimensionality reduction via the aforementioned 1×1 convolution. Finally, a PixelShuffle operation (upsampling factor r_up = 4) upsamples the resolution from 64×64 to 256×256, achieving a 2x super-resolution (128→256). This path has a computational cost of approximately 0.48 GMac, an inference latency of approximately 8 milliseconds, and a PSNR of approximately 28.5 dB. The feature map output from the last RRDB layer in this path is 64×64×64. Subsequently, a 1×1 convolutional layer projects the number of channels to 48, followed by PixelShuffle upsampling (r_up=4) to obtain the final SR image of 256×256×3.

[0099] StandardPath (Signal 1, Mid-Range Path): This path has a standard configuration and contains 16 RRDBs. The first 8 RRDBs (128 channels) share weights with FastPath and HighQualPath. After these 8 shared RRDBs, a 1×1 convolutional layer (denoted as conv_down_std) reduces the number of channels from 128 to 96, followed by an additional 8 RRDBs (96 channels, unique to StandardPath). To enhance global texture modeling capabilities, a window self-attention mechanism is inserted after these 8 unique RRDBs, with a window size of 8×8 and 8 attention heads. Finally, PixelShuffle upsampling is applied to 256×256. The computation is approximately 1.9 GMac, the latency is approximately 16 milliseconds, and the PSNR is approximately 29.2 dB, suitable for mid-range quality requirements. The feature map size output by the last RRDB is 64×64×96. It is then upsampled to 256×256×3 through a 1×1 convolutional layer (channel projection 96→48) and PixelShuffle upsampling.

[0100] HighQualPath (Signal 2, Near-field Path): This path is a high-fidelity configuration containing 23 RRDBs. The first 8 RRDBs (128 channels) are shared. Since this path requires high-fidelity reconstruction, channel dimensionality reduction is not performed; instead, it directly connects to an additional 15 RRDBs (128 channels, unique to HighQualPath). A sinc residual branch is added specifically for learning ringing artifact suppression: the input to this branch is the output feature map (size H×W×128) of the 23rd RRDB. The branch consists of two 3×3 convolutional layers; the first layer compresses the number of channels to 64, and the second layer restores it to 128. Finally, the output feature map of HighQualPath is fused with the main path's output through element-wise addition. The weights of the sinc branch are initialized to zero to ensure that it does not affect the main path during the initial training phase. The window size is expanded from the attention window to 12×12 to capture a wider range of structural dependencies, and the number of heads is increased accordingly to maintain computational efficiency. Finally, PixelShuffle upsampling is applied to 256×256. This path has a computational cost of approximately 3.2 GMac and a latency of approximately 24 milliseconds. However, since the update frequency of near-field tiles in the scene is typically below 10 Hz, the impact on the overall frame rate is limited. The feature map output by the last RRDB, after enhancement by the sinc residual branch, has a size of 64×64×128. Subsequently, a 1×1 convolutional layer (channel projection 128→48) and PixelShuffle upsampling are applied to obtain a size of 256×256×3.

[0101] Dynamic routing mechanism: In the RAGGenerator module, the three sub-networks are encapsulated using nn.ModuleDict. During forward propagation, the input tensors are grouped according to the gating signals of each sample within a batch, and then fed into the corresponding sub-networks for processing. Finally, the input tensors are merged and output according to their original indices. This mechanism avoids computational graph fragmentation caused by conditional branches, ensuring GPU parallel efficiency.

[0102] Weight sharing and training strategy: The weight sharing relationship among the three paths is clearly defined as follows:

[0103] Shared component: The first 8 RRDBs of all three paths (with a fixed number of channels of 128) share the same weight.

[0104] Channel adjustment layer (unique): The 1×1 dimensionality reduction convolution of FastPath (128→64) and the 1×1 dimensionality reduction convolution of StandardPath (128→96) are independent and not shared.

[0105] Unique RRDBs: FastPath has no unique RRDBs (all 8 of its RRDBs are shared); StandardPath has unique last 8 RRDBs (channel 96); HighQualPath has unique last 15 RRDBs (channel 128) and sinc residual branches.

[0106] During training, a course-based learning strategy is used to unlock the unique layers of each path in stages:

[0107] Phase 1 (0 ~ 200k iterations): Only StandardPath is trained. At this time, the shared RRDB is updated through backpropagation of StandardPath, and the unique layers of FastPath and HighQualPath (channel adjustment layer, back RRDB, sinc branch) are not activated.

[0108] Phase Two (200k ~ 350k iterations): Unlock the unique RRDBs (last 8) for StandardPath, the unique RRDBs (last 15) for HighQualPath, and the sinc branch, as well as the channel adjustment layers for each path. The learning rate is reduced to... And introduce a path balance regularization term:

[0109]

[0110] in , , These represent the number of samples for the gated signals 0, 1, and 2 in the current batch, respectively. This represents the L1 norm.

[0111] Phase 3 (350k ~ 500k iterations): Fix the parameters of the frustum classifier, and fine-tune the three paths completely independently. The learning rate is reduced to... .

[0112] Parameter initialization: The convolutional layers in RRDB are initialized using a Kaiming normal distribution with a variance of 2 / fan_in, where fan_in is the number of input channels. All 1×1 channel adjustment layers use the same initialization method. The weights of the last layer of the discriminator are initialized to zero. The weights of the two convolutional layers in the sinc residual branch are initialized to zero, and the biases are initialized to zero.

[0113] S2.1.3 U-Net Discriminator:

[0114] The U-Net discriminator consists of a 4-layer encoder and a 4-layer decoder. The number of encoder channels changes from 64 to 128, then to 256, and finally to 512. The number of decoder channels decreases symmetrically from 512 to 256, then to 128, and finally to 64. High-frequency details are passed between each encoder layer and its corresponding decoder layer via skip connections. Spectral normalization is applied after each convolutional layer to constrain the spectral norm of the weight matrix to 1. The spectral norm is estimated using a power-law iteration method with only one iteration (instead of the standard five) to balance training stability and inference speed. During inference, the spectral norm is frozen to the final training value and is not recalculated, resulting in a discriminator forward overhead of less than 2 milliseconds.

[0115] Specifically:

[0116] The encoder section consists of four downsampled convolutional layers, with the number of channels increasing from 64 to 512. Each convolutional layer has a 4×4 kernel size, a stride of 2, padding of 1, and is followed by LeakyReLU activation (negative slope of 0.2). Spectral normalization is achieved by calculating the spectral norm of the weight matrix and performing division, constraining the Lipschitz constant to 1 to prevent gradient explosion.

[0117] The decoder consists of four upsampled transposed convolutional layers, with the number of channels decreasing from 512 to 64. Each layer's input is a concatenation of the decoder's previous layer's output and the corresponding layer's features from the encoder (skip connections), preserving high-frequency details. The final layer outputs a single-channel confidence map, activated by a sigmoid function, where each pixel value represents the probability of being true or false at that location.

[0118] Input and output of the U-Net discriminator: During the training phase, the discriminator receives two types of input: the ground truth (HR) image (256×256×3) output by S1, and the ground truth (SR) image (256×256×3) output by the dynamic generator. The discriminator output is a pixel-wise confidence map (256×256×1), where each pixel value is activated by a sigmoid function, representing the probability that the input at that location is a real image.

[0119] S2.2 Cost Function Design:

[0120] The cost function consists of a weighted average of four loss terms, with some weights dynamically adjusted based on the gating signal.

[0121] S2.2.1 Pixel Loss (Charbonnier Loss)

[0122] Using Charbonnier loss, the expression is:

[0123]

[0124] In the formula: i is the index of each pixel in the image; HR_i is the i-th pixel value of the high-resolution ground truth image (normalized to the range [0,1]); SR_i is the i-th pixel value of the super-reconstructed image (also normalized to [0,1]); ε is a small constant, set as This is used to ensure that it is differentiable everywhere.

[0125] Based on the characteristics of remote sensing imagery, pixels with low brightness (dark areas) are assigned higher weights. Dark area mask is defined as follows:

[0126]

[0127] This involves generating a mask matrix of the same size as the high-resolution ground truth image HR, where pixels with values ​​less than the brightness threshold 30 / 255.0 (i.e., brightness values ​​below 30, approximately 0.1176 after normalization) are marked as 1.0, and all other pixels are marked as 0.0. The final pixel loss is then:

[0128]

[0129] torch.mean() means averaging all pixels in the entire image to obtain the final scalar loss value; torch.sqrt() means calculating the Charbonnier loss for each pixel.

[0130] This design prevents shadow areas from becoming overly smooth, thus improving the preservation of details in dark areas.

[0131] S2.2.2 Loss of physical perception

[0132] To overcome the feature distribution shift issue of general perceptual losses (such as VGG19 pre-trained on natural images) on remote sensing images, the general VGG19 perceptual loss is replaced with a remote sensing-specific feature extractor. This feature extractor uses ResNet-34 as its backbone network, removing its last fully connected layer and global average pooling layer, retaining only the first three residual blocks. Specifically, it is pre-trained on the Sentinel-2 satellite 10-meter resolution multispectral dataset (256×256 pixels) using an autoencoder: the encoder consists of the first three residual blocks of the aforementioned ResNet-34, and the decoder is a symmetrical transposed convolutional layer that reconstructs the original image and optimizes the mean squared error loss. After pre-training, the decoder is discarded, retaining only the encoder. The spatial resolution of the output feature map of the third residual block of this encoder is 1 / 4 of the input image (i.e., for a 256×256 input, the output feature map size is 64×64), with 256 channels. This encoder is denoted as φ_remote(·), and its parameters remain fixed during the super-resolution network training. Perceptual loss is defined as the L1 norm between the feature maps extracted from the ground truth image (HR) and the reconstructed image (SR) using φ_remote(·).

[0133]

[0134] In the formula: φ_remote() is the mapping function of the ResNet-34 encoder pre-trained on the Sentinel-2 dataset; ||·||1 is the L1 norm.

[0135] The weight λ2 of the loss is dynamically adjusted based on the gate signal signal: The weight is 1.0 for distant scenes (signal=0), 1.5 for mid-range scenes (signal=1), and 2.0 for close-up scenes (signal=2), thereby enhancing texture restoration in close-up areas.

[0136] S2.2.3 Adversarial Loss (RaGAN with Dynamic Weight)

[0137] This invention employs an adversarial loss in the form of a Relativistic Average Generative Adversarial Network (RaGAN). Unlike the discriminator in a standard GAN that estimates the absolute probability of whether a single sample is real, the RaGAN discriminator compares the relative realism between real and generated samples within a batch, thereby enabling the generator to learn edge and texture details more clearly.

[0138] Let the output of the discriminator D(·) be a real number logit (without Sigmoid activation), and let each batch contain N real images (HR) and N generated images (SR). Let the mean logit value of the real images within a batch be defined as... The mean logit value of batch endogenous images is , where E[·] represents the arithmetic mean of all samples in the batch.

[0139] The output of the RaGAN discriminator for real images is defined as follows: , which represents the probability that "the real image is more realistic than the average level of the batch of endogenous images".

[0140] The output of the RaGAN discriminator for the generated image is defined as follows: , which represents the probability that "the average level of the real images in the batch is more realistic than the generated image".

[0141] The total loss function of the RaGAN discriminator is:

[0142] L_adv_D = - E[ ​​log( D_real ) ] - E[ ​​log( D_fake ) ]

[0143] The loss function for the RaGAN generator is (used to train the generator to make the generated images more realistic):

[0144] L_adv_G = - E[ ​​log( 1 - D_fake ) ]

[0145] Where 1 - D_fake = 1 - σ( real_mean - D(SR) ) = σ( D(SR) - real_mean), that is, the generator attempts to maximize the probability that the generated image is more realistic than the average level of the real images in the batch.

[0146] In the total cost function of this invention, the adversarial loss L_adv used is L_adv_G.

[0147] In the formula: D(·) is the original logit value (real number) output by the discriminator; σ(·) is the Sigmoid function. real_mean indicates real-time calculation. E[·] represents the mathematical expectation.

[0148] The weight of the loss Dynamically adjust based on the gating signal:

[0149]

[0150] Distant view To reduce adversarial loss and prevent artifact flickering; in close-up (signal=2) To enhance adversarial loss and facilitate texture generation, the gradient norm of the discriminator is clipped to 1.0 during training to prevent gradient explosion.

[0151] S2.2.4 Spectral Angular Fidelity Loss

[0152] A dedicated spectral angle loss is added to remote sensing images, and its expression is:

[0153]

[0154] In the formula: HR_i and SR_i are the RGB three-channel vectors of the i-th pixel of the corresponding image, respectively; It is the sum of the dot products of the HR and SR vectors over all pixels; The L2 norm of all pixels in the HR image (i.e., the square root of the sum of squares of each channel). is the L2 norm of all pixels in the SR image.

[0155] This loss measures the spectral angle deviation between the reconstructed image and the ground truth image, effectively preventing the spectral characteristics of targets such as vegetation and water bodies from drifting. Weights The optimal value is 0.05. An approximate gradient formula is used during backpropagation to prevent the Arccos function from vanishing when the gradient approaches zero. This approximation is implemented using a custom `autograd.Function`, with a computational cost of less than 5%.

[0156] S2.2.5 Total Loss Function:

[0157] The total loss is the weighted sum of all losses:

[0158]

[0159] Fixed weights in the formula Set the value to 1.0. Dynamic adjustment strategy during training:

[0160] Warm-up period (0 ~ 50k iterations): Let... Only pixel loss and perceptual loss are trained;

[0161] The adversarial phase (50k ~ 400k iterations): The learning rate is dynamically adjusted according to the above signals, and cosine annealing is used.

[0162] Fine-tuning period (400k ~ 500k iterations): Increased to 0.1, learning rate decreased to .

[0163] Every 1000 iterations, the proportion of the four types of loss is recorded to ensure that L_pixel accounts for more than 40% and to prevent adversarial loss from dominating and causing training instability.

[0164] The offline training phase is now complete, and the trained RAG-SRNet model (FP32 accuracy) is obtained. We will now proceed to the online inference and deployment phase.

[0165] GPU-rendering co-acceleration optimization for S3 inference model:

[0166] The trained RAG-SRNet model (FP32 precision) cannot directly meet the latency and throughput requirements of real-time rendering on edge computing platforms (such as Jetson Orin NX). To address this, this invention designs a hardware-level collaborative acceleration scheme from model quantization and memory scheduling to asynchronous execution, achieving excellent performance metrics in super-resolution inference latency, frame rate, and power consumption from 512×512 input to 1024×1024 output. This scheme includes three sequentially dependent, progressively layered optimization steps: First, the trained model undergoes INT8 quantization and mixed-precision optimization to generate a lightweight and efficient TensorRT inference engine; then, based on the output dimension of this engine, a CUDA-OpenGL zero-copy memory mapping is established to eliminate CPU data movement; finally, an asynchronous inference thread pool, hardware semaphore synchronization, and a GPU-side double-buffering mechanism are constructed to achieve fully pipelined parallelism for inference and rendering. The above steps are executed in a strict sequence: only after obtaining the quantization engine in S3.1 can the memory allocation and interoperability registration in S3.2 be performed according to its output size; only after zero-copy resource establishment in S3.2 can asynchronous scheduling be started in S3.3. The input, processing, and output of each step are described in detail below.

[0167] S3.1 TensorRT INT8 Quantization and Mixed Precision Optimization:

[0168] The input for this step is:

[0169] The trained RAG-SRNet model weights (FP32 accuracy, typically saved in .pth or ONNX format) and the calibration dataset consist of 300 online synthesized physically degraded tiles. Specifically, 300 256×256 HR tiles are randomly selected from Level 17 of the 18-level LOD pyramid, and corresponding 128×128 LR tiles are generated online using the physical degradation module (optical MTF, atmospheric turbulence, quantization noise) and second-order sinc filtering described in S1. This process ensures that the degradation distribution of the calibration data is strictly consistent with that of the training data, avoiding accuracy loss due to distribution shift after quantization.

[0170] The processing procedure for this step is as follows:

[0171] 1. PTQ calibration:

[0172] PTQ (Post-Training Quantization) is a quantization technique that directly converts a pre-trained floating-point model (usually FP32) into a low-precision integer model (such as INT8) after the model has been fully trained, without requiring retraining or fine-tuning. To avoid privacy risks associated with real remote sensing data, this invention uses online synthesized physically degraded tiles as a calibration set. The specific implementation steps are as follows:

[0173] 300 256×256 HR tiles were randomly selected from Level 17 of the 18-level LOD pyramid;

[0174] Using the physical degradation module (optical MTF, atmospheric turbulence, quantization noise) described in S1 and a second-order sinc filter, corresponding 128×128 LR tiles are generated online. This process ensures that the degradation distribution of the calibration data is strictly consistent with that of the training data, avoiding a sharp drop in accuracy due to distribution shift after quantization.

[0175] The calibration algorithm employs TensorRT's built-in entropy calibration, based on the KL divergence minimization principle. It maps the FP32 activation value histogram to 256 discrete levels of INT8 and calculates the quantization parameters (scaling factor and zeros). Key parameter settings are as follows:

[0176] Number of calibration batches: 30 batches, 10 tiles per batch, totaling 300 tiles;

[0177] Statistics window: Each output channel of the RRDB dense block is statistically analyzed independently, rather than the entire graph is statistically analyzed globally, ensuring fine quantization granularity between channels.

[0178] Sensitivity threshold: Set the quantization sensitivity threshold to 0.95 for Charbonnier loss-sensitive layers (such as 1×1 convolutions before PixelShuffle). If the PSNR loss of this layer is greater than 0.05 dB after calibration, the layer will be automatically reverted to FP16.

[0179] 2. Mixed Precision Strategy: Since not all layers are suitable for INT8 quantization, this invention implements a layer-level mixed precision strategy, with the specific rules as follows:

[0180] Layers that enforce FP16 include: PixelShuffle layer: Upsampling operations are sensitive to quantization, and INT8 is prone to introducing checkerboard artifacts. This layer is forced to maintain FP16, and Q / DQ nodes (Quantize / Dequantize) are inserted between it and the preceding and following INT8 layers, with scaling factors automatically inserted by TensorRT; First and last convolutional layers: Convolutional layers that directly contact the data at input and output maintain FP16 to prevent quantization overflow caused by excessively large numerical ranges at the beginning and end; sinc residual branch: The sinc branch in HighQualPath has a wide dynamic range due to learning high-frequency compensation, and is forced to FP16.

[0181] INT8 layer: The computationally intensive layers such as the 3×3 dense convolution and discriminator coding layer in RRDB use INT8, enjoying a speedup of 2-4 times.

[0182] Rollback Decision: In TensorRT's command-line tool trtexec, the mixing precision can be specified using --precisionConstraints=prefer, and TensorRT will automatically perform layer fusion and rollback analysis; alternatively, the precision of each layer can be directly set using the setPrecision() method of the TensorRT C++ API. If INT8 quantization of a certain layer results in a final PSNR loss > 0.15 dB, then that layer will automatically roll back to FP16.

[0183] 3. Dynamic Shape Profile Configuration

[0184] The osgEarth rendering thread may suddenly request 1-4 tiles. This invention configures a dynamic Shape Profile during the TensorRT engine build process, setting three dimensions: minimum batch = 1; optimal batch = 2; maximum batch = 4. TensorRT pre-allocates GPU memory based on max_shape. During actual inference, the batch size is dynamically adjusted using TensorRT's IExecutionContext::setBindingDimensions() method, avoiding the duplication of compilation across multiple engines.

[0185] 4. Engine serialization: Compile to generate the TensorRT inference engine, which can be saved as a .engine file or reside directly in memory.

[0186] The output of this step is:

[0187] The TensorRT inference engine (INT8 / FP16 mixed precision) includes the network structure, quantization parameters, the shape of the output tensor (e.g., [batch, 3, 256, 256], i.e., RGB three-channel, 256×256 spatial resolution) and data type.

[0188] S3.2 GPU-OpenGL Zero-Copy Co-scheduling

[0189] The core innovation of this invention lies in the reuse of the inference output buffer and the video memory address of the OpenGL texture object, which completely eliminates the memory transfer overhead.

[0190] The input for this step is:

[0191] S3.1 outputs the TensorRT inference engine (the output tensor size needs to be obtained, for example, 1024×1024×3 bytes; for 256×256 output, it is 256×256×3 bytes, but in actual processing, the tile size can be larger; here, 256×256 is used as an example in the embodiment); and the OpenGL rendering context (the graphics environment provided by osgEarth).

[0192] The specific processing procedure is as follows:

[0193] 1. Device Memory Allocation and Alignment: Based on the engine output size, the CUDA Runtime API's cudaMalloc() function allocates device memory d_output, with a size of width × height × channels × sizeof(float) (half-precision allocation if the output is FP16). To optimize memory bandwidth, the CUDA Runtime API's cudaMallocPitch() function allocates memory with 256-byte alignment, conforming to the Jetson platform's L2 cache line size, avoiding bank conflicts, and improving bandwidth utilization by 15%. To implement double buffering, two device buffers are pre-allocated, denoted as buffer_ping and buffer_pong.

[0194] 2. CUDA-OpenGL Interoperability Resource Registration: Call the cudaGraphicsGLRegisterBuffer() function of the CUDA Runtime API to register the above device memory as an OpenGL buffer object (PBO, Pixel Buffer Object). Set the registration flag to cudaGraphicsRegisterFlagsWriteDiscard, indicating that the CUDA side will write data and the OpenGL side will read data.

[0195] 3. PBO Mapping Mechanism: Define the PixelBufferObject class and override its map() method to directly return the device address of d_output. The OpenGL driver detects that this address is in the unified memory space and automatically selects the EGL_NV_stream_dma extended zero-copy path of EGL (the interface between OpenGL and the native window system), instead of the traditional Host→Device copy path of glMapBuffer.

[0196] 4. Zero-copy texture upload process verification: After inference, CUDA writes to d_output and calls the cudaGraphicsMapResources() function of the CUDARuntime API to map the PBO to an OpenGL-accessible state (completed on the GPU side, with no data transfer). Then, it calls the glTexSubImage2D() function of the OpenGL core API, directly passing the device address of d_output as its pixel parameter. The OpenGL driver recognizes this address as device memory, bypassing host-side data copying, and uses the DMA (Direct Memory Access) engine to directly transfer data from the CUDA buffer to the GPU memory area where the texture object resides, taking <0.3 ms (compared to 6-8 ms for the traditional path).

[0197] The output of this step is:

[0198] The registered zero-copy double buffers (buffer_ping and buffer_pong), each in device memory shared by CUDA and OpenGL, can be used directly as a texture data source;

[0199] A CUDA external semaphore handle for hardware semaphore synchronization (initialized by calling the cudaImportExternalSemaphore() function of the CUDA RuntimeAPI to import an OpenGL semaphore object, which is created by the EGL_NV_sync extension of EGL and obtained as a handle of type cudaExternalSemaphore_t) is provided for use by S3.3.

[0200] S3.3 Inference-Rendering Asynchronous Decoupling and GPU-Side Double Buffering

[0201] The input for this step is:

[0202] The TensorRT inference engine output by S3.1;

[0203] S3.2 outputs zero-copy double buffers (buffer_ping, buffer_pong) and CUDA external semaphore handle (cudaExternalSemaphore_t).

[0204] The tile request task queue from the osgEarth plugin (containing LR image data, tile coordinates, etc.).

[0205] The processing procedure for this step is as follows:

[0206] 1. Create an independent inference thread pool: Create an independent inference thread pool (preferably 2 threads) in the osgEarth plugin, decoupled from the main rendering thread: Each inference thread has an independent CUDA stream, created through the cudaStreamCreate() function of the CUDA Runtime API, with a type of cudaStream_t.

[0207] 2. Task Queue and Lock-Free Management: After receiving a tile request in `createImage()`, the main thread pushes the task (containing a `cv::cuda::GpuMat` pointer (wrapping the LR image), tile key, semaphore handle, etc.) to the thread-safe `std::deque`. <task>The queue uses atomic operations to manage its capacity (limited to 64), then immediately returns nullptr, telling osgEarth "tiles will be served later", without blocking the rendering loop.

[0208] 3. Asynchronous inference and double-buffered switching:

[0209] The inference thread pops a task from the queue and selects either buffer_ping or buffer_pong as the output buffer based on the current frame index (alternating via the atomic boolean variable current_write_buffer).

[0210] Calling the `IExecutionContext::enqueueV3()` method of TensorRT (or the older `enqueueV2()` method) sends the input tensor (LR image, pre-mapped to GpuMat via page-locked memory, requiring no additional copying) and the output buffer pointer into the CUDA stream, initiating asynchronous inference (non-blocking). The inference thread then continues processing the next task without waiting for the current inference to complete.

[0211] 4. Hardware semaphore synchronization:

[0212] Before calling `enqueueV3()`, the CUDA external semaphore handle (`cudaExternalSemaphore_t`) is bound to the CUDA stream via the CUDA Runtime API's `cudaSignalExternalSemaphoresAsync()` function (or the driver-level `cuSignalExternalSemaphoresAsync`), and the semaphore parameters are set. After the last kernel layer of inference is completed, the GPU hardware automatically sets the semaphore state to `GL_SIGNALED`.

[0213] On the OpenGL side, a synchronization object (GLsync) is created using the glFenceSync() function of the OpenGL core API, and a callback is registered. When the semaphore state changes, the OpenGL driver automatically triggers a texture update operation: the contents of the output buffer are directly used as texture data to call the OpenGL glTexSubImage2D() function (using the zero-copy PBO DMA path established by S3.2) to update the corresponding tile. After the update is complete, the callback calls tile_key.setImage(hr_image), and osgEarth automatically sends the tile to the rendering queue. There is no CPU polling or blocking throughout the entire process.

[0214] 5. GPU-side dual-buffered pipeline:

[0215] The two pre-allocated buffers form a ping-pong buffer: the inference thread alternates between buffer_A and buffer_B for output. While buffer_A is being written to by TensorRT, the rendering thread reads the previous batch of results from buffer_B (accessed directly through previously registered texture objects).

[0216] Synchronization point: Call the CUDA Runtime API's cudaStreamSynchronize() function within the inference thread to ensure that the current write to the buffer is complete before switching to the next one. This synchronization call resides within the inference thread and does not block the main rendering thread.

[0217] Performance improvements: Double buffering effectively reduces latency from "inference time + texture upload time" to max(inference time, texture upload time), achieving near 100% GPU utilization at osgEarth's 60 Hz rendering frequency.

[0218] 6. Batch Inference Optimization: If multiple tile requests are received within 1 millisecond, they are automatically merged into batch=2 or batch=4 for inference. Inference latency is reduced from 16 ms per tile (StandardPath) to 9 ms for batch=2 and 6 ms for batch=4, resulting in an overall throughput improvement of 40%.

[0219] The output of this step is:

[0220] Asynchronous inference pipeline at runtime: The main rendering thread and the inference thread run in full parallel, and the super-resolution results are automatically updated to the graphics engine via hardware semaphores without CPU intervention;

[0221] Real-time performance metrics (obtained via monitoring API): average inference latency, rendering frame rate, double buffer hit rate, GPU utilization, cache hit rate, view frustum distribution, etc.

[0222] Through chained optimization of S3.1→S3.2→S3.3, the super-resolution inference of this invention achieves extreme edge efficiency on Jetson Orin NX with an average inference latency of 13ms, a frame rate increase of more than 25%, and a CPU usage reduction of 77%.

[0223] Asynchronous zero-copy integration architecture design for S4 graphics engine:

[0224] This invention achieves asynchronous inference-rendering across the entire chain in osgEarth (an open-source geospatial 3D rendering engine) through a plug-in architecture, zero-copy memory mapping, and hardware signal synchronization, resulting in significantly reduced latency and CPU usage.

[0225] S3 achieved model quantization and inference acceleration, but the core challenge in practical engineering implementation was how to seamlessly embed the optimized super-resolution capabilities into the existing graphics engine, enabling it to automatically trigger super-resolution upon tile requests and use the super-resolution results during rendering. S4 implemented a "plug-and-play" integration solution through a plugin architecture: it encapsulates the TensorRT inference engine, zero-copy buffer, and asynchronous scheduling mechanism built in S3 into a custom plugin class that inherits from osgEarth::TileSource, overriding its initialization and tile creation methods. This makes the system's upper layers completely transparent when using tiles, allowing them to obtain super-resolution enhancements without modifying the engine source code. The internal logic is detailed below.

[0226] S4.1 Plugin Architecture and Asynchronous Scheduling (Sequential Dependency of Resource Initialization)

[0227] The input for this step is:

[0228] The output of S3.1 is the TensorRT inference engine (serialized as a .engine file or a memory object); the output size information of the engine (e.g., 256×256×3 bytes, corresponding to the tile size after super-resolution); the system configuration file (.earth file or XML configuration), which includes the frustum activation distance (default 5000 meters), MTF parameters, device ID, etc.

[0229] The processing procedure for this step is as follows:

[0230] Plugin encapsulation and resource initialization: Create a `SuperResolutionTileSource` class, inheriting from the `TileSource` base class of `osgEarth` (the standard tile data source interface defined in the `osgEarth` official documentation; `ImageLayer` obtains tile data through the `createImage()` method). Override the following two methods:

[0231] The `initialize()` method is called once when the plugin loads, completing the registration of all one-time resources. Specifically, it includes:

[0232] The TensorRT engine file is loaded from the configuration file, and the `nvinfer1::IRuntime::deserializeCudaEngine()` method of the TensorRT C++ API is called to deserialize it and obtain the inference engine instance. The `cudaSetDevice()` method of the CUDA Runtime API is called to specify the GPU device (based on the device ID in the configuration file). Based on the engine output size, the zero-copy memory mapping process of S4.2 is called to allocate a double buffer and register it as an OpenGL graphics resource. The hardware semaphore and double buffer initialization process of S4.3 is called (creating CUDA external semaphore handles, starting the inference thread pool, etc.). The execution order of the `initialize()` method strictly follows the dependency relationship of S4.2 → S4.3, ensuring that all resources are ready before entering the runtime phase.

[0233] The `createImage()` method processes a single tile request each frame when called by `osgEarth`. Specifically, it calculates the distance between the tile center point and the camera viewpoint based on the passed camera view matrix and tile geographic bounding box. If the distance is greater than the frustum activation distance (default 5000 meters, dynamically adjustable via XML configuration), it calls the native data source interface of `osgEarth` to directly return the LR tile (without activating super-resolution to save computational power). For tiles requiring super-resolution, it wraps the tile coordinates (TileKey) and the corresponding LR image data (osg::Image*) into a task structure. It then calls the `cudaHostRegister()` function of the CUDARuntime API to register the pixel data pointers of the LR image as pinned memory for zero-copy GPU access. Finally, it pushes the task to the thread-safe `std::deque` through atomic operations. <task>Queue; immediately returns nullptr, informing osgEarth that the data for this tile will be provided asynchronously later (without blocking the rendering loop).

[0234] The output of this step is:

[0235] A fully initialized plugin instance, including a ready TensorRT engine, a registered zero-copy double buffer, an started inference thread pool, and bound hardware semaphores; a lock-free task queue for passing tile requests between the main rendering thread and the inference thread (queue capacity limited to 64, managed using lock-free atomic operations).

[0236] S4.2 Zero-copy memory mapping (resource allocation and registration based on engine output size):

[0237] The input for this step is:

[0238] The TensorRT engine output size obtained in S4.1 initialize() (e.g., the super-resolution tile resolution is 256×256, RGB 3 channels, and each pixel is 2 bytes when stored in FP16 half precision, for a total size of 256×256×3×2 = 393216 bytes); the current OpenGL rendering context (the graphics context provided by the osgEarth framework).

[0239] The processing procedure for this step is the same as that for step S3.2. After performing a resource readiness check on the same process as step S3.2, the process is executed again.

[0240] The output of this step is:

[0241] Two registered zero-copy double buffers (buffer_ping and buffer_pong), each in device memory shared by CUDA and OpenGL, can be used directly as a texture data source;

[0242] A CUDA external semaphore handle for hardware semaphore synchronization (initialized by calling the cudaImportExternalSemaphore() function of the CUDA RuntimeAPI to import an OpenGL semaphore object and obtain a handle of type cudaExternalSemaphore_t) is provided for use by S4.3.

[0243] S4.3 Hardware Semaphores and Double Buffer Initialization (GPU-side Synchronization Mechanism and Thread Pool Startup)

[0244] The input for this step is:

[0245] S4.2 output zero-copy double buffers (buffer_ping and buffer_pong); CUDA external semaphore handle (imported by cudaImportExternalSemaphore(), type cudaExternalSemaphore_t); configuration parameters for the inference thread pool (number of threads, preferably 2).

[0246] The processing procedure for this step is as follows:

[0247] 1. Inference Thread Pool Creation: Two independent inference threads are created using the `std::thread` function from the C++11 standard library. Each thread has its own independent CUDA stream (created via the `cudaStreamCreate()` function of the CUDA Runtime API, of type `cudaStream_t`). Within the thread's loop, tasks are continuously popped from the lock-free task queue created in S4.1 for consumption.

[0248] 2. CUDA External Semaphore Binding: Import an OpenGL synchronization object (created by the OpenGL glFenceSync() interface) using the cudaImportExternalSemaphore() function of the CUDA Runtime API to obtain a handle of type cudaExternalSemaphore_t (this functionality can also be implemented using the cuImportExternalSemaphore function of the CUDA Driver API). This handle supports cross-platform synchronization between CUDA and operating system-level semaphores. In the inference thread, call the cudaSignalExternalSemaphoresAsync() function to bind the semaphore to the corresponding CUDA stream, and rename the currently used semaphore to active_semaphore for easy tracking.

[0249] 3. Task Push and Callback Registration: Before each inference execution, the inference thread retrieves a task from the queue and selects either `buffer_ping` or `buffer_pong` as the output buffer based on the atomic boolean variable `current_buffer_index` (switched via lock-free atomic operations). It calls TensorRT's `IExecutionContext::enqueueV3()` to send the input tensor and output buffer pointer to the CUDA stream and returns immediately without waiting for execution to complete. The inference thread registers the context through the semaphore handle bound to the CUDA stream. When the inference kernel completes, the GPU hardware automatically signals this semaphore, notifying the OpenGL side that the buffer is ready.

[0250] 4. OpenGL Synchronization and Texture Update: The OpenGL rendering thread waits for semaphore signals through a synchronization object. Specifically, after mapping is complete, the OpenGL glFenceSync(GL_SYNC_GPU_COMMANDS_COMPLETE, 0) interface is called to create a fence synchronization object, which is then inserted into the OpenGL command stream. The completion status of the GPU command is checked using glWaitSync() or glClientWaitSync(). When the semaphore is set to the GL_SIGNALED state by the GPU hardware, the OpenGL driver automatically triggers a texture update: glTexSubImage2D() (pixels=0) is called to directly upload the inference results in the PBO to the video memory where the texture object resides. The entire synchronization process is fully automatic, without CPU polling or blocking.

[0251] 5. Seamless Double-Buffering Switching: `buffer_ping` and `buffer_pong` form a ping-pong buffer. When the inference thread writes to `buffer_ping`, the rendering thread reads the previous batch of texture data from `buffer_pong`. During each frame's rendering, the `nullptr` returned by `createImage()` does not affect the already rendered content. Once a buffer completes signal synchronization and updates its texture, osgEarth obtains the super-resolved texture the next time it references that tile. Lock-free switching is performed at the start of each frame's inference using the atomic boolean variable `current_write_buffer`, effectively reducing latency from "inference time + texture upload time" to `max(inference time, texture upload time)`, achieving near 100% GPU utilization at osgEarth's 60 Hz refresh rate. Synchronization checkpoints only require calling the CUDA Runtime API's `cudaStreamSynchronize()` before the double-buffering switch to ensure the current buffer is written to. This call resides within the inference thread and does not block the main rendering thread.

[0252] 6. Batch Inference Optimization: If multiple tile requests are received within 1 millisecond, the inference thread will automatically merge them into batch=2 or batch=4 for processing. The merged calls to the TensorRT engine utilize multi-batch parallelism to reduce inference latency from 16 milliseconds (StandardPath) for a single tile to 9 milliseconds for 2 tiles and 6 milliseconds for 4 tiles.

[0253] The output of this step is:

[0254] A fully initialized asynchronous inference pipeline: the main rendering thread and the inference thread run in parallel, and the super-resolution results are automatically updated to the graphics engine via hardware semaphores; real-time performance metrics (output via monitoring API): average inference latency, rendering frame rate, double buffer hit rate, GPU utilization, cache hit rate, and view frustum signal distribution, etc., supporting on-screen display and JSON serialization output.

[0255] In the `initialize()` method of S4.1, execution strictly follows the order of S4.2 (allocating and registering zero-copy double buffers) → S4.3 (importing hardware semaphores and starting the double-buffered inference thread pool) to ensure that tile requests are responded to only after resources are ready. The `createImage()` method of S4.1 uses the resources established in S4.2 and S4.3 to handle each tile request at runtime, automatically triggering texture updates through hardware semaphores to achieve zero-copy asynchronous inference-rendering across the entire chain. Through the sequential construction of S4.1–S4.3, this invention achieves a "plug-and-play" super-resolution integration solution without modifying the graphics engine source code. Physical parameters (such as MTF curves and atmospheric r0) can be dynamically configured through .earth files, significantly shortening the project deployment cycle.

[0256] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.< / task> < / task>

Claims

1. A neural network-based integrated visual scene super-resolution method, characterized in that, This includes an offline training phase and an online inference deployment phase; The offline training phase includes: Step S1: Based on the remote sensing data of the multi-level LOD pyramid structure, construct a training dataset driven by physical-data hybrid processing. The construction includes sequentially performing MTF-based optical resolution degradation, atmospheric turbulence model-based simulated degradation, quantization noise injection, and higher-order degradation and sinc artifact enhancement processing on high-resolution image tiles to form low-resolution-high-resolution image pairs. Step S2: Construct and train a dynamic super-resolution network for frustum perception. The network includes a frustum classifier, a dynamic generator, and a discriminator. The frustum classifier outputs discrete gating signals based on the input image. The dynamic generator contains multiple inference paths with different computational complexities and adaptively selects and activates one of the paths for super-resolution reconstruction based on the gating signals. The online inference deployment phase includes: Step S3: Perform INT8 quantization and mixed precision optimization on the trained network based on TensorRT, and realize the reuse of the memory address of the inference output buffer and the graphics texture object through zero-copy memory mapping technology between the GPU and the graphics API. Step S4: Integrate the optimized model into the graphics engine, and execute the tile request, super-resolution calculation and texture rendering process in parallel through the asynchronous inference thread pool and hardware semaphore synchronization mechanism.

2. The method of claim 1, wherein, Step S1 specifically includes: MTF curves were obtained from satellite technical manuals, and optical blur kernels were constructed using inverse Fourier transform. Based on the Kolmogorov turbulence model, the atmospheric coherence length is dynamically calculated according to meteorological visibility and shooting altitude to generate a long exposure point spread function kernel; The intensity of the second fuzzy ambiguity is adaptively adjusted based on the local variance for the first-order degradation result; The sinc filter with a dynamically changing cutoff frequency is executed with a preset probability.

3. The method of claim 1, wherein, The implementation method of the frustum classifier is as follows: Multi-layer convolutional layers are used to downsample and extract features from the input image, and three-dimensional logits are output through global average pooling and fully connected layers. Discrete gating signals are generated through the argmax operation. The gating signals include three levels corresponding to far-field, mid-field and near-field, respectively corresponding to screen projection areas of less than 4096 pixels², between 4096 and 65536 pixels², and greater than 65536 pixels². During the training phase, the parameters of the frustum classifier are fixed, and the gating signal is updated at a frequency lower than the rendering frame rate.

4. The method of claim 1, wherein, The dynamic generator is configured with multiple inference paths as follows: The FastPath path contains the first number of Residual Dense Blocks (RRDBs), with 64 channels. The mid-range path, StandardPath, contains more than the first number of RRDBs, has 96 channels, and incorporates a window self-attention mechanism. The HighQualPath for close-up shots contains more RRDBs than the mid-range path, has 128 channels, and adds a sinc residual branch to suppress artifacts. Among them, the first 8 RRDBs of the three paths share weights and the number of channels is uniformly set to 128. After sharing RRDBs, the far-field path and the mid-field path are reduced to their respective number of channels by 1×1 convolution, while the near-field path is not reduced in dimension. During training, a course learning strategy is adopted to unlock the unique layers of each path in stages.

5. The method of claim 1, wherein, The discriminator adopts a U-Net structure, which includes a 4-layer encoder and a 4-layer decoder. Features are passed through skip connections, and spectral normalization is applied after each convolution. The spectral norm is frozen during inference.

6. The method according to claim 1, characterized in that, The cost function used to train the network includes: Pixel loss is achieved using Charbonnier loss, which assigns double the weight to pixels with brightness below 30 / 255. The perceptual loss is calculated using the output of the third residual block of a ResNet-34 encoder pre-trained on the Sentinel-2 dataset, yielding an L1 distance with weights... ; Adversarial loss is achieved using a relatively average generative adversarial network (GAN) with weights... ; Spectral angle fidelity loss is used to constrain the spectral characteristics of the reconstructed image.

7. The method according to claim 1, characterized in that, Step S3 specifically includes: S3.1: Based on TensorRT, INT8 quantization and mixed precision optimization are performed. INT8 is used for the RRDB dense convolutional layer. FP16 is forced to be maintained for the PixelShuffle layer, the first and last convolutional layers and the sinc residual branch. Entropy calibration is adopted and a calibration dataset synthesized by physical degradation is used. Sensitivity threshold is set for automatic fallback. S3.2: During the initialization phase, cudaMalloc() is called to allocate the inference output buffer, and cudaGraphicsGLRegisterBuffer() is called to register the buffer as an OpenGL Pixel Buffer Object (PBO), with the registration flag set to cudaGraphicsRegisterFlagsWriteDiscard, so that CUDA and OpenGL share the same device memory. During inference, TensorRT writes the super-resolution results directly to the device memory, and then calls cudaGraphicsMapResources() to map the PBO to an OpenGL-accessible state. During texture upload, the PBO is bound as a pixel unpacking buffer, glTexSubImage2D() is called and the pixel shift is set to zero. The OpenGL driver recognizes that the data source is the PBO, automatically enables the DMA zero-copy path, and transfers the data directly from the CUDA buffer to the GPU memory area where the texture object is located, completely bypassing the CPU-side data transfer. S3.3: Construct an independent inference thread pool, GPU-side double buffering, and synchronization with CUDA external semaphores to achieve asynchronous pipeline parallelism.

8. The method according to claim 1, characterized in that, Step S4 specifically includes: S4.1: Create a plugin class that inherits from osgEarth::TileSource, override initialize() to complete the loading of the TensorRT engine, zero-copy resource registration, hardware semaphore initialization and inference thread pool startup; override createImage() to implement view frustum distance judgment, push the task into the lock-free queue for tiles that need super-resolution and immediately return a null pointer, without blocking rendering; S4.2: Based on the engine output size, allocate double buffers and register them as CUDA-OpenGL shared PBOs to achieve zero-copy texture upload; S4.3: Utilizes CUDA external semaphores to enable automatic OpenGL texture updates triggered upon GPU-side inference completion, combined with double-buffered ping-pong switching to achieve fully parallel rendering and inference, and provides an XML configuration interface and runtime monitoring API.

9. A comprehensive visual super-resolution system based on neural networks, characterized in that, include: The training data construction module is used to perform physical-data hybrid degradation processing to build the training dataset; The dynamic super-resolution network module includes a frustum classifier, a multi-path dynamic generator, and a discriminator, which are used to perform dynamic super-resolution inference. The inference acceleration module integrates the TensorRT engine and is used to load quantized models and perform zero-copy asynchronous inference. The graphics engine integration module is integrated into the osgEarth graphics engine as a plug-in, and is used to perform plug-in integration and asynchronous scheduling.

10. The system according to claim 9, characterized in that, The graphics engine integration module includes the SuperResolutionTileSource class, which inherits from osgEarth::TileSource and overrides the createImage() and initialize() methods. In initialize(), TensorRT engine deserialization, CUDA-OpenGL interoperability resource registration, and hardware semaphore import are completed. In createImage(), frustum distance judgment, asynchronous task submission, and zero-copy texture upload are implemented.

Images