Information density routing and gaze focused infrared video super-resolution method and system

Abstract

The application discloses an infrared video super-resolution method and system based on information density routing and gaze focusing, relates to the technical field of infrared image processing, and comprises the following steps: performing spatial up-sampling on a low-resolution infrared video and mapping the low-resolution infrared video to a latent space to generate a low-resolution latent representation; generating an information density map and a binary focus mask through LGDR; generating a latent feature by collaboratively recovering a token through GFC; performing density-aware modulation on the latent feature in an IDGC; and decoding the modulated latent feature through a decoder to obtain a high-resolution infrared video; and the method and system provided by the application solve the technical problems that the existing infrared video super-resolution method has high redundancy in global uniform calculation, has large diffusion reasoning time delay, does not adaptively focus on key regions by using the sparse characteristics of infrared video information, and thus leads to insufficient recovery of target edges and thermal anomaly zone details, poor background time sequence stability, and difficulty in balancing reconstruction quality and real-time deployment efficiency.

Description

Technical Field

[0001] This invention relates to the field of infrared image processing technology, and in particular to an infrared video super-resolution method and system for information density routing and gaze focusing. Background Technology

[0002] Infrared imaging systems are typically limited by factors such as sensor size, pixel pitch, imaging distance, optical system, and acquisition cost. The acquired videos generally suffer from insufficient spatial resolution, blurred edge contours, weak thermal texture, low local contrast, and thermal noise interference. In applications such as autonomous driving, security monitoring, industrial inspection, and maritime target observation, low-resolution infrared video directly affects the identification of target boundaries, the preservation of small targets, and the reliability of subsequent visual tasks. The goal of infrared video super-resolution technology is to recover high-resolution results based on low-resolution infrared video while ensuring the consistency of thermal structure between frames.

[0003] With the development of deep learning technology, current video super-resolution methods mostly use convolutional networks, loop propagation, deformable alignment, and Transformer temporal modeling to mine effective information in adjacent frames. In recent years, diffusion models and diffusion Transformers have also been applied to the field of video super-resolution, using generative priors and long-range spatiotemporal modeling capabilities to improve the detail reconstruction effect. At the same time, some studies have attempted to introduce diffusion models into the thermal infrared domain to optimize the texture and edge reconstruction performance of single-frame infrared images.

[0004] However, current infrared video super-resolution technologies still have many shortcomings and deficiencies: First, most existing methods adopt an approximately uniform processing strategy for all spatial regions, temporal frames, or visual tokens, failing to fully distinguish the differences in information content between target edges, thermal anomaly regions, moving targets, and large areas of smooth background within infrared videos. In contrast, most background regions in infrared videos change relatively smoothly over time, and the structures that truly require high-intensity reconstruction often occupy only a small spatial proportion. A uniform computation mode easily leads to computational redundancy. Second, while diffusion-based video super-resolution methods can improve reconstruction quality to some extent, multi-step sampling results in high inference latency. Even with a one-step or fewer-step solution scheme, if dense 3D attention is still applied to all tokens, it still results in high memory usage, computational load, and runtime, hindering the deployment of infrared video enhancement in real-time or near-real-time scenarios. Third, existing Transformer or DiT video restoration networks mostly focus on global spatiotemporal hybrid modeling, which does not fully meet the imaging characteristics of infrared video, which is characterized by "smooth background and sparse information regions." They lack a mechanism to adaptively delineate key restoration regions based on the differences between local and global thermal distribution. When dealing with dynamic targets, low-contrast targets, thermal boundaries, and local high-information-density regions, existing methods are prone to problems such as insufficient detail restoration, excessive edge smoothing, and unstable local thermal structures. Simply increasing the number of globally dense attention layers would significantly increase inference costs. Fourth, although some methods use sparse attention or fixed-window attention to reduce computation, their sparse regions are mostly determined by fixed rules or general visual priors. They cannot adaptively adjust the proportion of focus tokens according to changes in the information density of infrared video content, and they also cannot simultaneously ensure the temporal stability of smooth backgrounds and the detail enhancement effect of high-information regions. Summary of the Invention

[0005] The purpose of this invention is to provide an infrared video super-resolution method and system with information density routing and gaze focusing, which solves the technical problems of existing infrared video super-resolution methods, such as high redundancy of uniform computation across the entire domain, large diffusion inference delay, failure to utilize the sparse characteristics of infrared video information to adaptively focus on key areas, resulting in insufficient recovery of details at target edges and thermal anomaly areas, poor background temporal stability, and difficulty in balancing reconstruction quality and real-time deployment efficiency.

[0006] To achieve the above objectives, this invention provides an infrared video super-resolution method based on information density routing and gaze focusing, comprising the following steps: S1. Spatial upsampling of low-resolution infrared video and mapping to latent space to generate low-resolution latent representation; S2. Generate an information density map and a binary focusing mask using the Local-Global Information Density Routing (LGDR) module; S3. The gaze-focusing co-recovery module GFC performs co-recovery of the token based on the information density map and the binary focusing mask to generate latent features; S4. In the Information Density Sensing Modulation and Supervision Module IDGC, the information density map output by LGDR is used to perform density sensing modulation on the latent features. S5. The modulated latent features are decoded using the decoder of the video variational autoencoder (VAE) to obtain high-resolution infrared video. .

[0007] Preferably, the specific process for generating the low-resolution latent representation in S1 is as follows: S11, Input low-resolution infrared video ,right Spatial upsampling is performed to align it spatially with the target high-resolution video; S12. The spatially upsampled video is mapped to the latent space using a video variational autoencoder (VAE) to obtain a low-resolution latent representation. .

[0008] Preferably, the specific process for generating the information density map in S2 is as follows: S21. Represent the low-resolution latent representation In sequence view With spatiotemporal grid view Rearrange them between them. Represents the set of real numbers. This represents the total number of tokens in the sequence view, and , Represents the time dimension of the potential grid. , Both represent the spatial dimensions of the potential grid. and These represent the height and width of the potential mesh, respectively. Represents the feature dimension of the token; S22. Calculate the low-resolution latent representation. The scalar energy of an independent feature unit token is expressed as: ; In the formula, Indicates the first Spatial location in the frame The token at the location; Indicates the first Spatial location in the frame The scalar energy of the token at that location; It is a stable term; S23. After normalizing the energy values ​​within the same video segment, convert them to... Distribution of variable microhistograms over a fixed interval; S24. Calculate the average value of the soft histogram of each token in its local neighborhood to obtain the local distribution. The local distribution reflects the local thermal structure of the tokens; simultaneously, the average soft histogram of all tokens in the entire video segment is calculated to obtain the global distribution. The global distribution reflects the overall thermal response of the entire video segment; the spatial location is calculated by comparing the differences between the local and global distributions. The information density score of the token relative to the global background. An information density map is generated, and the information density score is represented using KL divergence: ; In the formula, The binning index of the soft histogram, with values ​​from 1 to K; This represents the probability that the global soft histogram is distributed in the k-th bin; Indicates the spatial position of the i-th frame. The probability that the local soft histogram of a given token is distributed in the k-th bin; The KL divergence represents the local distribution relative to the global distribution. It is used to measure the difference between the local thermal structure and the overall thermal response of the video. The KL divergence is the Kohlbek-Leibler divergence.

[0009] Preferably, the specific process for generating the binary focusing mask in S2 is as follows: S25. Extracting statistical features from the information density map. ,Will The input is given to an MLP network, which is activated by a sigmoid function to adaptively calculate the focus token ratio for the current video segment. The expression is: ; In the formula, and These represent the preset minimum and maximum routing ratios, respectively; This represents the Sigmoid activation function, used to map the output of a lightweight prediction network to the 0-1 range; This represents a lightweight prediction network; Indicates statistical characteristics; S26. For each frame in a video segment, use Top-K filtering to select the spatial locations with the highest information density within a single frame, obtaining the set of selected spatial locations for each frame. The expression is: ; In the formula, Indicates the first The set of selected Top-K spatial locations in a frame; This represents the Top-K index selection operator, used to return the largest value in the information density map. An index of a spatial location; Indicates the first The information density map of the frame; where, and These are the height and width of the potential frame space grid, respectively. Indicates rounding up; S27, Based on the selected position set per frame Generate a binary focusing mask The expression is: ; In the formula, This represents an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0010] Preferably, the specific process for generating latent features in S3 is as follows: S31. Perform calculations on all tokens in the spatiotemporal grid through the Global Look-Ahead Thermally Stable Path (GTS) to generate a global thermal canvas, and update the global thermal canvas through an adaptive temporal blending method. S32. Locally Focused Sparse 3D Attention Path (FS3A) uses a binary focusing mask to filter high-information-density tokens as focus tokens. A query is constructed from these focus tokens, and the complete token sequence provides the keys and values. High-capacity 3D attention enhancement is then performed to obtain the enhanced focus tokens. The specific operation of high-capacity 3D attention enhancement is as follows: the focus tokens are linearly mapped to form a query matrix. The complete token sequence is linearly mapped to form a key matrix. Sum matrix Using scaled dot product attention and Perform matching and apply the resulting attention weights to Weighting is applied to ensure that the result is only written to the focused index position; S33, Mask Backfilling Fusion: The enhanced focused token is scattered back to the complete token sequence according to its original spatiotemporal position, and written back to the global hot canvas generated by GTS through a mask to obtain the backfilling feature; S34. Correct the backfill features using a full-token dense consistency correction block to obtain the latent features. .

[0011] Preferably, the expression for updating the global hot canvas in S31 is: ; In the formula, and These represent the gaze state in the current frame and the previous frame, respectively; Indicates the current potential frame; This represents the temporal hybrid weights adaptively obtained based on potential inter-frame drift.

[0012] Preferably, the expression for mask backfill fusion in S33 is: ; In the formula, This represents the complete token sequence after the l-th GFC module completes mask backfilling, and serves as the input for the next module; Represents the global hot canvas; For the reason The binary mask obtained by flattening; This indicates the focusing enhancement effect after scattering back to the complete token sequence; This indicates element-wise multiplication.

[0013] Preferably, the specific process in S4 is as follows: S41. Normalize the information density map to obtain the value range as follows: Normalized density plot; S42. Based on the statistical characteristics of the information density map, generate modulation parameters using a lightweight prediction network. and The latent features output by GFC are then subjected to affine modulation to obtain the modulated latent features, expressed as: ; In the formula, It represents the latent features after information density-aware affine modulation and serves as the input to the video VAE decoder when IDGC is enabled; and These represent the scaling modulation parameters and offset modulation parameters generated by IDGC, respectively.

[0014] This invention also provides a system for an infrared video super-resolution method based on information density routing and gaze-focusing, comprising an input module, a video variational autoencoder, a local-global information density routing module (LGDR), a gaze-focusing collaborative recovery module (GFC), an information density-aware modulation and supervision module (IDGC), and a decoder. The input module is used to receive low-resolution infrared video and perform spatial upsampling operations; A video variational autoencoder is used to map spatially upsampled video to a latent space to generate a low-resolution latent representation. The Local-Global Information Density Routing (LGDR) module is used to generate information density maps and binary focusing masks. The gaze-focus co-recovery module (GFC) is used to generate latent features, including the global gaze thermally stable path (GTS), the local focus sparse 3D attention path (FS3A), and mask backfill fusion. A full-token dense consistency correction block is used to correct backfill characteristics; The Information Density-Aware Modulation and Supervision Module (IDGC) is used for density-aware modulation of latent features. A decoder is used to decode the modulated latent features.

[0015] Therefore, the infrared video super-resolution method and system based on information density routing and gaze focusing described above have the following beneficial effects: (1) Adaptively allocate computing resources based on infrared video content. By estimating the continuous information density map and predicting the focus ratio through LGDR, the model can select the number of focus tokens according to the thermal structure complexity and information sparsity of each video segment, so that the high-cost recovery computing can be used more for targets, edges and local thermal anomaly areas.

[0016] (2) Reduce redundant computation for smooth background regions. In infrared videos, large background areas usually have weak thermal textures and change gradually over time. This invention uses GTS to perform lightweight global stability modeling on the complete spatiotemporal mesh, and performs FS3A enhancement only on high information density tokens, which helps to reduce the computational overhead caused by dense 3D attention across all tokens.

[0017] (3) It helps to enhance key areas such as targets, edges and thermal boundaries. LGDR identifies relatively prominent thermal structures based on the difference between local and global distributions. FS3A performs high-capacity 3D attention recovery on these focused tokens, which can improve the detail representation of low-contrast targets, small targets, local thermal anomalies and motion boundaries.

[0018] (4) It can improve the temporal stability of infrared video recovery. GTS maintains a global thermal canvas through adaptive temporal blending, which can reduce inter-frame flicker in background and low information density areas, and retain the necessary current frame information when there is camera motion or thermal drift.

[0019] (5) The GFC collaborative mechanism balances global consistency and local detail recovery. Mask backfilling fusion maintains a globally thermally stable representation in non-focused regions and writes sparse 3D attention enhancement results into focused regions; a small number of full-token consistency correction blocks at the end further alleviate the boundary discontinuity or context fragmentation problems that may be caused by local enhancement.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is a diagram illustrating the overall framework of the infrared video super-resolution method based on information density routing and gaze focusing of the present invention. Figure 2 This is a schematic diagram of the GFC structure of the infrared video super-resolution method based on information density routing and gaze focusing according to the present invention. Detailed Implementation

[0022] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] Example 1 Please see Figures 1-2 An infrared video super-resolution method based on information density routing and gaze focusing includes the following steps: Step 1: Construct the system, including an input module, a video variational autoencoder, a Local-Global Information-Density Routing (LGDR) module, a Glance-Focus Collaborative Block (GFC) module, an Information-Density-Aware Gain Control (IDGC) module, and a full-token dense consistency correction block decoder. The input module receives low-resolution infrared video and performs spatial upsampling. The video variational autoencoder maps the spatially upsampled video to the latent space to generate a low-resolution latent representation. The LGDR generates an information density map and a binary focus mask. The GFC generates latent features, including a Global Glance Thermal Stabilization (GTS) path, a Local Focus Sparse 3D Attention (FS3A) path, and mask backfilling fusion. The full-token dense consistency correction block corrects the backfilled features. The Information-Density-Aware Gain Control (IDGC) module performs density-aware modulation on the latent features. The decoder decodes the modulated latent features.

[0024] Step 2: Input low-resolution infrared video ,right Spatially upsampled video is performed to align it spatially with the target high-resolution video. The spatially upsampled video is then mapped to the latent space using a video variational autoencoder (VAE) to obtain a low-resolution latent representation corresponding to the high-resolution output. .

[0025] Step 3: Restore the network in one step (G-FIRE) Video restoration is achieved through a single forward inference in the latent space, in a one-step restoration network. by As input, the continuous information density map and binary focusing mask are first estimated using LGDR; then, GFC is used to decompose the reconstruction computation into a global thermal stabilization path and a local focusing enhancement path; finally, IDGC is used to further improve the reconstruction quality of high-information regions, obtaining the modulated latent features. Through the above design, the high-cost 3D attention computation is concentrated on high-information-density tokens, while smoothing the background region mainly maintains temporal consistency through lower-cost global stabilization modeling, thereby reducing redundant computation while ensuring reconstruction quality.

[0026] Step 31: Generate an information density map and a binary focusing mask using the Local-Global Information Density Routing (LGDR) module.

[0027] Step 311: LGDR is used to estimate which locations are more suitable for high-capacity computation from the infrared video latent tokens, for low-resolution latent representations. In the sequence view With spatiotemporal grid view Rearrange them between them. Represents the set of real numbers. This represents the total number of tokens in the sequence view, and , Represents the time dimension of the potential grid. , Both represent the spatial dimensions of the potential grid. and These represent the height and width of the potential mesh, respectively. The feature dimension of the token is represented; LGDR uses a spatiotemporal grid view for calculation to maintain the temporal and spatial adjacency between infrared video tokens; Step 312: Calculate the low-resolution latent representation The scalar energy of an independent feature unit token, used to reflect the strength of the token's thermal structural response in the latent space, is expressed as: ; In the formula, Indicates the first Spatial location in the frame The token at the location; Indicates the first Spatial location in the frame The scalar energy of the token at that location; This is a stabilizing term used to avoid numerical instability; Step 313: Normalize the energy values ​​within the same video segment and convert them to... Distribution of variable microhistograms over a fixed interval; Step 314: Calculate the average value of the soft histogram of each token within its local neighborhood to obtain the local distribution. The local distribution reflects the local thermal structure near the token; simultaneously, the average soft histogram of all tokens in the entire video segment is calculated to obtain the global distribution. The global distribution reflects the overall thermal response of the entire video segment; the spatial location is calculated by comparing the differences between the local and global distributions. The information density score of the token relative to the global background. An information density map is generated, and the information density score is represented using KL divergence: ; In the formula, The binning index of the soft histogram, with values ​​from 1 to K; This represents the probability that the global soft histogram is distributed in the k-th bin; Indicates the spatial position of the i-th frame. The probability that the local soft histogram of a given token is distributed in the k-th bin; This represents the KL divergence of the local distribution relative to the global distribution, used to measure the difference between the local thermal structure and the overall thermal response of the video. Figure 1 middle, and These represent the local soft histogram distribution and the global soft histogram distribution, respectively. Step 315: Extract statistical features from the information density map. ,Will The input is given to an MLP network, which is activated by a sigmoid function to adaptively calculate the focus token ratio for the current video segment. The expression is: ; In the formula, and These represent the preset minimum and maximum routing ratios, respectively; This represents the Sigmoid activation function, used to map the output of a lightweight prediction network to the 0-1 range; This represents a lightweight prediction network; It represents statistical characteristics, including mean, standard deviation, 90th quantile, and entropy; in this way, video clips with sparser information regions can use a smaller focus ratio, while video clips with denser information regions or more complex motion can be assigned more focus tokens; Step 316: For each frame in the video segment, use Top-K filtering to select the spatial locations with the highest information density within a single frame, obtaining the set of selected spatial locations for each frame. The expression is: ; In the formula, Indicates the first The set of selected Top-K spatial locations in a frame; This represents the Top-K index selection operator, used to return the largest value in the information density map. An index of a spatial location; Indicates the first The information density map of the frame; this frame-by-frame Top-K selection method avoids a few high-density frames occupying the entire focus budget, ensuring that each frame retains the necessary key restoration area; among which... and These are the height and width of the potential frame space grid, respectively. Indicates rounding up; Step 317: Based on the set of selected positions in each frame Generate a binary focusing mask The expression is: ; In the formula, This represents an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0028] Step 32: The gaze-focus co-recovery module GFC performs co-recovery of the infrared video token based on the information density map and the binary focus mask to generate latent features; Step 321: GTS operates on the complete spatiotemporal token grid to form a globally thermally stable representation with low computational cost. Background regions in infrared video typically change smoothly over time, but may still experience slight flickering due to camera motion, thermal drift, or sensor noise. GTS estimates thermal drift based on the difference between the current frame's latent representation and the gaze state of the previous frame, and updates the global thermal canvas using an adaptive temporal blending approach, expressed as: ; In the formula, and These represent the gaze state in the current frame and the previous frame, respectively; Indicates the current potential frame; This indicates the temporal blending weights adaptively obtained based on potential inter-frame drift. When the drift is small, the proportion of historical gaze states can be increased to enhance the temporal stability of the background region. When the drift is large, the proportion of current frame information can be increased to adapt to scene changes. Step 322: FS3A, based on the binary focusing mask, collects only high-information-density tokens as focusing tokens and constructs queries using these focusing tokens. Simultaneously, the complete token sequence provides the keys and values, thereby performing high-capacity 3D attention enhancement to obtain enhanced focusing tokens. Compared to performing global dense 3D attention on all tokens, FS3A focuses computation primarily on areas such as target edges, thermal boundaries, local thermal anomalies, and moving targets. For unselected smooth background areas, it does not perform the same intensity of attention computation, thus reducing redundant computation. The specific operation of high-capacity 3D attention enhancement is as follows: focusing tokens are linearly mapped to form a query matrix. The complete token sequence is linearly mapped to form a key matrix. Sum matrix Using scaled dot product attention and Perform matching and apply the resulting attention weights to Weighting is applied, and the result is written only to the focused index position. If the total number of tokens is... Focus on the number of tokens The computational scale of this attention is approximately ,in Lower than the total token-intensive attention , Indicates the order of magnitude of the computational load; Step 323, Mask Backfilling Fusion: The enhanced focused token is scattered back to the complete token sequence according to its original spatiotemporal position, and then written back to the global heat canvas generated by GTS through a mask to obtain the backfilling features. The expression for mask backfilling fusion is: ; In the formula, This represents the complete token sequence after the l-th GFC module completes mask backfilling, and serves as the input for the next module; Represents the global hot canvas; For the reason The binary mask obtained by flattening; This indicates the focusing enhancement effect after scattering back to the complete token sequence; This indicates element-wise multiplication; this fusion method allows non-focused regions to maintain the globally stable representation of GTS, while allowing focused regions to be replaced or corrected by the enhanced results of FS3A, thereby simultaneously achieving background temporal stability and local detail recovery capabilities. Step 324: After multiple GFCs, set a small number of full-token dense consistency correction blocks at the end. Correct the backfill features using full-token dense consistency correction blocks to obtain potential features. The full attention DiT block is used as a full token dense consistency correction block to further eliminate the boundary discontinuity problem that may exist between the focused and non-focused regions before decoding, and to supplement long-range context relationships.

[0029] Step 33: In the Information Density Sensing Modulation and Supervision Module IDGC, the information density map output by LGDR is used to perform density sensing modulation on the recovered latent features to obtain the modulated latent features, so that the model applies stronger feature gain to high information density regions and only performs weaker modulation on smooth background regions. Step 331: Normalize the information density map to obtain the value range as follows: Normalized density plot; Step 332: Based on the statistical characteristics of the information density map, generate modulation parameters using a lightweight prediction network. and And the latent features of the GFC output are affine modulated, as expressed by: ; In the formula, It represents the latent features after information density-aware affine modulation and serves as the input to the video VAE decoder when IDGC is enabled; and These represent the scaling modulation parameters and offset modulation parameters generated by IDGC, respectively, which enable stronger modulation responses at locations such as target edges, thermal boundaries, and local thermal anomalies. During the training phase, high-resolution real-world videos As a ground truth label, it is used in conjunction with the high-resolution infrared video output from the model to calculate MSE, information density-weighted loss, and DISTS perceptual loss, thus updating the network parameters in reverse and guiding the model to learn infrared thermal texture, edge, and temporal consistency features. Simultaneously, it can upsample the normalized density map to the supervised resolution and increase the weight of high information density regions in the reconstruction loss, making the network focus more on key regions related to infrared target identification and temporal stability. IDGC can be enabled or disabled based on deployment computing power and performance requirements. If enabled, it will... Enter the decoder below; if closed, then directly... Input the following decoder for decoding. Figure 1 middle, , and These represent reconstruction loss, information density-weighted reconstruction loss, and DISTS perception loss, respectively.

[0030] Step 4: Decode the modulated latent features using the decoder of the Video Variational Autoencoder (VAE) to obtain high-resolution infrared video. .

[0031] Example 2 One-step network recovery It can be built based on the Wan2.1-T2V-1.3B diffusion Transformer backbone. Unlike text-to-video generation tasks, infrared video super-resolution tasks mainly use the low-resolution infrared video itself as the recovery condition. Therefore, the text-conditional cross-attention in the original backbone network can be replaced with a visual token-oriented focused sparse 3D attention path FS3A, while keeping the pre-trained video VAE encoder and decoder frozen to achieve the mapping between pixel space and latent space.

[0032] One-step network recovery includes A Transformer block, of which the first Each block adopts a GFC structure, with the end... Each block uses a fully attention-based DiT block. The energy stabilization term of LGDR can be set to... Token energy is normalized to min-max within the same video segment, and the local aggregation window can be set to... Number of soft histograms Histogram temperature The routing ratio range can be set to .

[0033] During training, standard methods can be used. Super-resolution settings. Each training sample includes 33 frames of low-resolution infrared video and their corresponding high-resolution target video, where the low-resolution input size is [size missing]. High-resolution target size is The model can be trained in a single stage on eight NVIDIA H20 GPUs, using the AdamW optimizer. , The weight decay is 0.01, the training iterations are 15000, and the initial learning rate is... The total batch size is 16.

[0034] Since the Top-K focus mask selection process is discrete and non-differentiable, a pass-through estimator can be used during LGDR training. Specifically, the hard binary mask obtained through Top-K selection is used in the forward propagation phase, while the gradient is transmitted to the route predictor through an approximate routing signal formed by continuous density fractions in the backpropagation phase. To improve routing robustness, during training, FS3A can use the focus mask generated by LGDR with a probability of 0.7 and a random mask that selects the same number of tokens with a probability of 0.3; in the inference phase, all focus masks are adaptively generated by LGDR.

[0035] Training objectives may include pixel MSE reconstruction loss, routing cost regularization term, information density-weighted reconstruction loss, and DISTS perceptual loss. And set... , , ,in, The weight of the routing cost regularization term is used to constrain the proportion of focused tokens; Weights for information density-weighted reconstruction loss; The weights are used for the DISTS perception loss. The above network structure, hyperparameters, and training configuration are only one specific implementation method used to illustrate the feasibility of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0036] Example 3 Comparison Analysis 1: The IRVA dataset ×4 infrared video super-resolution unified evaluation protocol was adopted. The low-resolution input was 33 frames (128×128), and the target / output was 33 frames (512×512). All comparison methods used the same LR-HR samples, data partitioning, and RealBasicVSR degradation process for training or fine-tuning. Test-time augmentation was not used during testing. Running time was averaged on a single NVIDIA H20 GPU after warm-up, based on the same input size. Peak memory usage and FLOPs were statistically analyzed or estimated under the same 33×512×512 output configuration. When performing ×4 super-resolution on 33 frames of infrared video with an output resolution of 512×512, the method of this invention ran for approximately 1.21 seconds on a single NVIDIA H20 GPU, lower than the approximately 3.29 seconds of the comparison methods FlashVSR and 7.83 seconds of DOVE. The estimated computational cost of this invention was approximately 214T FLOPs, lower than the approximately 369T FLOPs of FlashVSR. The above data is used to illustrate the trade-off between quality and efficiency under the same experimental settings and does not constitute a limitation on all hardware and scenarios.

[0037] Comparative Analysis 2: Ablation testing using the IRVA dataset ×4 was conducted. The input / output size, LR-HR construction method, degradation protocol, evaluation metrics, and runtime statistics were all consistent with those in Comparative Analysis 1. The default adaptive routing range was ρ∈[0.10,0.30], and the number of end-token consistency correction blocks R=2. This invention achieves a good quality-efficiency tradeoff. As shown in Table 1, the ablation experiment showed that the runtime of the complete model of this invention was 1.21s, PSNR was 22.03, LPIPS was 0.2580, and DOVER was 0.3421. When LGDR, GTS, IDGC, or information density-weighted supervision were removed, the reconstruction metrics or video perception quality metrics all decreased to varying degrees, indicating that information density routing, global thermal stability, focus enhancement, and density-aware supervision have complementary effects. The ablation results in Table 1 show that after removing LGDR, PSNR decreased from 22.03 to 21.62, while LPIPS increased from 0.2580 to 0.2735, indicating that local-global information density routing has a strong effect on the selection of key areas. After removing GTS, DOVER decreased from 0.3421 to 0.3275, indicating that the globally thermally stable path helps improve video perception quality and temporal stability. When IDGC or information density-weighted loss is removed, PSNR, LPIPS, and DOVER also decrease, indicating that density-aware modulation and supervision can further enhance the recovery of high information density areas. In addition, the routing budget experiment shows that the average routing ratio is about 0.21 and the running time is about 1.21s under the default range [0.10, 0.30]. When full token dense processing (ρ=1.0) is used, the running time increases to about 4.86s, while PSNR only increases from 22.03 to 22.11 and DOVER increases from 0.3421 to 0.3453, indicating that the quality improvement is relatively limited. End-point dense refinement experiments show that when R=0, the PSNR is 21.91 and DOVER is 0.3378. When R=2, the PSNR increases to 22.03 and DOVER to 0.3421. When R is further increased to R=3, the DOVER is only 0.3423 and the runtime increases to 1.26s. Therefore, R=2 is generally used as a compromise between quality and efficiency.

[0038] Table 1: Component Ablation Experiment Results

[0039] Therefore, this invention employs the aforementioned infrared video super-resolution method and system based on information density routing and gaze focusing. It estimates a continuous information density map and predicts the focusing ratio using LGDR. The model can select the number of focusing tokens based on the thermal structure complexity and information sparsity of each video segment, allowing high-cost recovery computations to focus more on targets, edges, and local thermal anomaly regions. This invention uses GTS to perform lightweight global stability modeling of the complete spatiotemporal grid, while performing FS3A enhancement only on high information density tokens, helping to reduce the computational overhead of dense 3D attention across all tokens. LGDR identifies relatively prominent thermal structures based on the difference between local and global distributions, and FS3A... High-capacity 3D attention restoration is performed on these focused tokens, improving detail representation of low-contrast targets, small targets, local thermal anomalies, and motion boundaries. GTS maintains a global thermal canvas through adaptive temporal blending, reducing inter-frame flicker in background and low-information-density regions and preserving necessary current frame information in the presence of camera motion or thermal drift. The GFC collaborative mechanism balances global consistency and local detail restoration, and mask backfilling fusion ensures that non-focused regions maintain a globally thermally stable representation, while focused regions are written into the sparse 3D attention enhancement results. A small number of full-token consistency correction blocks at the end further alleviate the boundary discontinuities or context fragmentation problems that may be caused by local enhancement.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An infrared video super-resolution method based on information density routing and gaze focusing, characterized in that, Includes the following steps: S1. Spatial upsampling of low-resolution infrared video and mapping to latent space to generate low-resolution latent representation; S2. Generate an information density map and a binary focusing mask using the Local-Global Information Density Routing (LGDR) module; S3. The gaze-focusing co-recovery module GFC performs co-recovery of the token based on the information density map and the binary focusing mask to generate latent features; S4. In the Information Density Sensing Modulation and Supervision Module (IDGC), the information density map output by LGDR is used to perform density sensing modulation on the latent features to obtain the modulated latent features. S5. The modulated latent features are decoded using the decoder of the video variational autoencoder (VAE) to obtain high-resolution infrared video. .

2. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 1, characterized in that, The specific process for generating low-resolution latent representations in S1 is as follows: S11, Input low-resolution infrared video ,right Spatial upsampling is performed to align it spatially with the target high-resolution video; S12. The spatially upsampled video is mapped to the latent space using a video variational autoencoder (VAE) to obtain a low-resolution latent representation. .

3. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 2, characterized in that, The specific process of generating the information density map in S2 is as follows: S21. Represent the low-resolution latent representation In sequence view With spatiotemporal grid view Rearrange them between them. Represents the set of real numbers. This represents the total number of tokens in the sequence view, and , Represents the time dimension of the potential grid. , Both represent the spatial dimensions of the potential grid. and These represent the height and width of the potential mesh, respectively. Represents the feature dimension of the token; S22. Calculate the low-resolution latent representation. The scalar energy of an independent feature unit token is expressed as: ； In the formula, Indicates the first Spatial location in the frame The token at the location; Indicates the first Spatial location in the frame The scalar energy of the token at that location; It is a stable term; S23. After normalizing the energy values ​​within the same video segment, convert them to... Distribution of variable microhistograms over a fixed interval; S24. Calculate the average value of the soft histogram of each token in its local neighborhood to obtain the local distribution. The local distribution reflects the local thermal structure of the tokens; simultaneously, the average soft histogram of all tokens in the entire video segment is calculated to obtain the global distribution. The global distribution reflects the overall thermal response of the entire video segment; the spatial location is calculated by comparing the differences between the local and global distributions. The information density score of the token relative to the global background. An information density map is generated, and the information density score is represented using KL divergence: ； In the formula, The binning index of the soft histogram, with values ​​from 1 to K; This represents the probability that the global soft histogram is distributed in the k-th bin; Indicates the first Frame space location The probability that the local soft histogram of a given token is distributed in the k-th bin; This represents the KL divergence of the local distribution relative to the global distribution, used to measure the difference between the local thermal structure and the overall thermal response of the video.

4. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 3, characterized in that, The specific process of generating the binary focusing mask in S2 is as follows: S25. Extracting statistical features from the information density map. ,Will The input is given to an MLP network, which is activated by a sigmoid function to adaptively calculate the focus token ratio for the current video segment. The expression is: ； In the formula, and These represent the preset minimum and maximum routing ratios, respectively; This represents the Sigmoid activation function, used to map the output of a lightweight prediction network to the 0-1 range; This represents a lightweight prediction network; Indicates statistical characteristics; S26. For each frame in a video segment, use Top-K filtering to select the spatial locations with the highest information density within a single frame, obtaining the set of selected spatial locations for each frame. The expression is: ； In the formula, Indicates the first The set of selected Top-K spatial locations in a frame; This represents the Top-K index selection operator, used to return the largest value in the information density map. An index of a spatial location; Indicates the first The information density map of the frame; where, and These are the height and width of the potential frame space grid, respectively. Indicates rounding up; S27, Based on the selected position set per frame Generate a binary focusing mask The expression is: ； In the formula, This represents an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

5. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 4, characterized in that, The specific process of generating latent features in S3 is as follows: S31. Perform calculations on all tokens in the spatiotemporal grid through the Global Look-Ahead Thermally Stable Path (GTS) to generate a global thermal canvas, and update the global thermal canvas through an adaptive temporal blending method. S32. Locally Focused Sparse 3D Attention Path (FS3A) uses a binary focusing mask to filter high-information-density tokens as focus tokens. A query is constructed from these focus tokens, and the complete token sequence provides the keys and values. High-capacity 3D attention enhancement is then performed to obtain the enhanced focus tokens. The specific operation of high-capacity 3D attention enhancement is as follows: the focus tokens are linearly mapped to form a query matrix. The complete token sequence is linearly mapped to form a key matrix. Sum matrix Using scaled dot product attention Match with K, and use the resulting attention weights to Weighting is applied to ensure that the result is only written to the focused index position; S33, Mask Backfilling Fusion: The enhanced focused token is scattered back to the complete token sequence according to its original spatiotemporal position, and written back to the global hot canvas generated by GTS through a mask to obtain the backfilling feature; S34. Correct the backfill features using a full-token dense consistency correction block to obtain the latent features. .

6. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 5, characterized in that, The expression for updating the global hot canvas in S31 is: ； In the formula, and These represent the gaze state in the current frame and the previous frame, respectively; Indicates the current potential frame; This represents the temporal hybrid weights adaptively obtained based on potential inter-frame drift.

7. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 6, characterized in that, The expression for mask backfill fusion in S33 is: ； In the formula, This represents the complete token sequence after the l-th GFC module completes mask backfilling, and serves as the input for the next module; Represents the global hot canvas; For the reason The binary mask obtained by flattening; This indicates the focusing enhancement effect after scattering back to the complete token sequence; This indicates element-wise multiplication.

8. The infrared video super-resolution method based on information density routing and gaze focusing according to claim 7, characterized in that, The specific process in S4 is as follows: S41. Normalize the information density map to obtain the value range as follows: Normalized density plot; S42. Based on the statistical characteristics of the information density map, generate modulation parameters using a lightweight prediction network. and The latent features output by GFC are then subjected to affine modulation to obtain the modulated latent features, expressed as: ； In the formula, It represents the latent features after information density-aware affine modulation and serves as the input to the video VAE decoder when IDGC is enabled; and These represent the scaling modulation parameters and offset modulation parameters generated by IDGC, respectively.

9. A system for an infrared video super-resolution method based on information density routing and gaze focusing as described in any one of claims 1-8, characterized in that, include: The input module is used to receive low-resolution infrared video and perform spatial upsampling operations; A video variational autoencoder is used to map spatially upsampled video to a latent space to generate a low-resolution latent representation. The Local-Global Information Density Routing (LGDR) module is used to generate information density maps and binary focusing masks. The gaze-focus co-recovery module (GFC) is used to generate latent features, including the global gaze thermally stable path (GTS), the local focus sparse 3D attention path (FS3A), and mask backfill fusion. A full-token dense consistency correction block is used to correct backfill characteristics; The Information Density-Aware Modulation and Supervision Module (IDGC) is used for density-aware modulation of latent features. A decoder is used to decode the modulated latent features.

Images