An image inpainting and super-resolution reconstruction system and method based on deep learning

Abstract

This invention proposes an image inpainting and super-resolution reconstruction system and method based on deep learning, belonging to the field of digital image processing technology. To address the problems of high computational complexity and resource consumption, limitations in long sequence processing, high training difficulty, and lack of texture and scene representation when using multi-scale residual networks based on the Transformer architecture for image resolution conversion, this invention comprises: an image preprocessing module that performs windowing and memory optimization on the input low-resolution image; a multi-layer fusion network that dynamically adjusts the features of the low-resolution image, captures channel information from different scenes and performs interactive fusion, performs comparative supervision, establishes information communication channels, and uses negative feedback to dynamically adjust and optimize parameters to obtain a super-resolution image; and a loss function module that maximizes the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image.

Description

Technical Field

[0001] This invention relates to a deep learning-based image inpainting and super-resolution reconstruction system and method, belonging to the field of digital image processing technology. Background Technology

[0002] Image super-resolution reconstruction has always been a cutting-edge technology in the field of digital image processing. New methods and technologies are constantly emerging in image processing research and innovation, resulting in leaps and bounds in image processing. Since the concept of super-resolution first appeared in the early 1980s, this technology has undergone three major innovations, continuously driving the progress and development of computer graphics.

[0003] In the first stage of technological evolution, traditional numerical analysis methods dominated the field of super-resolution reconstruction. However, due to their low processing power and limited ability to handle a wide range of image types, they were gradually phased out by later technologies. In the early stages of technological development, linear interpolation spatial domain reconstruction techniques were widely used due to their low computational complexity. However, because spatial domain reconstruction techniques suffer from insufficient sub-pixel information reconstruction capabilities, most processed images suffer from high-frequency detail loss and artifacts. To address this problem, researchers successively proposed multi-frame fusion algorithms and variational regularization, integrating temporal information by establishing probabilistic reconstruction models. Through continuous development, this technology has achieved breakthroughs in video quality enhancement and image quality improvement, laying the mathematical foundation for modern super-resolution technology.

[0004] In the early 21st century, the emergence of deep learning provided new methods for the field of digital image processing. Reinforcement learning methods, with convolutional neural networks at their core, can achieve nonlinear mapping from a low-dimensional observation space to a high-dimensional feature space by constructing a super-resolution neural network framework. This makes the originally complex nonlinear relationships linearly separable in high-dimensional space. Further applications of generative adversarial networks and residual networks enable image processing to achieve pixel-level fidelity and reconstruct texture details that better match visual perception.

[0005] The research breakthroughs in this field have multi-dimensional academic value. Building a deep model library provides a transferable feature representation system for subsequent image processing experiments. In later research, researchers innovatively integrated cutting-edge processing techniques such as attention mechanisms and meta-learning into the pixel reconstruction process, promoting the iterative upgrade of modeling theories related to visual cognition. High-precision super-resolution technology is becoming a key technology for breaking through the optical diffraction limit and is expected to provide powerful support for building a new generation of intelligent imaging systems.

[0006] At the current stage, multi-scale residual networks based on the Transformer architecture have significantly improved the PSNR metric. The optimization processing of large models has also evolved from auxiliary pixel enhancement tools to essential visual perception frameworks. This progress demonstrates the enormous advantages of deep learning in the field of image super-resolution and provides more ideas and challenges for future research in this field. Summary of the Invention

[0007] To address the issues of high computational complexity and resource consumption, limitations in long sequence processing, high training difficulty, and lack of texture scenes when using multi-scale residual networks based on the Transformer architecture for image resolution transformation, this invention proposes an image inpainting and super-resolution reconstruction system and method based on deep learning.

[0008] The technical solution adopted by this invention to solve the above problems is: This invention proposes an image inpainting and super-resolution reconstruction system based on deep learning, comprising:

[0009] Image preprocessing module, multilayer fusion network and loss function module;

[0010] The image preprocessing module is used to perform windowing and video memory optimization on the input low-resolution image;

[0011] Multilayer fusion networks are used to dynamically adjust the features of preprocessed low-resolution images, capture channel information in different scenes, perform interactive fusion of the captured features, conduct comparative supervision, establish information communication channels, and dynamically adjust and optimize parameters through negative feedback to obtain super-resolution images.

[0012] The loss function module is used to maximize the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image.

[0013] Furthermore, the image preprocessing module includes a lightweight convolutional network and a memory access optimization module;

[0014] Lightweight convolutional networks are used to parse parameters of various dimensions of input low-resolution images, evaluate the local complexity of input images, dynamically generate adaptive windows, perform differential region partitioning, and reconstruct tensors into multi-scale structures.

[0015] The video memory access optimization module performs tensor permutation to convert physical storage into row-major order storage, performs stride alignment, and partitions the processed window into... Output in the form of N=(H / K)×(W / K), where N is the total number of dynamic windows.

[0016] Furthermore, the multi-layer fusion network includes a feature extraction layer, a deep feature extraction layer, and an upsampling and reconstruction layer;

[0017] The feature extraction layer uses a two-dimensional convolutional layer with a kernel size of 3×3 and a padding pixel value of 1. The input data has 3 channels and its input is a preprocessed low-resolution image with a size of H×W×3, where H is the height of the image and W is the width of the image. The output is the shallow features of the low-resolution image with a size of H×W×embed_dim.

[0018] The deep feature extraction layer consists of multiple residual groups, each residual being composed of multiple enhanced dynamic aggregation Transformer blocks. The input is shallow features, and the output is deep features. The output of each enhanced dynamic aggregation Transformer block is added to the input through residual connections. A 1×1 or 3×3 convolution is used at the end of the residual group.

[0019] The upsampling and reconstruction layers use the PixelShuffle upsampling module. The input is deep features with a size of H×W×embed_dim, and the output is a super-resolution image with a size of H×upscale×W×upscale×3.

[0020] Furthermore, each enhanced dynamic aggregation Transformer block includes a multi-scale pyramid, an adaptive interaction module, a dynamic hyperparameter controller, a Tucker decomposition linear layer, and a dynamic spatial attention module;

[0021] The multi-scale pyramid consists of 1 / 2 downsampling convolutional layers, 1 / 4 downsampling convolutional layers, and multi-dilution convolutional layers. The 1 / 2 downsampling convolutional layers include Conv convolutional layers with GeLU activation functions, and the 1 / 4 downsampling convolutional layers include two sets of Conv convolutional layers with GeLU activation functions. The dilatancy of the multi-dilution convolutional layers is d=1,2,3. The multi-scale pyramid is used to extract features at the original, 1 / 2, and 1 / 4 scales, and the extracted multi-scale feature context is fused through the multi-dilution convolutional layers.

[0022] The adaptive interaction module is embedded in the lateral connections of the multi-scale pyramid. The input is the multi-scale features after fusing the context, including the cross-scale attention layer and the bilinear difference alignment feature scale layer. The cross-scale attention layer includes three Sigmoid weighted branches. The adaptive interaction module is used to dynamically fuse multi-scale features and output the weighted unified features.

[0023] The dynamic hyperparameter controller consists of two convolutional layers and one average pooling layer, which are used to dynamically adjust the number of attention heads, Tucker decomposition rank, and fusion weights of multi-scale and attention based on the weighted unified features of the input. The input of the dynamic hyperparameter controller is the weighted features, the output is the weighted unified features, and the output is the dynamic parameters, which include the number of attention heads, rank, and fusion weights.

[0024] The Tucker decomposition linear layer consists of a core matrix and a low-rank projection matrix. The rank of the core matrix is ​​rank×rank, and the low-rank projection matrix includes U and V matrices. The Tucker decomposition linear layer is used to reduce the computational complexity of the QKV projection through the core matrix and to compress parameters through the low-rank projection matrix.

[0025] The dynamic spatial attention module includes a deformable window attention layer and a channel-space interaction module, which are used to adjust the multi-head attention mechanism according to the number of attention heads output by the dynamic hyperparameter controller.

[0026] Furthermore, the loss function module employs a pre-trained multimodal image model. Through contrastive learning, it aligns the semantic feature distributions of the super-resolution image and the high-resolution image, and maximizes the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image.

[0027] A deep learning-based image inpainting and super-resolution reconstruction method includes:

[0028] Step 1: Select the low-resolution image to be processed I The input image preprocessing module performs preprocessing.

[0029] Step 2: Transfer the preprocessed low-resolution image I Input feature extraction layer to preprocess low-resolution image I Convert to preliminary feature map ,in, The height of the initial feature map, The width of the initial feature map. This represents the number of channels in the initial feature map;

[0030] Step 3: Calculate the query of the preliminary feature map ,key Sum And attention distribution;

[0031] Step 4: Dynamically fuse the multi-scale features of the initial feature map using a multi-scale pyramid and an adaptive interaction module, and output the weighted unified features;

[0032] Step 5: Optimize the weighted unified features using a dynamic hyperparameter controller and a dynamic spatial attention module. Perform a block approximate cross product on the weight tensor of the optimized features. Decompose the weight tensor of the optimized features using a Tucker decomposition linear layer to complete the positional self-attention processing.

[0033] Step 6: Combine class activation maps with contrastive learning through the loss function module to construct a semantically perceptual feature space, and set the given image pairs The contrastive loss function is used during training to train the multimodal image model. Image region category masks are generated through a pre-trained semantic segmentation model. The trained images are then processed by the multimodal image model, so that the original low-resolution images undergo structured processing and model optimization in sequence, and finally a super-resolution image is obtained.

[0034] Further, step 3 query ,key Sum The calculation formula is:

[0035] (1);

[0036] (2);

[0037] (3);

[0038] In formulas (1)-(3), , , The weight matrix is ​​a learnable matrix. This is the feature map extracted after the convolutional layer;

[0039] The formula for calculating attention distribution is:

[0040] (4);

[0041] In formula (2), For the dimensions of query and key, It is a normalized exponential function.

[0042] Furthermore, step 4 specifically includes:

[0043] The feature map after calculating the attention distribution is extracted using a multi-scale pyramid, and features at three scales (original, 1 / 2, and 1 / 4) are extracted. The extracted multi-scale feature context is then fused using a multi-dipty convolutional layer. Adaptive weights are generated based on the extracted feature content. According to adaptive weights Dynamically adjust fusion weights P i And through the cross-attention mechanism, it can interact in the channel or spatial dimensions;

[0044] Fusion weights P i The calculation formula is:

[0045] (5).

[0046] In formula (5), This represents a higher-level, more comprehensive fusion weight. For the feature map at the current scale i, For adaptive weights used to weight high-level features, These are the adaptive weights used to weight the features of the current layer.

[0047] Furthermore, step 5 specifically includes:

[0048] Step 5.1: Divide the hyperparameter space into three subspaces: structural parameters, regularization parameters, and learning parameters using a dynamic hyperparameter controller. Use a hierarchical optimization strategy to perform Bayesian optimization and gradient descent on the three subspaces respectively. For structural parameters, Bayesian optimization is used to search for the optimal combination of convolution kernel size and network depth, and an objective function is used. For regularization parameters, a dynamic mapping function is constructed based on meta-learning. For learning parameters, an adaptive gradient algorithm is used.

[0049] Step 5.2: Introduce differentiable NAS technology to construct a super network that covers all candidate structures. The super network covering all candidate structures includes constructing an operation pool, learning path weights by calculating the activation probability of each path through Gumbel-Softmax sampling, and adaptively selecting operation combinations based on the fusion frequency domain features of the image.

[0050] Step 5.3: Design a weighted loss function for multi-objective collaborative optimization and perform a global parameter search on the training dataset;

[0051] Step 5.4: Divide the weight tensor matrix of the optimized features into... After sub-blocks, approximate calculations are performed to complete the block approximate cross product, and the weight tensors of the optimized features are decomposed through the Tucker decomposition linear layer.

[0052] The expression for the objective function is:

[0053] (6);

[0054] In formula (6), This is the complexity penalty coefficient. This is the set of network architecture parameters for Bayesian optimization search; Is it using structural parameters? The model's prediction results for input X;

[0055] The expression for the dynamic mapping function is:

[0056] (7);

[0057] In formula (7), For characteristic statistics, These are dynamically generated regularization parameters;

[0058] The update rule for the adaptive gradient algorithm is:

[0059] (8);

[0060] In formula (8), This is the current loss function value. The cumulative amount of decay ( (attenuation rate) This is the gradient sensitivity coefficient (which controls the strength of the influence of the gradient norm on the learning rate). It represents the L2 norm of the gradient of the loss function, reflecting the stability of the current optimization direction.

[0061] The formula for calculating the path activation probability is:

[0062] (9);

[0063] In formula (9), It is Gumbel noise. For temperature parameters;

[0064] The expression for the weighted loss function is:

[0065] (10);

[0066] In formula (10), For the first Target feature embedding, For learnable weight vectors, T It is the transpose operator. This is the transpose of vector V;

[0067] The approximate calculation expression is:

[0068] (11);

[0069] In formula (11), The learnable importance coefficient, For Kronecker product, A and B The input matrix;

[0070] The expression for weighted tensor decomposition is:

[0071] (12);

[0072] In formula (12), For the Tucker core tensor, For Tensor Train chain tensors, Decompose the Tucker factor matrix.

[0073] Furthermore, the expression for the comparison loss function in step 6 is as follows:

[0074] (13);

[0075] In formula (13), For semantically relevant regions determined by CAM, For feature similarity, q is the feature vector (p is the sample point, q is the anchor point);

[0076] The multimodal image model employs the aforementioned multilayer fusion network. During training, it uses blur kernel combination and noise injection to simulate realistic degradation, performs downsampling, and uses bicubic interpolation to generate low-resolution input. Real-time augmentation of the training dataset is also performed, specifically including spatial manipulation, color perturbation, and texture destruction. A phased training strategy is adopted, which includes learning 0-100k basic features. The optimization objective uses Charbonnier. The Loss function is an adaptive robust loss function. The learning rate strategy uses cosine annealing, with dynamic activation during the 100k-200k iterations. A hybrid loss function is used: 60% L1 loss + 20% perceptual loss + 20% contrastive loss. The learning rate is reset to 1e-4, linearly decaying to 5e-6, and gradient clipping is introduced. Adversarial fine-tuning is performed during the 200k-300k iterations. The PatchGAN discriminator is loaded, an adversarial framework is constructed, and the loss weights are adjusted to 40% L1 loss + 30% adversarial loss + 30% contrastive loss. In the first 5k iterations, the generator parameters are fixed, and only the discriminator is trained, gradually increasing the adversarial loss weights and enabling mixed precision training.

[0077] The beneficial effects of this invention are:

[0078] 1. This invention establishes a complete technical system encompassing "feature parameter extraction - dynamic model optimization - global semantic constraints - hardware core acceleration." This system overcomes the technical bottleneck of traditional methods where quality and efficiency cannot be simultaneously achieved, effectively contributing to the advancement of image super-resolution technology and providing new ideas and opportunities for its future development. Furthermore, this model, through differentiable architecture search technology, can determine the most reasonable parameter configuration, enabling maximum restoration of the original image and generating high-quality images.

[0079] 2. This invention further processes the trained images using a multimodal image model, allowing the original low-resolution image to undergo sequential structured processing and model optimization, ultimately yielding a super-resolution image. This processing improves upon the distortion and color degradation issues inherent in conventional image restoration techniques, while preserving and enhancing image details during the restoration process, thus enhancing the quality of the restored image. Attached Figure Description

[0080] Figure 1 This invention provides a schematic diagram of the structure of an image inpainting and super-resolution reconstruction system based on deep learning.

[0081] Figure 2 This is a flowchart illustrating a deep learning-based image inpainting and super-resolution reconstruction method provided by the present invention. Detailed Implementation

[0082] Specific implementation method one: as follows Figure 1 As shown, the structure of the deep learning-based image inpainting and super-resolution reconstruction system described in this embodiment includes:

[0083] Image preprocessing module, multilayer fusion network and loss function module;

[0084] The image preprocessing module is used to perform windowing and video memory optimization on the input low-resolution image;

[0085] This implementation employs a dynamically adaptive preprocessing framework for data processing. By optimizing content-aware window partitioning and GPU memory size, it constructs an efficient data element for multi-scale feature extraction. The input image is first processed by a lightweight convolutional network to parse the parameters of each dimension of the image, generating tuples containing batch, channel, and size information. Then, the local texture complexity is evaluated, and adaptive windows ranging from 16×16 to 32×32 are dynamically generated. A differentiable region partitioning algorithm is used to reconstruct the tensor into (B, H / K, W / K, K). 2 The multi-scale structure of (C) is used, where the window size K is determined in real time by the entropy detector.

[0086] In terms of memory access optimization, a three-dimensional continuous reorganization strategy is designed: tensor permutation is performed to eliminate dimensional gaps, converting physical storage into row-major order storage; then, the contiguous method is called to achieve stride alignment, improving the L2 cache hit rate to 82%. The processed window partition is in (B×N,K) order. 2 Output in the form of C), where N = (H / K) × (W / K) is the total number of dynamic windows.

[0087] Multilayer fusion networks are used to dynamically adjust the features of preprocessed low-resolution images, capture channel information in different scenes, perform interactive fusion of the captured features, conduct comparative supervision, establish information communication channels, and dynamically adjust and optimize parameters through negative feedback to obtain super-resolution images.

[0088] The core network component of this implementation proposes a fusion-based multi-level network framework. Its key feature is the construction of a three-dimensional cross-cooperative mechanism across spatial, channel, and semantic dimensions. Through deep feature cross-processing and dynamic resource allocation, it achieves a combined improvement in both efficiency and quality in computer processing. The network architecture primarily employs a multi-layered path processing approach that coordinates different types of computational units, combined with global attention fusion. Specifically, it includes three core stages: an initial feature extraction layer, a dynamic spatial channel processing layer, and a semantic reconstruction layer. In the initial feature extraction stage, an improved spatial module and a heterogeneous channel module are used for parallel cross-processing. The former obtains spatial feature elements through a multi-level convolutional pyramid, while the latter uses a dynamically compressed channel attention mechanism to dynamically adjust the compression ratio of the input features, capturing channel information under different scenarios. After interactive fusion, a semantic module is used for comparative supervision, establishing information communication channels, and negative feedback dynamically adjusts and optimizes parameters, forming a multi-layered feature representation.

[0089] The dynamic processing stage innovatively constructs a spatial-channel alternating enhancement mechanism, introducing a master-slave attention architecture into the dual-spatial alternation module. The spatially dominant path adopts a dynamic head allocation strategy, calculating the image patch information entropy in real time based on the local complexity awareness module, and adaptively adjusting the computational resource allocation of the attention heads. The channel-assisted path divides the feature channels into independent computational groups through a grouped attention mechanism, combined with lightweight Tucker decomposition convolution, reducing the computational cost of the channel path by 56.3% while maintaining feature discriminative power. The two paths perform cyclic feature exchange through a bidirectional LSTM structure, and after each iteration, cross-dimensional features are fused through the final gating unit. This alternating enhancement mechanism effectively suppresses the error accumulation problem of a single path.

[0090] The semantic reconstruction stage utilizes an omnidirectional semantic attention module to integrate global features. Unlike traditional single-dimensional attention mechanisms, this module constructs a cube to simultaneously establish spatial, channel, and semantic three-dimensional attention dependencies. Specifically, for spatial processing, it employs position-sensitive self-attention to capture long-distance contextual relationships; for channel processing, it implements prototype-guided feature recalibration; and for semantic processing, it introduces contrastive learning constraints. A pre-trained DeepLabv3+ segmentation network generates regional semantic masks, constructing an optimization objective of aggregating features from similar regions and separating features from dissimilar regions. This module innovatively encodes prior knowledge of image segmentation into a contrastive loss function, improving image reconstruction accuracy while maintaining image naturalness.

[0091] Under the requirement of feature fusion, a three-modal interaction mechanism is designed for the processing network: In the feature extraction stage, a multi-dimensional pyramid is used for multi-scale feature fusion, and the feature extraction and pixel rearrangement of the convolutional neural network are used together with the learnable downsampling kernel to construct a bidirectional resolution transformation; then, a dynamic weight allocation mechanism is established, and the proportional weight of each level feature is calculated using an improved residual network module; a memory enhancement unit is added, which stores the features of typical patterns through a trainable initial feature library, thereby improving the robustness of feature reconstruction of degenerate types.

[0092] By performing alternating feature operations in both spatial and channel dimensions, a multi-branch feature fusion plan is constructed, which enhances the generality of the network model's features and improves the construction of complex visual features. Experimental results show that, after training, the model achieves better performance on multiple benchmark datasets compared to existing networks, demonstrating that this multi-dimensional interactive feature extraction design can significantly improve image accuracy and robustness.

[0093] The specific network framework includes: a feature extraction layer, a deep feature extraction layer, and an upsampling and reconstruction layer;

[0094] Feature extraction layer:

[0095] Structure: Conv2d(3,embed_dim,3,padding=1)

[0096] Input: Low-resolution image (LR, size H×W×3)

[0097] Output: Shallow features (H×W×embed_dim)

[0098] Function: Extracts initial local features and expands the number of channels through 3×3 convolution.

[0099] Deep feature extraction (residual set stacking):

[0100] It contains multiple residual groups, each of which consists of multiple enhanced dynamic aggregation Transformer blocks (ETBs).

[0101] Residual set structure:

[0102] Number of layers: 6 residual groups by default (depth=6), each group contains 2 EnhancedDATB blocks.

[0103] Core module: ETB module, which includes the following sub-modules:

[0104] Each enhanced dynamic aggregation Transformer block includes a multi-scale pyramid, an adaptive interaction module, a dynamic hyperparameter controller, a Tucker decomposition linear layer, and a dynamic spatial attention module;

[0105] Multi-Scale Pyramid

[0106] structure:

[0107] 1 / 2 downsampling convolution (Conv+GeLU)

[0108] 1 / 4 downsampling convolution (two Conv+GeLU operations)

[0109] Multi-hole convolution (d=1,2,3)

[0110] Function: Extracts features at three scales: original, 1 / 2, and 1 / 4, and fuses multi-scale context through dilated convolution.

[0111] Adaptive Interaction Module (AIM)

[0112] structure:

[0113] Cross-scale attention (3 Sigmoid weighted branches)

[0114] Bilinear interpolation aligned feature scale

[0115] Function: Dynamically fuses multi-scale features to generate weighted unified features.

[0116] Dynamic Hyperparameter Controller (DHPM)

[0117] structure:

[0118] Complexity predictor (Conv+AvgPool+Conv)

[0119] Output dynamic parameters: number of attention heads (4~8), rank (8~24), fusion weight (0.2~1.0)

[0120] Function: Dynamically adjusts the number of attention heads, Tucker decomposition rank, and fusion weights of multi-scale and attention based on input features.

[0121] Tucker decomposition of linear layers (TuckerLinear)

[0122] Structure: Core matrix (rank×rank) + low-rank projection (U, V matrices)

[0123] Purpose: To reduce the computational complexity of QKV projection by compressing parameters through low-rank decomposition.

[0124] Adaptive Spatial Attention

[0125] Structure: Deformable window attention (supporting dynamic window size and displacement) and channel-space interaction modules (CI and SI).

[0126] Function: Dynamically adjust the multi-head attention mechanism based on the number of attention heads predicted by DHPM.

[0127] Residual connection:

[0128] Each ETB output is added to the input via a residual connection to avoid gradient vanishing.

[0129] At the end of the residual group, 1×1 or 3×3 convolutions (controlled by the resi_connection parameter) are used to further fuse features.

[0130] Upsampling and Reconstruction Layer

[0131] Structure: Input to the PixelShuffle upsampling module: Deep features (H×W×embed_dim)

[0132] Process: Conv2d extended channels → PixelShuffle rearrangement → Conv2d output, supporting 2× / 3× / 4× upsampling (depending on the upscale parameter).

[0133] Output: Super-resolution image (SR, size H×upscale×W×upscale×3).

[0134] The loss function module is used to maximize the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image.

[0135] Loss function: Semantic Contrast Loss is used.

[0136] Structure: Pre-trained DeepLabV3 segmentation model (fixed parameters)

[0137] Contrastive Learning: Aligning the Semantic Feature Distributions of SR and HR

[0138] Function: By maximizing the similarity between SR and HR in the segmentation feature space, the accuracy of semantic reconstruction is improved.

[0139] Specific implementation method two: such as Figure 2 As shown, the steps of the deep learning-based image inpainting and super-resolution reconstruction method described in this embodiment include:

[0140] S1: Maintain global semantic awareness of the image;

[0141] This implementation employs a multi-dimensional self-attention mechanism, which uses a hybrid approach of deep and shallow networks to process information across layers. This enables the computation of the contextual correlation matrix between any pair of pixels, overcoming the limitations of the local receptive field in convolutional operations and achieving effective integration of global semantic information. Furthermore, this mechanism can capture long-distance dependencies, ensuring the complete preservation of the image's global structural information.

[0142] For the initial input image (Where H, W, and C represent the feature data of height, width, and number of channels obtained after image preprocessing, respectively.) A location-based self-attention mechanism needs to be applied first to capture the relationships between different locations and different dimensions. This process can be broken down into the following steps:

[0143] S101: Through one or more convolutional layers ( The input image I is converted into a preliminary feature map. ,in, The height of the initial feature map, The width of the initial feature map. This represents the number of channels in the initial feature map.

[0144] S102: Multi-head self-attention mechanism: For each head Calculate query ,key Sum :

[0145] (1);

[0146] (2);

[0147] (3);

[0148] In formulas (1)-(3), , , The weight matrix is ​​a learnable matrix. The feature map is extracted after the convolutional layer. The feature map is then transformed by applying the weight matrix. Q , K , V ;

[0149] S103: Attention Distribution Calculation: For each head, use formula (4) to obtain the attention distribution:

[0150] (4);

[0151] In formula (4), For the dimensions of query and key, It is a normalized exponential function.

[0152] S2: Parallelized, high-efficiency computation;

[0153] This implementation design features a parallelizable attention computation module. By replacing the full matrix multiplication in the cross product operation with low-rank tensor decomposition and combining it with grouped convolution, the number of parameters is reduced, achieving a time complexity of O(N) on a GPU cluster. 2 This method achieves a linear speedup compared to traditional convolutional neural network architectures, while maintaining the same number of parameters. Experimental data shows that compared to traditional convolutional neural network architectures, this method significantly improves training speed.

[0154] Traditional Khatri-Rao product operations often suffer from dimensionality explosion in tensor decomposition. We propose a block-approximated Khatri-Rao product method, which overcomes the trade-off between accuracy and efficiency in traditional tensor computation by partitioning the input matrix into... After the sub-block, perform the following approximation: (5);

[0155] In formula (5), The learnable importance coefficient, For Kronecker product, A and B The input matrix;

[0156] This strategy reduces computational complexity from Down to .

[0157] For a hierarchical implementation of hybrid tensor decomposition, the Tucker-Train hybrid decomposition model (TTM) is proposed, which decomposes the network weight tensors... Decomposed into:

[0158] (6);

[0159] In formula (6), For the Tucker core tensor, For Tensor Train chain tensors, Decompose the Tucker factor matrix.

[0160] This hybrid structure combines the features of two decomposition methods, enabling the capture of global feature interactions and optimization of high-dimensional fully connected layers.

[0161] S3: Multi-scale feature fusion;

[0162] This implementation introduces a multi-scale feature pyramid structure based on the cross-fusion of dual-space and dual-channel modules. It generates feature maps with resolutions ranging from 1 / 4 to 4 times through dilated convolution and adaptive downsampling. Then, it performs cross-scale feature fusion through an improved AIM module. This module can adaptively allocate the weights of features at different levels, thereby improving the expressive power of low-frequency structures in the image while preserving high-frequency details.

[0163] This implementation addresses the issues of fixed receptive field, feature redundancy, and loss of detail in traditional methods through multi-level feature fusion, dynamic weight allocation, and cross-modal interaction mechanisms. This module inherits and extends the classic feature pyramid architecture, employing a top-down path upsampling and lateral connections to construct the pyramid, with each layer exhibiting decreasing resolution.

[0164] An adaptive interaction module is embedded in the horizontal connections of the pyramid to dynamically adjust the fusion weights based on the input features. Its mathematical expression is:

[0165] (7).

[0166] In formula (7), This represents a higher-level, more comprehensive fusion weight. For the feature map at the current scale i, For adaptive weights used to weight high-level features, P is the fusion weight at scale i+1, used to weight the features of the current layer. Scale i+1 is coarser (lower resolution) than the current scale i.

[0167] Among them, adaptive weights It can be generated based on feature content.

[0168] This module achieves cross-scale fusion of multi-resolution features by introducing dilated convolution and residual connections. The pyramid provides multi-scale input to the adaptive module, and the adaptive module provides a dynamic optimization strategy for the pyramid, forming a closed loop. A cross-attention mechanism is designed to enable interaction between channel or spatial dimensions.

[0169] S4: Dynamic feature selection mechanism;

[0170] This implementation uses a dynamic hyperparameter control selector based on image content complexity. It automatically adjusts the number of attention heads, channel compression ratio, and fusion weights of the cross product operation according to local texture complexity. While the proposed self-attention mechanism can dynamically adjust the image context weights, fixed hyperparameters may lead to overfitting in simple regions and underfitting in complex regions. Experiments show that in natural images where high-frequency regions account for less than 30%, dynamic adjustment can save over 20% of computational resources. Therefore, this mechanism can effectively improve the feature responses of key regions, significantly enhancing the reconstruction quality of image edges and textures.

[0171] The hyperparameters of a Transformer are usually fixed, but the complexity of different regions in an image varies significantly. Applying hyperparameter optimization has drawbacks such as high computational cost and weak generalization ability. Therefore, the following four-layer architecture is used to achieve dynamic adjustment:

[0172] S401: Hyperparameter Space Partitioning and Hierarchical Optimization

[0173] This framework innovatively divides the hyperparameter space into three subspaces: structural parameters, regularization parameters, and learning parameters. A hierarchical optimization strategy is then used to perform Bayesian optimization and gradient descent on each of the three subspaces respectively.

[0174] Structural parameters: Bayesian optimization is used to search for the optimal combination of convolutional kernel size and network depth. The objective function is:

[0175] (8);

[0176] In formula (8), This is the complexity penalty coefficient. This is the set of network architecture parameters to be searched using Bayesian optimization, including convolutional kernel size and network depth. Is it using structural parameters? The model's prediction results for input X;

[0177] Regularization parameters: constructing dynamic mapping functions based on meta-learning ,in For feature statistics, referencing meta-learning strategies;

[0178] Learning parameters: An adaptive gradient algorithm is used, and its update rule is as follows:

[0179] (9);

[0180] In formula (9), The attenuation rate, is the gradient norm sensitivity coefficient.

[0181] S402: Neural Architecture Search (NAS) and Dynamic Reconfiguration

[0182] Differentiable NAS technology is introduced to construct a Supernet that covers all candidate architectures. Specific steps:

[0183] Candidate operation pool: includes 3×3 / 5×5 convolution, deformable convolution, dynamic convolution kernel, etc.;

[0184] Path weight learning: Calculate the activation probability of each path using Gumbel-Softmax sampling.

[0185] (10);

[0186] In formula (10), It is Gumbel noise. For temperature parameters;

[0187] Real-time dynamic adjustment: Based on the frequency domain characteristics of the input image, the operation combination is adaptively selected to realize the attention mechanism of the OSA module.

[0188] S403: Multi-objective collaborative optimization strategy

[0189] Design a weighted loss function to balance multiple optimization objectives:

[0190] (11);

[0191] In formula (11), For the first Target feature embedding, For learnable weight vectors, T It is the transpose operator. This is the transpose of vector V;

[0192] S404: Lightweight Online Inference Mechanism

[0193] Offline pre-training: Perform global parameter search on pre-training datasets such as COCO and REDS.

[0194] S5: A semantically guided contrastive learning framework;

[0195] This implementation introduces a semantic segmentation map as a priori initial design component and designs a comparatively reasonable loss function to force the model to maintain semantic consistency during the repair process. It emphasizes generating textures that conform to human visual perception but does not constrain semantic rationality. Contrastive learning can further reduce the feature distance between similar regions and increase the distance between dissimilar regions, effectively improving visual consistency.

[0196] This implementation innovatively combines Class Activation Maps (CAMs) with contrastive learning to construct a semantically perceptual feature space. Given image pairs... The contrast loss function is improved as follows:

[0197] (12);

[0198] In formula (12), This refers to the semantically relevant regions determined by CAM.

[0199] S6: Modular network architecture;

[0200] This implementation employs a hierarchical network design, including a feature extraction layer, a multi-scale attention layer, a residual connection layer, and an upsampling reconstruction layer. Each module allows for independent optimization and arbitrary combination, enabling rapid deployment and model customization across different task scenarios.

[0201] The multimodal image model employs a multi-layer fusion network, and the training process includes:

[0202] I. Data Preparation and Augmentation Strategies

[0203] 1. Dataset Construction

[0204] Base datasets: DIV2K (a benchmark dataset containing 800 training images and 100 validation images, covering complex texture scenes such as architecture and natural landscapes, using a YCbCr color space preprocessing workflow), and Flickr2K (introducing 2650 high-resolution network images to expand the diversity of data distribution, with a particular focus on enhancing the coverage of portraits and dynamic scenes).

[0205] Data registration: LR-HR pairing is constructed using bicubic downsampling. In order to prevent distortion of frequency domain information, physical degradation model fuzzing and degradation are used for joint modeling. Data augmentation is performed in HR space and then synchronous downsampling is performed to avoid LR-HR space conversion error.

[0206] 2. Degradation Model Construction

[0207] We use a combination of fuzzy kernels (Gaussian + motion blur) and noise injection (Poisson + salt and pepper noise) to simulate real degradation.

[0208] The downsampling ratio is set to 4 times, and bicubic interpolation is used to generate low-resolution input.

[0209] 3. Real-time enhancement strategy

[0210] Spatial transformation: random horizontal flip, ±15° rotation, 256×256 area cropping

[0211] Color perturbation: Replace linear adjustment with Sigmoid enhancement curve, Cb-Cr channel coupling offset, and random grayscale.

[0212] Texture destruction: Add random rectangular occlusions (stylized occlusion + content-aware occlusion).

[0213] 4. Dynamic construction of validation sets

[0214] Based on the DIV2K validation set, a dynamic validation set is constructed by periodically extracting 10% of the data from Flickr2K to monitor the risk of overfitting.

[0215] II. Model Architecture Initialization

[0216] 1. Backbone network configuration

[0217] Encoder: Improved ResNet50, removing the last two pooling layers to preserve spatial details, replacing the Stage 4 convolution kernels with dilated convolutions (dilation=2), increasing the feature map resolution to 1 / 16 of the input size while maintaining the receptive field.

[0218] Multi-scale pyramid: It adopts deformable convolution instead of fixed stride convolution to achieve spatial scale adaptive sampling, increases lateral connections to achieve cross-scale feature fusion, and improves multi-scale robustness by referencing the FPN architecture.

[0219] Attention module: Deploys dynamic head-count self-attention.

[0220] 2. Lightweight component loading

[0221] Matrix decomposition: Tucker decomposition is applied to the fully connected layer, with a compression ratio of 16:1.

[0222] Tucker decomposition is applied to the 1×1 convolution, and the compression formula is as follows:

[0223] (13);

[0224] In formula (13), For the original network weight tensor, Decompose the core tensor for Tucker. , , The Tucker decomposition factor matrix is ​​used, and the size of the core tensor G is controlled to be 1 / 16 of the original weights.

[0225] Grouped convolution: The number of channels is fixed at 8, reducing cross-channel computational dependencies.

[0226] 3. Semantic Guidance Branches

[0227] The number of DeepLab feature channels is compressed to a dimension matching that of the backbone network using 1×1 convolutions.

[0228] Constructing a semantic attention mask:

[0229] (14);

[0230] In formula (14), For semantic attention mask, The weight parameters are for a 1×1 convolutional layer. Feature maps extracted by the DeepLab segmentation network. This is the Sigmoid activation function.

[0231] Spatial weight map is generated by sigmoid activation.

[0232] 4. End-to-end training strategy

[0233] Multi-task loss combination:

[0234] in Robust L1-L2 hybrid

[0235] VGG-19 Feature Reconstruction Loss

[0236] Semantic segmentation consistency loss (KL divergence matching DeepLab output distribution)

[0237] III. Phased Training Strategy:

[0238] Phase 1: Basic Feature Learning (0-100k iterations);

[0239] Freezing the dynamic hyperparameter controller and semantic comparison branches

[0240] Optimization objective: Enhance robustness using the Charbonnier Loss adaptive robust loss function (ε=1e-3).

[0241] (15);

[0242] In formula (15), This is the Charbonnier loss function.

[0243] Learning rate strategy: cosine annealing (initial value 2e-4, minimum 1e-5), adding warm-up (Warmup=2kiter) to alleviate initial parameter oscillations.

[0244] Batch processing scale: 32 images per card, 8 cards for distributed training

[0245] Phase 2: Activation of dynamic mechanisms (100k-200k iterations);

[0246] Unfreeze the dynamic hyperparameter controller and initiate complexity-aware training.

[0247] Hybrid loss function: Multi-scale contrastive learning is adopted to construct positive and negative sample pairs in 1 / 2 and 1 / 4 resolution feature spaces, with L1 loss (60%) + perceptual loss (20%) + contrastive loss (20%).

[0248] The learning rate is reset to 1e-4, then linearly decays to 5e-6. Adaptive gradient pruning is introduced to prevent dynamic module oscillations.

[0249] Phase 3: Adversarial Fine-tuning (200k-300k iterations)

[0250] Load the PatchGAN discriminator and build an adversarial framework;

[0251] Loss weighting adjustment: L1 (40%) + adversarial loss (30%) + contrastive loss (30%)

[0252] Discriminator update frequency: For the first 5k iterations, the generator parameters are fixed, and only the discriminator is trained, gradually increasing the adversarial loss weights (linearly increasing from 0% to 30%).

[0253] Enabling mixed-precision training (FP16+FP32) reduces GPU memory usage by 40%.

[0254] IV. Key Training Techniques

[0255] 1. Hardware-level optimization

[0256] Video memory management: Gradient checkpointing technology + dynamic tensor loading

[0257] Disable checkpointing for Transformer modules (such as dynamic attention heads), apply checkpointing at the Stage level instead of the Block level, and balance computation and memory consumption.

[0258] Communication optimization: Use the PowerSGD algorithm to compress gradients (compression ratio up to 40x), and disable compression for BN layer parameters.

[0259] 2. Implementation of mixed precision

[0260] Backbone network: FP16 precision calculation, Batch Normalization (BN) layers are retained at FP32.

[0261] Loss calculation: FP32 precision accumulation to prevent underflow error.

[0262] Gradient scaling: Dynamic scaling factor (initial value 1024, adjusted per iteration)

[0263] 3. Convergence monitoring

[0264] Real-time metrics: Record training loss every iteration, and verify PSNR / SSIM every 5k iterations.

[0265] Early Stop Mechanism: Phase 1: Use the rate of loss decrease (e.g., loss decrease <1e-4 for 5 consecutive cycles).

[0266] Phase 2-3: Using the relative rate of change of PSNR (ΔPSNR < 0.02 dB)

[0267] Model archiving: Top-3 checkpoints are retained, and the parameter sensitivity-aware EMA is finally adopted to dynamically adjust the attenuation coefficient based on the parameter norm.

[0268] 4. Post-processing of multimodal models

[0269] The trained images are then processed by a multimodal image model, whereby the original low-resolution images undergo sequential structured processing and model optimization to ultimately obtain super-resolution images. This processing improves upon the distortion and color degradation issues present in conventional image restoration techniques, while preserving and enhancing image details during the restoration process, thus enhancing the quality of the restored images.

[0270] In summary, this invention establishes a complete technical system encompassing "feature parameter extraction - dynamic model optimization - global semantic constraints - hardware core acceleration." This system overcomes the technical bottleneck of traditional methods where quality and efficiency cannot be simultaneously achieved, effectively contributing to the advancement of image super-resolution technology and providing new ideas and opportunities for its future development. Furthermore, this model, through differentiable architecture search technology, can determine the most reasonable parameter configuration, enabling maximum restoration of the original image and generating high-quality images.

[0271] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A deep learning-based image inpainting and super-resolution reconstruction system, characterized in that, include: Image preprocessing module, multilayer fusion network and loss function module; The image preprocessing module is used to perform window division and video memory optimization on the input low-resolution image; The multi-layer fusion network is used to dynamically adjust the features of the preprocessed low-resolution image, capture channel information in different scenes, perform interactive fusion of the captured features, conduct comparative supervision, establish information communication channels, and dynamically adjust and optimize parameters through negative feedback to obtain a super-resolution image. The multi-layer fusion network includes a feature extraction layer, a deep feature extraction layer, and an upsampling and reconstruction layer; The feature extraction layer uses a two-dimensional convolutional layer with a kernel size of 3×3, a padding pixel value of 1, and 3 input channels. The input is a preprocessed low-resolution image with dimensions of H×W×3, where H is the height of the image and W is the width of the image. The output is the shallow features of the low-resolution image with dimensions of H×W×embed_dim. The deep feature extraction layer includes multiple residual groups, each residual consisting of multiple enhanced dynamic aggregation Transformer blocks. The input is shallow features, and the output is deep features. The output of each enhanced dynamic aggregation Transformer block is added to the input through residual concatenation. A 1×1 or 3×3 convolution is used at the end of the residual group. The upsampling and reconstruction layer uses the PixelShuffle upsampling module. The input is deep features with a size of H×W×embed_dim, and the output is a super-resolution image with a size of H×upscale×W×upscale×3. Each enhanced dynamic aggregation Transformer block includes a multi-scale pyramid, an adaptive interaction module, a dynamic hyperparameter controller, a Tucker decomposition linear layer, and a dynamic spatial attention module; The multi-scale pyramid includes a 1 / 2 downsampling convolutional layer, a 1 / 4 downsampling convolutional layer, and a multi-dilution rate convolutional layer. The 1 / 2 downsampling convolutional layer includes a Conv convolutional layer with a GeLU activation function, and the 1 / 4 downsampling convolutional layer includes two sets of Conv convolutional layers with GeLU activation functions. The dilatancy rate of the multi-dilution rate convolutional layer is d=1,2,3. The multi-scale pyramid is used to extract features at the original, 1 / 2, and 1 / 4 scales, and the extracted multi-scale feature context is fused through the multi-dilution rate convolutional layer. The adaptive interaction module is embedded in the lateral connection of the multi-scale pyramid. The input is the multi-scale features after fusing the context, including a cross-scale attention layer and a bilinear difference alignment feature scale layer. The cross-scale attention layer includes three Sigmoid weighted branches. The adaptive interaction module is used to dynamically fuse multi-scale features and output the weighted unified features. The dynamic hyperparameter controller includes two convolutional layers and one average pooling layer, which are used to dynamically adjust the number of attention heads, Tucker decomposition rank, and fusion weights of multi-scale and attention based on the weighted unified features of the input. The input of the dynamic hyperparameter controller is the weighted features, the output is the weighted unified features, and the output is dynamic parameters, including the number of attention heads, rank, and fusion weights. The Tucker decomposition linear layer includes a core matrix and a low-rank projection matrix. The rank of the core matrix is ​​rank×rank, and the low-rank projection includes U and V matrices. The Tucker decomposition linear layer is used to reduce the computational complexity of QKV projection through the core matrix and to compress parameters through the low-rank projection matrix. The dynamic spatial attention module includes a deformable window attention layer and a channel-space interaction module, which are used to adjust the multi-head attention mechanism according to the number of attention heads output by the dynamic hyperparameter controller. The loss function module is used to maximize the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image; The loss function module employs a pre-trained multimodal image model. Through comparative learning, it aligns the semantic feature distributions of the super-resolution image and the high-resolution image, and maximizes the similarity between the super-resolution image and the high-resolution image in the segmentation feature space to obtain the final super-resolution image.

2. The image inpainting and super-resolution reconstruction system based on deep learning according to claim 1, characterized in that, The image preprocessing module includes a lightweight convolutional network and a memory access optimization module; The lightweight convolutional network is used to parse the parameters of each dimension of the input low-resolution image, evaluate the local complexity of the input image, dynamically generate adaptive windows, perform differential region division, and reconstruct the tensor into a multi-scale structure. The memory access optimization module is used to perform tensor permutation to convert physical storage into row-major order storage, and to perform stride alignment, partitioning the processed window into... Output in the form of N=(H / K)×(W / K), where N is the total number of dynamic windows.

3. A deep learning-based image inpainting and super-resolution reconstruction method, applied to the deep learning-based image inpainting and super-resolution reconstruction system described in any one of claims 1-2, characterized in that, include: Step 1: Select the low-resolution image to be processed I The input image preprocessing module performs preprocessing. Step 2: Transfer the preprocessed low-resolution image I Input feature extraction layer to preprocess low-resolution image I Converted into feature maps extracted by convolutional layers ,in, The height of the initial feature map, The width of the initial feature map. This represents the number of channels in the initial feature map; Step 3: Calculate the query of the preliminary feature map ,key Sum And attention distribution; Step 4: Dynamically fuse the multi-scale features of the initial feature map using a multi-scale pyramid and an adaptive interaction module, and output the weighted unified features; Step 5: Optimize the weighted unified features using a dynamic hyperparameter controller and a dynamic spatial attention module. Perform a block approximate cross product on the weight tensor of the optimized features. Decompose the weight tensor of the optimized features using a Tucker decomposition linear layer to complete the positional self-attention processing. Step 6: Combine class activation maps with contrastive learning through the loss function module to construct a semantically perceptual feature space, and set the given image pairs The contrastive loss function is used during training to train the multimodal image model. Image region category masks are generated through a pre-trained semantic segmentation model. The trained images are then processed by the multimodal image model, so that the original low-resolution images undergo structured processing and model optimization in sequence, and finally a super-resolution image is obtained.

4. The image inpainting and super-resolution reconstruction method based on deep learning according to claim 3, characterized in that, Step 3 Query ,key Sum The calculation formula is: (1)； (2)； (3)； In formulas (1)-(3), , , The weight matrix is ​​a learnable matrix. This is the feature map extracted after the convolutional layer; The formula for calculating attention distribution is: (4)； In formula (2), For the dimensions of query and key, It is a normalized exponential function.

5. The image inpainting and super-resolution reconstruction method based on deep learning according to claim 3, characterized in that, Step 4 specifically includes: The feature map after calculating the attention distribution is extracted using a multi-scale pyramid, and features at three scales (original, 1 / 2, and 1 / 4) are extracted. The extracted multi-scale feature context is then fused using a multi-dipty convolutional layer. Adaptive weights are generated based on the extracted feature content. According to adaptive weights Dynamically adjust fusion weights P i And through the cross-attention mechanism, it can interact in the channel or spatial dimensions; Fusion weights P i The calculation formula is: (5)； In formula (5), This represents a higher-level, more comprehensive fusion weight. For the feature map at the current scale i, For adaptive weights used to weight high-level features, These are the adaptive weights used to weight the features of the current layer.

6. The image inpainting and super-resolution reconstruction method based on deep learning according to claim 3, characterized in that, Step 5 specifically includes: Step 5.1: Divide the hyperparameter space into three subspaces: structural parameters, regularization parameters, and learning parameters using a dynamic hyperparameter controller. Use a hierarchical optimization strategy to perform Bayesian optimization and gradient descent on the three subspaces respectively. For structural parameters, Bayesian optimization is used to search for the optimal combination of convolution kernel size and network depth, and an objective function is used. For regularization parameters, a dynamic mapping function is constructed based on meta-learning. For learning parameters, an adaptive gradient algorithm is used. Step 5.2: Introduce differentiable NAS technology to construct a super network that covers all candidate structures. The super network covering all candidate structures includes constructing an operation pool, learning path weights by calculating the activation probability of each path through Gumbel-Softmax sampling, and adaptively selecting operation combinations based on the fusion frequency domain features of the image. Step 5.3: Design a weighted loss function for multi-objective collaborative optimization and perform a global parameter search on the training dataset; Step 5.4: Divide the weight tensor matrix of the optimized features into... After sub-blocks, approximate calculations are performed to complete the block approximate cross product, and the weight tensors of the optimized features are decomposed through the Tucker decomposition linear layer. The expression for the objective function is: (6)； In formula (6), This is the complexity penalty coefficient. This is the set of network architecture parameters for Bayesian optimization search; Is it using structural parameters? The model's prediction results for input X; The expression for the dynamic mapping function is: (7)； In formula (7), These are characteristic statistics; The update rule for the adaptive gradient algorithm is: (8)； In formula (8), This is the current loss function value. This is the cumulative amount of decay. The attenuation rate, This is the gradient sensitivity coefficient, used to control the strength of the influence of the gradient norm on the learning rate. The L2 norm of the gradient of the loss function reflects the stability of the current optimization direction; The formula for calculating the path activation probability is: (9)； In formula (9), It is Gumbel noise. For temperature parameters; The expression for the weighted loss function is: (10)； In formula (10), For the first Target feature embedding, For learnable weight vectors, T It is the transpose operator. This is the transpose of vector V; The approximate calculation expression is: (11)； In formula (11), The learnable importance coefficient, For Kronecker product, A and B The input matrix; The expression for weighted tensor decomposition is: (12)； In formula (12), For the Tucker core tensor, For Tensor Train chain tensors, Decompose the Tucker factor matrix.

7. The image inpainting and super-resolution reconstruction method based on deep learning according to claim 3, characterized in that, The expression for the comparison loss function in step 6 is: (13)； In formula (13), This refers to the semantically relevant regions determined by CAM. The multimodal image model employs the aforementioned multilayer fusion network. During training, it uses blur kernel combination and noise injection to simulate realistic degradation, performs downsampling, and uses bicubic interpolation to generate low-resolution input. Real-time augmentation of the training dataset is also performed, specifically including spatial manipulation, color perturbation, and texture destruction. A phased training strategy is adopted, which includes learning 0-100k basic features. The optimization objective uses Charbonnier. The Loss function is an adaptive robust loss function. The learning rate strategy uses cosine annealing, with dynamic activation during the 100k-200k iterations. A hybrid loss function is used: 60% L1 loss + 20% perceptual loss + 20% contrastive loss. The learning rate is reset to 1e-4, linearly decaying to 5e-6, and gradient clipping is introduced. Adversarial fine-tuning is performed during the 200k-300k iterations. The PatchGAN discriminator is loaded, an adversarial framework is constructed, and the loss weights are adjusted to 40% L1 loss + 30% adversarial loss + 30% contrastive loss. In the first 5k iterations, the generator parameters are fixed, and only the discriminator is trained, gradually increasing the adversarial loss weights and enabling mixed precision training.

Images