Bidirectional adaptive video super-resolution method based on frame difficulty index

Abstract

The invention discloses a bidirectional adaptive video super-resolution method based on frame difficulty index, and belongs to the field of computer vision. The invention provides a frame-level dynamic reconstruction method for solving the problems that simple frame calculation is redundant and difficult frame reconstruction is insufficient due to the fact that an existing model adopts a fixed calculation strategy for video frames with different difficulties. According to the method, a motion detail decoupling propagation network is constructed, motion information is efficiently transmitted by utilizing a shallow forward propagation branch, and a deep backward propagation branch focuses on recovering texture details so as to decouple a time sequence propagation task; meanwhile, a frame reconstruction difficulty evaluation network is introduced to generate a global difficulty index, so that the receptive field weight of the adaptive time sequence fusion network and the refining depth of the dynamic refining network are regulated and controlled. According to the method, through explicit modeling frame-level reconstruction difficulty, adaptive matching of the model capacity and the video frame feature complexity is realized, and the video reconstruction performance is remarkably improved under limited computing power.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video image processing technology, specifically to a deep learning-based video super-resolution reconstruction technology, and particularly to a bidirectional adaptive video super-resolution method based on frame difficulty indexing. Background Technology

[0002] Video super-resolution (VSR) technology aims to reconstruct high-resolution continuous video frames by utilizing the spatiotemporal information contained in low-resolution video sequences. It has wide applications in fields such as security monitoring, high-definition display, restoration of old images, and streaming media transmission. With the rapid development of deep learning technology, video super-resolution methods based on convolutional neural networks have gradually replaced traditional interpolation methods and become the mainstream of current research.

[0003] Among numerous deep learning VSR architectures, bidirectional propagation mechanisms demonstrate significant technical advantages. These methods typically utilize optical flow networks to align features from adjacent frames and propagate these features bidirectionally along the time axis to maximize the aggregation of long-range temporal context information. Generally, such networks include an encoder for feature extraction, a recurrent neural network module for temporal feature aggregation, and an upsampling module for image reconstruction, achieving a good balance between performance and efficiency.

[0004] However, most existing bidirectional propagation-based video super-resolution methods employ deterministic models, meaning that regardless of the complexity of the input video frames, the model uses a fixed and uniform network depth and temporal receptive field. This static processing strategy ignores the objective differences in reconstruction difficulty between video frames, leading to a severe mismatch between model computational capacity and frame-level feature complexity: for simple, easily reconstructable frames, deep networks result in redundant computational resources and increased inference latency; while for complex, difficult-to-reconstruct frames, the fixed network capacity is often insufficient to fully capture high-frequency details, thus limiting reconstruction quality.

[0005] Furthermore, existing bidirectional propagation architectures typically do not decouple the propagation of motion information from the restoration of texture details. In traditional bidirectional propagation networks, low-frequency motion alignment information and high-frequency texture detail information are usually mixed and transmitted in the same feature channel. Since the two types of information differ significantly in frequency domain characteristics and optimization objectives, this feature coupling approach makes it difficult for the model to balance motion coherence and detail restoration accuracy in complex scenes. Deep features often lose some texture details due to excessive focus on motion alignment, limiting the model's feature representation capabilities.

[0006] Therefore, how to construct a video super-resolution method that can explicitly perceive frame-level difficulty to achieve adaptive allocation of computing resources and effectively decouple motion and detail feature processing under limited computing power constraints is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] The main objective of this invention is to provide a bidirectional adaptive video super-resolution method based on frame difficulty indexing, in order to solve the problems of uneven resource allocation caused by fixed model calculation strategies and limited reconstruction quality caused by the coupling of motion and texture features in the prior art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A bidirectional adaptive video super-resolution method based on frame difficulty indexing, comprising the following steps:

[0010] Step 1: Input the low-resolution video frame sequence to be reconstructed.

[0011] Step 2: Construct a motion detail decoupling propagation network. Input the low-resolution video frame sequence into the network, use the shallow forward propagation branch in the network to perform temporal transmission of motion information of the video frames, and use the deep backward propagation branch to restore and enhance the texture details of the video frames, so as to obtain forward propagation features and backward propagation features respectively.

[0012] Step 3: Construct a frame reconstruction difficulty assessment network. Input the low-resolution video frame input in Step 1 into the frame reconstruction difficulty assessment network, and generate a frame difficulty index that reflects the current frame reconstruction difficulty through the network.

[0013] Step 4: Construct an adaptive temporal fusion network and use the frame difficulty index to guide the adaptive temporal fusion network. Use the bidirectional propagation features obtained in Step 2 as the basic input and the frame difficulty index generated in Step 3 as the control signal. Use the index to dynamically generate fusion weights at each scale and perform multi-scale weighted fusion of the forward propagation features and backward propagation features to obtain the fused features.

[0014] Step 5: Construct a dynamic refinement network and use the frame difficulty index to guide the dynamic refinement network. Use the fusion features obtained in Step 4 as the basic input and the frame difficulty index generated in Step 3 as the control signal. Use the index to dynamically generate refinement weights at each depth and adaptively select the refinement depth of the network to refine the fusion features and obtain refined features.

[0015] Step 6: Input the refined features generated in Step 5 into the upsampling module to improve the spatial resolution and reconstruct a high-resolution video frame sequence.

[0016] This invention utilizes a motion detail decoupling propagation network and a frame reconstruction difficulty assessment network. By generating a frame difficulty index, it dynamically guides an adaptive temporal fusion network and a dynamic refinement network. Shallow forward propagation branches efficiently transmit motion information, while deep backward propagation branches focus on restoring texture details, achieving decoupling and adaptive processing of the temporal propagation task. This invention addresses the problems in existing technologies where models use fixed computational strategies for video frames of varying difficulty, leading to computational redundancy in simple frames and insufficient reconstruction of difficult frames. It also solves the problem of limited reconstruction quality due to the coupling of motion and texture features. This invention achieves an adaptive match between model capacity and video frame feature complexity, significantly improving video reconstruction performance under limited computing power. Attached Figure Description

[0017] Figure 1 This is the overall flowchart of the present invention.

[0018] Figure 2 This is a schematic diagram of the motion detail decoupling propagation network of the present invention.

[0019] Figure 3 This is a schematic diagram of the frame reconstruction difficulty assessment network of the present invention.

[0020] Figure 4 This is a schematic diagram of the adaptive temporal fusion network of the present invention.

[0021] Figure 5 This is a schematic diagram of the structure of the dynamic refining network of the present invention. Detailed Implementation

[0022] To make the objectives and technical solutions of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings.

[0023] Figure 1 The flowchart of this invention mainly includes five core steps: motion detail decoupling propagation, frame reconstruction difficulty assessment, adaptive temporal fusion, dynamic refinement, and upsampling reconstruction.

[0024] Decoupling and propagation of motion details, such as Figure 2 As shown, this method aims to decouple low-frequency motion information from high-frequency texture information in a video. The specific steps are as follows:

[0025] Step 1: Given a low-resolution video frame sequence First, a pre-trained optical flow estimation network is used to calculate the bidirectional optical flow between adjacent frames: including the optical flow from... Frame to Forward optical flow with frame shift and from Frame to Backward optical flow of frame shift .

[0026] Step 2: Construct a shallow forward propagation branch, i.e., a motion branch, to transmit motion information. This branch aims to utilize information from past time steps to assist in motion feature extraction of the current frame, focusing on ensuring motion continuity. For each time step... First, using forward optical flow right Forward hidden state features at time step Perform spatial transformations to achieve feature alignment:

[0027]

[0028] in, This represents a spatial transformation operation based on optical flow. This represents the forward optical flow field. This branch also includes a motion feature extraction module for explicitly extracting motion features.

[0029] Specifically, this module receives an optical flow field as input, passes it through two layers of 3×3 convolution and activation functions, and maps the optical flow into high-dimensional motion features. :

[0030]

[0031] in, Represents the ReLU activation function. This indicates a convolution operation with a kernel size of 3×3.

[0032] Finally, the current frame Aligned forward features and motion characteristics The data is concatenated along the channel dimension and then input into a shallow residual network. Given that motion information is primarily located in the low-frequency band, the network is configured to contain a small number of residual blocks, outputting the forward features at the current time step:

[0033]

[0034] Step 3: Construct a deep backpropagation branch, also known as the detail branch, to recover texture details. This branch aims to recover high-frequency details of the current frame using information from future time steps and correct alignment errors. For each time step... First, utilize backward optical flow right Backward hidden state characteristics at time step Alignment:

[0035]

[0036] To suppress artifacts caused by optical flow alignment errors, this branch includes a detail enhancement module. This module generates pixel-level attention masks using a 1×1 convolutional sigmoid activation function, and then performs weighted filtering on the aligned features.

[0037]

[0038]

[0039] in, Represents the ReLU activation function. This represents a convolution operation with a kernel size of 1×1. This represents the Sigmoid activation function. This indicates an element-wise multiplication operation.

[0040] Finally, the current frame Compared with enhanced features The data is concatenated along the channel dimension and then input into a deep residual network. Given that texture restoration requires complex nonlinear mappings, the network is configured to contain a large number of residual blocks, outputting the backward features at the current time step:

[0041]

[0042] Frame reconstruction difficulty assessment, such as Figure 3 As shown, the specific steps are as follows:

[0043] Step 1: Construct a frame reconstruction difficulty assessment network. This network receives the current low-resolution frame. As input, the feature channels are mapped and transformed through cascaded 1×1 convolutional layers to extract the complexity features of the image. The feature extraction process can be represented as:

[0044]

[0045] in, This represents a convolution operation with a kernel size of 1×1. This represents the ReLU activation function.

[0046] Step 2: Generate Frame Difficulty Index. Spatial features are compressed into feature vectors using a global average pooling layer. These vectors are then mapped through a fully connected layer and activated by a sigmoid function, outputting a scalar form of the frame difficulty index. :

[0047]

[0048] in , This represents the Sigmoid activation function. This indicates a global average pooling operation. This indicates a fully connected layer. When... When the value approaches 0, it indicates that the current frame is a simple frame; when... When the value approaches 1, it indicates that the current frame is a difficult frame. This index serves as a global control signal, guiding resource allocation for subsequent modules.

[0049] Adaptive timing fusion, such as Figure 4 As shown, this step aims to dynamically adjust the aggregation weights of forward and backward features based on the frame difficulty index. The specific steps are as follows:

[0050] Step 1: Construct the weight prediction network. This network receives the scalar form of the frame difficulty index generated in the preceding steps. As input, the goal is to establish a non-linear mapping between difficulty and feature weights at different scales. The network consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Softmax activation function, connected sequentially. The specific computation process is as follows: First, the first fully connected layer maps the one-dimensional difficulty index to a high-dimensional feature space. After adding a non-linear factor through the ReLU activation function, the second fully connected layer outputs the original weight values ​​corresponding to the three scales. Finally, the Softmax activation function is used for normalization to obtain the final weight vector. This process can be represented as:

[0051]

[0052] in, and These represent the first and second fully connected layers, respectively. Represents the ReLU activation function. This represents the Softmax activation function. , , These correspond to the fusion weights of the subsequent three convolutional branches.

[0053] Step 2: Construct a multi-scale feature aggregation module. This module receives forward propagation features. and backpropagation characteristics As input, the forward and backward features are first merged along the channel dimension using a channel concatenation operation to obtain the concatenated features. :

[0054]

[0055] Subsequently, the splicing feature Simultaneously, the input is fed into three parallel convolutional feature extraction branches, including:

[0056] It is equipped with 1×1 convolutional layers and ReLU activation function to extract pixel-level fusion features. :

[0057]

[0058] It is equipped with 3×3 convolutional layers and ReLU activation function to extract fusion features of local neighborhoods. :

[0059]

[0060] It is equipped with 5×5 convolutional layers and ReLU activation function to extract fusion features within a large receptive field. :

[0061]

[0062] Step 3: Perform weighted aggregation. Utilize the weight vector generated in Step 1. Features generated in step two at three scales Perform element-wise weighted summation to obtain the final fused feature map. This weighted aggregation process dynamically adjusts the contribution of receptive fields at different scales based on frame difficulty.

[0063]

[0064] Specifically, for difficult frames containing rapid motion or complex textures, the network automatically increases the weights of the 5×5 convolutional branches. The network utilizes a large receptive field to aggregate longer-range spatiotemporal context to supplement missing information; while for simple frames that are flat or static, the network automatically increases the weights of the 1×1 convolutional branches. It focuses on the precise fusion of local pixels, thereby ensuring reconstruction quality while avoiding the introduction of noise interference from a distance.

[0065] Dynamic refining Figure 5 As shown, this step aims to dynamically adjust the network's refinement depth based on the frame difficulty index, achieving an optimal balance between efficient allocation of computational resources and reconstruction quality. The specific steps are as follows:

[0066] Step 1: Construct the weight prediction network. This network receives the scalar form of the frame difficulty index generated in the preceding steps. As input, the goal is to map the difficulty score to execution weights for networks of different depths. The network consists of a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Softmax layer connected sequentially. The specific computation process is as follows: First, the frame difficulty index is mapped to the hidden space through the first fully connected layer and then undergoes a non-linear transformation using the ReLU activation function; subsequently, the second fully connected layer outputs the original weight values ​​corresponding to the three refined depths; finally, the Softmax function is used to normalize the weights, resulting in the final depth weight vector. This process can be represented as:

[0067]

[0068] in, and These represent the first and second fully connected layers, respectively. Represents the ReLU activation function. This represents the Softmax activation function. , , These correspond to the weights of the subsequent shallow, middle, and deep branches, respectively.

[0069] Step 2: Construct a multi-depth feature refinement module. This module receives the fused feature map. As input, the features are subjected to nonlinear mapping and refinement at different depths through three parallel residual network branches. These three branches are:

[0070] Shallow residual network branches configured with a small number of residual blocks It is used to process simple frames, avoiding overfitting and wasted computation. Its output features are: :

[0071]

[0072] Mid-level residual network branches configured with a moderate number of residual blocks It is used to process regular frames, and its output characteristics are: :

[0073]

[0074] Deep residual network branches with a large number of residual blocks It is used to process difficult frames, providing powerful feature abstraction and recovery capabilities, and its output features are :

[0075]

[0076] Step 3: Perform weighted aggregation. Utilize the depth weight vector generated in Step 1. Output features of the three branches generated in step two Perform element-wise weighted summation to obtain the final refined feature map. :

[0077]

[0078] Specifically, this weighted aggregation process achieves an adaptive match between model computational capacity and video frame feature complexity: for difficult-to-reconstruct frames with complex textures, the network automatically increases the weights of deep residual network branches. It utilizes deeper nonlinear mapping capabilities to fully recover high-frequency details; while for easily reconstructed frames with simple textures, the network automatically increases the weights of shallow residual network branches. This approach ensures reconstruction quality while avoiding over-computation of simple features, thus solving the problem of uneven distribution of computational resources in traditional fixed-depth models.

[0079] The specific steps for upsampling reconstruction are as follows:

[0080] Step 1: Residual Map Generation. This involves generating the refined features output by the aforementioned dynamic refining network. The input is fed into the upsampling module. This module employs a pixel-shuffle structure, which expands the number of channels and rearranges their spatial positions through convolution operations to improve spatial resolution, ultimately reconstructing a high-resolution residual image. The calculation formula is as follows:

[0081]

[0082] in, This represents a subpixel convolution operation. This indicates a convolution operation with a kernel size of 3×3.

[0083] Step 2: Image Fusion and Reconstruction. The original low-resolution input frame is reconstructed using bilinear interpolation. Magnification is performed to obtain the reference image. Subsequently, the baseline image is compared with the residual image generated in step one. By adding each pixel individually, the final high-resolution video frame is obtained. :

[0084]

[0085] This process uses residual learning to focus the network on restoring high-frequency details, thereby improving the edge sharpness of the reconstructed image.

[0086] The aforementioned networks (including the motion detail decoupling propagation network, frame reconstruction difficulty assessment network, adaptive temporal fusion network, dynamic refinement network, and upsampling reconstruction network) collectively constitute the overall neural network model of this invention. To enable this neural network model to achieve adaptive resource allocation and high-precision video reconstruction, a training method based on the above model is provided, with the specific steps as follows:

[0087] Step 1: Construct the training sample set. Obtain high-resolution video sequences as ground truth, and generate corresponding low-resolution input frame sequences through downsampling.

[0088] Step 2: Parameter Initialization. The parameters of the motion detail decoupling propagation network, frame reconstruction difficulty assessment network, weight prediction network, and dynamic refinement network involved in this invention are initialized.

[0089] Step 3: Calculate the loss function. Input the training samples into the network to obtain the reconstructed frames. The Charbonnier loss function is used to calculate the reconstructed frame and the true high-resolution frame. Error between:

[0090]

[0091] Where N is the total number of video frames, This is the tolerance constant.

[0092] Step 4: Parameter Update and Collaborative Optimization. Based on the error values, backpropagation of the gradient is performed, and the Adam optimizer is used to synchronously update the parameters of the entire network. This end-to-end training method enables the difficulty index generated by the frame reconstruction difficulty assessment network to accurately perceive the complexity of image features and guide subsequent modules to adaptively allocate computational resources until the model converges.

Claims

1. This invention discloses a bidirectional adaptive video super-resolution method based on frame difficulty indexing, characterized in that, The system comprises a motion detail decoupling propagation network, a frame reconstruction difficulty assessment network, an adaptive temporal fusion network, and a dynamic refinement network. Specifically, the motion detail decoupling propagation network efficiently transmits motion information using shallow forward propagation branches, while deep backward propagation branches focus on restoring texture details, thus decoupling the temporal propagation task. The frame reconstruction difficulty assessment network generates a frame difficulty index reflecting the reconstruction difficulty of the current frame. The adaptive temporal fusion network dynamically generates fusion weights at different scales using the frame difficulty index, performing multi-scale weighted fusion of forward and backward propagation features. The dynamic refinement network dynamically generates refinement weights at different depths using the frame difficulty index, adaptively selecting the network's refinement depth to refine the fused features.

2. The bidirectional adaptive video super-resolution method based on frame difficulty indexing according to claim 1, characterized in that, The steps performed by the motion detail decoupling propagation network are as follows: Step 1: Given a low-resolution video frame sequence First, a pre-trained optical flow estimation network is used to calculate the bidirectional optical flow between adjacent frames: including the optical flow from... Frame to Forward optical flow with frame shift and from Frame to Backward optical flow of frame shift . Step 2: Construct a shallow forward propagation branch, i.e., a motion branch, to transmit motion information. This branch aims to utilize information from past time steps to assist in motion feature extraction of the current frame, focusing on ensuring motion continuity. For each time step... First, using forward optical flow right Forward hidden state features at time step Perform spatial transformations to achieve feature alignment: in, This represents a spatial transformation operation based on optical flow. This represents the forward optical flow field. This branch also includes a motion feature extraction module for explicitly extracting motion features. Specifically, this module receives an optical flow field as input, passes it through two layers of 3×3 convolution and activation functions, and maps the optical flow into high-dimensional motion features. : in, Represents the ReLU activation function. This indicates a convolution operation with a kernel size of 3×3. Finally, the current frame Aligned forward features and motion characteristics The data is concatenated along the channel dimension and then input into a shallow residual network. Given that motion information is primarily located in the low-frequency band, the network is configured to contain a small number of residual blocks, outputting the forward features at the current time step: Step 3: Construct a deep backpropagation branch, also known as the detail branch, to recover texture details. This branch aims to recover high-frequency details of the current frame using information from future time steps and correct alignment errors. For each time step... First, utilize backward optical flow right Backward hidden state characteristics at time step Alignment: To suppress artifacts caused by optical flow alignment errors, this branch includes a detail enhancement module. This module generates pixel-level attention masks using a 1×1 convolutional sigmoid activation function, and then performs weighted filtering on the aligned features. in, Represents the ReLU activation function. This represents a convolution operation with a kernel size of 1×1. This represents the Sigmoid activation function. This indicates an element-wise multiplication operation. Finally, the current frame Compared with enhanced features The data is concatenated along the channel dimension and then input into a deep residual network. Given that texture restoration requires complex nonlinear mappings, the network is configured to contain a large number of residual blocks, outputting the backward features at the current time step:

3. The bidirectional adaptive video super-resolution method based on frame difficulty indexing according to claim 1, characterized in that, The steps performed by the frame reconstruction difficulty assessment network are as follows: Step 1: Construct a frame reconstruction difficulty assessment network. This network receives the current low-resolution frame. As input, the feature channels are mapped and transformed through cascaded 1×1 convolutional layers to extract the complexity features of the image. The feature extraction process can be represented as: in, This represents a convolution operation with a kernel size of 1×1. This represents the ReLU activation function. Step 2: Generate Frame Difficulty Index. Spatial features are compressed into feature vectors using a global average pooling layer. These vectors are then mapped through a fully connected layer and activated by a sigmoid function, outputting a scalar form of the frame difficulty index. : in , This represents the Sigmoid activation function. This indicates a global average pooling operation. This indicates a fully connected layer. When... When the value approaches 0, it indicates that the current frame is a simple frame; when... When the value approaches 1, it indicates that the current frame is a difficult frame. This index serves as a global control signal, guiding resource allocation for subsequent modules.

4. The bidirectional adaptive video super-resolution method based on frame difficulty indexing according to claim 1, characterized in that, The adaptive temporal fusion network performs the following steps: Step 1: Construct the weight prediction network. This network receives the scalar form of the frame difficulty index generated in the preceding steps. As input, the goal is to establish a non-linear mapping between difficulty and feature weights at different scales. The network consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Softmax activation function, connected sequentially. The specific computation process is as follows: First, the first fully connected layer maps the one-dimensional difficulty index to a high-dimensional feature space. After adding a non-linear factor through the ReLU activation function, the second fully connected layer outputs the original weight values ​​corresponding to the three scales. Finally, the Softmax activation function is used for normalization to obtain the final weight vector. This process can be represented as: in, and These represent the first and second fully connected layers, respectively. Represents the ReLU activation function. This represents the Softmax activation function. , , These correspond to the fusion weights of the subsequent three convolutional branches. Step 2: Construct a multi-scale feature aggregation module. This module receives forward propagation features. and backpropagation characteristics As input, the forward and backward features are first merged along the channel dimension using a channel concatenation operation to obtain the concatenated features. : Subsequently, the splicing feature Simultaneously, the input is fed into three parallel convolutional feature extraction branches, including: It is equipped with 1×1 convolutional layers and ReLU activation function to extract pixel-level fusion features. : It is equipped with 3×3 convolutional layers and ReLU activation function to extract fusion features of local neighborhoods. : It is equipped with 5×5 convolutional layers and ReLU activation function to extract fusion features within a large receptive field. : Step 3: Perform weighted aggregation. Utilize the weight vector generated in Step 1. Features generated in step two at three scales Perform element-wise weighted summation to obtain the final fused feature map. This weighted aggregation process dynamically adjusts the contribution of receptive fields at different scales based on frame difficulty. Specifically, for difficult frames containing rapid motion or complex textures, the network automatically increases the weights of the 5×5 convolutional branches. The network utilizes a large receptive field to aggregate longer-range spatiotemporal context to supplement missing information; while for simple frames that are flat or static, the network automatically increases the weights of the 1×1 convolutional branches. It focuses on the precise fusion of local pixels, thereby ensuring reconstruction quality while avoiding the introduction of noise interference from a distance.

5. The bidirectional adaptive video super-resolution method based on frame difficulty indexing according to claim 1, characterized in that, The feature refinement network performs the following steps: Step 1: Construct the weight prediction network. This network receives the scalar form of the frame difficulty index generated in the preceding steps. As input, the goal is to map the difficulty score to execution weights of networks of different depths. The network consists of a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Softmax layer connected sequentially. The specific computation process is as follows: First, the frame difficulty index is mapped to the hidden space through the first fully connected layer, and then undergoes a non-linear transformation using the ReLU activation function; Subsequently, the original weight values ​​corresponding to the three refined depths are output through the second fully connected layer; finally, the weights are normalized using the Softmax function to obtain the final depth weight vector. This process can be represented as: in, and These represent the first and second fully connected layers, respectively. Represents the ReLU activation function. This represents the Softmax activation function. , , These correspond to the weights of the subsequent shallow, middle, and deep branches, respectively. Step 2: Construct a multi-depth feature refinement module. This module receives the fused feature map. As input, the features are subjected to nonlinear mapping and refinement at different depths through three parallel residual network branches. These three branches are: Shallow residual network branches configured with a small number of residual blocks It is used to process simple frames, avoiding overfitting and wasted computation. Its output features are: : Mid-level residual network branches configured with a moderate number of residual blocks It is used to process regular frames, and its output characteristics are: : Deep residual network branches with a large number of residual blocks It is used to process difficult frames, providing powerful feature abstraction and recovery capabilities, and its output features are : Step 3: Perform weighted aggregation. Utilize the depth weight vector generated in Step 1. Output features of the three branches generated in step two Perform element-wise weighted summation to obtain the final refined feature map. : Specifically, this weighted aggregation process achieves an adaptive match between model computational capacity and video frame feature complexity: for difficult-to-reconstruct frames with complex textures, the network automatically increases the weights of deep residual network branches. It utilizes deeper nonlinear mapping capabilities to fully recover high-frequency details; while for easily reconstructed frames with simple textures, the network automatically increases the weights of shallow residual network branches. This approach ensures reconstruction quality while avoiding over-computation of simple features, thus solving the problem of uneven distribution of computational resources in traditional fixed-depth models.

Images