Video super-resolution method, device, equipment, storage medium and program product

Abstract

This application provides a video super-resolution method, apparatus, device, storage medium, and program product, which can be applied to the audio and video field. The method includes: acquiring an initial video sequence; dividing the initial video sequence into multiple groups along the time axis, each group containing a target frame and reference frames with different time spans from the target frame; extracting features from each frame in each group, aligning features through temporal multiple correspondence aggregation to obtain feature maps corresponding to multiple groups; sequentially inputting the feature maps corresponding to multiple groups into a first-layer feedback network for feedback fusion to obtain a fused feature sequence; inputting the fused feature sequence into a second-layer feedback network for reconstruction to obtain super-resolution residual information; and adding the super-resolution residual information to the upsampling result of the target frame to obtain the super-resolution reconstruction result.

Description

Technical Field

[0001] This application relates to the field of audio and video, and more specifically to a video super-resolution method, apparatus, device, storage medium, and program product. Background Technology

[0002] Video super-resolution technology aims to reconstruct high-resolution video frames from low-resolution video sequences, and has wide applications in video surveillance, medical imaging, high-definition video production, and digital entertainment. Existing deep learning-based video super-resolution methods typically utilize temporal information between adjacent frames to assist in target frame reconstruction. However, most of these methods only focus on motion compensation over short time spans (such as adjacent frames), making it difficult to effectively capture the temporal dependencies and complementary information between distant frames. When there is periodic motion, slowly changing background, or repetitive texture in the video, the rich details contained in distant frames are not fully utilized, resulting in significant deficiencies in high-frequency details and texture fidelity in the reconstructed image. Summary of the Invention

[0003] In view of the above problems, embodiments of this application provide a video super-resolution method, apparatus, device, medium, and program product.

[0004] According to a first aspect of this application, a video super-resolution method is provided, comprising: acquiring an initial video sequence; dividing the initial video sequence into multiple groups on a time axis, each group containing a target frame and a reference frame with a different time span from the target frame; extracting features from each frame in each group, aligning the features through temporal multiple correspondence aggregation, and obtaining feature maps corresponding to multiple groups; sequentially inputting the feature maps corresponding to multiple groups into a first-layer feedback network for feedback fusion, and obtaining a fused feature sequence; inputting the fused feature sequence into a second-layer feedback network for reconstruction, and obtaining super-resolution residual information; adding the super-resolution residual information to the upsampling result of the target frame to obtain a super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence.

[0005] According to an embodiment of this application, inputting the fused feature sequence into a second-layer feedback network for reconstruction includes: sequentially inputting the fused feature sequence into multiple cascaded feedback blocks for processing. The processing of each feedback block includes: upsampling the input features to increase the feature space resolution to the same level as the target frame; performing residual learning on the upsampled features; and feeding the output of the current feedback block back to the input of the current feedback block, thereby gradually optimizing the reconstruction quality through multiple iterations.

[0006] According to embodiments of this application, upsampling is achieved through subpixel convolution; residual learning is achieved through convolutional layers, batch normalization layers, and activation functions, and skip connections are used; the number of multiple cascaded feedback blocks is a preset number, and the reconstruction results are gradually refined through multiple iterative feedbacks.

[0007] According to an embodiment of this application, dividing the initial video sequence into multiple groups on the timeline includes: obtaining a preset number of consecutive initial video frames, and using one of these frames as the target frame; dividing the preset number of frames into multiple groups, each group containing the target frame and a reference frame with a different time span from the target frame.

[0008] According to the embodiments of this application, the preset quantity is an odd number, and the number of multiple groups is the preset quantity minus 1 and then divided by 2. Each group includes a target frame and reference frames symmetrically distributed on both sides of the target frame. The time span between the reference frames and the target frame is different in different groups.

[0009] According to an embodiment of this application, inputting feature maps corresponding to multiple groups into a first-layer feedback network for feedback fusion includes: inputting feature maps corresponding to multiple groups into multiple cascaded feedback fusion blocks for processing; each feedback fusion block fuses the current input feature with the output feature of the previous time step and adds residual connections; wherein, each feedback fusion block calculates fusion weights through a gating mechanism; the gating mechanism concatenates the current input feature with the output feature of the previous time step in the channel dimension, generates weights through convolutional layers and activation functions, and performs weighted fusion of the current input feature and the output feature of the previous time step according to the weights.

[0010] According to an embodiment of this application, feature alignment via temporal multiple correspondence aggregation includes: processing feature correspondences at different scales through multiple parallel deformable convolutional branches, each deformable convolutional branch learning the spatial offset of each position in the feature map, and adaptively aligning the features of the reference frame according to the offset; and concatenating the outputs of multiple branches in the channel dimension and fusing them through a convolutional layer to obtain the aligned temporal features.

[0011] According to an embodiment of this application, the method further includes a training step: calculating the difference between the super-resolution reconstruction result and the real frame using a loss function, and updating the network parameters according to the difference, so that the network learns the mapping relationship from the resolution of the initial video sequence to the resolution of the super-resolution reconstruction result; wherein the resolution of the real frame is equal to the resolution of the super-resolution reconstruction result.

[0012] According to a second aspect of this application, a video super-resolution apparatus is provided, comprising: a data acquisition module for acquiring an initial video sequence; a data partitioning module for dividing the initial video sequence into multiple groups on a time axis, each group containing a target frame and a reference frame with a different time span from the target frame; a feature extraction module for extracting features from each frame in each group, aligning features through temporal multiple correspondence aggregation, and obtaining feature maps corresponding to multiple groups; a feature fusion module for sequentially inputting the feature maps corresponding to multiple groups into a first-layer feedback network for feedback fusion, and obtaining a fused feature sequence; a feature reconstruction module for inputting the fused feature sequence into a second-layer feedback network for reconstruction, and obtaining super-resolution residual information; and a result acquisition module for adding the super-resolution residual information to the upsampling result of the target frame to obtain a super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence.

[0013] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0015] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 The illustrations depict application scenarios of video super-resolution methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0018] Figure 2 A flowchart illustrating a video super-resolution method according to an embodiment of this application is shown schematically.

[0019] Figure 3 A block diagram illustrating a video super-resolution method according to an embodiment of this application is shown schematically;

[0020] Figure 4 This illustration schematically shows a structural diagram of a second-layer feedback module according to an embodiment of this application;

[0021] Figure 5 This schematically illustrates a flowchart of dividing an initial video sequence into multiple groups on a timeline according to an embodiment of the present application;

[0022] Figure 6 A flowchart illustrating feature alignment via temporal multiple correspondence aggregation according to an embodiment of this application is shown schematically.

[0023] Figure 7 This illustration schematically shows another flowchart of a video super-resolution method according to an embodiment of the present application;

[0024] Figure 8 A flowchart illustrating the processing of each feedback block according to an embodiment of this application is shown schematically;

[0025] Figure 9 A schematic diagram illustrating the structure of a video super-resolution apparatus according to an embodiment of this application is shown.

[0026] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a video super-resolution method according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] As used in this article, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. In this article, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0031] Figure 1 The illustration shows an application scenario of the video super-resolution method according to an embodiment of this application.

[0032] like Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0034] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0035] It should be noted that the video super-resolution method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the video super-resolution device provided in this application embodiment can generally be set in server 105 and / or terminal devices 101-103.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] Figure 2 A flowchart illustrating a video super-resolution method according to an embodiment of this application is shown schematically. Figure 3 A block diagram illustrating a video super-resolution method according to an embodiment of this application is shown schematically.

[0038] like Figure 2 and Figure 3 As shown, the video super-resolution method 200 according to an embodiment of this application may include steps S210 to S260.

[0039] In step S210, the initial video sequence is obtained.

[0040] In this step, the initial video sequence to be processed is obtained as input data for super-resolution reconstruction. The initial video sequence contains multiple consecutive frames, each with an initial resolution and relatively insufficient detail. In practical applications, the initial video sequence can originate from video surveillance equipment, videos captured by mobile terminals, video streams transmitted over a network, or video files stored on storage media. To facilitate subsequent processing, the input video frames typically require preprocessing, including normalizing pixel values ​​to a preset range (e.g., [0,1] or [-1,1]) and standardizing the size format of each frame.

[0041] In step S220, the initial video sequence is divided into multiple groups on the time axis, each group containing a target frame and a reference frame with a different time span from the target frame.

[0042] In this step, the input initial video sequence is grouped along the timeline to extract temporal information across different time spans. Specifically, multiple consecutive initial video frames (e.g., 7 frames) are selected, with one frame used as the target frame to be reconstructed (usually an intermediate frame), and the remaining frames used as reference frames. The multiple frames are divided into several groups, each containing the target frame and reference frames with different time spans from the target frame.

[0043] For example, an exemplary grouping method is as follows: Seven consecutive frames are acquired, denoted as frame t-3, frame t-2, frame t-1, frame t, frame t+1, frame t+2, and frame t+3, where frame t is the target frame. These seven frames are divided into three groups: the first group contains frames t-3, t, and t+3 (a time span of 3 frames); the second group contains frames t-2, t, and t+2 (a time span of 2 frames); and the third group contains frames t-1, t, and t+1 (a time span of 1 frame). Through this grouping method, the distance between the reference frame and the target frame varies in each group, thus enabling the capture of motion information and texture details at different time scales. The target frame is reused repeatedly within each group, allowing for a weighted combination of multi-scale temporal features during subsequent fusion.

[0044] The core of this grouping strategy lies in the fact that short-span groups can capture subtle movements and rapid changes between adjacent frames, which is beneficial for reconstructing high-frequency details; while long-span groups can capture periodic movements and background information, which is beneficial for restoring global structure and suppressing noise. By extracting information from different time spans separately and then fusing them, the problems of insufficient or redundant information in a single span can be effectively avoided.

[0045] In step S230, features of each frame in each group are extracted, and feature alignment is performed through temporal multiple correspondence aggregation to obtain feature maps corresponding to multiple groups. .

[0046] In this step, feature extraction is first performed on the three frames in each group to obtain shallow feature representations for each frame. The feature extraction module can consist of multiple cascaded convolutional layers, each followed by an activation function (such as a linear rectified function) to introduce a nonlinear transformation. Through this module, the original image space is mapped to a high-dimensional feature space, preserving the structural information of the image while reducing noise interference.

[0047] For each group of three frames, the features of the target frame and the features of the reference frame are input into the temporal multiple correspondence aggregation module (TM-CAM) for feature alignment. The core of this module is to solve the problem of pixel position mismatch caused by motion between different frames, so that the useful information in the reference frame can be accurately compensated for the corresponding position in the target frame.

[0048] Temporal Multiple Correspondence (TM-CAM) employs a deformable convolution mechanism, processing feature correspondences at different scales through multiple parallel branches. Each deformable convolution branch learns the spatial offset of each location in the feature map, and adaptively aligns the reference frame features based on this offset, mapping the feature points in the reference frame to the target frame coordinate system. Specifically, for each location in the reference frame feature map, the deformable convolution predicts a two-dimensional offset, indicating the spatial displacement of the feature point relative to the corresponding location in the target frame. Then, the reference frame features are sampled based on this offset to obtain the aligned feature map.

[0049] After acquiring the alignment features, the correlation between the alignment features and the target frame features is calculated using the correspondence TM-CAM to determine which reference information is most valuable to the current target frame. Correlation calculation can use dot product similarity or cosine similarity metrics. Finally, the correlation information is fused with the original features to obtain the aligned temporal feature map. For the outputs of different scale branches, the channels are concatenated and then fused through a convolutional layer to obtain the corresponding final feature map.

[0050] Through the above processing, each group outputs a feature map, which integrates the temporal information of reference frames from different time spans within that group and is aligned to the target frame coordinate system. The feature maps of different groups correspond to temporal information at different time scales and are complementary.

[0051] In step S240, the feature maps corresponding to multiple groups are sequentially input into the first layer feedback network for feedback fusion to obtain a fused feature sequence.

[0052] In this step, the feature maps corresponding to the multiple groups obtained in step S230 (e.g., three groups of feature maps) are sequentially input into the first-layer feedback network for feedback fusion. The first-layer feedback network is used to gradually fuse temporal features at different time scales to form a more information-rich fused feature.

[0053] The first-layer feedback network consists of multiple cascaded feedback fusion blocks. Each feedback fusion block is responsible for fusing the current input features with the output features from the previous time step and adding residual connections. Specifically, let the current input features be F. in The output feature at the previous time step is F prev (Initial time F) prev If the value is zero, then the output F of the current block is zero. out The calculation is as follows: First, the fusion weights are calculated using a gating mechanism, and F is... in With F prev After concatenation along the channel dimension, a weight vector g is generated through a convolutional layer and activation function, with values ​​ranging from 0 to 1. Then, F is applied based on this weight. in With F prev Perform weighted fusion:

[0054] F fused =g⊙F in +(1-g)⊙F prev

[0055] Here, ⊙ represents element-wise multiplication. Finally, a residual connection is added to obtain the output feature: F. out =F fused +F in .

[0056] Through the aforementioned gating mechanism, the network can adaptively learn the relative importance of current group features and historical fusion features, dynamically adjusting the fusion weights based on the input content, thereby achieving on-demand fusion of temporal information. As multiple feedback fusion blocks are processed sequentially, features processed earlier can be passed to later stages, and features processed later can influence the fusion results of earlier stages through feedback connections, forming a closed loop of information flow. Through multiple iterations, the temporal features of different groups are fully fused, outputting a fused feature sequence.

[0057] In step S250, the fused feature sequence is input into the second-layer feedback network for reconstruction to obtain super-resolution residual information.

[0058] In this step, the fused feature sequence output from step S240 is input into the second-layer feedback network for reconstruction, outputting super-resolution residual information. The second-layer feedback network is used to progressively transform the fused features into high-frequency detail information, restoring fine structures such as edges and textures in the image.

[0059] The second-layer feedback network consists of multiple cascaded feedback blocks, each containing an upsampling unit, a residual unit, and a feedback connection unit. The upsampling unit upscales the feature map to the same spatial resolution as the target frame, typically using sub-pixel convolutions to efficiently map the initial resolution feature map to a higher resolution space through pixel rearrangement. The residual unit performs further non-linear transformations and detail enhancements on the upsampled features, usually containing two convolutional layers, two batch normalization layers, and two activation functions, with skip connections to preserve input features and prevent gradient vanishing. The feedback connection unit feeds the output of the current feedback block back to its input, allowing the network to iteratively optimize the reconstruction result through multiple iterations, refining and detailing the changes with each iteration.

[0060] Specifically, let H be the input of the i-th feedback block. i-1 Then its output H i The calculation is as follows: First, H is upsampled using the upsampling unit. i-1 Magnified to the target resolution, we obtain H up Then, H is processed through residual elements. up Feature extraction and nonlinear transformation are performed to obtain H. res Finally, H is connected via feedback. res With H after channel adjustment i-1 Add them together to get H. i =H res + Conv{1×1}H i-1 By cascading multiple feedback blocks, the network can gradually learn the process of reconstructing details from coarse to fine.

[0061] The core advantage of this feedback mechanism lies in its ability to iteratively optimize the network, gradually correcting errors from previous reconstructions while preserving recovered details, ultimately outputting high-quality residual information. Residual information represents the high-frequency differences between high-resolution frames (higher resolution than the initial video frame) and the upsampled target frame, i.e., missing edges, textures, and other details in the image.

[0062] In step S260, the super-resolution residual information is added to the upsampling result of the target frame to obtain the super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence.

[0063] In this step, the super-resolution residual information output in step S250 is added to the upsampling result of the target frame to obtain the final super-resolution reconstruction result. The upsampling of the target frame is achieved using traditional image interpolation methods such as bicubic interpolation, enlarging the target frame with its initial resolution to a target high-resolution size, thus obtaining an image containing low-frequency information. The residual information contains high-frequency details; after the two are added, the low-frequency structure and high-frequency details complement each other, forming a clear and realistic high-resolution image with a higher resolution than the initial resolution. The resolution of this super-resolution reconstruction result is greater than the resolution of the initial video sequence.

[0064] This design decomposes the super-resolution task into two parts: "low-frequency preservation" and "high-frequency recovery." The network only needs to learn the high-frequency residual information, without having to generate the entire image from scratch, thus reducing the learning difficulty. At the same time, the structural information of the target frame is fully preserved, ensuring the semantic consistency between the reconstructed result and the original content. Through this decomposition, the network can focus more on detail recovery, improving reconstruction quality and training efficiency.

[0065] Figure 4 The schematic diagram illustrates the structure of the second-layer feedback module according to an embodiment of this application.

[0066] like Figure 4 As shown, the second-layer feedback module consists of multiple cascaded feedback blocks, each with the same structure, which progressively optimize the reconstruction quality through an iterative feedback mechanism. Each feedback block includes an upsampling unit, a residual unit, and a feedback connection unit. Specifically, the upsampling unit is implemented using sub-pixel convolution to increase the spatial resolution of the input features to the same level as the target frame. The residual unit contains two convolutional layers, two batch normalization layers, and two activation functions, and uses skip connections to perform feature extraction and nonlinear transformation on the upsampled features. The feedback connection unit feeds the output of the current feedback block back to the input of the current feedback block after adjusting the number of channels through a 1×1 convolution, adding it to the input features to form a residual connection. Through the cascaded processing of multiple feedback blocks, the network can progressively refine the reconstruction results through multiple iterations, with each iteration correcting and enhancing details based on the previous iteration.

[0067] Figure 5 The flowchart illustrating the process of dividing an initial video sequence into multiple groups on a timeline according to an embodiment of this application is shown in the illustration.

[0068] like Figure 5 As shown, based on the aforementioned embodiments, step S220 may include steps S510 to S520.

[0069] In step S510, a preset number of consecutive initial video frames are obtained, and one of these frames is used as the target frame.

[0070] In this step, multiple consecutive frames are extracted from the initial input video sequence as the basis for subsequent grouping processing. The preset number of frames can be set according to the actual application scenario and network design requirements. An odd number of frames is usually used to ensure that the target frame is centered and to facilitate the symmetrical use of information from preceding and following frames.

[0071] Specifically, N consecutive initial video frames are acquired, where N is a preset positive integer. In one exemplary implementation, N is 7, meaning 7 consecutive initial video frames are acquired. For ease of description, these 7 frames are sequentially labeled as frame t-3, frame t-2, frame t-1, frame t, frame t+1, frame t+2, and frame t+3. Frame t is selected as the target frame, i.e., the initial resolution input corresponding to the high-resolution frame to be reconstructed. The target frame is the core object of super-resolution reconstruction; information from all subsequent reference frames will be used to assist in reconstructing this target frame.

[0072] It should be noted that the target frame is not limited to a strictly centered position. In some implementations, the target frame can be any frame in the sequence, and correspondingly, the selection range of reference frames is adjusted accordingly. For example, in real-time video stream processing scenarios, historical frames can be used as reference frames instead of relying on future frames to meet the needs of low-latency applications. This application does not specifically limit the position of the target frame, and those skilled in the art can flexibly choose according to the actual application scenario.

[0073] In step S520, a preset number of frames are divided into multiple groups, each group containing a target frame and a reference frame with a different time span from the target frame.

[0074] In the embodiments of this application, the preset quantity is an odd number, and the number of multiple groups is the preset quantity minus 1 and then divided by 2. Each group includes a target frame and reference frames symmetrically distributed on both sides of the target frame. The time span between the reference frames and the target frame is different in different groups.

[0075] In this step, the N consecutive frames obtained in step S510 are divided into multiple groups. Each group contains a target frame and a reference frame with a different temporal distance (i.e., time span) from the target frame. This grouping method allows for the extraction of temporal information at different time scales from the same video sequence, providing input for subsequent multi-scale feature fusion.

[0076] In some exemplary implementations, taking N=7 as an example, the 7 frames are divided into three groups, and the specific grouping method is as follows:

[0077] Group 1 (G1): Contains frames t-3, t, and t+3. The time span between the reference frame and the target frame in this group is 3 frames, which can capture motion trends and background information over a longer period, playing an important role in restoring global structure and suppressing noise.

[0078] The second group (G2) contains frames t-2, t, and t+2. The reference and target frames in this group span two frames in time, which can capture motion information at a medium time scale and helps to recover mid-frequency details.

[0079] The third group (G3) includes frames t-1, t, and t+1. The time span between the reference frame and the target frame in this group is one frame, i.e., adjacent frames. It can capture subtle movements and rapid changes within a short period of time, which is crucial for reconstructing high-frequency details and edge textures.

[0080] By employing the aforementioned grouping method, reference frames in each group are symmetrically distributed on both sides of the target frame, ensuring the symmetry and integrity of temporal information. The target frame is reused repeatedly within each group, enabling information at different time scales to be aligned and fused with the target frame as a reference during subsequent feature extraction and fusion processes, thus avoiding information loss due to missing reference frames.

[0081] The core advantage of this grouping strategy lies in the fact that different groups capture motion features and texture information at different time scales, forming a complementary set of temporal information. The short-time span group (G3) is sensitive to rapid motion and can accurately capture local detail changes; the medium-time span group (G2) can smooth short-term fluctuations and provide stable structural information; and the long-time span group (G1) can capture periodic motion and background texture, which helps to restore global consistency. The combination of these three provides a rich source of information for subsequent temporal alignment and feature fusion.

[0082] It should be noted that the above grouping method is merely an example and does not constitute a limitation on the scope of protection of this application. In practical applications, the preset number of frames and the specific grouping method can be flexibly adjusted according to the characteristics of the video content, the speed of motion, and the needs of network design. For example, when the video motion is relatively slow, groups with larger time spans can be added to make full use of periodic information; when the video motion is rapid, the weight of groups with shorter time spans can be increased to ensure alignment accuracy. Those skilled in the art can make adaptive adjustments based on the concepts disclosed in this application according to specific application scenarios.

[0083] Figure 6 The flowchart illustrating feature alignment via temporal multiple correspondence aggregation according to an embodiment of this application is shown in the illustration.

[0084] like Figure 6 As shown, based on the aforementioned embodiments, step S230 may include steps S610 to S620.

[0085] In step S610, feature correspondences at different scales are processed by multiple parallel deformable convolutional branches. Each deformable convolutional branch learns the spatial offset of each position in the feature map and adaptively aligns the features of the reference frame according to the offset.

[0086] In this step, the reference frame features and target frame features in each group are temporally aligned to eliminate pixel position mismatches caused by object motion, camera shake, or scene changes in the video. To achieve high-precision adaptive alignment, this application employs multiple parallel deformable convolutional branches to process feature correspondences at different scales.

[0087] The core idea of ​​deformable convolution is to introduce a learnable spatial offset on top of the standard convolution kernel, enabling the convolution operation to adaptively adjust the sampling position according to the image content, thereby better capturing the deformation and motion of objects. For feature alignment tasks, deformable convolution learns the spatial mapping relationship between the feature maps of the reference frame and the target frame, accurately "moving" the feature points in the reference frame to the corresponding positions in the target frame, achieving pixel-level precise alignment.

[0088] Specifically, each deformable convolutional branch contains an offset prediction network and a deformable convolutional layer. The offset prediction network takes the concatenation of reference frame features and target frame features as input and predicts the spatial offset Δp at each sampling location through multiple convolutional layers. For each location p on the reference frame feature map, the offset prediction network outputs a two-dimensional offset vector Δp ​​= (Δx, Δy), indicating the spatial displacement of the feature point at that location relative to the corresponding location in the target frame.

[0089] Then, the deformable convolutional layer samples and aggregates the reference frame features based on the predicted offset. Let the reference frame features be F. r The target frame feature is F t The deformable convolution kernel size is K×K. For each position p on the output feature map, the deformable convolution... The calculation formula can be expressed as:

[0090]

[0091] Among them, w k p represents the convolution kernel weights. k For a predefined sampling offset (such as the grid position of a standard convolutional kernel), Δpk is the predicted offset corresponding to the k-th sampling point. Through this operation, feature points in the reference frame are adaptively "moved" to the target frame coordinate system, achieving precise alignment.

[0092] To capture motion information at different scales, this application employs multiple parallel deformable convolutional branches, each using a different receptive field or feature scale. For example, three parallel deformable convolutional branches can be used to process feature maps with different downsampling ratios, or convolutional kernels with different dilation rates can be employed. Branches with small receptive fields are sensitive to local motion and can accurately align small-scale motions; branches with large receptive fields can capture global motion trends and have better robustness to large-scale motions. Through multi-scale parallel processing, the alignment requirements of motions at different scales can be simultaneously addressed, improving alignment accuracy.

[0093] In step S620, the outputs of multiple branches are concatenated along the channel dimension and then fused through a convolutional layer to obtain aligned temporal features.

[0094] In this step, the output feature maps of multiple parallel branches in step S610 are concatenated along the channel dimension to form a fused feature map with a multiplied number of channels. Then, feature fusion is performed through a convolutional layer to obtain the aligned temporal features.

[0095] Specifically, let F be the output feature map of the i-th deformable convolutional branch. i Its dimensions are H×W×C i C i Let F be the number of channels in this branch. Concatenate the outputs of the M branches along the channel dimension to obtain the concatenated feature F. cat Its dimensions are H×W×(C1+C2+...+C M The splicing operation preserves the feature information independently learned by each branch, avoiding information loss.

[0096] Subsequently, the concatenated feature maps are input into a convolutional layer for feature fusion. This convolutional layer typically uses 1×1 or 3×3 convolutions, and its function is to interact and integrate the information from each branch, extracting more discriminative fused features. 1×1 convolutions can effectively perform linear combination between channels, reducing the number of channels while retaining key information from each branch; 3×3 convolutions can further fuse spatial neighborhood information, enhancing the expressive power of the features. Through this fusion convolutional layer, the finally aligned temporal feature map F is output. out Its dimensions are H×W×C out C out It can be set according to network design requirements (e.g., to be consistent with the number of feature channels of the target frame).

[0097] This multi-branch parallel processing and fusion mechanism captures local detailed motion and global large motion by setting deformable convolutional branches of different scales, achieving scale-adaptive and accurate alignment and avoiding the limitations of a single scale. The offset information learned by each branch is spliced ​​and fused to form complementarity, significantly improving the alignment robustness in complex motion scenarios. At the same time, the entire system is end-to-end differentiable and can be jointly optimized through backpropagation, enabling each branch to adaptively learn the optimal alignment strategy.

[0098] Through the above steps S610 and S620, this application achieves adaptive, multi-scale, and high-precision alignment of reference frame features, providing accurate temporal feature input for subsequent feedback fusion.

[0099] Figure 7 Another flowchart of a video super-resolution method according to an embodiment of this application is illustrated schematically.

[0100] like Figure 7 As shown, based on the foregoing embodiments, step S240 may include step S710.

[0101] In step S710, the feature maps corresponding to multiple groups are sequentially input into multiple cascaded feedback fusion blocks for processing. Each feedback fusion block fuses the current input feature with the output feature of the previous time step and adds residual connections. Each feedback fusion block calculates fusion weights through a gating mechanism. The gating mechanism concatenates the current input feature and the output feature of the previous time step in the channel dimension, generates weights through convolutional layers and activation functions, and performs weighted fusion of the current input feature and the output feature of the previous time step according to the weights.

[0102] In this step, the feature maps corresponding to multiple groups obtained in step S230 (e.g., three groups of feature maps, each corresponding to a different time span) are sequentially input into the first-layer feedback network for feedback fusion. The first-layer feedback network is used to gradually fuse temporal features from different time scales to form a more information-rich and compact fused feature sequence.

[0103] The first-layer feedback network consists of multiple cascaded feedback fusion blocks. "Cascaded" means that multiple feedback fusion blocks are connected sequentially according to the input order, with the output of the previous block serving as part of the input of the next block, forming a serial processing chain. "Feedback" means that when processing the current input, each feedback fusion block incorporates the output characteristics of the previous time step (i.e., the previous block), allowing information to circulate between blocks and achieve gradual correction and fusion.

[0104] Suppose the first-layer feedback network contains K feedback fusion blocks, where K equals the number of feature maps output from step S230 (e.g., 3). For the i-th feedback fusion block, its input includes two parts: the current input feature F... in,i(i.e., the feature map corresponding to the i-th group) and the output feature F from the previous time step prev,i (i.e., the output of the (i-1)th block, with F at the initial time) prev,1 (The value is zero). The processing of this block includes fusion operations and residual connections, ultimately outputting the feature F of the current block. out,i .

[0105] Each feedback fusion block calculates fusion weights through a gating mechanism to achieve adaptive fusion of the current input features and the output features from the previous time step. The core idea of ​​the gating mechanism is to dynamically determine the relative importance of the current features and historical features based on the input content, avoiding poor fusion results caused by fixed weights.

[0106] Specifically, the calculation process of the gating mechanism is as follows:

[0107] First, the current input feature F in Compared with the output feature F of the previous time step prev The concatenation is performed along the channel dimension to obtain the concatenated feature F. cat Let F in The dimensions are H×W×C, F prev If the dimensions are H×W×C, then after splicing, F cat The dimensions are H×W×2C.

[0108] Then, the splicing feature F cat The input is a convolutional layer, typically using a 1×1 kernel, which linearly combines the channel information, compressing the 2C channels into a C channel. Following the convolutional layer is an activation function, which introduces a non-linear transformation to enhance the expressive power of the gating mechanism. The output value of this activation function is constrained between 0 and 1, serving as the fusion weights.

[0109] Through the convolution and activation operations described above, we obtain the gating weights g, which have dimensions H×W×C. Each element of the gating weights g represents the relative importance of the current input feature and historical features at the corresponding spatial location and channel. Specifically, the closer g is to 1, the more important the current input feature is; the closer g is to 0, the more important the historical feature is.

[0110] Next, the current input feature F is adjusted according to the gating weight g. in and the output feature F of the previous time step prev Perform weighted fusion:

[0111]

[0112] Here, ⊙ represents element-wise multiplication. This weighted fusion operation achieves an adaptive combination of current and historical information: when g is close to 1, current information dominates; when g is close to 0, historical information is retained; when g is around 0.5, the two are fused with nearly equal weights.

[0113] Finally, to further alleviate the vanishing gradient problem and preserve the original information, residual connections are added. The fused features are added to the current input features to obtain the output F of this feedback fusion block. out :

[0114]

[0115] Through the aforementioned gating mechanism and residual connections, each feedback fusion block achieves adaptive fusion of current and historical features while maintaining the fluidity and stability of information.

[0116] Taking three sets of feature maps as an example, let the input feature map sequences be F1, F2, and F3, corresponding to time spans 1, 2, and 3 respectively. The processing flow of the first-layer feedback network is as follows:

[0117] First feedback fusion block: input is F1 and initial zero features (F prev =0). The gating mechanism calculates weights based on F1 and zero features, since F prev Since the weight is zero, the weight g is close to 1, therefore the output F is zero. out,1 ≈ F1 + F1 = 2 F1 (actually an enhanced version of F1 after gating and residuals). At this point, the current output mainly retains the information of the first set of features.

[0118] The second feedback fusion block: inputs are F2 and F out,1 The gating mechanism is based on F2 and F out,1 Calculate the fusion weights and adaptively determine F2 and F. out,1 The contribution ratio. Output F out,2 It combines information from F2 and F1 and adds residuals.

[0119] The third feedback fusion block: inputs are F3 and F out,2 The gating mechanism is based on F3 and F... out,2 Calculate the fusion weights and output F. out,3 It integrates all the information from F3, F2 and F1.

[0120] After the above three iterations, the fused feature sequence (F) output by the first layer feedback network is... out,1 F out,2 F out,3 The information from more and more groups is successively integrated, and the final output is an enhanced feature sequence that fully integrates multi-scale temporal features.

[0121] Through step S710 above, this application achieves adaptive and progressive fusion of features at different time scales, providing a rich and fully expressive fusion feature sequence for subsequent second-layer feedback reconstruction. This fusion mechanism effectively overcomes the limitations of traditional simple splicing or fixed-weight fusion methods, significantly improving the quality and robustness of video super-resolution reconstruction.

[0122] Figure 8 A flowchart illustrating the processing of each feedback block according to an embodiment of this application is shown.

[0123] like Figure 8 As shown, based on the aforementioned embodiment, step S250 may include sequentially inputting the fused feature sequence into multiple cascaded feedback blocks for processing, and the processing of each feedback block includes steps S810 to S830.

[0124] In step S810, the input features are upsampled to improve the feature space resolution to the same level as the target frame. Upsampling is achieved through sub-pixel convolution.

[0125] In this step, the input features are upsampled to enlarge the spatial size of the feature map to the same resolution as the target frame. Since the spatial resolution of the input feature map is typically low (e.g., 64×64), while the target frame needs to achieve a high resolution (e.g., 256×256) after upsampling, the feature map needs to be enlarged by a factor of 4. This step uses subpixel convolution (also known as pixel reassembly) to achieve upsampling. This method uses learnable convolutional kernels to efficiently map the initial resolution feature map to a higher resolution spatial space than the initial resolution.

[0126] The principle of subpixel convolution is as follows: Let the size of the input feature map be H×W×C, then expand the number of channels to r through a convolutional layer. 2 ×C, where r is the upsampling factor (r=4 in this embodiment). The resulting size is H×W×(r). 2 After obtaining the intermediate feature map of ×C), a pixel recombination operation is performed to reconstruct the r at each position. 2 The channels are rearranged into r×r spatial blocks, resulting in an output feature map of size (r×H)×(r×W)×C. This process can be represented as:

[0127]

[0128] PixelShuffle is the pixel reassembly operation, Conv 3×3 This is a convolutional layer used for channel expansion.

[0129] Subpixel convolution offers several advantages over traditional interpolation upsampling (such as bilinear interpolation): First, its parameters are learnable, allowing it to adaptively learn the optimal upsampling mode based on task requirements. Second, subpixel convolution combines the upsampling process with feature extraction, avoiding the blurring effects introduced by interpolation. Third, it boasts high computational efficiency, maintaining feature richness while amplifying resolution. Through subpixel convolution, this step elevates the spatial resolution of the input features to the same size as the target frame, providing a spatial alignment foundation for subsequent residual learning.

[0130] In step S820, residual learning is performed on the upsampled features. The residual learning is implemented through convolutional layers, batch normalization layers, and activation functions, and skip connections are used.

[0131] In this step, residual learning is performed on the upsampled features. High-frequency detail information is extracted through multiple convolutional layers, batch normalization layers, and activation functions. Skip connections are used to preserve the input features, forming a residual block structure.

[0132] Specifically, the structure of the residual learning module (residual block) is as follows: First, the upsampled features F... up The first convolutional layer, typically using a 3×3 kernel, is input to extract local spatial features. Following this convolution is a batch normalization layer to standardize the features, making their distribution more stable and accelerating network convergence. Then, a non-linear transformation is introduced through an activation function, such as ReLU. Next, the activated features are input to a second convolutional layer (also a 3×3 convolution) for further feature extraction, followed by another batch normalization layer. Finally, skip connections are used to connect the input features F. up The output of the second batch normalization layer is added to form a residual connection. The entire residual block F res The calculation process can be expressed as:

[0133]

[0134] The design of residual learning has the following technical advantages: First, it alleviates the vanishing gradient problem, enabling deep networks to be trained stably; second, it reduces the learning difficulty, as the network only needs to learn the difference between the input and output (residual), rather than the complete mapping function; third, it helps to preserve low-frequency information, allowing the network to focus on learning high-frequency details and improve reconstruction quality.

[0135] In step S830, the output of the current feedback block is fed back to the input of the current feedback block. The reconstruction quality is gradually optimized through multiple iterations. The number of multiple cascaded feedback blocks is a preset number. The reconstruction result is gradually refined through multiple iterations.

[0136] In this step, the output of the current feedback block is fed back to the input of the current feedback block through feedback connections, forming an iterative optimization mechanism. This feedback mechanism enables the network to gradually refine the reconstruction results through multiple iterations, with each iteration correcting and enhancing details based on the previous one.

[0137] Specifically, let H be the input of the i-th feedback block. i-1 After upsampling in step S610 and residual learning in step S620, the intermediate feature F is obtained. tmp Then, F is connected via a feedback link. tmp Adding this to the channel-adjusted input features yields the output Hi of the current feedback block:

[0138]

[0139] Where Conv1×1 is a 1×1 convolution used to adjust the number of channels of the input features to be similar to F. tmp Matching. This residual connection allows the output of the current block to simultaneously contain newly learned details and original input features, forming an information accumulation.

[0140] Multiple feedback blocks are cascaded to form the second layer of the feedback network. Let there be a total of L feedback blocks (L is a preset number, such as 3). The processing is as follows: First feedback block: input is a feature map H0 from the fused feature sequence, output is H1; Second feedback block: input is H1, output is H2; Third feedback block: input is H2, output is H3.

[0141] Through the cascaded structure described above, each iteration refines the features based on the previous one. The first iteration learns coarse details; the second iteration corrects and enhances the results of the first iteration; and the third iteration further refines the features, generating the final super-resolution residual information.

[0142] The core advantages of the feedback mechanism are: First, iterative optimization: By iterating multiple times, the optimal reconstruction result is gradually approached, with each iteration making corrections based on the previous one, effectively improving reconstruction quality. Second, information accumulation: Through feedback connections, subsequent blocks can obtain the processing results of preceding blocks, achieving information accumulation and transmission, avoiding the problem of insufficient information utilization in a single forward propagation. Third, controllable complexity: The number L of feedback blocks can be adjusted according to task requirements, achieving a balance between accuracy and computational efficiency. The larger L is, the higher the reconstruction quality, but the computational load also increases accordingly; the smaller L is, the higher the computational efficiency, suitable for real-time application scenarios. Fourth, end-to-end trainability: The entire feedback network is end-to-end differentiable, enabling joint optimization through backpropagation, allowing the network to automatically learn the optimal number of iterations and iteration strategy.

[0143] The cascading processing of multiple feedback blocks allows the second-layer feedback network to progressively learn the detail reconstruction process from coarse to fine. Taking L=3 as an example, the first feedback block primarily recovers the low-frequency structure and general outline of the image; the second feedback block enhances edge and texture details based on this; and the third feedback block further refines and corrects the image, outputting the final residual information. This hierarchical optimization approach enables the network to better handle features at different scales, improving reconstruction quality.

[0144] exist Figure 2 Based on the method shown, the method according to the embodiments of this application may further include a training step: calculating the difference between the super-resolution reconstruction result and the real frame using a loss function, and updating the network parameters according to the difference, so that the network learns the mapping relationship from the resolution of the initial video sequence to the resolution of the super-resolution reconstruction result; wherein, the resolution of the real frame is equal to the resolution of the super-resolution reconstruction result. The training process includes four stages: forward propagation, loss calculation, backpropagation, and parameter update.

[0145] First, a training dataset is constructed. A large number of high-resolution video frames, denoted as ground frames I, are obtained from a high-resolution video dataset with a higher resolution than the initial video frames. HR To simulate real-world application scenarios, a downsampling method is used to generate the corresponding initial video frame I. LR The downsampling factor is usually 4 times, that is, the high-resolution frame size is H×W, and the initial frame size is (H / 4)×(W / 4).

[0146] In one exemplary implementation, high-resolution video frames are obtained from a publicly available video dataset as high-resolution video sources. High-resolution video frames of size 256×256 are randomly selected from the dataset as learning targets, and downsampled by a factor of 4 to obtain initial video frames of size 64×64. All data are normalized, mapping pixel values ​​to the [0,1] or [-1,1] interval to ensure training stability.

[0147] To enhance the model's generalization ability, online data augmentation is performed on the input data during training. Data augmentation methods include, but are not limited to, random horizontal flipping, random vertical flipping, random rotation, and random cropping. Data augmentation can effectively increase the diversity of training samples and prevent model overfitting.

[0148] The preprocessed initial video sequence is input into the aforementioned video super-resolution network, and the forward propagation process of steps S210 to S260 is executed.

[0149] Specifically, N consecutive initial video frames (e.g., 7 frames) are taken as input, where the t-th frame is the target frame and the rest are reference frames. The network divides the input frames into multiple groups according to the aforementioned grouping strategy, performs feature alignment through temporal multiple correspondence aggregation, performs feature fusion through a first-layer feedback network, performs reconstruction through a second-layer feedback network, and finally outputs the super-resolution reconstruction result I. SR .

[0150] During the training phase, the network uses the exact same structure and parameters as the inference phase to ensure consistency between training and inference.

[0151] The super-resolution reconstruction result I is calculated using a loss function. SR With real frame I HR The difference between the two is used as the optimization objective. The choice of loss function directly affects the reconstruction quality. This application adopts the L1 loss function (mean absolute error) as the main loss term.

[0152] The formula for calculating the L1 loss function is as follows:

[0153]

[0154] Where N is the batch size, and ||·||1 represents the L1 norm, which is the sum of absolute errors per pixel.

[0155] In some implementations, perceptual loss or adversarial loss can be introduced as auxiliary loss terms to further improve the perceptual quality of the reconstructed image. Perceptual loss extracts features through a pre-trained VGG network and calculates the distance between the reconstructed result and the real image in the feature space, making the reconstructed result more consistent with human visual perception. Adversarial loss introduces a discriminator network to make the generated result closer to the distribution of the real image, enhancing the realism of the image. However, to simplify the training process and ensure stability, this application uses L1 loss as the primary loss function.

[0156] The gradients calculated from the loss function are used to propagate the error back through each layer of the network via backpropagation, calculating the gradients of the parameters at each layer. Specifically, the chain rule is used to calculate the partial derivatives of the loss function with respect to the learnable parameters of each layer, yielding the gradient information. A gradient descent optimization algorithm is then employed to update the network parameters based on the calculated gradients.

[0157] During training, model performance is periodically evaluated on the validation set. The validation loss is used to determine if the model has converged, preventing overfitting. Training is stopped and the network model parameters are saved when the preset number of training iterations is reached or the validation loss converges to a preset threshold. The trained network model can then be used for real-world video super-resolution reconstruction tasks.

[0158] Based on the above-described video super-resolution method, embodiments of this application also provide a video super-resolution apparatus. The following will be combined with... Figure 9 The device is described in detail.

[0159] Figure 9 A schematic block diagram of a video super-resolution apparatus according to an embodiment of this application is shown.

[0160] like Figure 9 As shown, the video super-resolution device 900 of this embodiment includes a data acquisition module 910, a data partitioning module 920, a feature extraction module 930, a feature fusion module 940, a feature reconstruction module 950, and a result acquisition module 960.

[0161] The data acquisition module 910 is used to acquire an initial video sequence. In one embodiment, the data acquisition module 910 can be used to perform step S210 described above, which will not be repeated here.

[0162] The data partitioning module 920 is used to divide the initial video sequence into multiple groups on the time axis, each group containing a target frame and a reference frame with a different time span from the target frame. In one embodiment, the data partitioning module 920 can be used to perform step S220 described above, which will not be repeated here.

[0163] The feature extraction module 930 is used to extract features from each frame in each group, and perform feature alignment through temporal multiple correspondence aggregation to obtain feature maps corresponding to multiple groups. In one embodiment, the dual-stream coding module 930 can be used to perform step S230 described above, which will not be repeated here.

[0164] The feature fusion module 940 is used to sequentially input the feature maps corresponding to multiple groups into the first-layer feedback network for feedback fusion to obtain a fused feature sequence. In one embodiment, the feature fusion module 940 can be used to execute step S240 described above, which will not be repeated here.

[0165] The feature reconstruction module 950 is used to input the fused feature sequence into the second-layer feedback network for reconstruction to obtain super-resolution residual information. In one embodiment, the feature reconstruction module 950 can be used to perform step S250 described above, which will not be repeated here.

[0166] The result acquisition module 960 is used to add the super-resolution residual information to the upsampling result of the target frame to obtain the super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence. In one embodiment, the result acquisition module 960 can be used to perform step S260 described above, which will not be repeated here.

[0167] According to an embodiment of this application, inputting the fused feature sequence into a second-layer feedback network for reconstruction includes: sequentially inputting the fused feature sequence into multiple cascaded feedback blocks for processing. The processing of each feedback block includes: upsampling the input features to increase the feature space resolution to the same level as the target frame; performing residual learning on the upsampled features; and feeding the output of the current feedback block back to the input of the current feedback block, thereby gradually optimizing the reconstruction quality through multiple iterations.

[0168] According to embodiments of this application, upsampling is achieved through subpixel convolution; residual learning is achieved through convolutional layers, batch normalization layers, and activation functions, and skip connections are used; the number of multiple cascaded feedback blocks is a preset number, and the reconstruction results are gradually refined through multiple iterative feedbacks.

[0169] According to an embodiment of this application, dividing the initial video sequence into multiple groups on the timeline includes: obtaining a preset number of consecutive initial video frames, and using one of these frames as the target frame; dividing the preset number of frames into multiple groups, each group containing the target frame and a reference frame with a different time span from the target frame.

[0170] According to the embodiments of this application, the preset quantity is an odd number, and the number of multiple groups is the preset quantity minus 1 and then divided by 2. Each group includes a target frame and reference frames symmetrically distributed on both sides of the target frame. The time span between the reference frames and the target frame is different in different groups.

[0171] According to an embodiment of this application, inputting feature maps corresponding to multiple groups into a first-layer feedback network for feedback fusion includes: inputting feature maps corresponding to multiple groups into multiple cascaded feedback fusion blocks for processing; each feedback fusion block fuses the current input feature with the output feature of the previous time step and adds residual connections; wherein, each feedback fusion block calculates fusion weights through a gating mechanism; the gating mechanism concatenates the current input feature with the output feature of the previous time step in the channel dimension, generates weights through convolutional layers and activation functions, and performs weighted fusion of the current input feature and the output feature of the previous time step according to the weights.

[0172] According to an embodiment of this application, feature alignment via temporal multiple correspondence aggregation includes: processing feature correspondences at different scales through multiple parallel deformable convolutional branches, each deformable convolutional branch learning the spatial offset of each position in the feature map, and adaptively aligning the features of the reference frame according to the offset; and concatenating the outputs of multiple branches in the channel dimension and fusing them through a convolutional layer to obtain the aligned temporal features.

[0173] According to an embodiment of this application, the device further includes a training module: used to calculate the difference between the super-resolution reconstruction result and the real frame using a loss function, and update the network parameters according to the difference, so that the network learns the mapping relationship from the first resolution to the second resolution; wherein the second resolution is greater than the first resolution, and the resolution of the real frame is equal to the second resolution.

[0174] According to embodiments of this application, any multiple modules among the data acquisition module 910, data partitioning module 920, feature extraction module 930, feature fusion module 940, feature reconstruction module 950, and result acquisition module 960 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 910, data partitioning module 920, feature extraction module 930, feature fusion module 940, feature reconstruction module 950, and result acquisition module 960 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 910, data partitioning module 920, feature extraction module 930, feature fusion module 940, feature reconstruction module 950, and result acquisition module 960 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0175] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a video super-resolution method according to an embodiment of this application.

[0176] like Figure 10 As shown, an electronic device 1000 according to an embodiment of this application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage portion 1008 into a random access memory 1003. The processor 1001 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0177] Random access memory 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, read-only memory 1002, and random access memory 1003 are interconnected via bus 1004. Processor 1001 executes various steps of the method flow according to embodiments of this application by executing programs in read-only memory 1002 and / or random access memory 1003. It should be noted that programs may also be stored in one or more memories other than read-only memory 1002 and random access memory 1003. Processor 1001 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0178] According to embodiments of this application, the electronic device 1000 may further include an input / output interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card, such as a local area network card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0179] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0180] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. According to embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 1002, and / or random access memory 1003, and / or one or more memories other than read-only memory 1002 and random access memory 1003 described above.

[0181] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0182] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0183] According to embodiments of this application, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0184] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A video super-resolution method, characterized in that, include: Obtain the initial video sequence; The initial video sequence is divided into multiple groups on the timeline, each group containing a target frame and a reference frame with a different time span from the target frame; Features of each frame in each group are extracted separately, and feature alignment is performed by temporal multiple correspondence aggregation to obtain feature maps corresponding to multiple groups. The feature maps corresponding to the multiple groups are sequentially input into the first-layer feedback network for feedback fusion to obtain a fused feature sequence. The fused feature sequence is input into the second-layer feedback network for reconstruction to obtain super-resolution residual information. The super-resolution residual information is added to the upsampling result of the target frame to obtain the super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence.

2. The method according to claim 1, characterized in that, The step of inputting the fused feature sequence into the second-layer feedback network for reconstruction includes: sequentially inputting the fused feature sequence into multiple cascaded feedback blocks for processing, wherein the processing of each feedback block includes: Upsample the input features to increase the feature space resolution to the same level as the target frame; Residual learning is performed on the upsampled features; The output of the current feedback block is fed back to the input of the current feedback block, and the reconstruction quality is gradually optimized through multiple iterations.

3. The method according to claim 2, characterized in that, The upsampling is achieved through subpixel convolution; The residual learning is implemented through convolutional layers, batch normalization layers, and activation functions, and uses skip connections; The number of the multiple cascaded feedback blocks is a preset number, and the reconstruction result is gradually refined through multiple iterative feedbacks.

4. The method according to claim 1, characterized in that, The step of dividing the initial video sequence into multiple groups on the time axis includes: Obtain a preset number of consecutive initial video frames, and use one of them as the target frame; The preset number of frames are divided into multiple groups, each group containing the target frame and a reference frame with a different time span from the target frame.

5. The method according to claim 4, characterized in that, The preset quantity is an odd number, and the number of the multiple groups is the preset quantity minus 1 and then divided by 2. Each group includes a target frame and reference frames symmetrically distributed on both sides of the target frame. The time span between the reference frames and the target frame is different in different groups.

6. The method according to claim 1, characterized in that, The feature maps corresponding to the multiple groups are sequentially input into the first-layer feedback network for feedback fusion, including: The feature maps corresponding to the multiple groups are sequentially input into multiple cascaded feedback fusion blocks for processing. Each feedback fusion block fuses the current input features with the output features of the previous time step and adds residual connections. Each feedback fusion block calculates fusion weights through a gating mechanism. The gating mechanism concatenates the current input features and the output features from the previous time step along the channel dimension, generates weights through a convolutional layer and an activation function, and performs weighted fusion of the current input features and the output features from the previous time step based on the weights.

7. The method according to claim 1, characterized in that, The feature alignment via temporal multiple correspondence aggregation includes: The feature correspondence at different scales is processed by multiple parallel deformable convolutional branches. Each deformable convolutional branch learns the spatial offset of each position in the feature map and performs adaptive alignment of the reference frame features based on the offset. The outputs of multiple branches are concatenated along the channel dimension and then fused through a convolutional layer to obtain aligned temporal features.

8. The method according to claim 1, characterized in that, It also includes a training step: using a loss function to calculate the difference between the super-resolution reconstruction result and the real frame, and updating the network parameters according to the difference, so that the network learns the mapping relationship from the resolution of the initial video sequence to the resolution of the super-resolution reconstruction result; wherein, the resolution of the real frame is equal to the resolution of the super-resolution reconstruction result.

9. A video super-resolution device, characterized in that, The device includes: The data acquisition module is used to acquire the initial video sequence; The data partitioning module is used to divide the initial video sequence into multiple groups on the time axis. Each group includes a target frame and a reference frame with a different time span from the target frame. The feature extraction module is used to extract the features of each frame in each group, and perform feature alignment through temporal multiple correspondence aggregation to obtain feature maps corresponding to multiple groups. The feature fusion module is used to sequentially input the feature maps corresponding to the multiple groups into the first layer feedback network for feedback fusion to obtain a fused feature sequence. The feature reconstruction module is used to input the fused feature sequence into the second-layer feedback network for reconstruction to obtain super-resolution residual information. The result acquisition module is used to add the super-resolution residual information to the upsampling result of the target frame to obtain the super-resolution reconstruction result, wherein the resolution of the super-resolution reconstruction result is greater than the resolution of the initial video sequence.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images