An inter-frame rate and inter-resolution image alignment super-resolution method and system

Abstract

The present application provides a kind of different frame rate different resolution image alignment super-resolution method and system, the steps of the method include: each image sequence is extracted by feature extraction network, and a feature sequence is obtained corresponding to each image sequence;Feature sequence is processed by self-attention network, and first attention sequence is obtained;Image sequence is classified as reference sequence or reference sequence based on frame rate, the first attention sequence corresponding to reference sequence is constructed with the first attention sequence corresponding to each reference sequence as sequence group input into cross-attention network, and second attention sequence is obtained;The weight of each reference sequence is determined based on the second attention sequence and the preset learnable fusion Token sequence, and the reference sequence is fused based on the weight of reference sequence, and the fusion feature sequence is obtained;Fusion feature sequence is input into preset super-resolution reconstruction network, and fusion image sequence is output.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a super-resolution method and system for aligning images with different frame rates and resolutions. Background Technology

[0002] With the rapid development of high-definition video, ultra-high-definition video and ubiquitous video surveillance, people's requirements for video quality and resolution are increasing day by day. Although existing single-frame super-resolution methods can improve the spatial details of each frame, they cannot make full use of temporal information and are difficult to eliminate motion blur and loss of detail.

[0003] Multi-frame super-resolution technology can significantly improve reconstruction results by fusing image information from adjacent frames, but it usually assumes that the input sequence has a uniform frame rate and the same resolution, and cannot be directly applied to videos from different cameras or with different sampling parameters.

[0004] In practical applications, video sequences from different sources often have differences in frame rate and resolution, leading to timestamp misalignment and spatial scale mismatch. Directly fusing multiple frames can easily result in artifacts, misalignment, and information redundancy or loss. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a super-resolution method and system for aligning images with different frame rates and resolutions, in order to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a super-resolution method for aligning images with different frame rates and resolutions, the method comprising the following steps: Multiple image sequences are acquired, and features are extracted from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence. Each feature sequence is processed through a self-attention network to obtain the first attention sequence; Based on the frame rate of the video sequence, the image sequence is classified into a baseline sequence or a reference sequence. The first attention sequence corresponding to the baseline sequence and the first attention sequence corresponding to each reference sequence are constructed into a sequence group. Each sequence group is input into the cross-attention network to obtain the second attention sequence. The weights of each reference sequence are determined based on the second attention sequence and the preset learnable fusion token sequence. The reference sequences are then fused based on their weights to obtain a fused feature sequence. The fused feature sequence is input into a preset super-resolution reconstruction network, which includes a multi-level upsampling module, a residual block, and a fusion layer. A fused image sequence is constructed based on the super-resolution reconstruction network.

[0007] The above scheme first achieves simultaneous multi-scale feature extraction through a feature extraction network, taking into account both global semantics and local details; then, by combining self-attention and cross-sequence attention, high-precision spatiotemporal alignment and information complementarity can be achieved in the deep feature space; the learnable fusion token mechanism realizes adaptive weighted fusion of multi-path features, avoiding the limitations of manually designed weights; the frame rate adjustment and super-resolution reconstruction modules work together to significantly improve the visual clarity and detail restoration capability of the video.

[0008] In some embodiments of the present invention, in the step of extracting features from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence, the feature extraction network uses a ResNet101 network as the backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

[0009] In some embodiments of the present invention, the step prior to the step of extracting features from each image sequence through a feature extraction network further includes unifying the images in the image sequence to a preset scale.

[0010] In some embodiments of the present invention, the feature sequence includes a sub-feature sequence corresponding to each image in the image sequence. In the step of processing each feature sequence through a self-attention network to obtain a first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image, mapping it to a query vector, a key vector, and a value vector. Attention weights are calculated by dot product and normalized by Softmax, and then the value vectors are weighted and summed. Finally, the first attention sequence is obtained by connecting it to the residual through a feedforward network, thereby enhancing the internal spatiotemporal dependency information.

[0011] In some embodiments of the present invention, in the step of classifying image sequences into baseline sequences or reference sequences based on the frame rate of video sequences, the image sequence with the highest frame rate among the various video sequences is used as the baseline sequence, and the remaining image sequences are used as reference sequences.

[0012] In some embodiments of the present invention, in the step of inputting each sequence group into the cross-attention network to obtain the second attention sequence, the first attention sequence corresponding to the benchmark sequence is used as the query vector, and the first attention sequence corresponding to the reference sequence is used as the key vector and value vector, and input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution, performs gating control on the cross-sequence attention output, and outputs the second attention sequence.

[0013] In some embodiments of the present invention, in the step of determining the weight of each reference sequence based on the second attention sequence and the preset learnable fusion token sequence, the similarity between each second attention sequence and the preset learnable fusion token sequence is calculated, and the weight of the reference sequence corresponding to each second attention sequence is determined based on the similarity.

[0014] In some embodiments of the present invention, when the fused feature sequence is input into a preset super-resolution reconstruction network, the super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolutional layer and a sub-pixel rearrangement layer; the residual block includes a convolutional layer, an activation function layer, and a convolutional skip layer; the fusion layer includes a stitching layer and a fusion convolutional layer.

[0015] In some embodiments of the present invention, in the step of constructing a fused image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further includes a reconstruction convolutional layer, the residual block outputs a fused feature sequence, and the fused feature sequence is input into the reconstruction convolutional layer to obtain a fused image sequence including multiple fused images.

[0016] A second aspect of the present invention also provides a super-resolution system for image alignment at different frame rates and resolutions. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0017] A third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned heterogeneous frame rate and heterogeneous resolution image alignment super-resolution method.

[0018] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0019] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0021] Figure 1 This is a schematic diagram of one implementation of the super-resolution method for aligning images with different frame rates and resolutions according to this scheme. Figure 2 This is a schematic diagram of another implementation of the super-resolution method for aligning images with different frame rates and resolutions according to this scheme. Figure 3 This is a schematic diagram of the feature extraction network processing architecture of this scheme; Figure 4 This is a schematic diagram of the processing architecture of the self-attention network in this solution; Figure 5 This is a schematic diagram of the processing architecture of the cross-attention network in this scheme; Figure 6 This is a schematic diagram of the gating mechanism processing architecture in the cross-attention network of this scheme; Figure 7 This diagram illustrates the fusion of multiple sequences into a learnable Token sequence in this scheme. Figure 8 This is a schematic diagram illustrating the adjustment of the fused feature sequence length in this scheme; Figure 9 This is a schematic diagram of the processing architecture of the super-resolution reconstruction network in this scheme; Figure 10 This is a schematic diagram of the hardware architecture of this solution. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0023] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0024] Existing temporal alignment methods mostly rely on optical flow estimation or block matching, and their accuracy is easily affected by fast motion and low-texture regions. While interpolation-based frame rate conversion can smooth frame rate differences, it is difficult to maintain spatiotemporal consistency in high-dimensional feature spaces. Simple feature stitching or weighted fusion lacks an adaptive weight allocation mechanism, resulting in unsatisfactory fusion effects.

[0025] like Figure 1 and 2As shown, this invention proposes a super-resolution method for aligning images with different frame rates and resolutions. The steps of this method include: Step S100: Obtain multiple image sequences, extract features from each image sequence through a feature extraction network, and obtain a feature sequence for each image sequence; In its implementation, this scheme addresses multiple image sequences from different frame rates and resolutions. First, it unifies the input dimensions of each frame in each sequence and then feeds it into a shared-parameter feature extraction network. This feature extraction network is a convolutional neural network (CNN) with a ResNet101 backbone, followed by Feature Pyramid Network (FPN) modules at each stage. The FPN modules generate multi-scale feature maps through bottom-up and top-down paths and lateral connections, preserving both the deep semantic information of the backbone network and the shallow detail information. The final output is a multi-level feature pyramid sequence with different spatial resolutions, i.e., a feature sequence, which serves as the basis for subsequent spatiotemporal alignment and fusion.

[0026] Step S200: Process each feature sequence through a self-attention network to obtain the first attention sequence; In the specific implementation process, a multi-head self-attention mechanism is applied within each feature sequence. Specifically, positional encoding is added to the sub-feature sequences at each time step of the feature sequence, mapping them to query, key, and value vectors; attention weights are calculated through dot product and normalized by Softmax, then the value vectors are weighted and summed, and finally connected to the residual through a feedforward network to obtain a self-attention feature sequence that enhances the internal spatiotemporal dependency information, i.e., the first attention sequence.

[0027] Step S300: Based on the frame rate of the video sequence, classify the image sequence into a base sequence or a reference sequence, construct a sequence group by combining the first attention sequence corresponding to the base sequence and the first attention sequence corresponding to each reference sequence, and input each sequence group into the cross-attention network to obtain the second attention sequence; In the specific implementation, among the multiple self-attention output first attention sequences, the sequence with the highest frame rate is selected as the baseline sequence, and the remaining sequences are used as reference sequences in sequence. For each reference sequence, the self-attention features of the baseline sequence are used as the query, and the self-attention features of the reference sequences are used as the key and value, respectively, and input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution and performs gating control on the cross-sequence attention output, realizing information supplementation and depth complementarity between the baseline sequence and each reference sequence. For each reference sequence, a cross-sequence attention feature sequence with the same length as the baseline sequence is output, which is the second attention sequence.

[0028] Step S400: Determine the weight of each reference sequence based on the second attention sequence and the preset learnable fusion token sequence, and fuse the reference sequences based on the weights of the reference sequences to obtain the fused feature sequence. In the specific implementation process, this scheme introduces a learnable fusion token sequence with the same length as the second attention sequence to calculate the similarity with the second attention sequence, perform dot product, and then obtain the weight of each reference sequence through softmax; this fusion token is learned in parallel with each feature during the training process, and the attention weight with each feature is calculated through the attention mechanism to autonomously allocate the degree of attention; then the features of each feature are weighted and superimposed according to the learned weights to form a unique and unified spatiotemporal feature sequence, which effectively integrates the complementary information of multiple sequences.

[0029] like Figure 8 As shown, the method further includes performing frame rate matching processing on the fused feature sequence output in step S400. When the target number of frames is less than the length of the fused feature sequence, a uniform interval sampling strategy is adopted; when the target number of frames is equal to the length of the fused feature sequence, the sequence is directly used; when the target number of frames is greater than the length of the fused feature sequence, temporal interpolation (such as bilinear interpolation or cubic interpolation) is performed on the fused feature sequence to supplement new frames, thereby generating a fused feature sequence that meets the target frame rate requirements.

[0030] Step S500: Input the fused feature sequence into a preset super-resolution reconstruction network. The super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. Construct a fused image sequence based on the super-resolution reconstruction network.

[0031] In the specific implementation process, the fused feature sequence is input into the super-resolution reconstruction network. This network includes multi-level upsampling modules, residual blocks and fusion layers. By progressively amplifying the spatial resolution and refining the image details, the spatiotemporal features are transformed into a high-resolution, high-quality image frame sequence. Finally, the output is a super-resolution video sequence that meets both resolution and frame rate requirements, namely the fused image sequence.

[0032] The above scheme first achieves simultaneous multi-scale feature extraction through a feature extraction network, taking into account both global semantics and local details; then, by combining self-attention and cross-sequence attention, high-precision spatiotemporal alignment and information complementarity can be achieved in the deep feature space; the learnable fusion token mechanism realizes adaptive weighted fusion of multi-path features, avoiding the limitations of manually designed weights; the frame rate adjustment and super-resolution reconstruction modules work together to significantly improve the visual clarity and detail restoration capability of the video.

[0033] like Figure 3As shown, in some embodiments of the present invention, in the step of extracting features from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence, the feature extraction network uses a ResNet101 network as the backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

[0034] In some embodiments of the present invention, the step prior to the step of extracting features from each image sequence through a feature extraction network further includes unifying the images in the image sequence to a preset scale.

[0035] In the specific implementation process, the feature extraction network uses ResNet101 as the backbone and connects to the Feature Pyramid Network (FPN) module in parallel after each stage. Through bottom-up and top-down paths and lateral connections, a multi-scale feature pyramid is generated, which not only preserves deep semantic information but also takes into account shallow details, laying a solid foundation for subsequent alignment and fusion.

[0036] Specifically, ResNet101 consists of 101 convolutional layers. Specifically, Conv1 has one convolutional layer; Conv2_x consists of three sets of convolutions (each set follows the sequence: 1×1,64→3×3,64→1×1,256); Conv3_x consists of four sets of convolutions (each set follows the sequence: 1×1,128→3×3,128→1×1,512); Conv4_x consists of 23 sets of convolutions (each set follows the sequence: 1×1,256→3×3,256→1×1,1024); Conv5_x consists of three sets of convolutions (each set follows the sequence: 1×1,512→3×3,512→1×1,2048); and finally, there is a 1000-d fully connected layer (FC). In this embodiment, the final FC is not used; instead, the FPN is constructed from the outputs of Conv2_x, Conv3_x, Conv4_x, and Conv5_x.

[0037] The FPN module adopts a bottom-up and top-down decoder structure.

[0038] like Figure 4 As shown, in some embodiments of the present invention, the feature sequence includes a sub-feature sequence corresponding to each image in the image sequence. In the step of processing each feature sequence through a self-attention network to obtain a first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image, mapping it to a query vector, a key vector, and a value vector. Attention weights are calculated by dot product and normalized by Softmax, and then the value vectors are weighted and summed. Finally, the first attention sequence with enhanced internal spatiotemporal dependency information is obtained through a feedforward network and a residual connection.

[0039] In the specific implementation process, each obtained first attention sequence is flattened in the time dimension and input into the multi-head self-attention module. The features of each frame are mapped into query, key, and value vectors. The correlation weights between each time step within the sequence are calculated and normalized by Softmax. Finally, the feedforward network and residual connection are combined to generate a self-attention feature sequence that strengthens its own spatiotemporal dependencies.

[0040] In some embodiments of the present invention, in the step of classifying image sequences into baseline sequences or reference sequences based on the frame rate of video sequences, the image sequence with the highest frame rate among the various video sequences is used as the baseline sequence, and the remaining image sequences are used as reference sequences.

[0041] like Figure 5 As shown, in some embodiments of the present invention, in the step of inputting each sequence group into the cross-attention network to obtain the second attention sequence, the first attention sequence corresponding to the benchmark sequence is used as the query vector, and the first attention sequence corresponding to the reference sequence is used as the key vector and value vector, and input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution, performs gating control on the cross-sequence attention output, and outputs the second attention sequence.

[0042] In the specific implementation, the first attention sequence with the highest frame rate is used as the baseline query, and the self-attention outputs of other sequences are used as keys and values, which are then input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution in a high-dimensional feature space, realizing the deep information complementarity and feature enhancement between the baseline sequence and each reference sequence, and outputs a multi-path cross-sequence attention feature sequence with the same length as the baseline sequence.

[0043] like Figure 6 As shown, in the specific implementation process, a gating mechanism is introduced into the cross-attention network of this scheme. The gating mechanism includes: a. concatenating the output of the first attention sequence of the baseline sequence and the reference sequence calculated by the self-attention network by channel; b. obtaining the gating vector Gr by Sigmoid activation; c. finally multiplying the cross-sequence attention features of the reference sequence by Gr and performing weighted fusion; d. adding the gating feature sequence to the first attention sequence after the self-attention network and performing LayerNorm.

[0044] like Figure 7As shown, in some embodiments of the present invention, in the step of determining the weight of each reference sequence based on the second attention sequence and the preset learnable fusion token sequence, the similarity between each second attention sequence and the preset learnable fusion token sequence is calculated, and the weight of the reference sequence corresponding to each second attention sequence is determined based on the similarity.

[0045] In practice, the learnable fusion token sequence is initialized using a uniform Xavier distribution during the initial stage of network training, and the update rate is constrained during training using the following regularization terms: .

[0046] The above scheme introduces a learnable fusion token sequence with the same time length as the second attention sequence output by the cross-attention network. This token is updated in parallel with the features of each path during training, and is automatically learned and assigned fusion weights through attention or dot product mechanisms. The features of each path are weighted and superimposed to generate a single and unified spatiotemporal feature representation, realizing adaptive fusion of information from multiple sources.

[0047] like Figure 9 As shown, in some embodiments of the present invention, when the fused feature sequence is input into a preset super-resolution reconstruction network, the super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolutional layer and a sub-pixel rearrangement layer; the residual block includes a convolutional layer, an activation function layer, and a convolutional skip layer; the fusion layer includes a stitching layer and a fusion convolutional layer.

[0048] In some embodiments of the present invention, in the step of constructing a fused image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further includes a reconstruction convolutional layer, the residual block outputs a fused feature sequence, and the fused feature sequence is input into the reconstruction convolutional layer to obtain a fused image sequence including multiple fused images.

[0049] Using the above scheme, the network combines multi-level upsampling modules (such as subpixel convolution or transposed convolution), residual blocks and fusion layers. By progressively amplifying the spatial resolution and refining image details, it maps spatiotemporal features into a high-resolution image frame sequence, outputting a super-resolution video that meets both resolution and frame rate requirements.

[0050] Example: This solution is suitable for image sequences with multiple frame rates and resolutions. 1. First, resize the images to unify the resolution of all images to be processed (e.g., ...). Then, a shared parameter network with ResNet101 as the backbone, combined with a Feature Pyramid Network (FPN), is used for multi-scale feature extraction. This multi-scale feature extraction captures feature information from different levels in the original image, resulting in a unified-dimensional feature representation. Given K input sequences, sequence k is denoted as... in Represents the color image of the k-th sequence at time t; In order to extract multi-scale features of image sequences with different frame rates and resolutions, the feature extraction network in this embodiment uses ResNet101 as the backbone network and embeds FPN modules at the outputs of Stage 2 to Stage 5. ResNet101 consists of 101 convolutional layers. Specifically, Conv1 has 1 convolution; Conv2_x consists of 3 sets of convolutions (each set follows the sequence: 1×1,64 → 3×3,64 → 1×1,256); Conv3_x consists of 4 sets of convolutions (each set follows the sequence: 1×1,128 → 3×3,128 → 1×1,512); Conv4_x consists of 23 sets of convolutions (each set follows the sequence: 1×1,256 → 3×3,256 → 1×1,1024); Conv5_x consists of 3 sets of convolutions (each set follows the sequence: 1×1,512 → 3×3,512 → 1×1,2048); and finally, there is a 1000-d fully connected layer (FC). In this embodiment, the final FC is not used; instead, the FPN is constructed from the outputs of Conv2_x, Conv3_x, Conv4_x, and Conv5_x. The FPN module employs a bottom-up and top-down decoder structure. Specifically, each image frame is first preprocessed using standardization (mean and standard deviation normalization), and then input into ResNet101 to obtain feature maps at different stages. Stage 2: Feature map size is C3×H / 4×W / 4, C3=256; Stage 3: Feature map size is C3×H / 8×W / 8, C3=512; Stage 4: Feature map size is C4×H / 16×W / 16, C4=1024; Stage 5: Feature map size is C5×H / 32×W / 32, C5=2048.

[0051] In the FPN part, features at each scale are first projected to a fixed dimension D using 1×1 lateral convolutions. Then, upsampling (bilinear interpolation) combined with lateral connections is used to fuse multi-scale features into a series of feature pyramids, where each layer is convolved with a 3×3 convolution before outputting the final features. This process not only captures global semantics but also takes into account local details, ensuring that downstream modules obtain rich feature descriptions. Each scale's features are then subjected to global average pooling, resulting in four 256-dimensional feature vectors. These four feature vectors from different scales are concatenated to obtain a 1024-dimensional feature vector for this image. The four feature maps at different scales have different resolutions, which can better adapt to objects of different sizes in the image. Will Input the ResNet101 backbone network and extract convolutional feature maps from the outputs of Stages 2–5: ; in ; First, perform a 1×1 lateral convolution on the feature maps of each layer and project them to a uniform number of channels. Then, a feature pyramid is generated by connecting the top-down path with the horizontal path; in, They are 1×1 and 3×3 convolution kernels, respectively. This indicates that bilinear interpolation amplifies the data by a factor of 2, resulting in a multi-scale feature map. ; Feature mapping for each scale Apply global average pooling: get Concatenate the pooling vectors of each scale along the channel direction: As an image The final eigenvector.

[0052] As mentioned above, each frame of all sequences is mapped to a sequence of uniform dimensions. And finally, the feature sequence is obtained; Subsequent modules can directly input this feature sequence for spatiotemporal alignment and fusion of self-attention and cross-sequence attention.

[0053] 2. Perform self-attention calculation for each image feature sequence. In each image sequence, some frames may be blurry, but this blurriness can be repaired by using the contextual information of the frame in the sequence, that is, by taking information from those clear frames as supplementary information.

[0054] For each sequence, a learnable positional encoding is introduced into the self-attention computation module. , so that the input feature sequence Updated to Subsequently, a multi-head attention mechanism is employed to perform a linear transformation of query, key, and value for each input. Specifically, the number of attention heads is set to [number missing]. The dimensions of each head are For the h-th head, we have: ; Attention Score The final attention output is obtained by calculating the dot product and scaling operations, and then adding Dropout (with a scaling factor of 0.1). Finally, residual connections and LayerNorm are used to effectively improve the feature transfer and expressive power.

[0055] Let the first The frame feature matrix of the image sequence is The row index corresponds to the time frame, and the column vector is the feature extracted by the aforementioned FPN+GAP.

[0056] To preserve timing information, positional encoding is added, first fixing or learningable positional codes. Add to it: ; Linear mapping to multi-head Q, K, V Let the number of long positions be Dimensions per head ,satisfy .

[0057] Define the projection matrix respectively; ; Then the first The header's query, key, and value; ; in .

[0058] Calculate attention weights For the First, calculate the unnormalized attention score matrix: ; If causal or local constraints are required, they can be found in Add mask Finally through With optional Dropout, the attention distribution is obtained: ; Weighted summation produces the header output; .

[0059] Multi-head splicing and output mapping will The output of each item is concatenated by channel: Then through linear mapping Add Dropout: Residual connections and layer normalization are used to add the multi-head attention results to the input and apply LayerNorm: Feedforward network, followed by two fully connected feedforward layers (with GeLU activation and Dropout) with residuals and layer normalization: in .

[0060] Thus, the first The output of the self-attention module for the image sequence is It can be used for subsequent cross-sequence alignment and fusion.

[0061] 3. Cross Sequence Attention Network After completing the self-attention for each image sequence, we obtained... The self-attention module for each sequence is based on the global content-based mechanism in Transformer, enabling it to capture long-range temporal dependencies within the sequence and ensuring the stability of information flow and gradient propagation through residuals and layer normalization. Self-attention is introduced to allow each element in the sequence to query information from its own context, thereby strengthening feature representation. However, single-path self-attention can only utilize the context of frames within the same path. For frames lacking information due to motion blur or exposure issues—information unavailable to the current camera stream due to hardware limitations—compensation is impossible. Considering that different cameras or different frame rates record the same target from multiple perspectives / rates, frames from different sequences at the same or adjacent moments often contain complementary spatial details. Therefore, cross-sequence attention is introduced, allowing the "baseline sequence" (highest frame rate, richest temporal sequence) to query information from other "supplementary sequences" to enhance the feature representation of each frame.

[0062] To overcome the shortcomings of a single sequence in processing ambiguous frame information, a cross-sequence attention mechanism is proposed. Specifically, the sequence with the highest frame rate (denoted as the baseline sequence) is first selected. The first sequence is used as the main sequence, and the remaining sequences are used as reference sequences. For each reference sequence... Let the output of the reference sequence be The reference sequence is By using linear projection, we obtain... (by (for input) and and (by (As input), and calculate the attention score: Subsequently, a Softmax operation is performed on the attention score of each head to obtain the attention distribution, and then compared with... Weighted fusion is used to generate cross-sequence enhanced features. A gating mechanism is introduced to suppress noise interference. Finally, the cross-sequence attention fusion result is expressed as: in, This represents element-wise multiplication, enabling efficient and selective supplementation of reference information.

[0063] Selecting the reference and baseline sequences, let the sequence number with the most frames be... (Base sequence), the remaining sequences Reference sequences. Baseline sequence output: Linear projection and multi-head splitting for each reference sequence Learn about projection separately: Definition of the first Header query, key, value Attention Calculation and Attention Distribution Calculate unnormalized attention for each head: Then perform Softmax on each row (optionally add a mask). (Restricting the query time range): Weighted summation and long position splicing Output per head: And then map after splicing: Gating fusion and residual Considering that not all reference information is helpful for every frame, a gating vector is introduced. The gating mechanism aims to automatically assign a [0,1] "trust coefficient" for each time step and each channel, suppressing unreliable information. The process here is as follows: the self-attention sequence is concatenated with the cross-sequence attention sequence, a linear mapping is performed to transform the size, and an element-wise sigmoid function is used to scale each transformed element to the [0,1] interval. Then, it is multiplied element-wise with the cross-sequence attention sequence, and finally added to the output of the original self-attention sequence. After passing through LayerNorm, the gating is completed. in Indicates channel splicing. This represents the element-wise Sigmoid.

[0064] Final Fusion: Finally, for each reference sequence, a gated cross-sequence attention output is obtained. That is, in the original input, there is a base sequence with the most frames and N reference sequences with a less than the most frames. After this cross-sequence attention step, N feature sequences are obtained, each with the same length as the base sequence.

[0065] Theoretical basis Self-attention, based on Transformer (Vaswani et al. 2017), focuses on content relevance and can dynamically focus on advantageous positions within a time series, capturing global context and long-range dependencies.

[0066] Cross-sequence attention is similar to multimodal or multi-view fusion (e.g., ViLT, Florence). Through the "Query-Key-Value" mechanism, the features of each moment in the baseline sequence "question" other sequences and "borrow" their semantic and spatial details. It is an effective compensation strategy for information loss problems such as shooting shake and motion blur.

[0067] Introducing a gating mechanism can dynamically suppress information from noise or flickering frames, while ensuring that supplementary information can be fully integrated when it is reliable, thereby improving the robustness and representation integrity of downstream tasks to blurred frames.

[0068] In summary, this section constructs a "multi-view compensation" channel for each frame in the baseline sequence through cross-sequence attention, making it more robust and richer in spatiotemporal feature representation.

[0069] 4. Learnable Token Fusion feature sequences for each path graph Thus, the enhancement features of each reference sequence to the baseline sequence were obtained through multi-path cross-sequence attention. common The road. In order to... The supplementary information is "converged" into a unified spatiotemporal feature sequence, and a "learnable fusion token sequence" is introduced. Using these as queries, and the fused graph features obtained by cross-sequence attention fusion for each path as keys and values, attention fusion is performed on all reference path features.

[0070] To integrate information obtained from multi-path cross-sequence attention, this module introduces a learnable token sequence with the same length as the baseline sequence. Token The network is initialized using a uniform Xavier distribution in the initial stage and continuously optimized adaptively during training. The concatenated multi-path features are then... and Obtain the multi-head attention mechanism through linear mapping respectively. , , The matrix is ​​calculated as follows: For each attention head ,have: Attention distribution is calculated as follows: The fused output is: After concatenating the outputs of all the heads, a linear transformation is performed. Then, perform a residual join and LayerNorm with the original Token to obtain the final fused feature: Furthermore, additional regularization terms can be introduced to constrain the token update rate, thereby improving the stability and efficiency of cross-sequence information fusion.

[0071] The definition sequence length of the fusion token is the same as that of the base sequence: Each of them All are trainable vectors, and are related to time... Feature alignment.

[0072] splicing multiple features, Path enhancement features are concatenated along the feature dimension: Multi-head cross-sequence attention mapping, let the number of multi-heads be Each head has a dimension of ,satisfy Projection matrices are defined separately for fusing tokens and splicing features. No. After head projection: Attention weights and weighted fusion Calculate and normalize the attention: Then weight the sum and combine the long positions: Residual connectivity and layer normalization Add the residuals to the original token and normalize: at this time That is to be The fusion feature sequence after the path is fused into a single path.

[0073] Theory and Function Adaptive weight allocation: By fusing tokens, attention weights are dynamically allocated to multi-path features at all times, and the network can automatically learn which time and which path information is most valuable.

[0074] Temporal alignment is preserved: the token sequence corresponds one-to-one with the temporal frame, eliminating the need for additional temporal scaling or interpolation, which facilitates subsequent convolution or classification head processing.

[0075] End-to-end learning: The integration of tokens and attention projection matrices optimizes the entire network without requiring a pre-defined pooling strategy.

[0076] Parallel and efficient: Based on the Transformer multi-head mechanism, long sequence attention can be computed in parallel on GPUs.

[0077] Method source: Adapted from Vit While incorporating ideas such as tokens and Induced Set Attention from Set Transformer, this approach offers greater flexibility by using equal-length token sequences to output spatiotemporal features.

[0078] In summary, the "Learnable Token Fusion" module introduces trainable tokens that match the temporal length and uses them as queries to adaptively fuse multi-path cross-sequence attention features, generating a unified and discriminative spatiotemporal feature sequence, which provides strong representation support for downstream tasks such as action recognition and temporal segmentation.

[0079] 5. Frame-Rate Alignment After completing the "learnable token fusion", a spatiotemporal feature sequence of the same length as the original baseline sequence was obtained. To lay the foundation for the next step of reconstructing new graph feature sequences based on graph feature sequences using super-resolution, it is necessary to consider the target... Compared with the original length The size relationship is used to adjust the length of the original sequence. There are three main situations: After obtaining the spatiotemporal feature sequence of multi-path fusion Afterwards, due to differences in the frame rate of the source video, it is necessary to... Adjust to target output frame rate The specific steps are as follows: downsampling ( < ): Using uniform interval sampling index To prevent high-frequency information aliasing, firstly... A one-dimensional Gaussian filter is performed, and the standard deviation of the filter kernel is adaptively adjusted according to the sequence length and sampling ratio.

[0080] Remain unchanged ( = ): Direct reuse No additional processing is required.

[0081] Upsampling ( > ): Define target mesh ,in For each new moment The following methods can be used: Linear interpolation: Let ,but Cubic interpolation: Select four adjacent points and use the cubic spline kernel wmα for interpolation to improve the smoothness and accuracy of interpolation.

[0082] Learnable upsampling: The interpolation kernel is adaptively learned through deconvolution or local attention networks to achieve the end-to-end interpolation process.

[0083] The above method can effectively unify the time length of multi-source features, providing a consistent data foundation for subsequent tasks.

[0084] The target number of frames is less than the original length (downsampling). when < At that time, sampling at uniform intervals is used: This ensures that the beginning and end are aligned (when k=0, the first position is taken). Frame, k= -1 hour take the first -1 frame), and then evenly distributed in the middle.

[0085] Sampling by index: Optionally, a one-dimensional low-pass filter (such as Gaussian convolution) can be applied to Ufused before downsampling to suppress high-frequency noise and avoid aliasing artifacts.

[0086] The target frame count is equal to the original length (remaining unchanged), when = At that time, the fusion result can be directly reused: The target number of frames is greater than the original length (upsampling). when > At this time, a new frame needs to be inserted between adjacent feature frames. Let the target time grid be... in For real number indices. Common interpolation methods include: Linear interpolation: Let , but Cubic spline interpolation: taking four adjacent points , Using cubic spline kernel calculate Learnable upsampling: Treating it as a one-dimensional signal, an interpolation kernel is learned through a deconvolutional layer (Transposed Conv1D) or a local attention network to achieve differentiable and adaptive temporal expansion.

[0087] Comprehensive Plan and Precautions Maintain boundary alignment: Ensure accurate mapping between the first and last frames, preserving motion start and end information; Anti-aliasing filtering: Low-pass filtering is applied before downsampling, and slight smoothing is achieved after upsampling to suppress interpolation artifacts; Real-time performance and differentiability: Linear / cubic interpolation is simple, efficient and differentiable; learnable interpolation is more flexible, but computational overhead must be weighed. Batch optimization: A unified sampling / interpolation grid is constructed for multiple sequences in a batch, and matrix operations and GPU parallel acceleration are used. Final output After adjustment, the fused feature sequence of the target length is obtained. , It can be directly input into the classification head, detection head, or other downstream networks to meet various application scenarios with different frame rate requirements.

[0088] 6. Super-resolution reconstruction After the preceding module completes the frame rate adjustment, a spatiotemporal feature sequence of the target length is obtained. This fused feature sequence integrates aggregated information obtained through multi-scale feature extraction, self-attention, cross-sequence attention, and learnable token fusion. To further recover high-quality image frame sequences that meet resolution and frame rate requirements, a super-resolution reconstruction network is designed. This network mainly consists of the following key modules, and overall optimization is achieved through end-to-end training. Multi-level upsampling module Original spatiotemporal characteristics In order to increase spatial resolution, this module introduces a multi-level upsampling strategy after downsampling or feature compression in the spatial dimension: (PixelShuffle): First generated through convolutional layers One channel, among which The upsampling factor, The target output channel number is determined. Then, the channel information of the convolution result is rearranged into high-resolution spatial information through sub-pixel rearrangement (PixelShuffle). The mathematical description is as follows: in For convolution kernel parameters, This represents the spatial dimensions after the previous dimensionality reduction.

[0089] Transposed convolution upsampling: To supplement upsampling methods, transposed convolutional layers can also be used in the network. Transposed convolutional layers can directly learn upsampling kernels, and their mathematical description is as follows: in These are the parameters for the transposed convolutional layer.

[0090] Multi-stage cascade design: To achieve progressive scaling, construct There are several upsampling levels. Assuming each level can use a different upsampling factor, then the... The upsampling result is represented as follows: The initial input is This cascading design helps to gradually restore details and reduces the information loss caused by a large one-time expansion.

[0091] Residual block refinement While upsampling improves spatial resolution, it can introduce noise and loss of detail. Therefore, refining the magnified features using residual blocks is an important method for restoring image texture and detail.

[0092] Single residual block structure: for input features A single residual block utilizes the following computational process: in and For convolution kernel parameters, and For the corresponding bias term, The output represents the activation function (such as ReLU or GeLU), and the residual output... Then, the original features are superimposed on the convolutional output through skip connections.

[0093] Stacking multiple residual blocks: To fully recover local textures and global detail information, multiple residual blocks are stacked. Let's assume a common stacking... If there are 1 residual block, the overall residual output is: This residual learning mechanism not only effectively alleviates the gradient vanishing problem in deep network training, but also better restores image details.

[0094] Fusion layer and multi-scale feature integration Multi-level upsampling and residual blocks generate rich features at different scales. Effectively fusing this information is key to improving reconstruction quality. Therefore, a fusion layer is introduced into the network to achieve adaptive integration and dynamic weighting of features.

[0095] Feature splicing: Information is extracted from the outputs of different upsampling stages or residual blocks and then concatenated along the channel dimension: in It is the selected hierarchical index.

[0096] Fusion convolution operations: The concatenated features are fused using convolution, and then dynamically weighted to combine global and local information. in and These are the kernel parameters and biases of the fused convolutional layers, respectively. This operation enables the network to automatically learn the optimal feature combination, thereby better improving the coordination between local image details and overall structure.

[0097] High-resolution image reconstruction The fused feature tensor It contains rich multi-scale information, but it is still in the feature space. Next, these feature maps are transformed into the final high-resolution image frames by reconstructing the convolutional layers.

[0098] Reconstructing convolutional layers: Using single or multiple convolutional layers to transform fused features into pixel-level image representations, mathematically described as follows: in and To reconstruct the convolutional layer parameters, the following were obtained: Represents a high-resolution image sequence (usually) (Represents an RGB color image).

[0099] Detail enhancement strategies: To further ensure image detail and global consistency, additional non-linear transformations and pixel-level adaptive adjustments can be introduced before and after the reconstruction convolutional layer. For example, an activation function can be added before convolutional reconstruction, and edge enhancement or detail compensation modules can be used for post-processing on the output image. in This indicates a post-processing operation used to mitigate artifacts during upsampling and further enhance image quality.

[0100] End-to-end optimization and advantages The super-resolution reconstruction network, together with the preceding spatiotemporal feature extraction, alignment, and fusion modules, constitutes a complete end-to-end system. The entire system jointly optimizes the parameters of each module through a backpropagation algorithm. Its advantages are mainly reflected in: Gradual Refinement Recovery: Through multi-level upsampling and layer-by-layer refinement of residual blocks, the network can gradually recover delicate texture information, effectively reducing the information loss caused by a single upsampling.

[0101] Adaptive feature fusion: The fusion layer uses learnable convolutional kernels to automatically weight and fuse features across multiple scales and sequences, improving the consistency between local details and overall structure of the image.

[0102] Flexible scalability: Regardless of whether PixelShuffle or transposed convolution is used, each module supports different upsampling factors and network structure designs, which facilitates targeted optimization in different application scenarios such as real-time video super-resolution and high-quality video restoration.

[0103] Summarize In summary, the super-resolution reconstruction module of this invention will convert the spatiotemporal feature sequence after frame rate adjustment... Through a series of steps including multi-level upsampling, residual block refinement, feature fusion, and reconstruction convolution, a high-quality image frame sequence that meets the resolution and frame rate requirements is gradually recovered. This design not only has significant advantages in detail recovery and global consistency, but also makes full use of multi-scale information, providing a solid foundation for subsequent visual tasks such as action recognition, object detection and video enhancement.

[0104] In summary, the processing flow of this scheme includes: Step 1 (feature extraction): After unifying the size of multiple image sequences from different frame rates and resolutions, input them into a ResNet101-FPN network with shared parameters, and output a multi-scale feature pyramid sequence; Step 2 (self-attention operation): After flattening each feature pyramid sequence in the time dimension, add position encoding, apply multi-head self-attention respectively, and pass it through a feedforward network and residual connection to obtain enhanced temporal self-attention features; Step 3 (cross-sequence attention): Use the feature sequence with the highest frame rate as the Query and other sequences as Key / Value, input it into the multi-head cross-sequence attention module, and introduce a gating mechanism to output multi-path cross-sequence enhanced features with the same length as the baseline sequence; Step 4 (learnable token fusion): The multi-path cross-sequence attention features from Step 3 are then integrated into the multi-head cross-sequence attention module. Step 5 (Frame Rate Adjustment): Based on the relationship between the target frame number and the original sequence length, uniform sampling, direct reuse, or interpolation (linear, cubic, or learnable interpolation) strategies are adopted to adjust the fused feature sequence to the target length. Step 6 (Super-Resolution Reconstruction): The spatiotemporal feature sequence with the frame rate aligned is input into the super-resolution reconstruction network. This network consists of multi-level upsampling (PixelShuffle or transposed convolution), residual blocks, fusion layers, and reconstruction convolutions, which gradually enlarge the spatial resolution and refine the image details, outputting a high-quality image sequence that meets both resolution and frame rate requirements.

[0105] like Figure 10 As shown, this embodiment of the invention also provides a super-resolution system for image alignment with different frame rates and resolutions. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0106] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned heterogeneous frame rate and heterogeneous resolution image alignment super-resolution method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0107] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0108] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0109] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A super-resolution method for aligning images with different frame rates and resolutions, characterized in that, The steps of this method include: Multiple image sequences are acquired, and features are extracted from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence. Each feature sequence is processed through a self-attention network to obtain the first attention sequence; Based on the frame rate of the video sequence, image sequences are classified into baseline sequences or reference sequences. The image sequence with the highest frame rate in each video sequence is used as the baseline sequence, and the remaining image sequences are used as reference sequences. The first attention sequence corresponding to the baseline sequence and the first attention sequence corresponding to each reference sequence are used to construct a sequence group. Each sequence group is input into a cross-attention network to obtain a second attention sequence. The first attention sequence corresponding to the baseline sequence is used as the query vector, and the first attention sequence corresponding to the reference sequence is used as the key vector and value vector, respectively. These are input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution, performs gating control on the cross-sequence attention output, and outputs the second attention sequence. The weight of each reference sequence is determined based on the second attention sequence and the preset learnable fusion token sequence. The similarity between each second attention sequence and the preset learnable fusion token sequence is calculated. The weight of the reference sequence corresponding to each second attention sequence is determined based on the similarity. The reference sequences are fused based on the weights of the reference sequences to obtain the fused feature sequence. The fused feature sequence is input into a preset super-resolution reconstruction network, which includes a multi-level upsampling module, a residual block, and a fusion layer. A fused image sequence is constructed based on the super-resolution reconstruction network.

2. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of extracting features from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence, the feature extraction network uses a ResNet101 network as the backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

3. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, The steps preceding the step of extracting features from each image sequence through a feature extraction network also include unifying the images in the image sequence to a preset scale.

4. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, The feature sequence includes a sub-feature sequence corresponding to each image in the image sequence. In the step of processing each feature sequence through a self-attention network to obtain the first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image and map it into a query vector, a key vector, and a value vector. Attention weights are calculated by dot product and normalized by Softmax. Then, the value vectors are weighted and summed. Finally, the first attention sequence is obtained by connecting it to the residual through a feedforward network, which enhances the internal spatiotemporal dependency information.

5. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of inputting the fused feature sequence into a preset super-resolution reconstruction network, the super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolutional layer and a sub-pixel rearrangement layer; the residual block includes a convolutional layer, an activation function layer, and a convolutional skip layer; the fusion layer includes a stitching layer and a fusion convolutional layer.

6. The super-resolution method for aligning images with different frame rates and resolutions according to any one of claims 1 to 5, characterized in that, In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further includes a reconstruction convolutional layer, the residual block outputs a fused feature sequence, and the fused feature sequence is input into the reconstruction convolutional layer to obtain a fused image sequence including multiple fused images.

7. A super-resolution system for image alignment at different frame rates and resolutions, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 6.

Images