Different-frame-rate and different-resolution image alignment super-division method and system

Abstract

The invention provides a different-frame-rate and different-resolution image alignment super-division method and system, and the method comprises the steps: carrying out the feature extraction of each image sequence through a feature extraction network, and obtaining a feature sequence corresponding to each image sequence; processing the feature sequence through a self-attention network to obtain a first attention sequence; classifying the image sequence into a reference sequence or a reference sequence based on the frame rate, constructing a first attention sequence corresponding to the reference sequence and a first attention sequence corresponding to each reference sequence into a sequence group, and inputting the sequence group into a cross attention network to obtain a second attention sequence; determining the weight of each reference sequence based on the second attention sequence and a preset learnable fusion Token sequence, and fusing the reference sequences based on the weights of the reference sequences to obtain a fused feature sequence; and inputting the fused feature sequence into a preset super-resolution reconstruction network, and outputting a fused image sequence.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a super-resolution method and system for aligning images with different frame rates and resolutions. Background Technology

[0002] With the rapid development of high-definition video, ultra-high-definition video and ubiquitous video surveillance, people's requirements for video quality and resolution are increasing day by day. Although existing single-frame super-resolution methods can improve the spatial details of each frame, they cannot make full use of temporal information and are difficult to eliminate motion blur and loss of detail.

[0003] Multi-frame super-resolution technology can significantly improve reconstruction results by fusing image information from adjacent frames, but it usually assumes that the input sequence has a uniform frame rate and the same resolution, and cannot be directly applied to videos from different cameras or with different sampling parameters.

[0004] In practical applications, video sequences from different sources often have differences in frame rate and resolution, leading to timestamp misalignment and spatial scale mismatch. Directly fusing multiple frames can easily result in artifacts, misalignment, and information redundancy or loss. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a super-resolution method and system for aligning images with different frame rates and resolutions, in order to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a super-resolution method for aligning images with different frame rates and resolutions, the method comprising the following steps: Multiple image sequences are acquired, and features are extracted from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence. Each feature sequence is processed through a self-attention network to obtain the first attention sequence; Based on the frame rate of the video sequence, the image sequence is classified into a baseline sequence or a reference sequence. The first attention sequence corresponding to the baseline sequence and the first attention sequence corresponding to each reference sequence are constructed into a sequence group. Each sequence group is input into the cross-attention network to obtain the second attention sequence. The weights of each reference sequence are determined based on the second attention sequence and the preset learnable fusion token sequence. The reference sequences are then fused based on their weights to obtain a fused feature sequence. The fused feature sequence is input into a preset super-resolution reconstruction network, which includes a multi-level upsampling module, a residual block, and a fusion layer. A fused image sequence is constructed based on the super-resolution reconstruction network.

[0007] By adopting the scheme, the scheme first realizes synchronous extraction of multi-scale features through a feature extraction network, giving consideration to global semantics and local details; then through the combination of self-attention and cross-sequence attention, high-precision space-time alignment and information complementation can be completed in a deep feature space; the Token mechanism can be learned and fused to realize adaptive weighted fusion of multiple features, avoiding the limitations of artificial design of weights; the frame rate adjustment and super-resolution reconstruction module work collaboratively, significantly improving the visual clarity and detail restoration capability of the video.

[0008] In some embodiments of the present application, in the step of extracting features of each image sequence through a feature extraction network to obtain a feature sequence corresponding to each image sequence, the feature extraction network takes a ResNet101 network as a backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

[0009] In some embodiments of the present application, the step of extracting features of each image sequence through a feature extraction network is preceded by a step of unifying the images in the image sequence to a preset scale.

[0010] In some embodiments of the present application, the feature sequence includes a sub-feature sequence corresponding to each image in the image sequence, and in the step of processing each feature sequence through a self-attention network to obtain a first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image to map it into a query vector, a key vector and a value vector; the attention weight is calculated through dot product and normalized by Softmax, and then the value vector is weighted and summed, finally the first attention sequence with strengthened internal space-time dependent information is obtained through a feedforward network and a residual connection.

[0011] In some embodiments of the present application, in the step of classifying the image sequences into reference sequences or reference sequences based on the frame rate of the video sequences, the image sequence with the highest frame rate in each video sequence is taken as the reference sequence, and the remaining image sequences are taken as the reference sequences.

[0012] In some embodiments of the present application, in the step of inputting each sequence group into a cross-attention network to obtain a second attention sequence, the first attention sequence corresponding to the reference sequence is taken as the query vector, and the first attention sequences corresponding to the reference sequences are taken as the key vector and the value vector, which are input into the cross-attention network; the cross-attention network calculates the cross-sequence attention distribution and controls the cross-sequence attention output through gating, and outputs the second attention sequence.

[0013] In some embodiments of the present application, in the step of determining the weight of each reference sequence based on the second attention sequence and the preset learnable fusion Token sequence, the similarity between each second attention sequence and the preset learnable fusion Token sequence is calculated, and the weight of the corresponding reference sequence of each second attention sequence is determined based on the similarity.

[0014] In some embodiments of the present application, in the step of inputting the fusion feature sequence into the preset super-resolution reconstruction network, the super-resolution reconstruction network comprises a multi-level upsampling module, a residual block and a fusion layer, and in the step of constructing a fusion image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolution layer and a sub-pixel rearrangement layer; the residual block comprises a convolution layer, an activation function layer and a convolution jump layer; and the fusion layer comprises a splicing layer and a fusion convolution layer.

[0015] In some embodiments of the present application, in the step of constructing a fusion image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further comprises a reconstruction convolution layer, the residual block outputs a fusion feature sequence, the fusion feature sequence is input into the reconstruction convolution layer, and a fusion image sequence comprising a plurality of fusion images is obtained.

[0016] The second aspect of the present application also provides a heterogeneous frame rate and heterogeneous resolution image alignment super-resolution system, which comprises a computer device, the computer device comprises a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system realizes the steps realized by the method as described above.

[0017] The third aspect of the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps realized by the aforementioned heterogeneous frame rate and heterogeneous resolution image alignment super-resolution method are realized.

[0018] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings, or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0019] Those skilled in the art will understand that the objects and advantages of the present application are not limited to the above specifically described, and the above and other objects that can be achieved by the present application will be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0021] Figure 1 A schematic diagram of an embodiment of the image alignment and super-resolution method for images with different frame rates and different resolutions according to the present application; Figure 2 A schematic diagram of another embodiment of the image alignment and super-resolution method for images with different frame rates and different resolutions according to the present application; Figure 3 A schematic diagram of the processing architecture of the feature extraction network according to the present application; Figure 4 A schematic diagram of the processing architecture of the self-attention network according to the present application; Figure 5 A schematic diagram of the processing architecture of the cross-attention network according to the present application; Figure 6 A schematic diagram of the processing architecture of the gating mechanism in the cross-attention network according to the present application; Figure 7 A schematic diagram of the learning of the fusion of Token sequences to fuse multiple sequences according to the present application; Figure 8 A schematic diagram of the adjustment of the length of the fused feature sequence according to the present application; Figure 9 A schematic diagram of the processing architecture of the super-resolution reconstruction network according to the present application; Figure 10 A schematic diagram of the hardware architecture according to the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and the drawings. Here, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application but are not intended to limit the present application.

[0023] It should also be noted that, in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0024] Existing temporal alignment methods mostly rely on optical flow estimation or block matching, and the accuracy is easily affected by fast motion and low-texture regions; frame rate conversion based on interpolation can smooth the frame rate difference, but it is difficult to maintain spatio-temporal consistency in high-dimensional feature space; and simple feature splicing or weighted fusion lacks adaptive weight distribution mechanism, and the fusion effect is not ideal.

[0025] As Figure 1 and 2As shown, the present application proposes a method for image alignment and super-resolution with different frame rates and different resolutions, and the steps of the method include: Step S100, a plurality of image sequences are obtained, and each image sequence is subjected to feature extraction by a feature extraction network to obtain a feature sequence corresponding to each image sequence; In the specific implementation process, for the plurality of image sequences from different frame rates and different resolutions, after the images in each sequence are uniformly input into a feature extraction network with shared parameters, the feature extraction network is a convolutional neural network, which takes ResNet101 as the backbone and is connected with a feature pyramid network (FPN) module after each stage. The FPN module generates multi-scale feature maps through two paths from bottom to top and from top to bottom and horizontal connection, which not only preserves the semantic information of the deep layers of the backbone network, but also preserves the detailed information of the shallow layers. Finally, a multi-level and different spatial resolution feature pyramid sequence, i.e., a feature sequence, is output as the basis for subsequent spatio-temporal alignment and fusion.

[0026] Step S200, each feature sequence is processed by a self-attention network to obtain a first attention sequence; In the specific implementation process, a multi-head self-attention mechanism is applied within each feature sequence. Specifically, position encoding is added to each sub-feature sequence of the feature sequence at each time, which is mapped to query (Query), key (Key), and value (Value) vectors. The attention weight is calculated by dot product and normalized by Softmax, and then the value vector is weighted and summed. Finally, via a feedforward network and a residual connection, a self-attention feature sequence with enhanced internal spatio-temporal dependency information, i.e., a first attention sequence, is obtained.

[0027] Step S300, based on the frame rate of the video sequence, the image sequences are classified into a reference sequence or a reference sequence, the first attention sequence corresponding to the reference sequence is constructed with the first attention sequence corresponding to each reference sequence to form a sequence group, and each sequence group is input into a cross-attention network to obtain a second attention sequence; In the specific implementation process, among the first attention sequences output by the plurality of self-attention, the sequence with the highest frame rate is selected as the reference sequence, and the remaining sequences are sequentially selected as the reference sequences. For each reference sequence, the self-attention feature of the reference sequence is taken as Query, and the self-attention feature of the reference sequence is taken as Key and Value, which are input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution and controls the cross-sequence attention output by gating, realizes the information supplement and deep complementarity between the reference sequence and each reference sequence, and outputs a cross-sequence attention feature sequence with the same length as the reference sequence for each reference sequence, i.e., a second attention sequence.

[0028] Step S400, determining the weight of each reference sequence based on the second attention sequence and a preset learnable fusion Token sequence, fusing the respective reference sequences based on the weight of the reference sequence to obtain a fused feature sequence; In the specific implementation process, the scheme introduces a learnable fusion Token sequence with the same length as the second attention sequence to calculate the similarity with the second attention sequence, point multiplication, and then obtain the weight of each reference sequence through softmax. The fusion Token learns in parallel with each feature in the training process, and the attention mechanism is used to calculate the attention weight of each feature to autonomously allocate the attention degree. Then, each feature is weighted and superimposed according to the learned weight to form a unique and unified spatio-temporal feature sequence, which effectively integrates the complementary information of multiple sequences.

[0029] As shown in Figure 8 The steps of the method further include frame rate matching processing on the fused feature sequence output by step S400. When the target frame number is less than the length of the fused feature sequence, a uniform interval sampling strategy is used; when the target frame number is equal to the length of the fused feature sequence, the sequence is directly used; and when the target frame number is greater than the length of the fused feature sequence, time series interpolation (such as bilinear interpolation or cubic interpolation) is performed on the fused feature sequence to complete the new frame, thereby generating a fused feature sequence that meets the target frame rate requirement.

[0030] Step S500, inputting the fused feature sequence into a preset super-resolution reconstruction network, the super-resolution reconstruction network including a multi-level upsampling module, a residual block and a fusion layer, and constructing a fused image sequence based on the super-resolution reconstruction network.

[0031] In the specific implementation process, the fused feature sequence is input into the super-resolution reconstruction network, which includes a multi-level upsampling module, a residual block and a fusion layer. Through layer-by-layer amplification of the spatial resolution and refinement of the image details, the spatio-temporal features are converted into a high-resolution and high-quality image frame sequence, and finally a super-resolution video sequence that meets the dual requirements of resolution and frame rate, i.e., a fused image sequence, is output.

[0032] By using the above scheme, the scheme first realizes synchronous extraction of multi-scale features through a feature extraction network, taking into account global semantics and local details. Then, through the combination of self-attention and cross-sequence attention, high-precision spatio-temporal alignment and information complementation can be completed in the deep feature space. The learnable fusion Token mechanism realizes adaptive weighted fusion of multiple features, avoiding the limitations of manually designed weights. The frame rate adjustment and super-resolution reconstruction module work together to significantly improve the visual clarity and detail restoration ability of the video.

[0033] As shown in Figure 3As shown, in some embodiments of the present invention, in the step of extracting features from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence, the feature extraction network uses a ResNet101 network as the backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

[0034] In some embodiments of the present invention, the step prior to the step of extracting features from each image sequence through a feature extraction network further includes unifying the images in the image sequence to a preset scale.

[0035] In the specific implementation process, the feature extraction network uses ResNet101 as the backbone and connects to the Feature Pyramid Network (FPN) module in parallel after each stage. Through bottom-up and top-down paths and lateral connections, a multi-scale feature pyramid is generated, which not only preserves deep semantic information but also takes into account shallow details, laying a solid foundation for subsequent alignment and fusion.

[0036] Specifically, ResNet101 consists of 101 convolutional layers. Specifically, Conv1 has one convolutional layer; Conv2_x consists of three sets of convolutions (each set follows the sequence: 1×1,64→3×3,64→1×1,256); Conv3_x consists of four sets of convolutions (each set follows the sequence: 1×1,128→3×3,128→1×1,512); Conv4_x consists of 23 sets of convolutions (each set follows the sequence: 1×1,256→3×3,256→1×1,1024); Conv5_x consists of three sets of convolutions (each set follows the sequence: 1×1,512→3×3,512→1×1,2048); and finally, there is a 1000-d fully connected layer (FC). In this embodiment, the final FC is not used; instead, the FPN is constructed from the outputs of Conv2_x, Conv3_x, Conv4_x, and Conv5_x.

[0037] The FPN module adopts a bottom-up and top-down decoder structure.

[0038] like Figure 4 As shown, in some embodiments of the present invention, the feature sequence includes a sub-feature sequence corresponding to each image in the image sequence. In the step of processing each feature sequence through a self-attention network to obtain a first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image, mapping it to a query vector, a key vector, and a value vector. Attention weights are calculated by dot product and normalized by Softmax, and then the value vectors are weighted and summed. Finally, the first attention sequence with enhanced internal spatiotemporal dependency information is obtained through a feedforward network and a residual connection.

[0039] In the implementation process, each obtained first attention sequence is flattened in the time dimension and is respectively input into a multi-head self-attention module. Each frame feature is mapped into a query (Query), a key (Key) and a value (Value) vector, the correlation weight between each time in the sequence is calculated and is normalized by Softmax, finally, a self-attention feature sequence with strengthened self-temporal and spatial dependence is generated by combining a feedforward network and a residual connection.

[0040] In some embodiments of the present application, in the step of classifying the image sequences into reference sequences or reference sequences based on the frame rate of the video sequences, the image sequence with the highest frame rate in each video sequence is taken as the reference sequence, and the remaining image sequences are taken as the reference sequences.

[0041] As shown in the figure, Figure 5 In some embodiments of the present application, in the step of inputting each sequence group into a cross-attention network to obtain a second attention sequence, the first attention sequence corresponding to the reference sequence is taken as a query vector, and the first attention sequence corresponding to the reference sequence is taken as a key vector and a value vector, which are input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution and controls the cross-sequence attention output by gating, and outputs the second attention sequence.

[0042] In the implementation process, the first attention sequence with the highest frame rate is taken as a reference query (Query), and the self-attention output of the other sequences is taken as a key (Key) and a value (Value), which are input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution in a high-dimensional feature space, realizes the deep information complementation and feature enhancement between the reference sequence and each reference sequence, and outputs a multi-channel cross-sequence attention feature sequence with the same length as the reference sequence.

[0043] As shown in the figure, Figure 6 In the implementation process, a gating mechanism is introduced into the cross-attention network of the present scheme, which includes: a. The first attention sequence outputs of the reference sequence and the reference sequence calculated by the self-attention network are spliced by channel; b. The gating vector Gr is obtained by Sigmoid activation; c. Finally, the cross-sequence attention features of the reference sequence are multiplied by Gr for weighted fusion; d. The gated feature sequence is added to the first attention sequence after the self-attention network, and LayerNorm is performed.

[0044] As shown in the figure, Figure 7As shown, in some embodiments of the present invention, in the step of determining the weight of each reference sequence based on the second attention sequence and the preset learnable fusion token sequence, the similarity between each second attention sequence and the preset learnable fusion token sequence is calculated, and the weight of the reference sequence corresponding to each second attention sequence is determined based on the similarity.

[0045] In practice, the learnable fusion token sequence is initialized using a uniform Xavier distribution during the initial stage of network training, and the update rate is constrained during training using the following regularization terms: .

[0046] The above scheme introduces a learnable fusion token sequence with the same time length as the second attention sequence output by the cross-attention network. This token is updated in parallel with the features of each path during training, and is automatically learned and assigned fusion weights through attention or dot product mechanisms. The features of each path are weighted and superimposed to generate a single and unified spatiotemporal feature representation, realizing adaptive fusion of information from multiple sources.

[0047] like Figure 9 As shown, in some embodiments of the present invention, when the fused feature sequence is input into a preset super-resolution reconstruction network, the super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolutional layer and a sub-pixel rearrangement layer; the residual block includes a convolutional layer, an activation function layer, and a convolutional skip layer; the fusion layer includes a stitching layer and a fusion convolutional layer.

[0048] In some embodiments of the present invention, in the step of constructing a fused image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further includes a reconstruction convolutional layer, the residual block outputs a fused feature sequence, and the fused feature sequence is input into the reconstruction convolutional layer to obtain a fused image sequence including multiple fused images.

[0049] Using the above scheme, the network combines multi-level upsampling modules (such as subpixel convolution or transposed convolution), residual blocks and fusion layers. By progressively amplifying the spatial resolution and refining image details, it maps spatiotemporal features into a high-resolution image frame sequence, outputting a super-resolution video that meets both resolution and frame rate requirements.

[0050] Example: This solution is suitable for image sequences with multiple frame rates and resolutions. 1. First, resize the images to unify the resolution of all images to be processed (e.g., ...). ), and then a ResNet101 combined with a feature pyramid network (FPN) is used as a shared parameter network to perform multi-scale feature extraction. Different levels of feature information in the original image are captured through multi-scale features to obtain a unified dimensional feature representation. K input sequences are provided, and sequence k is represented as wherein represents a color image of the kth sequence at time t; To extract multi-scale features of image sequences with different frame rates and resolutions, the feature extraction network of the embodiment uses ResNet101 as the backbone network, and embeds the FPN module at the output ends of Stage2 to Stage5. ResNet101 is composed of 101 convolutional layers. Specifically, there is one convolution in Conv1; Conv2_x is composed of three groups of convolutions (the process of each group is: 1x1, 64→3x3, 64→1x1, 256); Conv3_x is composed of four groups of convolutions (the process of each group is: 1x1, 128→3x3, 128→1x1, 512); Conv4_x is composed of 23 groups of convolutions (the process of each group is: 1x1, 256→3x3, 256→1x1, 1024); Conv5_x is composed of three groups of convolutions (the process of each group is: 1x1, 512→3x3, 512→1x1, 2048); and finally there is a 1000-d fully connected layer FC. In the embodiment, the last FC is not used, and only the outputs of Conv2_x, Conv3_x, Conv4_x, and Conv5_x are used to construct the FPN; The FPN module adopts a bottom-up and top-down decoder structure. Specifically, first, each image frame is preprocessed by standardization (mean and standard deviation normalization), and then input into ResNet101 to obtain feature maps of different stages; Stage2: the feature map size is C3xH / 4xW / 4, and C3=256; Stage3: the feature map size is C3xH / 8xW / 8, and C3=512; Stage4: the feature map size is C4xH / 16xW / 16, and C4=1024; and Stage5: the feature map size is C5xH / 32xW / 32, and C5=2048.

[0051] In the FPN part, features at each scale are first projected to a fixed dimension D using 1×1 lateral convolutions. Then, upsampling (bilinear interpolation) combined with lateral connections is used to fuse multi-scale features into a series of feature pyramids, where each layer is convolved with a 3×3 convolution before outputting the final features. This process not only captures global semantics but also takes into account local details, ensuring that downstream modules obtain rich feature descriptions. Each scale's features are then subjected to global average pooling, resulting in four 256-dimensional feature vectors. These four feature vectors from different scales are concatenated to obtain a 1024-dimensional feature vector for this image. The four feature maps at different scales have different resolutions, which can better adapt to objects of different sizes in the image. Will Input the ResNet101 backbone network and extract convolutional feature maps from the outputs of Stages 2–5: ; in ; First, perform a 1×1 lateral convolution on the feature maps of each layer and project them to a uniform number of channels. Then, a feature pyramid is generated by connecting the top-down paths with the horizontal connections. in, They are 1×1 and 3×3 convolution kernels, respectively. This indicates that bilinear interpolation amplifies the data by a factor of 2, resulting in a multi-scale feature map. ; Feature mapping for each scale Apply global average pooling: get Concatenate the pooling vectors of each scale along the channel direction: As an image The final eigenvector.

[0052] As mentioned above, each frame of all sequences is mapped to a sequence of uniform dimensions. And finally, the feature sequence is obtained; Subsequent modules can directly input this feature sequence for spatiotemporal alignment and fusion of self-attention and cross-sequence attention.

[0053] 2. Perform self-attention calculation for each image feature sequence. In each image sequence, some frames may be blurry, but this blurriness can be repaired by using the contextual information of the frame in the sequence, that is, by taking information from those clear frames as supplementary information.

[0054] For each sequence, a learnable positional encoding is introduced into the self-attention computation module. , so that the input feature sequence Updated to Subsequently, a multi-head attention mechanism is employed to perform a linear transformation of query, key, and value for each input. Specifically, the number of attention heads is set to [number missing]. The dimensions of each head are For the h-th head, we have: ; Attention Score The final attention output is obtained by calculating the dot product and scaling operations, and then adding Dropout (with a scaling factor of 0.1). Finally, residual connections and LayerNorm are used to effectively improve the feature transfer and expressive power.

[0055] Let the first The frame feature matrix of the image sequence is The row index corresponds to the time frame, and the column vector is the feature extracted by the aforementioned FPN+GAP.

[0056] To preserve timing information, positional encoding is added, first fixing or learningable positional codes. Add to it: ; Linear mapping to multi-head Q, K, V Let the number of long positions be Dimensions per head ,satisfy .

[0057] Define the projection matrix respectively; ; Then the first The header's query, key, and value; ; in .

[0058] Calculate attention weights For the First, calculate the unnormalized attention score matrix: ; If causal or local constraints are required, they can be found in Add mask Finally through With optional Dropout, the attention distribution is obtained: ; Weighted summation produces the header output; .

[0059] Multi-head splicing and output mapping will The output of the head is concatenated channel-wise: Then it goes through a linear mapping and Dropout: Residual connection and layer normalization are applied to add the multi-head attention output to the input and perform LayerNorm: Feed-Forward, followed by two fully connected feed-forward networks (with GeLU activation and Dropout) and residual connection, layer normalization: where .

[0060] So far, the self-attention module output of the th image sequence is which can be used for subsequent cross-sequence alignment and fusion.

[0061] 3. Cross-Sequence Attention After completing the self-attention of each image sequence, we get The self-attention module of each sequence is based on the global content-based mechanism in the Transformer, which can capture long-range temporal dependencies within the sequence and ensure stable information flow and gradient transmission through residual connection and layer normalization. The introduction of self-attention is to let each element in the sequence query information from its own context to strengthen the feature representation; However, the self-attention of a single path can only utilize the context of its own frames. For frames with missing information due to motion blur or exposure problems, the information that the camera stream cannot obtain due to hardware sources cannot be compensated. Considering that different camera streams or different frame rates are multi-view / multi-rate recordings of the same target, the frames of different sequences at the same time or adjacent time often contain complementary spatial detail information. Based on this, cross-sequence attention is introduced to let the "reference sequence" (with the highest frame rate and the most abundant temporal information) query information from other "complementary sequences" to strengthen the feature representation of each frame.

[0062] To overcome the shortcomings of a single sequence in processing blurred frame information, a cross-sequence attention mechanism is proposed. Specifically, first, the sequence with the highest frame rate (denoted as the reference sequence ) is taken as the main sequence, and the remaining sequences are taken as reference sequences. For each reference sequence , let the output of the reference sequence be , the reference sequence is By linear projection, we obtain (e.g. input ) and and (e.g. input ), and compute the attention scores: Then, we apply Softmax to the attention scores of each head to obtain the attention distribution, and weight it with to generate the cross-sequence augmented feature. To suppress noise interference, we introduce a gating mechanism: Finally, the cross-sequence attention fusion result is expressed as: where denotes element-wise multiplication, which realizes efficient and selective supplement of reference information.

[0063] Select the sequence number with the most frames as the (base sequence) among the benchmark and reference sequences, and the remaining sequences are reference sequences (reference sequences). The output of the base sequence is: Linear projection and multi-head splitting learn the projection for each reference sequence Define the query, key, and value of the head as Attention calculation and attention distribution Compute the unnormalized attention for each head: Then do Softmax by row (optionally add mask to limit the query time domain range): Weighted summation and multi-head splicing The output of each head is: and then map after splicing: Gating fusion and residual To consider that not all reference information is helpful for each frame, a gating vector is introduced. The gating mechanism aims to automatically assign a [0,1] "trust coefficient" to each channel for each time, suppressing unreliable information. The process here is to concatenate the sequence of self-attention with the sequence of cross-sequence attention, complete the size transformation through linear mapping, and scale each element of the transformed element to the [0,1] interval through element-level Sigmoid. Then multiply element by element with the cross-sequence attention sequence, and finally add the original self-attention sequence output, pass through LayerNorm, and complete the gating. wherein represents channel splicing, represents element-wise Sigmoid.

[0064] Final fusion: Finally, for each reference sequence, a gated cross-sequence attention output result is obtained. That is, in the original input, there is one reference sequence with the most frames and N reference sequences with the most frames. After this cross-sequence attention step, N feature sequences are obtained, each with the same length as the reference sequence.

[0065] Theoretical basis Self-attention (Self-Attention) is based on Transformer (Vaswani et al. 2017), with content correlation as the core, which can dynamically focus on beneficial positions within the time sequence, capturing global context and long-range dependencies.

[0066] Cross-sequence attention is similar to multi-modal or multi-view fusion (e.g. ViLT, Florence), which allows each feature in the reference sequence to "ask" and "borrow" semantics and spatial details from other sequences through the "Query-Key-Value" mechanism. It is an effective compensation strategy for information loss caused by camera shake, motion blur, etc.

[0067] The gating (Gating) mechanism introduced can dynamically suppress noise or flicker frame information, and on the other hand, ensure that when the supplementary information is reliable, it can be fully integrated to improve the robustness and completeness of the blurred frame representation for downstream tasks.

[0068] In summary, this section constructs a "multi-view compensation" channel for each frame in the reference sequence through cross-sequence attention, making it more robust and detailed in spatial and temporal feature representation.

[0069] 4. Learnable Token Fusion (Learnable Token Fusion) of various image feature sequences So far, the enhanced features of each reference sequence to the reference sequence are obtained by multi-path cross-sequence attention Common To "converge" the multi-path supplementary information into a unified spatio-temporal feature sequence, a "learnable fusion Token sequence" is introduced , and they are used as Query. Each path fuses the features obtained by cross-sequence attention into a fused graph feature as Key and Value, and performs attention fusion on all reference path features.

[0070] To integrate the information obtained by multi-path cross-sequence attention, this module introduces a learnable Token sequence consistent with the length of the reference sequence . The Token is initialized by Xavier uniform distribution in the initial stage of the network, and is continuously self-adaptively optimized in the training process. The spliced multi-path features and are respectively mapped by linear mapping to obtain the , , matrices required by the multi-head attention mechanism, and the specific calculation is as follows: For each attention head , we have: The attention distribution calculation is: The fusion output is: After splicing the outputs of all heads, linear transformation is performed, and residual connection and LayerNorm are performed with the original Token to obtain the final fused features: In addition, an additional regularization term can be further introduced to constrain the update rate of the Token, thereby improving the stability and efficiency of cross-sequence information fusion.

[0071] The definition sequence length of the fusion Token is the same as that of the reference sequence: Where each is a trainable vector, which is aligned with the features at time .

[0072] Splice multi-path features, and splice path enhanced features in the feature dimension: ​ Multi-head cross-sequence attention mapping, the number of multi-head is , the dimension of each head is , satisfying The projection matrix is defined for the fusion of Token and spliced features The first After the first projection: Attention weight and weighted fusion Calculate attention and normalize: Re-weighted sum and splice multi-head: Residual connection and layer normalization Residual addition and normalization with original Token: At this time is the fusion feature sequence after fusing path to a single path.

[0073] Theory and function Adaptive weight distribution: the fusion Token dynamically allocates attention weights to multi-path features at all times, and the network can automatically learn which time and which path information is most valuable.

[0074] Timing alignment preservation: Token sequence and timing frame correspond one by one, without additional timing scaling or interpolation, which is convenient for subsequent convolution or classification head processing.

[0075] End-to-end learning: the fusion Token and attention projection matrix are optimized together in the entire network, without pre-set pooling strategy.

[0076] Parallel and efficient: based on the multi-head mechanism of Transformer, long sequence attention can be calculated in parallel on GPU.

[0077] Method source: draw on the ideas of token, induced set attention of Set Transformer, etc. in ViT, but use equal-length Token sequence to output space-time features more flexibly. In summary, the "learnable Token fusion" module introduces trainable Tokens matching the timing length, and uses them as Query to adaptively fuse multi-path cross-sequence attention features, generating a unified and discriminative space-time feature sequence, providing strong representation support for downstream action recognition, timing segmentation, etc.

[0078] ​

[0079] 5. Frame-Rate Alignment After the completion of the "learnable Token fusion", the spatio-temporal feature sequence with the same length as the original reference sequence is obtained In order to reconstruct a new graph feature sequence based on the graph feature sequence in the next step, it is necessary to adjust the original sequence length to according to the size relationship between the target and the original length , which mainly includes three cases: After obtaining the multi-channel fused spatio-temporal feature sequence , due to the difference in the frame rate of the source video, it is necessary to adjust to the target output frame number . The specific operation is as follows: Down-sampling ( < ) : Uniformly spaced sampling index is adopted. In order to prevent aliasing of high-frequency information, first perform one-dimensional Gaussian filtering on , and the standard deviation of the filter kernel is adaptively adjusted according to the sequence length and the sampling ratio.

[0080] Keep unchanged ( = ) : Directly reuse , without additional processing.

[0081] Up-sampling ( > ) : Define the target grid , where . For each new time , the following methods can be used: Linear interpolation: Let , then Cubic interpolation: Select the adjacent 4 points, and use the cubic spline kernel wmα for interpolation to improve the smoothness and accuracy of the interpolation.

[0082] Learnable up-sampling: Through deconvolution or local attention network, the interpolation kernel is adaptively learned to realize the end-to-end interpolation process.

[0083] The above methods can effectively unify the time length of multi-source features and provide a consistent data basis for subsequent tasks.

[0084] Target frame number less than original length (down-sampling) When < When the target number of frames is equal to the original length (unchanged), when This ensures that the first and last frames are aligned (the first frame is taken when k = 0, and the last frame is taken when k = -1), and the frames are evenly distributed in the middle.

[0085] Sampling by index: Optionally, a one-dimensional low-pass filter (such as Gaussian convolution) is applied to Ufused before downsampling to suppress high-frequency noise and avoid aliasing artifacts.

[0086] When the target number of frames is equal to the original length (unchanged), when When the target number of frames is greater than the original length (upsampling), when When the target number of frames is greater than the original length (upsampling), when When the target number of frames is greater than the original length (upsampling), when When the target number of frames is greater than the original length (upsampling), when where is a real index. Common interpolation methods include: Linear interpolation: let , , Cubic spline interpolation: take the adjacent four points , , , Learnable upsampling: treat as a one-dimensional signal, learn the interpolation kernel through a deconvolution layer (Transposed Conv1D) or a local attention network, to achieve a differentiable and adaptive temporal expansion.

[0087] Comprehensive solutions and considerations Maintain boundary alignment: ensure that the first and last frames are accurately mapped, preserving the start and end information of the action; Anti-aliasing filter: low-pass before downsampling, and slightly smooth after upsampling to suppress interpolation artifacts; Real-time and differentiability: linear / cubic interpolation is simple, efficient, and differentiable; learnable interpolation is more flexible, but requires a trade-off in computational overhead; ​​​​​Batch optimization: unified construction of sampling / interpolation grid for multiple sequences in batch, using matrix operation and GPU parallel acceleration; Final output After adjustment, the target length of the fusion feature sequence is obtained , It can be directly input to the classification head, detection head or other downstream network to meet various frame rate requirement application scenarios.

[0088] 6. Super-resolution reconstruction After the frame rate adjustment is completed by the pre-sequence module, the target length of the spatio-temporal feature sequence is obtained The fusion feature sequence fuses the aggregated information obtained after multi-scale feature extraction, self-attention, cross-sequence attention, and learnable Token fusion. In order to further restore high-quality image frame sequences that meet the resolution and frame rate requirements, a super-resolution reconstruction network is designed. The network mainly includes the following key modules, and the overall optimization is realized through end-to-end training. Multi-level up-sampling module Original spatio-temporal features In the spatial dimension, after down-sampling or feature compression, in order to realize the enlargement of spatial resolution, this module introduces a multi-level up-sampling strategy: (PixelShuffle): first generate channels through the convolution layer, where is the up-sampling factor, is the target output channel number. Then rearrange the channel information of the convolution result to high-resolution spatial information through sub-pixel rearrangement (PixelShuffle). The mathematical description is as follows: Where is the convolution kernel parameter, is the spatial size after pre-sequence dimension reduction.

[0089] Transposed convolution up-sampling: To supplement the up-sampling means, the network can also use a transposed convolution layer. The transposed convolution layer can directly learn the up-sampling kernel, and its mathematical description is as follows: Where is the transposed convolution layer parameter.

[0090] Multi-level cascade design: In order to realize step-by-step enlargement, construct up-sampling levels. Assuming that different up-sampling factors can be used for each level, the level up-sampling result is represented as: Where the initial input is This cascade design is conducive to gradually restoring details and reducing information loss caused by one-time large-scale expansion.

[0091] Residual block refinement Although the upsampling operation improves the spatial resolution, it may introduce noise and loss of details. Therefore, refining the enlarged features through the residual block is an important means to restore image texture and details.

[0092] Single residual block structure: input feature The single residual block uses the following calculation process: where and are the convolution kernel parameters, and are the corresponding bias terms, denotes the activation function (such as ReLU or GeLU), and the residual output is obtained by stacking the original feature and the convolution output through a jump connection.

[0093] Stack multiple residual blocks: To fully restore local texture and global detail information, multiple residual blocks are stacked. Let's stack residual blocks, then the total residual output is: This residual learning mechanism not only effectively alleviates the gradient vanishing problem in deep network training, but also better restores image details.

[0094] Fusion layer and multi-scale feature integration Multi-level upsampling and residual blocks generate rich features at different scales. How to effectively integrate these information is the key to improving reconstruction quality. Therefore, a fusion layer is introduced in the network to realize adaptive integration and dynamic weighting of features.

[0095] Feature concatenation: Extract information from the output of different upsampling stages or residual blocks and concatenate them in the channel dimension: where is the selected level index.

[0096] Fusion convolution operation: Fuse the concatenated features through convolution to dynamically weight global and local information: where and are the kernel parameters and bias of the fusion convolutional layer, respectively. This operation enables the network to automatically learn the optimal feature combination manner, thus better improving the coordination between local details and overall structure of the image.

[0097] High-resolution image reconstruction Fused feature tensor contains rich multi-scale information, but is still in the feature space. Next, these feature maps are converted into the final high-resolution image frames through the reconstruction convolutional layer.

[0098] Reconstruction convolutional layer: single or multi-layer convolution is used to convert the fused features into pixel-level image representation, which is mathematically described as: where and are the reconstruction convolutional layer parameters, and the obtained represents a high-resolution image sequence (usually represents an RGB color image.

[0099] Detail enhancement strategy: to further ensure the consistency of image details and the overall, additional nonlinear transformations and pixel-level adaptive adjustments can be introduced before and after the reconstruction convolutional layer. For example, an activation function can be added before the convolution reconstruction, and an edge enhancement or detail compensation module can be used for post-processing on the output image: where represents the post-processing operation, which is used to alleviate the artifact problem in the upsampling process and further improve the image quality.

[0100] End-to-end optimization and advantages The super-resolution reconstruction network and the previous spatiotemporal feature extraction, alignment, and fusion module form a complete end-to-end system, and the parameters of each module are jointly optimized through the backpropagation algorithm. Its advantages mainly include: Step-by-step refinement recovery: through multi-level upsampling and residual block layer-by-layer refinement, the network can gradually recover delicate texture information, effectively reducing the information loss caused by one-time upsampling.

[0101] Adaptive feature fusion: the fusion layer uses learnable convolution kernels to automatically weight and fuse multi-scale and cross-sequence features, improving the consistency of image local details and overall structure.

[0102] Flexible scalability: Whether using PixelShuffle or transposed convolution, each module supports different up-sampling factors and network structure designs, facilitating targeted optimization in different application scenarios such as real-time video super-resolution and high-quality video restoration.

[0103] Summary In summary, the super-resolution reconstruction module of the application gradually recovers a high-quality image frame sequence that meets the resolution and frame rate requirements by sequentially passing the spatio-temporal feature sequence after frame rate adjustment through multiple levels of up-sampling, residual block refinement, feature fusion, and reconstruction convolution In summary, the super-resolution reconstruction module of the application gradually recovers a high-quality image frame sequence that meets the resolution and frame rate requirements by sequentially passing the spatio-temporal feature sequence after frame rate adjustment through multiple levels of up-sampling, residual block refinement, feature fusion, and reconstruction convolution This design not only has significant advantages in detail recovery and global consistency, but also fully utilizes multi-scale information, providing a solid foundation for subsequent visual tasks such as action recognition, target detection, and video enhancement.

[0104] In summary, the processing flow of the present scheme includes: Step 1 (feature extraction): After uniformly sizing multiple image sequences from different frame rates and different resolutions, input the ResNet101-FPN network with shared parameters, and output a multi-scale feature pyramid sequence; Step 2 (self-attention operation): After flattening each feature pyramid sequence in the time dimension, add position encoding, apply multi-head self-attention respectively, and connect with the feedforward network and residual connection to obtain enhanced time sequence self-attention features; Step 3 (cross-sequence attention): Take the highest frame rate feature sequence as Query, and the other sequences as Key / Value, input the multi-head cross-sequence attention module, and introduce a gating mechanism, output a multi-channel cross-sequence enhanced feature consistent with the length of the reference sequence; Step 4 (learnable Token fusion): Concatenate the multi-channel cross-sequence attention features of Step 3, introduce a learnable Token sequence with the same length as the sequence, take Token as Query, and multi-channel features as Key / Value, apply multi-head cross-sequence attention and residual connection to obtain a unified spatio-temporal feature sequence; Step 5 (frame rate adjustment): According to the size relationship between the target frame number and the original sequence length, respectively adopt uniform sampling, direct reuse or interpolation (linear, cubic or learnable interpolation) strategy to adjust the fused feature sequence to the target length; Step 6 (super-resolution reconstruction): Input the spatio-temporal feature sequence with aligned frame rate into the super-resolution reconstruction network, which is composed of multiple levels of up-sampling (PixelShuffle or transposed convolution), residual blocks, fusion layers, and reconstruction convolution, gradually enlarging the spatial resolution and refining image details, and output a high-quality image sequence that meets the dual requirements of resolution and frame rate.

[0105] For example, Figure 10As shown, the embodiment of the present application also provides a system for image alignment and super-resolution of images with different frame rates and different resolutions, which comprises a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, and the system implements the steps of the method as described above when the computer instructions are executed by the processor.

[0106] The embodiment of the present application also provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the image alignment and super-resolution method of images with different frame rates and different resolutions as described above. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0107] It should be apparent to those skilled in the art that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination thereof. The choice of hardware or software implementation is dependent on the particular application and design constraints imposed on the overall system. Skilled artisans can employ a variety of approaches to implement the described functionality, and the application is not limited by the choice of hardware or software implementations. When implemented in hardware, the functionality can be provided as an electronic circuit, a special-purpose integrated circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave in a transmission medium or communication link.

[0108] It should be apparent to those skilled in the art that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination thereof. The choice of hardware or software implementation is dependent on the particular application and design constraints imposed on the overall system. Skilled artisans can employ a variety of approaches to implement the described functionality, and the application is not limited by the choice of hardware or software implementations. When implemented in hardware, the functionality can be provided as an electronic circuit, a special-purpose integrated circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave in a transmission medium or communication link.

[0109] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0110] The above merely illustrates the preferred embodiments of the present application, and is not used to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of protection of the present application.

Claims

1. A super-resolution method for aligning images with different frame rates and resolutions, characterized in that, The steps of this method include: Multiple image sequences are acquired, and features are extracted from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence. Each feature sequence is processed through a self-attention network to obtain the first attention sequence; Based on the frame rate of the video sequence, the image sequence is classified into a baseline sequence or a reference sequence. The first attention sequence corresponding to the baseline sequence and the first attention sequence corresponding to each reference sequence are constructed into a sequence group. Each sequence group is input into the cross-attention network to obtain the second attention sequence. The weights of each reference sequence are determined based on the second attention sequence and the preset learnable fusion token sequence. The reference sequences are then fused based on their weights to obtain a fused feature sequence. The fused feature sequence is input into a preset super-resolution reconstruction network, which includes a multi-level upsampling module, a residual block, and a fusion layer. A fused image sequence is constructed based on the super-resolution reconstruction network.

2. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of extracting features from each image sequence through a feature extraction network to obtain a feature sequence for each image sequence, the feature extraction network uses a ResNet101 network as the backbone network, and a feature pyramid network is connected to each stage of the ResNet101 network.

3. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, The steps preceding the step of extracting features from each image sequence through a feature extraction network also include unifying the images in the image sequence to a preset scale.

4. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, The feature sequence includes a sub-feature sequence corresponding to each image in the image sequence. In the step of processing each feature sequence through a self-attention network to obtain the first attention sequence, a self-attention mechanism is used to add position encoding to the sub-feature sequence of each image and map it into a query vector, a key vector, and a value vector. Attention weights are calculated by dot product and normalized by Softmax. Then, the value vectors are weighted and summed. Finally, the first attention sequence is obtained by connecting it to the residual through a feedforward network, which enhances the internal spatiotemporal dependency information.

5. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of classifying image sequences into baseline sequences or reference sequences based on the frame rate of video sequences, the image sequence with the highest frame rate among all video sequences is used as the baseline sequence, and the remaining image sequences are used as reference sequences.

6. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of inputting each sequence group into the cross-attention network to obtain the second attention sequence, the first attention sequence corresponding to the benchmark sequence is used as the query vector, and the first attention sequence corresponding to the reference sequence is used as the key vector and value vector, which are input into the cross-attention network. The cross-attention network calculates the cross-sequence attention distribution, performs gating control on the cross-sequence attention output, and outputs the second attention sequence.

7. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of determining the weight of each reference sequence based on the second attention sequence and the preset learnable fusion token sequence, the similarity between each second attention sequence and the preset learnable fusion token sequence is calculated, and the weight of the reference sequence corresponding to each second attention sequence is determined based on the similarity.

8. The super-resolution method for aligning images with different frame rates and resolutions according to claim 1, characterized in that, In the step of inputting the fused feature sequence into a preset super-resolution reconstruction network, the super-resolution reconstruction network includes a multi-level upsampling module, a residual block, and a fusion layer. In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the multi-level upsampling module is provided with a convolutional layer and a sub-pixel rearrangement layer; the residual block includes a convolutional layer, an activation function layer, and a convolutional skip layer; the fusion layer includes a stitching layer and a fusion convolutional layer.

9. The super-resolution method for aligning images with different frame rates and resolutions according to any one of claims 1 to 8, characterized in that, In the step of constructing a fused image sequence based on the super-resolution reconstruction network, the super-resolution reconstruction network further includes a reconstruction convolutional layer, the residual block outputs a fused feature sequence, and the fused feature sequence is input into the reconstruction convolutional layer to obtain a fused image sequence including multiple fused images.

10. A super-resolution system for image alignment at different frame rates and resolutions, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 9.

Images