Track sensing converter for video super-resolution

CN118922854APending Publication Date: 2024-11-08MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093320.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-29
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing video super-resolution technologies are difficult to effectively utilize information from long-distance frames, resulting in high computational cost and insufficient long-term modeling capabilities when processing large image sequences.

Method used

The trajectory perception transformer is used to generate trajectories through visual marker embedding networks and motion estimation networks, calculate the similarity values ​​between image frames, select the most relevant frames for reconstruction, reduce computing costs and improve long-range feature modeling capabilities.

Benefits of technology

It significantly reduces computing costs, improves the quality of video super resolution, can effectively utilize information from long-distance frames, improves long-range feature modeling capabilities, and performs better than other technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922854A_ABST
    Figure CN118922854A_ABST
Patent Text Reader

Abstract

A computing system includes a processor and a memory storing instructions executable by the processor to obtain a sequence of image frames in a video. Target image frames of the sequence are input into a visual marker embedding network of a trajectory aware transducer to output a plurality of query markers. A plurality of different image frames are input into a motion estimation network, a visual marker embedding network, and a value embedding network to respectively output a location map, a plurality of key markers, and a plurality of value embedding for each image frame. An image frame having a closest similarity value from among the plurality of key markers is selected from among the plurality of different image frames, and a super-resolution image frame is generated at a target time step according to an index query marker, a value embedding of the selected frame, and a function of the target image frame.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Super-resolution techniques aim to restore a high-resolution (HR) image frame from its low-resolution (LR) image frame. For example, video super-resolution techniques attempt to discover detailed textures from various frames in an LR image sequence, which can be used to restore the target frame and enhance the video quality. However, processing large sequences of images can be challenging. Therefore, there is a technical challenge to utilize information from distant frames to increase the resolution of the target frame. Summary of the invention

[0002] To address this challenge, according to one aspect of the present disclosure, a computing system is disclosed, which includes a processor and a memory storing instructions executable by the processor to obtain a sequence of image frames in a video. Each image frame of the sequence corresponds to a time step in a plurality of time steps. A target image frame of the sequence is input into a visual marker embedding network of a trajectory-aware transformer, so that the visual marker embedding network outputs a plurality of query markers. A plurality of different image frames are input into a motion estimation network of the trajectory-aware transformer, so that the motion estimation network outputs a position map for each image frame, the position map indicating a position within the image frame corresponding to an index position within the target image frame, the index position having moved along a trajectory between the image frame and the target image frame. A plurality of different image frames are input into the visual marker embedding network, so that the visual marker embedding network outputs a plurality of key markers. A plurality of different image frames are input into a value embedding network of the trajectory-aware transformer, so that the value embedding network outputs a plurality of value embeddings. For each key marker along the trajectory, the computing system is configured to calculate a similarity value with the query marker at the index position. An image frame having the closest similarity value from among the plurality of key markers is selected from the plurality of different image frames. A super-resolution image frame is generated at a target time step as a function of the query label, the value embedding of the selected frame at a position corresponding to the index position, the closest similarity value, and the target image frame.

[0003] This summary is provided to introduce in simplified form some concepts that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An example of a computing system for generating super-resolution image frames from a sequence of low-resolution images is shown.

[0005] Figure 2 It shows that it can be Figure 1 An example of an image sequence used by a computing system of Figure 1 An example of a super-resolution image output by a computing system.

[0006] Figure 3 It shows that the Figure 1 An example of a system for computing trajectories using a sequence of images and a corresponding position map.

[0007] Figure 4 It shows that it can be Figure 1 Examples of super-resolution image frames generated by a computing system and image frames generated using other methods.

[0008] FIG. 5A to FIG. 5C A flow chart illustrating an example method for generating super-resolution image frames from a sequence of low-resolution images.

[0009] Figure 6 A schematic diagram of an example computing system is shown according to an example embodiment.

[0010] Non-patent literature

[0011] [NPL1] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen ChangeLoy. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPRW, 2019.

[0012] [NPL2] Yapeng Tian, ​​Yulun Zhang, Yun Fu, and Chenliang Xu. TDAN: Temporally-deformable alignment network for video super-resolution. In CVPR, pp. 3360–3369, 2020.

[0013] [NPL3] Sheng Li, Fengxiang He, Bo Du, Lefei Zhang, Yonghao Xu, and Dacheng Tao; Fast spatio-temporal residual network for video super-resolution, in CVPR, pp. 10522–10531, 2019.

[0014] [NPL4] Takashi Isobe, Songjiang Li, Xu Jia, Shanxin Yuan, Gregory Slabaugh, Chunjing Xu, Ya-Li Li, Shengjin Wang, and Qi Tian; Video super-resolution with temporal group attention, in CVPR, pp. 8008–8017, 2020.

[0015] [NPL5] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi; Real-time video super-resolution with spatio-temporal networks and motion compensation, in CVPR, pp. 4778–4787, 2017.

[0016] [NPL6] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown; Frame-recurrent video super-resolution, in CVPR, pp. 6626–6634, 2018.

[0017] [NPL7] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita; Recurrent back-projection network for video super-resolution, in CVPR, pp. 3897–3906, 2019.

[0018] [NPL8] Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li, Shengjin Wang, and Qi Tian; Video super-resolution with recurrent structure-detail network, in ECCV, pp. 645–660, 2020.

[0019] [NPL9] Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, Tao Lu, Xin Tian, ​​and Jiayi Ma; Omniscient video super-resolution, arXiv:2103.15683, 2021.

[0020] [NPL10] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen ChangeLoy; BasicVSR: The search for essential components in video super-resolution and beyond, in CVPR, pp. 4947–4956, 2021.

[0021] [NPL11]Alexey Dosovitskiy,Lucas Beyer,Alexander Kolesnikov,DirkWeissenborn,Xiaohua Zhai,Thomas Unterthiner,Mostafa Dehghani,MatthiasMinderer,Georg Heigold,Sylvain Gelly,et al.; An image is worth 16x16 words: transformers for image recognition at scale,arXiv:2010.11929,2020.

[0022] [NPL12] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo; Learning texture transformer network for image super-resolution, in CVPR, pp. 5791–5800, 2020.

[0023] [NPL13] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool; Video super-resolution transformer, arXiv:2106.06847, 2021.

[0024] [NPL14] Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia; MuCAN: Multi-correspondence aggregation network for video super-resolution, in ECCV, pp. 335–351, 2020.

[0025] [NPL15] Anurag Ranjan and Michael J Black; Optical flow estimation using a spatial pyramid network, in CVPR, pp. 4161–4170, 2017.

[0026] [NPL16] Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, and Jiayi Ma; Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations, in ICCV, pp. 3106–3115, 2019.

[0027] [NPL17] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu; Image super-resolution using very deep residual channel attention networks, in ECCV, pp. 286–301, 2018.

[0028] [NPL18] Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S Huang, and Honghui Shi; Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining, in CVPR, pp. 5690–5699, 202.

[0029] [NPL19] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William TFreeman; Video enhancement with task-oriented flow, IJCV, 127(8): 1106–1125, 2019.

[0030] [NPL20] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim; Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation, in CVPR, pp. 3224–3232, 2018.

[0031] [NPL21] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee; NTIRE 2019 challenge on video deblurring and super-resolution: Dataset and study, in CVPRW, 2019.

[0032] [NPL22] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William TFreeman; Video enhancement with task-oriented flow, IJCV, 127(8): 1106–1125, 2019.

[0033] [NPL 23] Ce Liu and Deqing Sun; On bayesian adaptive video superresolution, IEEE TPAMI, 36(2): 346–360, 2013. DETAILED DESCRIPTION

[0034] As introduced above, super-resolution techniques can be used to output high-resolution (HR) image frames given a sequence of low-resolution (LR) images. For example, video super-resolution (VSR) techniques attempt to construct a sequence of HR image frames by assembling detailed textures found in individual frames in a sequence of LR images. This technique is valuable in many applications, such as video surveillance, high-definition cinematography, and satellite imaging.

[0035] In some examples, VSR methods attempt to align temporal features using adjacent frames (e.g., a sliding window of 5 to 7 frames adjacent to the target frame) as input, either implicitly or explicitly. VSR methods mainly focus on using two-dimensional (2D) or three-dimensional (3D) convolutional neural networks (CNNs), as well as optical flow estimation or deformable convolutions, to design advanced alignment modules and fuse detailed textures from adjacent frames. For example, Enhanced Deformable Video Restoration (EDVR-[NPL1]) and Temporal Deformable Alignment Network (TDAN-[NPL2]) employ deformable convolutions to align adjacent frames and capture features within a sliding window. To exploit complementary information across frames, Fast Spatiotemporal Residual Network (FSTRN-[NPL3]) employs 3D convolutions. Temporal Group Attention (TGA-[NPL4]) divides the input into several groups and incorporates temporal information in a hierarchical manner. To align adjacent frames, VESCPN ([NPL5]) introduces a spatiotemporal sub-pixel convolutional network and combines motion compensation and VSR algorithms. However, since extending the sliding window to include more frames will significantly increase the computational cost, utilizing textures at other time steps with this technique may be challenging, especially from relatively distant frames (e.g., greater than 5 to 7 frames from the target frame).

[0036] In other examples, methods based on recurrent structures use hidden states to convey relevant information in previous frames, rather than aggregating information from adjacent frames. For example, frame recurrent video super-resolution (FRVSR-[NPL6]) uses previous super-resolution (SR) frames to recover subsequent frames. Inspired by back-projection, recurrent back-projection networks (RBPN-[NPL7]) treat each frame as a separate source, which is combined in an iterative refinement framework. Recurrent structure detail networks (RSDN-[NPL8]) divide the input into structural and detailed components, and learn textures using two-stream structure detail blocks. Omniscient video super-resolution (OVSR-[NPL9]), BasicVSR, and Icon-VSR ([NPL10]) fuse bidirectional hidden states from the past and the future for reconstruction. This technique attempts to fully utilize the entire sequence and update the hidden state synchronously by the weights of the reconstruction network. Nevertheless, due to the gradient vanishing problem, recurrent networks lose long-term modeling capabilities to a certain extent.

[0037] In other examples, transformer models are used to model long-term sequences. In the field of computer vision, transformers model the relationship between tags in image-based tasks such as image classification, object detection, image restoration, and image super-resolution. For example, ViT ([NPL11]) expands the image into small blocks as tags for attention to capture long-range relationships in high-level vision. TTSR ([NPL12]) uses texture transformers in low-level vision to search for relevant texture patches from reference images to apply to LR images. In the VSR task, VSR transformers (VSR-T-[NPL13]) and MuCAN ([NPL14]) attempt to use attention mechanisms for aligning different frames. However, due to the heavy computational cost of attention calculations on videos, this mechanism aggregates information from relatively narrow time windows. Therefore, there is a technical challenge to exploit information from time-distance frames for VSR.

[0038] To address this problem, an example involving the use of a trajectory-aware transformer to achieve effective video representation learning for VSR (TTVSR) is disclosed. A motion estimation network is used to formulate a video frame as several pre-aligned trajectories including persistent visual markers. Self-attention is learned on relevant visual markers along the spatiotemporal trajectory for a query marker. Compared to traditional visual transformers, this approach significantly reduces the computational cost and enables the transformer to model long-range features. In addition, and as described in more detail below, a cross-scale feature tokenization module is used to address scale changes that may occur in long-range videos. Experimental results show that TTVSR outperforms other techniques on 4 VSR benchmarks.

[0039] Figure 1 The method for generating super-resolution image frames is shown. In some examples, computing system 102 includes a server computing system (e.g., a cloud-based server or multiple distributed cloud servers). In other examples, computing system 102 may include any other suitable type of computing system. Other examples of suitable computing systems include, but are not limited to, desktop computers and laptop computers. Figure 6 Additional aspects of computing system 102 are described in greater detail.

[0040] The computing system 102 is optionally configured to convert the super-resolution image frame Output to client 104. In some examples, client 104 includes a computing system separate from computing system 102. Some examples of suitable computing systems include, but are not limited to, a desktop computing device, a laptop computing device, or a smart phone. Client 104 may include a processor that executes an application (e.g., a video player or a video conferencing application). Figure 6 Additional aspects of client 104 are described in greater detail.

[0041] To generate super-resolution image frames The computing system 102 is configured to obtain a sequence of image frames in a video. Each image frame of the sequence corresponds to a time step in a plurality of time steps. In some examples, the sequence of image frames includes a video, such as a pre-recorded video, a streaming video, or a video conference. Each image frame of the sequence is a super-resolution image frame. Given a sequence of LR image frames, the computing system 102 is configured to use the LR image frames represented herein as One or more target frames (e.g., Its super-resolution image frame corresponding) HR version (for example, ).

[0042] Figure 2 Another example of an image sequence 202 comprising a plurality of image frames is shown. Figure 2 Also shown are portions of super-resolution images 204 to 208 generated for a box-shaped region 210 in a target frame using TTVSR (204) relative to other methods (MuCAN (206) and Icon-VSR (208)) and a ground truth (GT) HR image 212. Figure 2 As shown in the example in FIG. 2 , in relatively distant frames (eg, frames 57, 61, and 64) tracked by trajectory 216, finer textures are introduced into the super-resolution image from the corresponding frame-shaped region 214. Figure 2As depicted in the example of , the quality of image 204 constructed using TTVSR is more similar to ground truth 212 on a qualitative basis than image 206 and image 208 constructed using MuCAN and Icon-VSR, respectively.

[0043] Reference again Figure 1 , and to generate super-resolution image frames The computing system 102 is configured to convert a target image frame for a target time step of the sequence The visual tag embedding network Φ is input into the trajectory-aware transformer, so that the visual tag embedding network Φ outputs multiple query tags Q. In some examples, the visual tag embedding network Φ is used to extract the query tags Q through a sliding window method. Additional aspects of the visual tag embedding network Φ, including training, are described in more detail below. The query tag Q is represented as

[0044] The computing system 102 is further configured to convert a plurality of different image frames IL R The visual tag embedding network Φ is input into the visual tag embedding network Φ, so that the visual tag embedding network Φ outputs multiple key tags K. The visual tag embedding network Φ is the same network Φ used to generate multiple query tags Q. The use of the same network Φ enables comparison between the query tags Q and the key tags K. In some examples, the visual tag embedding network Φ is used to extract the key tags K by a sliding window method.

[0045] The key token K is represented as

[0046] Multiple different image frames I LR The value embedding network is also fed into the trajectory-aware transformer So that the value is embedded in the network Output multiple value embeddings V. In some examples, the value embedding network is used to extract the value embedding V through a sliding window method. Value Embedding Network Additional aspects of , including training, are described in more detail below. The value embedding V is denoted as

[0047] In some examples, computing system 102 is configured to span image feature tags (e.g., q, k, and v) across scales. In long-range videos, complex motion may be accompanied by changes in scale. It will be appreciated that textures from larger scales can help recover textures at smaller scales. Thus, spanning scales allows for extraction of tags from multiple scales.

[0048] In some examples, to extract the token, consecutive expand and collapse operations are used to expand the receptive field of the feature. Second, features from different scales are scaled to the same scale through a pooling operation. Third, features are split through an expansion operation to obtain the output token. This process can extract features from larger scales while maintaining the size of the output token, which simplifies attention calculation and token integration.

[0049] As mentioned above, along the space-time trajectory Learning self-attention on related visual tags. can be formulated as the trajectory τ i A collection of , where each trajectory τ i is the sequence of coordinates over time, and the trajectory τ i The endpoint and query tag q i The coordinates are associated with:

[0050] (1)

[0051] Here, and represents the trajectory τ at time t i H and W represent the height and width of the feature map, respectively.

[0052] From the perspective of trajectory, the input to the trajectory-aware transformer can also be expressed as Visual markers for alignment:

[0053] (2)

[0054] Some methods that compute the trajectory of an object over space and time (such as feature alignment and global optimization) are time consuming and inefficient. This is especially true for trajectories that are updated over time, where the computational cost can explode. Therefore, and in a potential advantage of the present disclosure, a collection of position graphs is used for trajectory generation, where the position graph is represented as a set of matrices over time. In this way, trajectory generation can be expressed in terms of matrix operations that are computationally efficient and easy to implement in the model described herein.

[0055] In order to obtain the position map, the computing system 102 is configured to convert a plurality of different image frames I LR Input to the motion estimation network H of the trajectory-aware transformer, so that the motion estimation network H for each image frame Output location map The location map Indicates the index position within the image frame and the target image frame The corresponding position The index position Already along the image frame With the target image frame The trajectory between i Move. Equation (3) shows the position diagram An example formulation of , where time is fixed to T for simplicity:

[0056] (3)

[0057] Here, each location map A matrix of positions within the corresponding image frames, each position corresponding to a target index position within the target image frame. As described in more detail below, the position map can be used to calculate the trajectory τ using matrix operations i This allows the generation of trajectories τ in a lightweight and computationally efficient manner relative to the use of other techniques such as feature alignment and global optimization i .

[0058] In location map , the target index position The position of the corresponding element in the matrix is ​​indicated. represents the coordinates at time t in the trajectory ending at (m, n) at time T. Time T is related to the target image frame The time step corresponds to the position diagram. The trajectory τ of equation (1) i The relationship between can also be expressed as:

[0059] (4)

[0060] Here, m∈[1,H] and n∈[1,W].

[0061] Figure 3 Shown for Figure 1 The trajectory τ of the sequence of images at time t is shown in i and the corresponding location map The frame 108 with coordinates (3, 3) at time T includes the target image frame Therefore, for the target image frame Initialize the location map Make location map Evaluates to (3, 3) at position (3, 3).

[0062] At time t, located in the target image frame The index feature at (3, 3) in has moved relative to the field of view of the image frame to the second frame 110 with coordinates (4, 3). Therefore, the position map at time t Evaluates to (4, 3) at position (3, 3), where position (3, 3) of the position map represents the target index position of the index feature at time T Similarly, at time 1, the target image frame The index feature at (3, 3) in the position map is located in the third box 112 with coordinates (5, 5). At position (3, 3) it is evaluated to (5, 5). Advantageously, the index feature along the trajectory τ can be determined by reading the position map at a position corresponding to the position of the index feature at time T. i location.

[0063] When moving from time T to time T+1, the new position graph at time T+1 is initialized. Based on equation (4), represents the coordinates of the trajectory ending at (m, n) at time T+1, which can be expressed as:

[0064] (5)

[0065] Existing location map is updated accordingly.

[0066] In order to establish the connection between the trajectory at time T and time T+1, the motion estimation network H calculates arrive Backward flow O T+1 . The process can be formulated as:

[0067] (6)

[0068] Here, H is a motion estimation network with parameters θ and an average pooling operation. Average pooling is used to ensure that the output of the motion estimation network is consistent with Same size.

[0069] In some examples, the motion estimation network H comprises a neural network. An example of a suitable neural network includes, but is not limited to, a spatial pyramid network such as SPYNET. As described in more detail below, the neural network is configured to output an optical flow (e.g., O) between a runtime input image frame and a successive runtime input image frame. T+1 The optical flow output by the spatial pyramid network indicates the motion of image features (e.g., objects, edges, or patches comprising parts of image frames) between a runtime input image frame and a successive runtime input image frame. T+1 The established spatial correlation can track the position map backward from time T+1 to time T Since the correlations in the stream can be floating point numbers, the position map can be obtained by interpolating between adjacent coordinates. The updated coordinates in .

[0070] (7)

[0071] Here, S represents a spatial sampling matrix operation that can be integrated with the motion estimation network H. The spatial sampling matrix operation is configured to transform the coordinates in the position map corresponding to the consecutive runtime input image frames to generate an updated position map corresponding to the runtime input image frames based on the optical flow. An example of a suitable spatial sampling matrix operation includes, but is not limited to, grid sampling in PYTORCH provided by Meta Platforms, Inc. of Menlo Park, California. T+1 In the matrix Executing S on results in an updated position graph for time T+1. Therefore, and in one potential advantage of the present disclosure, the trajectory can be efficiently computed and maintained by a parallel matrix operation (e.g., operation S)

[0072] Compared to the traditional attention mechanism that takes a weighted sum of time keys, the track-aware attention module uses hard attention to select the most relevant markers along the track. This can reduce the ambiguity introduced by the weighted sum method. As described in more detail below, soft attention is used to generate confidences for relevant patches. This can reduce the influence of irrelevant markers. The following paragraphs provide example formulations for hard attention and soft attention calculations.

[0073] To calculate hard attention and soft attention, Figure 1 The computing system 102 is configured to calculate the i Each key tag Calculates the query token at the index position In some examples, as well as Figure 1 As depicted in , computing the similarity value includes computing the query token at the index position Along the trajectory τ i Each key tag The cosine similarity value between . The calculation process can be formulated as:

[0074] (8)

[0075] Here, and Represent the results of hard attention and soft attention respectively. Hard attention operation is configured to select from a plurality of different image frames having from a plurality of key markers The image frame having the closest similarity value among the query tags. The closest similarity value may be the highest or maximum similarity value among the multiple key tags. The image frame includes the query tag The most similar texture, and therefore the most relevant image frame in the sequence, is used to obtain information for reconstructing the portion of the target image frame represented by the query key. The selected image frame may be dedicated to the selected query tag within the target frame. It will also be appreciated that the frame with the most similar key may be different for different query tags within the target frame.

[0076] After the attention calculation, the computing system 102 is further configured to generate a value embedding of a frame selected at a position corresponding to the index position according to the query tag at the target time step The closest (e.g., largest) similarity value and the target image frame Function to generate super-resolution image frames Super-resolution image frames A function is generated based on all query tokens in the target image frame, the value embedding of the selected frame at the position corresponding to the index position of each query token, and the closest similarity value at the target time step. th HR Frame The process can also be expressed as:

[0077] (9)

[0078] Here, T traj A represents the trajectory-aware transformer. traj represents track-aware attention. R represents a reconstruction network followed by a pixel shuffling layer, which is operably configured to resize the feature map to a desired size. U represents an upsampling operation (e.g., a bicubic upsampling operation). Figure 1 An example architecture of a trajectory-aware attention module (including example values ​​of q, k, and v) is shown, which is followed by a reconstruction network R and an upsampling operation U, which generates an HR frame ⊙ and Indicates multiplication and element-wise addition, respectively. τ i instruct By introducing the trajectory into the transformer in TTVSR, the computational overhead of attention calculation can be significantly reduced because the transformer in TTVSR can avoid spatial dimension calculation compared with ordinary visual transformers.

[0079] Based on Equation (8), the attention calculation in Equation (9) can be formulated as:

[0080] (10)

[0081] Here, the trajectory-aware attention result A traj Based on query tag In the query tag The index position corresponding to the trajectory τ i The selected frame at the position The value of embedding and the closest (e.g., largest) similarity value In the example of equation (10), the query tag Similarity value and the value of the selected frame is embedded The operator ⊙ represents multiplication. C represents the join operation. As introduced above, the soft attention value (close to, for example, the maximum similarity value) ) Attention result A traj The weighting is performed to reduce the influence of less relevant tags that have relatively low similarity values ​​when compared to the query tag, while increasing the contribution of tags that are closer to the query tag. Overall, features from the entire image sequence are integrated in the track-aware attention. This allows focusing the attention computation along the spatiotemporal track, thereby alleviating the computational cost.

[0082] As described above, the trajectory-aware attention result is output to the image reconstruction network R, so that the image reconstruction network outputs an image feature map for the super-resolution image frame. In addition, in some examples, the computing system 102 is configured to upsample the target image frame and map the output of the image reconstruction network to the upsampled target image frame to generate a super-resolution image frame. Since the position map in equation (4) is the trajectory τ in equation (9) i The interchangeable formulation of , so TTVSR can also be expressed as:

[0083] (11)

[0084] Here, m∈[1,H], n∈[1,W], and t∈[1,T-1]. In this formulation, the coordinate system in the transformer is transformed from the coordinate system defined by the trajectory to a set of aligned matrices (e.g., a position map). This design has two advantages: First, the position map provides a more efficient way to enable TTVSR to directly utilize information from distant video frames. Second, since trajectory is a widely used concept in video, the methods and devices disclosed in this article can be applied to increase the efficiency and capabilities of other video tasks.

[0085] The following paragraphs provide information on Figure 1 In some examples, the image reconstruction network R, the visual label embedding network Φ used to generate the query label Q and the key label K, and the value embedding network Φ used to generate the value embedding V In some such examples, during a training phase, computing system 102 is configured to receive training data comprising a training sequence of low-resolution image frames as input and a corresponding sequence of high-resolution image frames as ground truth output. Computing system 102 is also configured to train a visual label embedding network Φ, a value embedding network Φ on the training data. and an image reconstruction network R to output a runtime super-resolution image frame based on a runtime input image sequence (e.g., ). Therefore, and in one potential advantage of the present disclosure, the visual label embedding network Φ and the value embedding network φ are trained together, as opposed to training these networks independently. And the image reconstruction network R can lead to higher resolution output and reduced training time.

[0086] In an example where the motion estimation network H comprises a neural network, the computing system 102 is configured to train the neural network by receiving training data during a training phase, the training data comprising a training sequence of image frames as input, and a ground truth optical flow between image frames in the training sequence as a ground truth output. The neural network is trained on the training data to output an optical flow (e.g., O) between a runtime input image frame and a successive runtime input image frame. T+1 ). In this way, the neural network is trained to output a representation of the motion of an object between consecutive image frames.

[0087] In some examples, training the neural network includes obtaining a neural network pre-trained for motion estimation (e.g., SPYNET), and fine-tuning the pre-trained neural network. Fine-tuning the pre-trained neural network can be less computationally demanding than training the neural network from scratch, and the fine-tuned neural network can outperform a randomly initialized neural network.

[0088] In order to utilize the entire sequence, a bidirectional propagation scheme is adopted, in which features in different frames can be back-propagated and forward-propagated separately. In order to reduce the consumption in time and memory, visual markers of different scales are generated from different frames. Features from adjacent frames are finer, so markers of size 1×1 are generated. Features from long distances are coarser, so these frames are selected at a certain time interval, and markers of size 4×4 are generated. Kernels of sizes 4×4, 6×6, and 8×8 are used for cross-scale feature tokenization. During training, a cosine annealing scheme and an Adam optimizer with β1=0.9 and β2=0.99 are used. The learning rates for motion estimation and other parts are set to 1.25×10 -5 and 2×10 -4 The batch size is set to 8, and the input patch size is set to 64×64. For comparison purposes, the training data is augmented with random horizontal flipping, vertical flipping, and 90-degree rotation. To achieve long-range sequence capability, sequences with length 50 are used as input. The Charbonnier penalty loss is applied to the ground truth image I HR and the restored SR frame I SR All frames between To stabilize the training of TTVSR, the weights of the motion estimation module are fixed in the first 5K (5 thousand) iterations and then made trainable. The total number of iterations is 400K (400,000).

[0089] The following paragraphs provide additional details about an example implementation of TTVSR. TTVSR is evaluated and compared in performance with other methods on two datasets: REDS ([NPL21]) and VIMEO-90K ([NPL22]). REDS contains a total of 300 video sequences, of which 240 video sequences are used for training, 30 video sequences are used for validation, and 30 video sequences are used for testing. Each sequence contains 100 frames with a resolution of 720×1280. In order to create training and test sets, four sequences are selected as the test set, which is called "REDS4". The training and validation sets are selected from the remaining 266 sequences. VIMEO VIMEO-90K contains 64612 sequences for training and 7824 sequences for testing. Each sequence contains 7 frames with a resolution of 448×256. For comparison purposes, TTVSR was evaluated using two degradations using 4× downsampling: 1) bicubic downsampling (hereinafter “BI”) in MATLAB provided by MathWorks, Natick, Massachusetts, and 2) Gaussian filter and downsampling with σ=1.6 standard deviation (hereinafter “BD”). BI degradation was applied to REDS4, and BD degradation was applied to VIMEO VIMEO-90K-T, Vid4 ([NPL23]), and UDM10 ([NPL16]). Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation metrics.

[0090] TTVSR is compared with 15 other methods. These methods can be summarized into three categories: single image super-resolution (SISR), sliding window-based methods, and recurrent structure-based methods. To facilitate comparison, the corresponding performance parameters are obtained from the original public release associated with each technique, or the results are reproduced using the original formally released model.

[0091] The proposed TTVSR technique described in this paper is compared with other SOTA methods on the REDS dataset. As shown in Table 1, the method is classified according to the frame used in each reasoning. Among these methods, the performance of the SISR method is relatively low due to the use of one LR frame. MuCAN and VSR-T use an attention mechanism in the sliding window, which leads to a significant improvement in performance compared to the SISR method. However, they do not fully utilize all the texture information available in the sequence. BasicVSR and IconVSR attempt to model the entire sequence through hidden states. Nevertheless, gradient disappearance poses a challenge for long-term modeling, resulting in information loss at a certain distance. In contrast, TTVSR links related visual markers together along the same trajectory in an effective way. TTVSR also uses the entire sequence to restore the lost texture. Therefore, TTVSR achieves a result of 32.12dB PSNR, and significantly outperforms Icon-VSR 0.45dB (PSNR) on REDS4. This proves the ability of TTVSR in long-range modeling.

[0092] Table 1. Quantitative comparison (PSNR↑ and SSIM↑) on the REDS4 dataset for 4× video super-resolution. Results are tested on RGB channels. The two strongest performing models are underlined. #frames indicates the number of input frames used to perform inference, and “r” indicates that a recurrent structure is employed.

[0093]

[0094]

[0095] *[NPL17].**[NPL18].***[NPL19]. [NPL20].

[0096] In order to further verify the generalization ability of TTVSR, TTVSR is trained on the VIMEO-90K dataset, and the results are evaluated on the Vid4 dataset, the UDM10 dataset, and the VIMEO-90K-T dataset. As shown in Table 2, on the Vid4 test set, the UDM10 test set, and the VIMEO-90K-T test set, TTVSR achieved results of 28.40dB, 40.41dB, and 37.92dB in PSNR, respectively, which is better than other methods (results). Specifically, TTVSR outperforms Icon-VSR by 0.36dB (PSNR) and 0.38dB (PSNR) on the Vid4 dataset and the UDM10 dataset, respectively. At the same time, it is noted that compared with the evaluation on VIMEO-90K-T with 7 frames in each test sequence, TTVSR outperforms other methods by a larger margin on datasets with at least 30 frames per video. These results verify that TTVSR has strong generalization ability and is good at modeling information in long-range sequences.

[0097] Table 2. Quantitative comparison (PSNR↑ and SSIM↑) of the Vid4 dataset, UDM10 dataset, and VIMEO-90K-T dataset for 4× video super-resolution. B C R All results are calculated on the Y channel of the color space. The two strongest performing models are underlined.

[0098]

[0099]

[0100] To further compare the visual quality of different methods, Figure 4 Visual results generated by TTVSR and other methods on four different test sets are shown. For comparison purposes, the original publicly available SR images published for a given method are obtained, or the results are generated using officially released models. It can be observed that TTVSR greatly improves the visual quality relative to other methods, especially for areas with detailed textures. For example, in Figure 4 In the fourth row of , TTVSR recovers more stripe details from the stonework in the painting. These results verify that TTVS can exploit textures from relevant markers to produce more refined results than other methods.

[0101] In many applications, model size is balanced for computational cost. To avoid disparities between devices using different hardware, two hardware-independent metrics are used, including the number of parameters (#Param) and floating-point operations per second (FLOP). As shown in Table 3, FLOP is calculated using LR input of size 180×320 and ×4 upsampling setting. Compared to IconVSR, TTVSR achieves higher performance while maintaining similar #Params and FLOP. In addition, TTVSR is more lightweight than MuCAN, which is another attention-based method. The improved performance mainly benefits from the use of trajectories in the attention calculation, which significantly reduces the computational cost.

[0102] Table 3. Comparison of parameters, FLOPs, and numbers. FLOPs are calculated on one LR frame with size 180×320 and ×4 upsampling on the REDS4 dataset.

[0103]

[0104] Reference now FIG. 5A to FIG. 5C , a flowchart depicting an example method 500 for generating a super-resolution image frame from a sequence of low-resolution image frames is illustrated. Figures 1 to 4 and Figure 6 The following description of method 500 is provided with reference to the software components and hardware components shown in FIG. Figures 1 to 4 and Figure 6 The method steps in method 500 are described in detail with reference to the corresponding parts of the method 500. It will be understood that method 500 may also be performed in other contexts using other suitable hardware components and software components.

[0105] It will be understood that the following description of method 500 is provided by way of example, and is not meant to be limiting. It will be understood that various steps of method 500 may be omitted or performed in a different order than described, and that method 500 may include additional steps relative to the method 500 without departing from the scope of the present disclosure. FIG. 5A to FIG. 5C Additional steps and / or alternative steps to those shown in .

[0106] First reference Figure 5A , method 500 includes steps performed in a training phase 502 and a runtime phase 504. As introduced above, in some examples, a motion estimation network (e.g., Figure 1The motion estimation network (F) includes a neural network. Therefore, during the training phase 502, the method 500 may include receiving training data at 506. The training data includes a training sequence of image frames as input, and a ground truth optical flow between image frames in the training sequence as a ground truth output. Step 506 also includes training the neural network on the training data to output an optical flow between a runtime input image frame and a consecutive runtime input image frame. In this way, the neural network is trained to output a representation of the motion of an object between consecutive image frames.

[0107] In some examples, during the training phase 502, method 500 includes, at 508, training a visual marker embedding network (e.g., Figure 1 Visual tag embedding network Φ), value embedding network (e.g., Figure 1 The value of embedding network ), and image reconstruction networks (e.g., Figure 1 To train these networks, step 508 includes receiving training data comprising a training sequence of low-resolution image frames as input and a corresponding sequence of high-resolution image frames as ground truth output. The visual label embedding network, the value embedding network, and the image reconstruction network are trained on the training data to output a runtime super-resolution image frame (e.g., ). In some examples, training one or more of the networks together can result in higher resolution output and reduced training time relative to training each network independently.

[0108] In the run-time phase, at 510, method 500 includes obtaining a sequence of low-resolution image frames, wherein each image frame of the sequence corresponds to a time step of a plurality of time steps. Given the sequence of low-resolution image frames, method 500 is configured to use the image frames from one or more different image frames. The restored image texture is used to generate one or more target frames (e.g., ) (e.g., ).

[0109] At 512, method 500 includes inputting a target image frame for a target time step of the sequence into a visual tag embedding network of a track-aware transformer, so that the visual tag embedding network outputs a plurality of query tags. For example, Figure 1 The visual tag embedding network Φ is used to embed the target image frame Extract the query tag Q. As described above, the query tag Q is compared with a plurality of key tags K extracted from a plurality of different image frames to identify relevant textures from the different image frames, which can be assembled to generate a super-resolution frame

[0110] Reference now Figure 5B , the method 500 further includes: at 514, inputting the plurality of different image frames into a motion estimation network of the trajectory-aware transformer, so that the motion estimation network outputs a position map for each image frame, the position map indicating a position within the image frame corresponding to an index position within the target image frame, the index position having moved along the trajectory between the image frame and the target image frame. For example, the computing system 102 is configured to input the plurality of different image frames I LR The input is sent to the motion estimation network H, which estimates the motion of each image frame. Generate location map Relative to the use of other techniques such as feature alignment and global optimization, the position graph enables trajectories to be computed in a lightweight and computationally efficient manner.

[0111] In some examples, at 516, the motion estimation network includes a neural network and a spatial sampling matrix operation. In such examples, the method also includes receiving an optical flow indicating the motion of an object between a runtime input image frame and a consecutive runtime input image frame from the neural network. The spatial sampling operation is performed to transform the coordinates in the position map corresponding to the consecutive runtime input image frames, thereby generating an updated position map corresponding to the runtime input image frame based on the optical flow. For example, Figure 1 The motion estimation network H is configured to output the optical flow O T+1 , which can sample the optical flow using grid sampling in PYTORCH to generate an updated position map *L t In this way, a position map can be generated using simple matrix operations.

[0112] At 518, method 500 includes inputting a plurality of different image frames into a visual marker embedding network, causing the visual marker embedding network to output a plurality of key markers. As described above, the visual marker embedding network Φ is the same network Φ used to generate a plurality of query markers Q. This enables the query markers Q to be directly compared with the key markers K to identify the key markers K used to generate the super-resolution frame. Related textures.

[0113] The method 500 further includes: at 520, inputting a plurality of different image frames into a value embedding network of the trajectory-aware transformer, so that the value embedding network outputs a plurality of value embeddings. Figure 1 The computing system 102 is configured to convert the image frame I LR Input to the value embedding network In order to generate the value embedding V. The value embedding V includes the value embedded in the HR image frame. characteristics.

[0114] At 522, method 500 includes: for each key tag along the track, calculating a similarity value with the query tag at the index position. For example, Figure 1 The computing system 102 is configured to compare the query token with the key token to respectively calculate the hard attention and soft attention hard attention The most relevant image frame in the sequence is selected to be used to reconstruct the queried portion of the target image frame. is used to weight the influence of a tag by its relevance to the queried portion of the target image frame.

[0115] Reference now Figure 5C The method 500 further includes: at 524, selecting an image frame having a closest (eg, largest) similarity value from among the plurality of key tags from the plurality of different image frames. The image frame includes a similarity value to the query tag The most similar texture, and therefore the most relevant image frame in the sequence, is used to obtain information for reconstructing the portion of the target image frame represented by the query key.

[0116] At 526, method 500 also includes generating a super-resolution image frame at the target time step based on the query token, the value embedding of the selected frame at the position corresponding to the index position, the closest (e.g., maximum) similarity value, and the function of the target image frame. For example, Figure 1 The computing system 102 is configured to query the With query tag The index position corresponding to the trajectory τ i The selected frame at the position The value of embedding The closest (e.g., largest) similarity value and the target image frame To generate HR frames

[0117] In some examples, at 528, generating the super-resolution image frame includes generating a track-aware attention result based on the query tag, the value embedding of the selected frame at a location along the track corresponding to the index location of the query tag, and the closest (e.g., largest) similarity value. Value Embedding and the closest (e.g., largest) similarity value To generate the trajectory-aware attention result A of equation (10) trajThe track-aware attention result is output to an image reconstruction network (e.g., reconstruction network R), so that the image reconstruction network outputs an image feature map for a super-resolution image frame. In some examples, the output of the image reconstruction network is mapped to an upsampled target image frame to generate a super-resolution image frame.

[0118] The systems and methods described above can be used to generate super-resolution image frames from a sequence of low-resolution image frames. Trajectories are introduced into the transformer model to reduce the computational overhead of generating super-resolution image frames by calculating the attention on a subset of key tags aligned with the query tag along the trajectory. This enables the computing device to avoid consuming resources on less relevant parts of the image frame. Additionally, the position map is used to generate the trajectory using lightweight and efficient matrix operations. Compared with other techniques (such as feature alignment and global optimization), this enables the trajectory to be generated in a less intensive manner. Additionally, the systems and methods described above can outperform other systems and methods at least on video sequence datasets in video super-resolution applications.

[0119] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as a computer application or service, an application program interface (API), a library, and / or other computer program products.

[0120] Figure 6 An example of a computing system 600 in which one or more of the devices and methods described above may be implemented is schematically shown. The computing system 600 is shown in simplified form. The computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), wearable computing devices (such as smart watches and head-mounted augmented reality devices), and / or other computing devices. In some examples, the computing system 600 may embody Figure 1 computing system 102 and / or client 104 .

[0121] The computing system 600 includes a logic processor 602, a volatile memory 604, and a non-volatile storage device 606. The computing system 600 may optionally include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or Figure 6 Other components not shown.

[0122] Logical processor 602 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise produce a desired result.

[0123] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic instructions or firmware instructions. The processor of the logical processor 602 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. The individual components of the logical processor may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for collaborative processing. Aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it will be understood that these virtualization aspects are run on different physical logical processors of various different machines.

[0124] The non-volatile storage device 606 includes one or more physical devices that are configured to store instructions executable by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 606 can be transformed—for example, to store different data.

[0125] The non-volatile storage device 606 may include removable and / or built-in physical devices. The non-volatile storage device 606 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technology. The non-volatile storage device 606 may include a non-volatile device, a dynamic device, a static device, a read / write device, a read-only device, a sequential access device, a location addressable device, a file addressable device, and / or a content addressable device. It will be understood that the non-volatile storage device 606 is configured to save instructions even when the power to the non-volatile storage device 606 is cut off.

[0126] The volatile memory 604 may include a physical device that includes random access memory. The volatile memory 604 is typically utilized by the logical processor 602 to temporarily store information during the processing of software instructions. It will be understood that when power to the volatile memory 604 is cut off, the volatile memory 604 typically does not continue to store instructions.

[0127] Aspects of logic processor 602, volatile memory 604, and non-volatile storage device 606 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program-specific integrated circuits and application-specific integrated circuits (PASIC / ASIC), program-specific standard products and application-specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0128] The terms "module" and "program" may be used to describe aspects of a computing system 600 that is typically implemented in software by a processor to perform a specific function using portions of volatile memory that involves transformative processing that specifically configures the processor to perform that function. Thus, a module or program may be instantiated via a logical processor 602 that executes instructions stored by a non-volatile storage device 606 using portions of volatile memory 604. It will be understood that different modules and / or programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module and / or program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module" and "program" may cover individuals or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0129] When including a display subsystem 608, the display subsystem 608 can be used to present a visual representation of the data stored by the non-volatile storage device 606. The visual representation can take the form of a GUI. As described herein, the methods and processes change the data stored by the non-volatile storage device, and thus transform the state of the non-volatile storage device, and can also transform the state of the display subsystem 608 to visually represent the changes in the underlying data. The display subsystem 608 may include one or more display devices utilizing virtually any type of technology. Such a display device can be combined with the logical processor 602, the volatile memory 604, and / or the non-volatile storage device 606 in a shared enclosure, or such a display device can be a peripheral display device.

[0130] When an input subsystem 610 is included, the input subsystem 610 may include or interface with one or more user input devices, such as a keyboard, a mouse, a touch screen, or a game controller. In some examples, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be processed on-board or off-board. Example NUI components may include microphones for speech recognition and / or voice recognition; infrared cameras, color cameras, stereo cameras, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0131] When included, the communication subsystem 612 can be configured to communicatively couple the various computing devices described herein to each other, as well as to communicatively couple to other devices. The communication subsystem 612 can include wired communication devices and / or wireless communication devices compatible with one or more different communication protocols. For example, the communication subsystem can be configured for communication via a wireless telephone network, or a wired or wireless local area network or wide area network. In some examples, the communication subsystem can allow the computing system 600 to send messages to other devices and / or receive messages from other devices via a network (such as the Internet).

[0132] The following paragraphs provide additional support for the claims of the present application. On the one hand, a computing system is provided, the computing system comprising: a processor; and a memory storing instructions executable by the processor to obtain a sequence of image frames in a video, wherein each image frame of the sequence corresponds to a time step in a plurality of time steps; input a target image frame for a target time step of the sequence into a visual tag embedding network of a trajectory-aware transformer, so that the visual tag embedding network outputs a plurality of query tags; input a plurality of different image frames into a motion estimation network of the trajectory-aware transformer, so that the motion estimation network outputs a position map for each image frame, the position map indicating a position within the image frame corresponding to an index position within the target image frame, the index position having been moved along the image frame and a target image frame; inputting a plurality of different image frames into a visual marker embedding network so that the visual marker embedding network outputs a plurality of key markers; inputting a plurality of different image frames into a value embedding network of a trajectory-aware transformer so that the value embedding network outputs a plurality of value embeddings; for each key marker along the trajectory, calculating a similarity value with a query marker at an index position; selecting an image frame having the closest similarity value from among the plurality of key markers from the plurality of different image frames; and generating a super-resolution image frame at a target time step as a function of the query marker, the value embedding of the selected frame at a position corresponding to the index position, the closest similarity value, and the target image frame. In this aspect, the motion estimation network additionally or alternatively includes a neural network, and the instructions additionally or alternatively executable to: receive training data during a training phase, the training data including a training sequence of image frames as input, and a ground truth optical flow between image frames in the training sequence as a ground truth output; and train the neural network on the training data to output an optical flow between a runtime input image frame and consecutive runtime input image frames. In this aspect, the instructions executable to train the neural network additionally or alternatively include instructions executable to fine-tune a pre-trained neural network. In this aspect, the neural network additionally or alternatively includes a spatial pyramid network. In this aspect, the motion estimation network additionally or alternatively includes a neural network and a spatial sampling matrix operation, wherein the neural network is configured to output an optical flow indicating motion of image features between a runtime input image frame and a consecutive runtime input image frame; and wherein the spatial sampling matrix operation is configured to transform coordinates in a position map corresponding to the consecutive runtime input image frames, thereby generating an updated position map corresponding to the runtime input image frame based on the optical flow.In this aspect, the instructions are additionally or alternatively executable to generate a track-aware attention result based on a query tag, a value embedding of a selected frame at a position along the track corresponding to an index position of the query tag, and a closest similarity value; and output the track-aware attention result to an image reconstruction network, so that the image reconstruction network outputs an image feature map for a super-resolution image frame. In this aspect, the instructions are additionally or alternatively executable to upsample a target image frame and map the output of the image reconstruction network to the upsampled target image frame to generate a super-resolution image frame. In this aspect, the instructions are additionally or alternatively executable to, during a training phase: receive training data, the training data comprising a training sequence of low-resolution image frames as input, and a corresponding sequence of high-resolution image frames as ground truth outputs; and train a visual tag embedding network, a value embedding network, and an image reconstruction network on the training data to output a runtime super-resolution image frame based on a runtime input image sequence. In this aspect, instructions executable to generate a track-aware attention result additionally or alternatively include instructions executable to join a query marker with a similarity value and a product of a value embedding of a selected frame at a position along the track corresponding to an index position of the query marker. In this aspect, the sequence of image frames additionally or alternatively includes a pre-recorded video, a streaming video, or a video conference. In this aspect, instructions additionally or alternatively are executable to output a super-resolution image frame to a client. In this aspect, instructions executable to calculate a similarity value additionally or alternatively are executable to include instructions executable to calculate a cosine similarity value between a query marker at an index position and each key marker along the track. In this aspect, each position map additionally or alternatively includes a matrix of positions within a corresponding image frame, each position corresponding to a target index position within a target image frame, and wherein the target index position is indicated by the position of the corresponding position element in the matrix. In this aspect, instructions additionally or alternatively are executable to label image features across scales.

[0133] In another aspect, a method for generating a super-resolution image frame from a sequence of low-resolution image frames at a computing system is provided, the method comprising: obtaining a sequence of low-resolution image frames, wherein each image frame of the sequence corresponds to a time step of a plurality of time steps; inputting a target image frame for a target time step of the sequence into a visual tag embedding network of a trajectory-aware transformer, so that the visual tag embedding network outputs a plurality of query tags; inputting a plurality of different image frames into a motion estimation network of the trajectory-aware transformer, so that the motion estimation network outputs a position map for each image frame, the position map indicating a position within the image frame corresponding to an index position within the target image frame, the index position having been moved along the image frame and the target time step. The method comprises: providing a method for detecting a super-resolution image frame by detecting a trajectory of a target image frame; inputting a plurality of different image frames into a visual marker embedding network so that the visual marker embedding network outputs a plurality of key markers; inputting a plurality of different image frames into a value embedding network of a trajectory-aware transformer so that the value embedding network outputs a plurality of value embeddings; for each key marker along the trajectory, calculating a similarity value with a query marker at an index position; selecting an image frame having the closest similarity value from among the plurality of key markers from the plurality of different image frames; and generating a super-resolution image frame at a target time step according to a function of the query marker, the value embedding of the frame selected at a position corresponding to the index position, the closest similarity value, and the target image frame. In this aspect, the motion estimation network additionally or alternatively includes a neural network and a spatial sampling matrix operation, and the method additionally or alternatively includes: receiving an optical flow indicating the motion of an object between a runtime input image frame and a consecutive runtime input image frame from the neural network; and performing a spatial sampling operation to transform coordinates in a position map corresponding to the consecutive runtime input image frames, thereby generating an updated position map corresponding to the runtime input image frame based on the optical flow. In this aspect, the motion estimation network additionally or alternatively includes a neural network, and the method additionally or alternatively includes, during a training phase: receiving training data, the training data including a training sequence of image frames as input, and ground truth optical flow between image frames in the training sequence as ground truth output; and training the neural network on the training data to output optical flow between runtime input image frames and consecutive runtime input image frames. The method additionally or alternatively includes generating a track-aware attention result based on a query tag, a value embedding of a selected frame at a position along the track corresponding to an index position of the query tag, and a closest similarity value; and outputting the track-aware attention result to an image reconstruction network, so that the image reconstruction network outputs an image feature map for a super-resolution image frame.The method additionally or alternatively includes, during training: receiving training data, the training data comprising a training sequence of low-resolution image frames as input, and a corresponding sequence of high-resolution image frames as ground truth output; and training a visual label embedding network, a value embedding network, and an image reconstruction network on the training data to output runtime super-resolution image frames based on the runtime input image sequence.

[0134] On the other hand, a computing system is provided, the computing system comprising: a processor; and a memory storing instructions executable by the processor to obtain a sequence of image frames including a video conference, wherein each image frame of the sequence corresponds to a time step of a plurality of time steps; input a target image frame for a target time step of the sequence into a visual tag embedding network of a trajectory-aware transformer, so that the visual tag embedding network outputs a plurality of query tags; input a plurality of different image frames into a motion estimation network of the trajectory-aware transformer, so that the motion estimation network outputs a position map for each image frame, the position map indicating a position within the image frame corresponding to an index position within the target image frame, the index position having been moved along the image frame to the target image frame moving along a trajectory between; inputting a plurality of different image frames into a visual tag embedding network so that the visual tag embedding network outputs a plurality of key tags; inputting a plurality of different image frames into a value embedding network of a trajectory-aware transformer so that the value embedding network outputs a plurality of value embeddings; for each key tag along the trajectory, calculating a similarity value with a query tag at an index position; selecting an image frame having a closest similarity value from among the plurality of key tags from the plurality of different image frames; generating a super-resolution image frame at a target time step according to a function of the query tag, the value embedding of the frame selected at the position corresponding to the index position, the closest similarity value, and the target image frame; and outputting the super-resolution image frame to a client.

[0135] It will be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because numerous variations are possible. Specific routines or methods described herein can represent one or more processing strategies in any number of processing strategies. Therefore, the illustrated and / or described various actions can be executed in the illustrated and / or described order, in other orders, in parallel, or omitted. Similarly, the order of the process described above can be changed.

[0136] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

[0137] Additionally, it will be understood that the terms "includes," "including," "has," "contains," variations thereof, and other similar words used in the detailed description or claims are intended to be inclusive in a manner similar to the term "comprising" as an open transitional word and do not exclude any additional or other elements.

Claims

1. A computing system comprising: processor; as well as A memory storing instructions executable by the processor to: Obtaining a sequence of image frames in a video, wherein each image frame of the sequence corresponds to a time step in a plurality of time steps; inputting a target image frame for a target time step of the sequence into a visual tag embedding network of a trajectory-aware transformer, such that the visual tag embedding network outputs a plurality of query tags; inputting a plurality of different image frames into a motion estimation network of the trajectory-aware transformer, such that the motion estimation network outputs, for each image frame, a position map indicating positions within the image frame corresponding to index positions within the target image frame, the index positions having moved along a trajectory between the image frame and the target image frame; inputting the plurality of different image frames into the visual tag embedding network so that the visual tag embedding network outputs a plurality of key tags; inputting the plurality of different image frames into a value embedding network of the trajectory-aware transformer, so that the value embedding network outputs a plurality of value embeddings; For each key tag along the trajectory, calculating a similarity value with the query tag at the index position; selecting, from the plurality of different image frames, an image frame having a closest similarity value from among the plurality of key markers; as well as A super-resolution image frame is generated at the target time step based on the query token, the value embedding of the selected frame at the position corresponding to the index position, the closest similarity value, and a function of the target image frame.

2. The computing system of claim 1 , wherein the motion estimation network comprises a neural network, and wherein the instructions are further executable to, during a training phase: receiving training data, the training data comprising: A training sequence of image frames as input, and outputting as ground truth a ground truth optical flow between image frames in the training sequence; and The neural network is trained on the training data to output an optical flow between a runtime input image frame and consecutive runtime input image frames.

3. The computing system of claim 2, wherein the instructions executable to train the neural network include instructions executable to fine-tune a pre-trained neural network.

4. The computing system of claim 2, wherein the neural network comprises a spatial pyramid network.

5. The computing system of claim 1 , wherein the motion estimation network comprises a neural network and a spatial sampling matrix operation; wherein the neural network is configured to output an optical flow indicating motion of image features between a runtime input image frame and a consecutive runtime input image frame; as well as The spatial sampling matrix operation is configured to transform coordinates in a position map corresponding to the consecutive runtime input image frames, thereby generating an updated position map corresponding to the runtime input image frames based on the optical flow.

6. The computing system of claim 1, wherein the instructions are further executable to: generating a track-aware attention result based on the query tag, the value embedding of the selected frame at the location along the track corresponding to the index location of the query tag, and the closest similarity value; and The trajectory-aware attention result is output to an image reconstruction network, so that the image reconstruction network outputs an image feature map for the super-resolution image frame.

7. The computing system of claim 6, wherein the instructions are further executable to: upsample the target image frame and map the output of the image reconstruction network to the upsampled target image frame to generate the super-resolution image frame.

8. The computing system of claim 6, wherein the instructions are further executable to, during a training phase: receiving training data, the training data comprising: a training sequence of low-resolution image frames as input, and a corresponding sequence of high-resolution image frames as ground truth outputs; and training the visual tag embedding network, the value embedding network, and the image reconstruction network on the training data to output runtime super-resolution image frames based on a runtime input image sequence.

9. The computing system of claim 6, wherein the instructions executable to generate the trajectory-aware attention result include instructions executable to join the query marker with the product of the similarity value and the value embedding of the selected frame at the position along the trajectory corresponding to the index position of the query marker.

10. The computing system of claim 1, wherein the sequence of the image frames comprises a pre-recorded video, a streaming video, or a video conference.

11. The computing system of claim 1, wherein the instructions are further executable to output the super-resolution image frame to a client.

12. The computing system of claim 1, wherein the instructions executable to calculate the similarity value include instructions executable to calculate a cosine similarity value between the query token at the index location and each key token along the trajectory.

13. The computing system of claim 1 , wherein each position map comprises a matrix of positions within a corresponding image frame, each of the positions corresponding to a target index position within the target image frame, and wherein the target index position is indicated by the position of the corresponding position element in the matrix.

14. The computing system of claim 1, wherein the instructions are further executable to label image features across scales.

15. A method for generating a super-resolution image frame from a sequence of low-resolution image frames at a computing system, the method comprising: Obtaining a sequence of the low-resolution image frames, wherein each image frame of the sequence corresponds to a time step in a plurality of time steps; inputting a target image frame for a target time step of the sequence into a visual tag embedding network of a trajectory-aware transformer, such that the visual tag embedding network outputs a plurality of query tags; inputting a plurality of different image frames into a motion estimation network of the trajectory-aware transformer, such that the motion estimation network outputs, for each image frame, a position map indicating positions within the image frame corresponding to index positions within the target image frame, the index positions having moved along a trajectory between the image frame and the target image frame; inputting the plurality of different image frames into the visual tag embedding network so that the visual tag embedding network outputs a plurality of key tags; inputting the plurality of different image frames into a value embedding network of the trajectory-aware transformer, so that the value embedding network outputs a plurality of value embeddings; For each key tag along the trajectory, calculating a similarity value with the query tag at the index position; selecting, from the plurality of different image frames, an image frame having a closest similarity value from among the plurality of key markers; as well as The super-resolution image frame is generated at the target time step based on the query token, the value embedding of the selected frame at the position corresponding to the index position, the closest similarity value, and a function of the target image frame.