High-magnification face video super-resolution processing method based on transformer

Abstract

The application discloses a high-multiple face video super-resolution processing method based on a transformer and belongs to the technical field of video super-resolution. The application comprises the following contents: first, a multi-dimensional feature representation method for multiple channels; second, a time uniformity method based on an inter-frame point-by-point mask attention module; and third, an attention mechanism parameter optimization method based on low-rank decomposition. The application fully utilizes the above-mentioned methods in the processing of information in and between video frames, combines the idea of spatiotemporal consistency of the video super-resolution task, realizes efficient reconstruction of high-multiple face video frames, and effectively improves the overall video super-resolution quality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video super-resolution, and particularly relates to a high-multiplying face video super-resolution processing method based on a Transformer. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, the research in the field of video processing has made significant progress. In particular, in the field of video super-resolution (VSR) technology, improving video quality and details has become one of the research hotspots. Given the increasingly widespread application of face images in social media, surveillance, and video communication scenarios, face video super-resolution technology is particularly important.

[0003] Traditional video super-resolution methods rely heavily on interpolation techniques and filtering algorithms, which can improve video quality to some extent, but often struggle to restore high-frequency details and have limited effectiveness in handling dynamic scenes. In recent years, deep learning-based super-resolution technology has gradually emerged, using models such as convolutional neural networks (CNNs) to effectively capture complex features and structural information in images, thereby achieving more refined image enhancement.

[0004] However, when faced with high-multiplying face video super-resolution tasks, existing deep learning methods still face technical challenges. First, traditional CNNs are prone to information loss and blurring when processing long-time sequence videos, especially during the detail restoration process. In addition, the temporal characteristics of video data require super-resolution technology not only to handle single-frame images but also to consider the temporal correlation between frames to maintain video clarity and feature consistency.

[0005] To address these issues, Transformer-based architectures have been introduced to video super-resolution research in recent years. The self-attention mechanism of Transformer can effectively capture long-range dependencies, making it suitable for processing time-series data. By applying Transformer to face video super-resolution tasks, we can enhance spatial details while capturing temporal information between frames. This approach not only improves the super-resolution effect of single-frame images but also achieves higher clarity and feature consistency in video sequences, enhancing the overall quality of the video.

[0006] For standard Transformer architectures, computational complexity has always been a pain point in the process of handling long sequences. Through an analysis of computational complexity, the application proposes a low-rank decomposition-based attention mechanism parameter optimization method, which effectively alleviates the computational complexity problem of the model.

[0007] Based on the attention mechanism based on low-rank decomposition technology, the application constructs a framework for high-multiple face video super-resolution technology.

[0008] Firstly, a multi-channel information extraction module is constructed to obtain multi-channel spatial information, secondly, an inter-frame point-by-point mask attention module is constructed to obtain inter-frame motion trajectories, then a low-rank decomposition-based frame reconstruction module is used to integrate time and space dimensions, finally, the proposed Gan loss, inter-frame point-by-point loss and mean square error loss are used to improve the detail recovery and generation capability of high-multiple face video, and more real and clear face image reconstruction is realized.

[0009] Through this novel technical architecture, the development of face video super-resolution technology can be effectively promoted, and higher quality face video processing solutions are provided for various application scenarios. SUMMARY

[0010] The technical problem to be solved by the application is the insufficient reconstruction quality and instability caused by large-scale information loss and feature drift in high-multiple super-resolution (6, 8, 10) tasks of face video. To this end, a high-multiple face video super-resolution technology framework based on Transformer is proposed. First, a multi-channel information extraction module is constructed to extract the spatial information contained in the input video frames, ensuring the diversity of spatial information, then a frame-by-frame mask attention module is used to integrate the motion trajectories between different frames to obtain time information and strengthen the consistency of inter-frame features, finally, a low-rank decomposition-based frame reconstruction module is used to integrate the spatial information while reducing the calculation parameters of the attention mechanism, improving the inference speed.

[0011] To achieve the above purpose, the application adopts the following technical scheme:

[0012] The high-multiple face video super-resolution technology framework based on Transformer includes the following contents:

[0013] Firstly, a multi-dimensional feature representation method for multi-channel is provided.

[0014] In the face video super-resolution task, the application inputs 5 adjacent frames, denoted as n-2, n-1, n, n+1 and n+2. For the size of each frame, three dimensions of channel (Channel), height (Height) and width (Weight) are usually used for representation. For an 8x face video super-resolution task, the size of the input frame is 3x64x64, where 3 represents the channel, 64 represents the height and 64 represents the width, and the output is the predicted intermediate frame np with a size of 3x512x512.

[0015] For the input frame, first expand it to 3x256x256 by bilinear interpolation and data enhancement technology, and then change the representation of the input frame to 48x64x64 by vector transformation operation. Then, the input frame is respectively divided into 4 12x64x64, 2 24x64x64 and 1 48x64x64 according to the channel dimension. The channel is expanded to 64x64x64 for each 12x64x64 using convolution, the long-distance dependence of the features in each 64x64x64 is constructed using the attention mechanism, and then each 64x64x64 is spliced into 256x64x64. Similar operations are performed on the 2 24x64x64 and 1 48x64x64, and the output of 256x64x64 is also obtained, as shown in formula 1.

[0016] Channel_Enhance(n+i)=λUpsample(n+i)+Attention (1)

[0017] Where i takes the values -2, -1, 0, 1, 2, representing the current input frame position, λ takes the values 4, 2, 1, representing the number of samples divided by channel, Upsample represents the up-sampling method of the application, and Attention represents the attention mechanism after parameter optimization of the application.

[0018] Finally, the mean of the three groups of 256x64x64 is calculated to effectively integrate information from different sources and generate a representative output to provide sufficient spatial information support for the subsequent frame-by-frame mask attention module and frame reconstruction module.

[0019] Second: time consistency method based on frame-by-frame mask attention module

[0020] In the face video super-resolution task, 5 consecutive input images are denoted as n-2, n-1, n, n+1, and n+2, and the output is the predicted intermediate frame denoted as np. According to formula 1, enhanced 5-frame image information can be obtained. The key problem to be solved in the face video super-resolution task is how to integrate the 5-frame image information into the predicted intermediate frame np. To solve this problem, a time consistency method based on a frame-by-frame mask attention module is designed, as shown in formulas 2 and 3.

[0021] Difference(i)=Pixel(n+i)-Pixel(n) (2)

[0022] Mask:Pixel(n+i)-Pixel(n)≤0.00001 (3)

[0023] Where formula 2 is a method for calculating the different points between frames, and formula 3 is a Mask identification standard.

[0024] In the interaction attention module in the standard Transformer architecture, the input contains queries, keys and values, and the information of the input intermediate frame n is used as the query in the application, and the pixel difference is calculated between the n-2 frame, the n-1 frame, the n+1 frame and the n+2 frame and the n frame. For the positions with pixel difference absolute value less than 0.00001, mask identification is performed to guide the model to pay attention to the point-by-point information between frames and strengthen the attention distribution.

[0025] Third: parameter optimization method of attention mechanism based on low-rank decomposition;

[0026] In the self-attention mechanism, the input is X, and the output is calculated through the query (Query), key (Key)

[0027] and value (Value) matrix. The calculation formula of the attention mechanism is:

[0028]

[0029] In the standard attention mechanism calculation, Q is equal to the product of the input X and the parameter W, and the low-rank decomposition technology is performed on W, so that:

[0030]

[0031] wherein,

[0032] W∈R embed_dim×embed_dim ,(5)

[0033]

[0034] W low ∈R rank×embed_dim ,

[0035] In the application, the size of embed_dim is 256, and the rank is 64. Therefore, in the standard attention mechanism, the parameter size corresponding to Q is the product of 256 and 256, which is 65536, and after adopting the low-rank decomposition technology, the parameter size corresponding to Q is the product of 2 and 64 and 256, which is 32768, and the parameter amount is reduced by one time. Similarly, after performing low-rank decomposition on the parameters corresponding to K and V, the parameter amount is also reduced by one time respectively.

[0036] By adjusting the rank, the multiple of parameter reduction can be freely controlled, and the specific reduction scale Scale can be calculated by formula 6.

[0037] Scale=embed_dim÷(rank×2) (6)

[0038] The attention mechanism parameter optimization method based on low-rank decomposition can provide different parameter compression ratios to meet the actual application calculation requirements. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The application mainly realizes a high-multiple face video super-resolution technology framework based on Transformer.

[0040] Figure 2 The application mainly realizes a high-multiple face video super-resolution technology framework based on Transformer.

[0041] Figure 3 The application mainly realizes a high-multiple face video super-resolution technology framework based on Transformer. DETAILED DESCRIPTION

[0042] The application mainly realizes a high-multiple face video super-resolution technology framework based on Transformer.

[0043] Specifically, the high-multiple face video super-resolution technology framework based on Transformer includes the following aspects: first, a multi-dimensional feature representation method for multi-channel; second, a time consistency method based on an inter-frame point-by-point mask attention module; and third, an attention mechanism parameter optimization method based on low-rank decomposition. Figure 1

[0044] The attention mechanism parameter optimization method based on low-rank decomposition.

[0045] First, the multi-dimensional feature representation method for multi-channel:

[0046] Step 1: input a video frame sequence and read a parameter file;

[0047] Step 2: divide the information of the channel dimension according to formula 1, and perform different up-sampling processing on different channel information;

[0048] Step 3: use the attention mechanism module optimized by formula 5 to perform data enhancement on the up-sampled information.

[0049] Step 4: merge the data processed and enhanced according to the multi-channel.

[0050] Second, the time consistency method based on the inter-frame point-by-point mask attention module:

[0051] Step 5: calculate the inter-frame point-by-point information by formula 2;

[0052] Step 6: identify the pixel points that meet the conditions according to formula 3.

[0053] Third, the attention mechanism parameter optimization method based on low-rank decomposition:​

[0054] Step 7: According to formula 5, the parameter calculation in formula 4 is improved, and according to the analysis in formula 6, the rank value is appropriately adjusted, the parameter scale is controlled, and in the experiment, the rank value is 64;

[0055] Step 8: A frame reconstruction module is constructed by a parameter optimization method based on a low-rank decomposition attention mechanism to obtain a high-quality intermediate frame np.

[0056] The above specific embodiments are only used to illustrate the technical solutions of the present application, and not to limit it. Those skilled in the art should understand that the above embodiments do not limit the present application in any form, and any similar technical solutions obtained by equivalent replacement or equivalent transformation, etc. belong to the protection scope of the present application.

Claims

1. A method for high-resolution face video processing based on Transformer, characterized in that, Includes the following steps: Step 1: Multi-dimensional feature representation for multiple channels; In the face video super-resolution task, the input consists of 5 adjacent frames, denoted as n-2, n-1, n, n+1, n+2. The size of each frame is represented by three dimensions: Channel, Height, and Weight. Step 2, based on the temporal consistency of the inter-frame pointwise masking attention module; In the face video super-resolution task, the input consists of 5 consecutive frames of images, denoted as n-2, n-1, n, n+1, n+2, and the output is the predicted intermediate frame, denoted as np. Step 3: Parameter optimization of the attention mechanism based on low-rank decomposition; In the self-attention mechanism, the input is X, and the output is calculated by querying the matrix Q, key K, and value V. In the interactive attention module of the standard Transformer architecture, the input includes the query, key, and value. The information of the intermediate frame n is used as the query, and the pixel difference between the n-2, n-1, n+1, and n+2 frames and the n frame is calculated respectively. For positions where the absolute value of the pixel difference is less than 0.00001, a mask is applied to guide the model to focus on the inter-frame point-by-point information and strengthen the attention distribution. In step 1, for the 8x face video super-resolution task, the input frame size is 3×64×64, where 3 represents the channel, 64 represents the height and 64 represents the width, and the output is the predicted intermediate frame np, with a size of 3×512×512. For the input frame, it is first expanded to 3×256×256 through bilinear interpolation and data augmentation, and then the representation of the input frame is changed to 48×64×64 through vector transformation operation. The input frame is divided into four 12×64×64, two 24×64×64, and one 48×64×64 according to the channel dimension. Convolution is used to expand the channels of each 12×64×64 to 64×64×64. An attention mechanism is used to construct the long-distance dependency of the features in each 64×64×64. Then each 64×64×64 is concatenated to 256×64×64. Similar operations are performed on the two 24×64×64 and one 48×64×64 to obtain the output of 256×64×64, as shown in Equation (1). (1)； Where Channel_Enhance represents the enhanced channel information, n represents the input intermediate frame, i takes values ​​of -2, -1, 0, 1, 2, representing the current input frame position, which respectively represent the two frames before the intermediate frame, the one frame before the intermediate frame, the current frame, the one frame after the intermediate frame, and the two frames after the intermediate frame; λ takes values ​​of 4, 2, 1, representing the number of samples segmented by channel; Upsample represents the upsampling method; Attention represents the attention mechanism after parameter optimization. Finally, the average of the three 256×64×64 values ​​is calculated to fuse information from different sources and generate a representative output, providing sufficient spatial information support for the subsequent inter-frame pointwise masking attention module and frame reconstruction module.

2. The high-resolution face video super-resolution processing method based on Transformer according to claim 1, characterized in that, In step 2, the enhanced 5-frame image information is obtained according to equation (1). In the face video super-resolution task, these 5-frame image information are integrated into the predicted intermediate frame np. The time consistency method based on the inter-frame point-by-point mask attention module is adopted, as shown in equation (2) and equation (3). (2)； (3)； Equation (2) is the calculation method for different pixel points between frames, and Equation (3) is the Mask identification standard; Difference represents the point-by-point difference between frames, Pixel represents the pixel point of the current frame, and Mask represents the masking of unimportant pixel points. The determination condition for whether a pixel point is important is shown in Equation (3).

3. The high-resolution face video super-resolution processing method based on Transformer according to claim 2, characterized in that, The formula for calculating attention mechanism is: (4)； Where Q, K, and V are vectors in the attention mechanism, their sizes determined by the size of the input image and the size of the vector representation of each pixel; the Softmax function is a mathematical function commonly used in machine learning, d k K represents the dimension of vector K. T This indicates that the vector K is transposed, and T represents the transpose of a matrix.

4. The high-resolution face video super-resolution processing method based on Transformer according to claim 3, characterized in that, In standard attention mechanism computation, Q equals the product of input X and parameter W. A low-rank decomposition of W results in: (5)； The size of embed_dim is 256, and the rank is 64. In the standard attention mechanism, the parameter size of Q is the product of 256 and 256, which is 65536. After using the low-rank decomposition technique, the parameter size of Q is the product of 2 and 64 and 256, which is 32768, reducing the number of parameters by half. After performing low-rank decomposition on the parameters corresponding to K and V, their parameter sizes are also reduced by half. By adjusting the rank, the reduction factor of the parameter can be freely controlled, and the specific reduction scale is calculated by equation (6). (6)； Attention mechanism parameter optimization methods based on low-rank decomposition offer different parameter compression ratios to meet the computational needs of practical applications.

Images