A video super-resolution method without spatiotemporal coding
Through the video super-resolution method without spatiotemporal encoding, the cross-attention neural network and diffusion principle are used to extract the correlation between video frames, solving the data dispersion and content deviation problems caused by spatiotemporal encoding in the prior art, achieving high Quality high-definition image reconstruction.
Patent Information
- Application Number
- CN202410521060.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-28
AI Technical Summary
The large model based on the Transformer attention mechanism in the prior art cannot correlate the built-in logical relationship between pixels at different locations in the video super-segment algorithm, resulting in spatiotemporal encoding that leads to the data probability distribution being too dispersed and difficult to converge, and the generated content and the original image content. Details vary greatly.
Using the video super-resolution method without spatiotemporal encoding, the cross-attention neural network is constructed by generating image summary and local features, and the model is established using the diffusion principle to extract the correlation between pixel blocks, avoid spatiotemporal encoding, and realize high-definition image generation .
It effectively solves the problems of local image details and content deviation caused by spatiotemporal encoding, improves the video quality after video overscore, and realizes high-definition image reconstruction with small deviations from the original image content.
Smart Images

Figure CN118446890B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a video super-resolution method without spatiotemporal coding. Background Art
[0002] In film and television and advertising production, super-resolution algorithms are often needed to pre-process videos, remove image noise, and improve video resolution, or to improve the clarity of early standard-definition film and television works. In recent years, image generation and video generation technologies have been changing with each passing day, and AIGC technology applications based on large models have emerged in an endless stream, such as ChatGPT human-computer dialogue, StableDiffusion image generation, Sora video generation and other products. The quality of the generated graphic content has been qualitatively improved compared to the previous generation of products. Therefore, video super-resolution algorithms based on large model generation technology are also constantly being tried and applied and continuously developed, such as the StableSR super-resolution algorithm and the Stable Diffusion XL super-resolution algorithm.
[0003] The main problems existing in the prior art and the difficulty of solving the problem are: the large model based on the Transformer attention mechanism cannot associate the built-in logical relationship between pixels at different positions within the model. Therefore, the prior art generally adopts spatiotemporal coding to encode the input image data in time and space, and the encoded data can be input into the neural network, so that the data at different positions of different images can be different, that is, to tell the neural network the current pixel position. At the same time, it is necessary to ensure that the probability distribution of the data is not too much to prevent the data from being too scattered and causing the result to fail to converge. It is generally controlled between [-1.0, 1.0]. While the spatiotemporal coding changes the original RGB data distribution, the position coding of the high-definition image with more than 1920*1080 pixels will cause the data probability distribution to be too scattered and difficult to converge. As a compromise solution, the diffusion principle based on the latent space is currently adopted to generate content in a 64*64 area, and then upsample after generation, but this will cause the details of the generated content image to deviate too much from the details of the original image content.
[0004] The significance of solving the above problem is to solve the problem that the data probability distribution is too dispersed due to the spatiotemporal coding of the high-resolution image reconstruction process in the existing method. Summary of the invention
[0005] The present invention provides a video super-resolution method without spatiotemporal coding, which effectively solves the problem that the generated content of the old method deviates too much from the original content due to spatiotemporal coding, and realizes the improvement of video quality after video super-resolution.
[0006] The technical solution of the present invention is as follows:
[0007] The video super-resolution method without spatiotemporal coding of the present invention comprises the following steps: S1. generating an image summary: uniformly adjusting the input video frames to images of the same resolution, and then inputting them into a variational autoencoder and a fully connected network respectively, to obtain an image summary C and an image local feature map X respectively; S2. fusing the image summary with the image local features: constructing a cross-attention neural network without spatiotemporal coding, using the cross-attention neural network to fuse the image summary and the image local feature map, and extracting the correlation between pixel blocks in space and time; S3. local feature diffusion modeling: establishing a model based on the diffusion principle, and improving the generation capability of high-definition images through its diffusion process and generation process.
[0008] Optionally, in the above-mentioned video super-resolution method without spatiotemporal coding, in step S2, the local feature map X of the image is divided into n pixel blocks, and the pixel blocks are moved to obtain the feature map X1; at the same time, the image summary C is processed into the image summary C1 using a fully connected network; finally, X, X1, and C1 are cross-self-attention calculated to obtain F1; the above steps are repeated n times to obtain the output result F n .
[0009] Optionally, in the above-mentioned video super-resolution method without spatiotemporal coding, in step S2, a corresponding neural network is constructed through the following calculation process, which is called a cross-attention neural network (CrossNet), and the calculation process includes T1: the image summary C obtained in step S1 is passed through the fully connected neural network L1 to obtain the image summary C1; at the same time, the first pixel block of the local feature X of the image is moved to the last position of the pixel block in a circular position to obtain the local feature X1 of the image; then X, X1, and C1 are input into the cross-self-attention neural network CA1 to calculate F1, wherein the image summary C1 provides the correlation between image blocks, and the local feature X1 of the image provides the relationship between adjacent pixels. The calculation process is f=X1·C1, and then f is used as the data source of q in the standard cross-attention calculation, and X1 is used as the data source of k and v; T2: After T1 is completed, enter the T2 calculation process. The difference between the T2 calculation process and T1 is that in the cross-self-attention calculation, the input is F1, X2, and C2, wherein F1 is obtained by the previous T1 calculation, and so on. The above steps are repeated n times to obtain the output result F n .
[0010] Optionally, in the above-mentioned video super-resolution method without spatiotemporal coding, in step S3, noise is added m times to the local image feature X in step S2 to obtain Xm, the image summary C remains unchanged, and the local image feature Xm after adding noise is input into a cross attention neural network (CrossNet) to obtain an output result Will Enter CrossNet again to calculate the feature map Repeat the calculation m times to get Represents the predicted final noise; input image frame minus the final noise The final high-definition image is obtained.
[0011] According to the technical solution of the present invention, the beneficial effects produced are:
[0012] Aiming at the problem that the traditional super-resolution algorithm performs spatiotemporal coding on the input data, which leads to the loss of local details of the image, the present invention constructs an image generation model based on cross-attention, which effectively solves the problem that the old method causes a large deviation between the content of the super-resolution image and the original image content due to spatiotemporal coding. In the prior art, the input data is generally added to the spatiotemporal coding, which will cause the probability distribution value range of the input data to differ too much, making it difficult to establish non-overlapping spatiotemporal coding information on the high-definition image, and ultimately leading to the loss of local details of the image, and the output image has a large deviation from the original image content. The method of the present invention, based on the data association characteristics of the cross-attention mechanism, calculates the association between each block of the image in pairs, and the neural network automatically extracts the spatiotemporal information of the pixel block, thereby avoiding the problem that the old method needs to perform spatiotemporal coding on the input data, which leads to deviation in the image content, and finally realizes the video super-resolution function with a small deviation in image content.
[0013] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.
[0015] Figure 1 is a flow chart of a video super-resolution method without spatiotemporal coding of the present invention;
[0016] Figure 2 is a schematic diagram of a generated image summary of the present invention;
[0017] Figure 3 is a schematic diagram of image summary and feature fusion of the present invention;
[0018] Figure 4 It is a schematic diagram of the diffusion process and the generation process of the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.
[0020] The method of the present invention utilizes the cross-attention mechanism to construct a new video super-resolution algorithm model. The model supports data input without spatiotemporal coding. The neural network inside the algorithm model can automatically extract the correlation between the data and use the correlation information to reconstruct a high-resolution image, thereby solving the problem of the model losing local details of the image due to spatiotemporal coding in the existing method.
[0021] The method of the present invention constructs a neural network algorithm model without spatiotemporal coding, allowing the neural network to automatically extract the correlation between pixels in each area of the image using a cross-attention mechanism, and generate and complete the content according to the association between pixel blocks and frames to achieve super-resolution. In particular, based on the cross-attention mechanism of the neural network, a neural network algorithm model based on cross-attention is designed, allowing the model to support the simultaneous input of global information and local detail features. The neural network effectively associates the pixels of each block without spatiotemporal coding, and finally realizes the function of reconstructing a high-resolution image using the correlation of pixel blocks. This solves the problem of large deviations in image content caused by spatiotemporal coding in the old method, and solves the problem of large deviations between local details of the video super-resolution image and the original image content.
[0022] The principle of the present invention is: utilizing the data association characteristics of the cross-attention mechanism to associate image summary information with local feature information, realizing an algorithm model that can extract the pairwise correlation between each area of the image, and then realizing a video super-resolution method, thereby eliminating the problem of local detail deviation (model loses local image details) caused by spatiotemporal coding, and then realizing a video super-resolution function based on a generative model.
[0023] like Figure 1 As shown, the video super-resolution method without spatiotemporal coding of the present invention comprises the following steps from the beginning to the end:
[0024] S1. Generate image summary: The input video frame Frame is uniformly adjusted to the image frame of the same resolution, and then the image of the same resolution is input into the variational autoencoder and the fully connected network respectively to obtain the image summary C and the image local feature map X respectively.
[0025] Specifically, Figure 2As shown, it is a schematic diagram of generating an image summary of the present invention. First, a frame of image Frame is read from a video file, and then the image is adjusted to a uniform size, such as 1920*1080, to obtain the resized image frame (Frame represents the video frame before resizing; frame represents the video frame after resizing); then the image frame is input into a variational autoencoder VAE and a fully connected neural network L0 respectively, and the image summary C and the image local feature map X are output respectively, wherein the variational autoencoder VAE is a neural network model that has been trained on a large public data set. These VAEs can be trained by themselves, or they can be publicly available existing variational autoencoders, such as the trained variational autoencoder Autoencoder KL provided by StabilityAI in the Stable Diffusion project. In particular, multiple adjacent frames before and after the video frame Frame are read from the video file, and these multiple frames are spliced into a large picture, such as reading 7 frames of images and splicing them into a size of 1920*7560, which is used as the frame input.
[0026] S2. Fusion of image summary and local image features: A cross-attention neural network without spatiotemporal encoding is constructed, which is used to fuse the image summary and the local image feature map to extract the spatial and temporal correlations between pixel blocks.
[0027] The local feature map X of the image is divided into n pixel blocks, and the pixel blocks are moved to obtain the feature map X1; at the same time, the image summary C is processed into the image summary C1 using a fully connected network; finally, X, X1, and C1 are cross-attention calculated to obtain F1; repeat the above steps n times to obtain the output result F n .
[0028] Specifically, Figure 3The figure shows a schematic diagram of the image summary and feature fusion of the present invention. First, the image summary C obtained in step S1 is passed through the fully connected neural network L1 to obtain the image summary C1; at the same time, the first pixel block of the local feature X of the image is moved to the last position of the pixel block in a circular moving position manner to obtain the local feature X1 of the image; then X, X1, and C1 are input into the cross self-attention neural network CA1 to calculate F1, wherein the image summary C1 provides the correlation between image blocks, and the local feature X1 of the image provides the relationship between adjacent pixels. It is worth noting that the standard cross attention calculation method can only accept 2 data source inputs, while X, X1, and C1 are 3 data sources. Therefore, the standard cross attention calculation method is improved here, and the calculation process is f=X1·C1, and then f is used as the data source of q in the standard cross attention calculation, and X1 is used as the data source of k and v. The above process is collectively referred to as T1. After T1 is completed, the T2 calculation process is entered. The T2 calculation process is basically the same as T1. The only difference is that in the cross self-attention calculation, the input is F1, X2, and C2, where F1 is obtained by the previous T1 calculation. And so on, repeat the above steps n times to get the output result Fn. In the whole process, the image summary always provides global image information for each image block, and after repeating the calculation n times, the correlation between all image blocks is also calculated once, so there is no need to add spatiotemporal coding like the traditional neural network based on the attention mechanism, and the neural network itself can capture the spatiotemporal information of the pixel block. The neural network corresponding to the construction of the above calculation process of the present invention is called CrossNet, that is, the present invention names the neural network with the above calculation process as the cross-attention neural network (CrossNet), which is the core invention of the present invention.
[0029] S3. Local feature diffusion modeling. Establish a model based on the diffusion principle, and improve the generation capability of high-definition images through its diffusion process and generation process.
[0030] Based on the diffusion principle modeling, the diffusion process is to add m noises to X in step S2 to obtain X m The generation process is to convert X m The noise is calculated through the CrossNet neural network m times. X minus noise The final high-definition image Image is obtained.
[0031] Specifically, in the process of generating video super-resolution from low resolution to high resolution, the neural network algorithm model needs to have strong data generation capabilities. The algorithm model based on the stable diffusion principle has been proven to be effective in mainstream algorithms such as StableDiffusion. Figure 4As shown in FIG. 1 , it is a schematic diagram of the diffusion process and the generation process of the present invention. The local feature X of the image in step S2 is added with noise m times to obtain X m , the image summary C remains unchanged, and the local features X of the image after adding noise (noise) m Input to CrossNet to get the output result Will Enter CrossNet again to calculate the feature map Repeat the calculation m times to get ( Figure 4 , CrossNet prediction noise), Represents the predicted final noise; input image frame minus the final noise The final high-definition image Image is obtained. The traditional stable diffusion model only directly upsamples the image summary C to obtain the final output image without the input of the local image feature X, which will cause a large deviation between the generated image content and the initial input image content. The present invention does not have such a problem.
[0032] The present invention utilizes the correlation characteristics of the cross-attention mechanism to realize a video super-resolution generation algorithm without spatiotemporal coding. By constructing an image generation model that supports the simultaneous input of image summary and image local features (using image summary information and image local feature maps as input data of the algorithm model at the same time), the model does not need to perform spatiotemporal coding on the input data, but extracts the correlation of pixels in each region of the image through the cross-attention mechanism to obtain a generator that reconstructs high-resolution images that does not rely on spatiotemporal coding. The present invention utilizes the fusion ability of the cross-attention mechanism neural network to construct a generation network for reconstructing high-resolution images, thereby realizing a video super-resolution function for reconstructing high-resolution images without spatiotemporal coding.
[0033] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.
Claims
1. A video super-resolution method without spatiotemporal coding, characterized in that: The following steps are involved: S1. Generate image summary: The input video frames are uniformly adjusted to images of the same resolution, and then input into the variational autoencoder and the fully connected network respectively to obtain the image summary C and the image local feature map X respectively; S2. Fusion of image summary and local image features: constructing a cross-attention neural network without spatiotemporal coding, using the cross-attention neural network to fuse the image summary and the local image feature map, extracting the correlation between pixel blocks in space and time, specifically, Divide the local feature map X of the image into n pixel blocks, and move the pixel blocks to obtain the feature map ; At the same time, the image summary C is processed into an image summary using the fully connected network ;at last , , Perform cross self-attention calculation to obtain ; Repeat the above steps n times to get the output result , The corresponding neural network constructed through the following calculation process is called Cross Attention Neural Network (CrossNet), which includes: T1: The image summary C obtained in step S1 is passed through a fully connected neural network , get the image summary ; At the same time, the local features of the image The first pixel block is moved to the last position of the pixel block in a circular manner to obtain the local features of the image. ; Then , , Input to the cross self-attention neural network CA1 to calculate , where image summary Provides the correlation between image blocks and local features of the image Provides the relationship between adjacent pixels, the calculation process is , and then As the standard criss-cross attention calculation The data source, As and Data source; T2: After T1 is completed, enter the T2 calculation process. The difference between the T2 calculation process and T1 is that during the cross self-attention calculation, the input is , , ,in The image summary C in step S1 is calculated by T1 above. Get image summary , and the local features of the image The first and second pixel blocks are moved to the penultimate and last positions of the pixel blocks respectively in a circular moving manner to obtain the local features of the image ; And so on, repeat the above steps n times to get the output result ; S3. Local feature diffusion modeling: Establish a model based on the diffusion principle, and improve the generation capability of high-definition images through its diffusion process and generation process.
2. The video super-resolution method without spatiotemporal coding according to claim 1, characterized in that: In step S3, m noises are added to the local image feature X in step S2 to obtain , the image summary C remains unchanged, and the local features of the image after adding noise Input into the CrossNet neural network to get the output result ;Will Input the cross attention neural network (CrossNet) again to get the feature map , repeat the calculation m times to get , represents the predicted final noise; Input image frame minus the final noise The final high-definition image is obtained.
Citation Information
Patent Citations
Three-dimensional reconstruction method and system based on deep learning
CN115330935A
Monocular depth prediction algorithm based on multi-scale progressive interaction and aggregation cross attention features
CN116485860A
Abdomen multi-organ segmentation method based on Transform and U-Net fusion network
CN116580036A
Generating neural network outputs by enriching potential embedding using self-attention and cross-attention operations
CN116888603A
Video restoration method based on diffusion model
CN117219104A
