A method and system for real-time reconstruction of spatiotemporal super-resolution

Abstract

The present invention discloses a method and system for real-time reconstruction of spatiotemporal super-resolution. The method comprises: constructing a network model; constructing the network model comprises: extracting features and splicing features of an input frame to obtain a feature map of the input frame and a spliced ​​feature map of the input frame; performing offset field compensation based on the feature map of the input frame and the spliced ​​feature map of the input frame to obtain a feature map after offset field compensation; performing feature fusion based on the feature map after offset field compensation and the feature map of the input frame to obtain a fused feature map; generating a super-resolution reconstructed image based on the fused feature map. The present invention can effectively solve the problems of large computational load, poor real-time performance, and artificial severance of the internal connection between VFI and VSR in the current VFI+VSR "two-step method" application scenario, and finally obtain high-resolution and high-frame-rate video output in real time.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a spatiotemporal super-resolution real-time reconstruction method and system. Background Art

[0002] Video super-resolution (VSR) technology is a technical means to transform low-resolution video into high-resolution video, and video frame interpolation (VFI) technology is a technical means to interpolate low-frame rate video into high-frame rate video. It is gradually being used in video, game and other scenarios.

[0003] For scenes where both interpolation and frame interpolation are required, the traditional approach is to use a "two-step method" (such as DAIN+EDVR and DAIN+RBPN), that is, to perform VFI interpolation and VSR interpolation in sequence. However, this method cannot fully utilize the high intrinsic correlation between temporal interpolation and spatial interpolation in video processing tasks. In addition, the most advanced VFI or VSR networks require frame synthesis or reconstruction modules for predicting high-quality video frames, respectively, which makes the "two-step method" have a large model size, resulting in high computing resource consumption and slow computing speed. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a real-time reconstruction method and system for spatiotemporal super-resolution, which can effectively solve the problems of huge computational complexity, poor real-time performance and artificial separation of the internal connection between VFI and VSR in the current VFI+VSR "two-step method" application scenario. Finally, high-resolution and high-frame rate video output can be obtained in real time.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present invention, a method for real-time reconstruction of spatiotemporal super-resolution is provided, including constructing a network model; constructing the network model includes: extracting features and splicing features of an input frame to obtain a feature map of the input frame and a spliced ​​feature map of the input frame; performing offset field compensation based on the feature map of the input frame and the spliced ​​feature map of the input frame to obtain a feature map after offset field compensation; performing feature fusion based on the feature map after offset field compensation and the feature map of the input frame to obtain a fused feature map; generating a super-resolution reconstructed image based on the fused feature map.

[0006] In some embodiments, offset field compensation is performed based on a feature map of an input frame and a spliced ​​feature map of the input frame to obtain a feature map after offset field compensation, including: for a current input frame F(i), based on the spliced ​​feature map of the input frame, respectively calculating an offset field OF(i-2t,i) of each frame in other input frames F(i-2t) relative to the current input frame F(i), where t=1,...,N; and respectively superimposing the offset field OF(i-2t,i) of each frame in other input frames F(i-2t) relative to the current input frame F(i) and the corresponding feature map FM(i-2t) of other input frames F(i-2t), to obtain N feature maps FM'(i-2t,i) of the current input frame after motion compensation based on other input frames F(i-2t).

[0007] In some embodiments, offset field compensation is performed based on a feature map of an input frame and a spliced ​​feature map of the input frame to obtain a feature map after offset field compensation, and the method further includes: for an intermediate frame F(i-1), based on the spliced ​​feature map of the input frame, respectively calculating the offset field of each input frame relative to the intermediate frame F(i-1); and respectively superimposing the offset field of each input frame relative to the current intermediate frame F(i-1) and the feature map of the corresponding input frame to obtain N+1 feature maps of the current intermediate frame after motion compensation based on the input frame.

[0008] In some embodiments, feature fusion is performed based on the feature map after offset field compensation and the feature map of the input frame to obtain the fused feature map, including: splicing all N feature maps FM'(i-2t,i) of the current input frame after motion compensation based on other input frames F(i-2t) with the feature map FM(i) of the current input frame F(i), and then calculating the fused feature map of the current input frame F(i) through feature fusion based on the convolution block; splicing all N+1 feature maps of the intermediate frames after motion compensation based on the input frame, and then calculating the fused feature map of the intermediate frame F(i-1) through feature fusion based on the convolution block.

[0009] In some embodiments, the method also includes constructing a data set; constructing the data set includes: acquiring video data, downsampling the resolution of the video data, dividing the downsampled video data into a training set and a test set according to a predetermined ratio, or dividing the downsampled video data and the acquired video data into a training set and a test set according to a predetermined ratio.

[0010] In some embodiments, the method also includes training a network model; training the network model includes: using image data with initial resolution and target resolution in a training set, calculating a loss value according to a preset loss function, and completing the training of the network model when the loss value and / or the accuracy of the training set and / or the accuracy of the test set reaches a preset standard.

[0011] According to another aspect of the present invention, a spatiotemporal super-resolution real-time reconstruction method is provided, wherein the network model trained by the above method is quantized and then deployed on a hardware platform to achieve the target function.

[0012] According to another aspect of the present invention, a spatiotemporal super-resolution real-time reconstruction system is provided, comprising a network model construction module; the network model construction module comprises: a feature extraction module, for performing feature extraction on each frame in an input frame to obtain a feature map of the input frame; a feature splicing module, for splicing the feature maps of all input frames to obtain a spliced ​​feature map of the input frame; a motion compensation module, for performing offset field compensation based on the feature map of the input frame and the spliced ​​feature map of the input frame to obtain a feature map after offset field compensation; the feature splicing module is also used to perform feature splicing based on the feature map after offset field compensation and the feature map of the input frame; a feature fusion module, for performing feature fusion on an image obtained by feature splicing based on the feature map after offset field compensation and the feature map of the input frame to obtain a fused feature map; a pixel rearrangement module, for performing pixel rearrangement on the fused feature map to generate a super-resolution reconstructed image.

[0013] According to another aspect of the present invention, there is provided an electronic device comprising the above-mentioned spatiotemporal super-resolution real-time reconstruction system.

[0014] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor; a memory communicatively connected to the processor; the memory storing instructions executable by the processor, the instructions being executed by the processor so that the processor can execute the above method.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the above method is implemented.

[0016] In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

[0017] 1. The traditional "two-step method" is abolished in super-resolution and interpolation tasks, and both super-resolution reconstruction and interpolation are realized in one neural network, which greatly reduces the demand for computing resources;

[0018] 2. Fusion of features of multiple input frames, filling and enhancing the image quality and details of each frame, the reconstruction and interpolation results are high quality and rich in details;

[0019] 3. Low latency. In scenarios that require real-time performance, such as live broadcasting, only the current frame can be reconstructed with super-resolution, and the intermediate frame between the current frame and the previous frame can be super-resolution generated. The minimum design latency is 0.017s when interpolating from 30fps to 60fps, and only 0.008s when interpolating from 60fps to 120fps. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic flow chart of a method for real-time spatiotemporal super-resolution reconstruction according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a process of constructing a data set according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a process of building a network model according to an embodiment of the present invention;

[0023] Figure 4 yes Figure 3 The specific implementation process diagram of the process of building a network model is shown;

[0024] Figure 5 yes Figure 4 The specific implementation process diagram of the process of building a network model is shown;

[0025] Figure 6 is a schematic structural diagram of a spatiotemporal super-resolution real-time reconstruction system according to an embodiment of the present invention;

[0026] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. As those skilled in the art can appreciate, the described embodiments can be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0028] like Figure 1 As shown, the spatiotemporal super-resolution real-time reconstruction method of the embodiment of the present invention comprises the following steps:

[0029] Step 1: Build a dataset.

[0030] Specifically, video data is collected, the resolution of the video data is downsampled, and the downsampled video data is divided into a training set and a test set according to a predetermined ratio, or the downsampled video data and the collected video data are divided into a training set and a test set according to a predetermined ratio.

[0031] In some embodiments, the collected video data constitutes a first data set, having a first resolution and a first frame rate, and the downsampled video data constitutes a second data set, having a second resolution and a second frame rate, wherein the second resolution is 1 / 2 of the first resolution, and the second frame rate is 1 / 2 of the first frame rate. The first data set and the second data set are divided into a training set and a test set according to a predetermined ratio. The second resolution is the initial resolution, the second frame rate is the initial frame rate, the second data set is the initial data set, the first resolution is the target resolution, the first frame rate is the target frame rate, and the first data set is the target data set. Figure 2 As shown, for example, for super-resolution reconstruction with an input of 1080P, a frame rate of 30fps and an output of 2160P, a frame rate of 60fps, the first resolution is 2160P, the first frame rate is 60fps, the second resolution is 1080P, and the second frame rate is 30fps, and the first data set and the second data set are divided into 2K training sets and 2K test sets in a ratio of 7:3.

[0032] In some embodiments, the downsampled video data constitutes a third data set, having a third resolution and a third frame rate, the third resolution is 1 / 2 of the first resolution, and the third frame rate is equal to the first frame rate; the downsampled video data also constitutes a fourth data set, having a fourth resolution and a fourth frame rate, the fourth resolution is 1 / 4 of the first resolution, and the fourth frame rate is 1 / 2 of the first frame rate. The third data set and the fourth data set are divided into a training set and a test set according to a predetermined ratio. The fourth resolution is the initial resolution, the fourth frame rate is the initial frame rate, the fourth data set is the initial data set, the third resolution is the target resolution, the third frame rate is the target frame rate, and the third data set is the target data set. Figure 2 As shown, for example, for super-resolution reconstruction with an input of 540P, a frame rate of 30fps and an output of 1080P, a frame rate of 60fps, the third resolution is 1080P, the third frame rate is 60pfs, the fourth resolution is 540P, and the fourth frame rate is 30fps, and the third data set and the fourth data set are divided into a 1080P training set and a 1080P test set in a ratio of 7:3.

[0033] In some embodiments, the data proportion of the training set is greater than that of the test set. For example, the initial data set is divided into an initial training set and an initial test set at a ratio of 7:3, and the target data set is divided into a target training set and a target test set at a ratio of 7:3, the data in the target training set corresponds to the data in the initial training set, the target test set corresponds to the data in the initial test set, the target training set and the initial training set constitute the training set, and the target test set and the initial test set constitute the test set.

[0034] Step 2: Build a network model.

[0035] The input frame includes a current frame F(i) and N historical frames F(i-2t), where t=1,...,N, t is an integer, N is a natural number, and the historical frame F(i-2t) is the previous t frames of the current frame F(i).

[0036] Perform feature extraction and feature splicing on the input frame to obtain a feature map of the input frame and a spliced ​​feature map of the input frame; perform offset field compensation based on the feature map of the input frame and the spliced ​​feature map of the input frame to obtain a feature map after offset field compensation; perform feature fusion based on the feature map after offset field compensation and the feature map of the input frame to obtain a fused feature map; based on the fused feature map, generate a super-resolution reconstructed image of the current input frame and a super-resolution reconstructed image of an intermediate frame between the current input frame and a frame previous to the current input frame.

[0037] Specifically, Figure 3 As shown, feature extraction is performed on each frame in the input frame to obtain feature maps of N+1 input frames, and the feature maps of all N+1 input frames are spliced ​​to obtain a spliced ​​feature map of the input frame. Specifically, it includes: feature extraction is performed on the current frame F(i) to obtain the feature map FM(i) of the current frame, feature extraction is performed on each frame in the N historical frames F(i-2t) to obtain the feature maps FM(i-2t) of the N historical frames, and the feature map FM(i) of the current frame is spliced ​​with the feature maps FM(i-2t) of the N historical frames to obtain the spliced ​​feature map of the input frame.

[0038] The frame rate of the input frame is low and does not contain any intermediate frames. For example, the input frame adjacent to F(i) is F(i-2), and there are no other input frames between F(i) and F(i-2). Therefore, the intermediate frame F(i-1) is a virtual frame and does not contain any information.

[0039] refer to Figure 3 , for the intermediate frame F(i-1), the steps of super-resolution generation are as follows:

[0040] Based on the spliced ​​feature map of the input frame, the offset field of each input frame relative to the intermediate frame F(i-1) is calculated. Specifically, it includes: calculating the offset field OF(i,i-1) of the current frame F(i) relative to the intermediate frame F(i-1), and the offset field OF(i-2t,i-1) of each frame in the N historical frames F(i-2t) relative to the intermediate frame F(i-1);

[0041] Superimpose the offset field of each input frame relative to the intermediate frame F(i-1) and the feature map of the corresponding input frame to obtain N+1 feature maps of the current intermediate frame after motion compensation of the input frame. Specifically, it includes: superimposing the offset field OF(i,i-1) of the current frame F(i) relative to the intermediate frame F(i-1) and the feature map FM(i) of the current frame to obtain the feature map FM'(i,i-1) of the intermediate frame after motion compensation of the current frame, superimposing the offset field OF(i-2t,i-1) of each frame in the N historical frames F(i-2t) relative to the intermediate frame F(i-1) and the feature map FM(i-2t) of the corresponding historical frame to obtain N feature maps FM'(i-2t,i-1) of the intermediate frames after motion compensation of each historical frame;

[0042] After splicing all N+1 feature maps of the intermediate frames after motion compensation based on the input frame, the fused feature map of the intermediate frame F(i-1) is calculated by feature fusion based on the convolution block;

[0043] The fused feature map of the intermediate frame F(i-1) is pixel-shuffled to generate a super-resolution high-quality intermediate frame F'(i-1).

[0044] refer to Figure 3 , for the current input frame F(i), the steps of super-resolution reconstruction are as follows:

[0045] Based on the spliced ​​feature map of the input frame, the offset field OF(i-2t,i) of each frame in other input frames F(i-2t) relative to the current input frame F(i) is calculated respectively.

[0046] Superimpose the offset field OF(i-2t,i) of each frame in the other input frames F(i-2t) relative to the current input frame F(i) and the corresponding feature map FM(i-2t) of the other input frames F(i-2t) to obtain N feature maps FM'(i-2t,i) of the current input frame after motion compensation based on the other input frames F(i-2t);

[0047] After concatenating all N motion-compensated feature maps FM'(i-2t,i) of the current input frame based on other input frames F(i-2t) with the feature map FM(i) of the current input frame F(i), the fused feature map of the current input frame F(i) is calculated by feature fusion based on convolution blocks;

[0048] The pixels of the fused feature map of the current input frame F(i) are rearranged to obtain a super-resolution reconstructed image of the current input frame, that is, a super-resolution high-quality output frame F'(i).

[0049] It can be seen that the super-resolution reconstructed image includes not only the super-resolution high-quality output frame F'(i) corresponding to the current input frame F(i), but also the super-resolution high-quality intermediate frame F'(i-1) that did not originally exist in the input.

[0050] Depending on the actual usage, only the current frame can be reconstructed with super resolution, and the intermediate frame between the current frame and the previous frame can be super-resolution generated. That is to say, in the above method, N = 1, the input frame includes the current frame F(i) and 1 historical frame F(i-2), the super-resolution reconstruction object is limited to the current frame F(i), and does not include the historical frame F(i-2), and the intermediate frame F(i-1) is super-resolution generated based on the current frame F(i) and the historical frame F(i-2). In this way, the repeated output of historical frames can be reduced, and the consumption of computing resources can be reduced.

[0051] In some embodiments, constructing the network model further includes performing pixel correction on the pixel rearranged image. For example, after performing pixel correction on the pixel rearranged image using a convolution kernel without an activation function, a super-resolution reconstructed image of the input frame and the intermediate frame is obtained.

[0052] The spatiotemporal super-resolution real-time reconstruction method of the present invention is described in detail below with reference to specific examples. The resolution magnification is set to u, the number of basic convolution kernels is n, the size of the basic convolution kernel is k, the convolution step length a=1, and the activation function is AFunc, which is a nonlinear function.

[0053] like Figure 4 As shown, the activation function AFunc is used to perform n-core convolution with a kernel size of k*k on the input frames F(i) and F(i-2) respectively to extract nonlinear features, and obtain nonlinear feature maps of two input frames. The nonlinear feature maps of the two input frames are spliced ​​to obtain a spliced ​​feature map of the input frame.

[0054] For the intermediate frame F(i-1), the steps of super-resolution generation are as follows: use the activation function AFunc to perform n-core convolution with a kernel size of k*k on the spliced ​​feature map of the input frame, and calculate the offset field of the current frame F(i) relative to the intermediate frame F(i-1), as well as the offset field of the historical frame F(i-2) relative to the intermediate frame F(i-1); superimpose the offset field of the current frame F(i) relative to the intermediate frame F(i-1) and the feature map of the current frame to obtain a feature map of the intermediate frame after motion compensation based on the current frame; superimpose the offset field of the historical frame F(i-2) relative to the intermediate frame F(i-1) and the feature map of the historical frame to obtain a feature map of the intermediate frame after motion compensation based on the historical frame; after splicing the two feature maps based on the intermediate frame after motion compensation of the input frame, use the activation function AFunc to perform 3au convolution with a kernel size of k*k. 2 The kernel convolution is used to perform feature fusion and calculate the fused feature map of the intermediate frame F(i-1). Finally, the pixels of the fused feature map of the intermediate frame F(i-1) are rearranged to generate the intermediate frame for super-resolution reconstruction.

[0055] For the input frame F(i), the steps of super-resolution reconstruction are as follows: use the activation function AFunc to perform n-core convolution with a convolution kernel size of k*k on the spliced ​​feature map of the input frame, and calculate the offset field of the input frame F(i-2) relative to the input frame F(i); superimpose the offset field of the input frame F(i-2) relative to the input frame F(i) and the feature map of the input frame F(i-2) to obtain the feature map of the input frame F(i) after motion compensation based on the input frame F(i-2); splice the feature map of the input frame F(i) after motion compensation of the input frame F(i-2) with the feature map of the input frame F(i), and then use the activation function AFunc to perform 3au convolution with a convolution kernel size of k*k. 2 The kernel convolution is used to perform feature fusion and calculate the fused feature map of the input frame F(i); finally, the fused feature map of the input frame F(i) is rearranged by pixels to obtain a super-resolution reconstructed image of the input frame F(i).

[0056] In some implementations, the nonlinear function may be, for example, tanh, sigmoid, or relu.

[0057] At this point, the STSR network is built. The STSR network uses the current frame and the historical N frames for feature extraction (N satisfies ≥ 1). In the process of nonlinear convolution, the number of basic convolution kernels n is not limited (generally n = 2 a , a≥4, or n=3a*u 2 , a is a positive integer), Figure 4Several convolution blocks are omitted in the ellipsis, which can be added or deleted in the actual design. The STSR network is easy to modify. The simplified version can be used for mobile or embedded devices, or convolution blocks can be added to the previous version for use in workstations, theaters and other scenes with demanding quality requirements. It can complete any multiple of super-resolution and interpolation tasks, and obtain video streams of any specifications including but not limited to 1080P 200fps, 2K 144fps, 4K 58fps, etc.

[0058] The STSR network retrieves the images of the previous and next related frames, and establishes a nonlinear mapping relationship between each frame and the current frame within the specified frame range. The precise motion compensation calculated by the global feature map makes the feature alignment effect obvious. Due to the constraints between the network structures, the jitter and artifacts of the picture details are effectively suppressed, and the picture quality performance is improved. Since the network has both interpolation and super-resolution functions, N+1 frames of low-resolution images can be used as input to obtain 2N+1 frames of high-resolution output, providing an efficient solution for low-frame rate and low-resolution imaging and display systems to display high-quality video streams. In addition, the STSR network takes into account the spatiotemporal correlation of interpolation and super-resolution tasks, reduces the complexity of the traditional "two-step method", avoids the disadvantage of running two steps of similar steps (such as feature extraction) in the traditional "two-step method", and effectively improves the efficiency of interpolation + super-resolution tasks.

[0059] Step 3: Train the network model.

[0060] Using the image data of the initial resolution and target resolution in the training set, the loss value is calculated according to the preset loss function. When the loss value and the accuracy of the training set and the test set reach the preset standard, the model is retained and the model training is completed.

[0061] In some embodiments, the Huber loss function is used to calculate the loss value to enhance the robustness of the square error loss function to outliers. The expression of the loss value is as follows:

[0062]

[0063] Where y is the image data of the target resolution in the training set, that is, the target output. x is the image data of the initial resolution in the training set, the f() function is the model calculation process, and f(x) is the super-resolution reconstructed image result calculated by the neural network model. δ is a custom global parameter. When the deviation value |yf(x)| is less than δ, the square error is used. When the deviation value |yf(x)| is greater than δ, the linear error is used.

[0064] In some embodiments, the training set constructed in step 1 is used to train the network model constructed in step 2 using a back propagation algorithm, that is, the initial resolution (low resolution) data of the training set is input into the model, the model calculates and outputs a predicted high-resolution image, and then the predicted high-resolution image and the target resolution image of the training set (e.g., Ground Truth) are substituted into the loss function to calculate the loss value (loss), and then the loss value is back-propagated to adjust the parameters of the model, and the above calculation process is repeated, and iterated continuously until the loss value and the accuracy of the training set and the test set reach the preset standard, and the model training is completed.

[0065] In some embodiments, the accuracy is the ratio of the number of pixels that can be reconstructed from the initial resolution to the target resolution using a neural network model to the total number of pixels. Specifically, the accuracy of the training set is to input the initial resolution data in the training set into the model, and compare the resolution data output by the model with the corresponding target resolution data in the training set, where the number of pixels with the same resolution is divided by the total number of pixels in the training set, which is the accuracy of the training set. Similarly, the accuracy of the test set is to input the initial resolution data in the test set into the model, and compare the resolution data output by the model with the corresponding target resolution data in the test set, where the number of pixels with the same resolution is divided by the total number of pixels in the test set, which is the accuracy of the test set.

[0066] The accuracy of the test set is similar to the accuracy in the real scene, which can predict the model performance in the real scene. In addition, the accuracy of the test set can also reflect the robustness of the model. When the accuracy of the test set is similar to the accuracy of the training set, the robustness of the model is strong; when the accuracy of the test set is significantly different from the accuracy of the training set, the robustness of the model is poor.

[0067] In some embodiments, the preset criteria include: loss value ≤ 10 -5 , the accuracy of the training set is ≥ 99.9%, and the accuracy of the test set is ≥ 99.9%.

[0068] Step 4: Network model deployment.

[0069] After quantizing the network model trained in step 3 and deploying it on the hardware platform, the target function can be achieved.

[0070] The following describes in detail the spatiotemporal super-resolution real-time reconstruction method of the present invention by taking 540P resolution (original resolution) super-resolution to 1080P resolution (target resolution), using 1 frame of historical frame (N=1), and the King of Glory game as an example.

[0071] The resolution magnification u=2, the number of basic convolution kernels n=32, the basic convolution kernel size k=3, the convolution step size a=1, and the activation function AFunc is relu.

[0072] Combination Figure 1 The specific implementation process of this example is as follows:

[0073] Step 1: Build a dataset.

[0074] like Figure 2 As shown, first collect high-resolution (2160P) high-frame-rate (60fps) video data, downsample the resolution to 1 / 2 of the original specification (1080P), downsample to 1 / 4 of the original specification (540P), and retain the frame rate to 60fps and 1 / 2 of the original specification (30fps). At this time, the user has four sets of data: 2160P 60fps, 1080P 30fps, 1080P 60fps, and 540P 30fps. The combination can obtain two data sets with a resolution and frame rate that are 2 times the relationship, for example Figure 2 The 540P 30fps and 1080P 60fps data in the dataset constitute the 1080P dataset, and the 1080P 30fps and 2160P 60fps constitute the 2K dataset. After that, the dataset is divided into a training set and a test set according to the conventional 7:3 ratio. Of course, the division ratio can be selected according to the actual situation.

[0075] Step 2: Build a network model.

[0076] like Figure 3 As shown, feature extraction is performed on the current frame and the historical frames to obtain feature maps, and all feature maps are spliced. For the intermediate frame, the spliced ​​feature map is used to calculate the offset field OF of each input frame to the current intermediate frame for all N+1 input frames, and the feature map FM of each input frame is added with the corresponding offset field OF to perform feature motion compensation of the input frame to the current intermediate frame. After splicing all the motion-compensated feature maps FM', feature fusion is performed, and then pixels are rearranged to obtain the intermediate frame reconstructed by super resolution; for the current input frame, the spliced ​​feature map is used to calculate the offset field OF of other input frames to the current input frame, and the feature maps FM of other frames are added with the corresponding offset field OF to perform feature motion compensation, and then all the motion-compensated feature maps FM' are fused with the feature map of the current input frame, and finally pixels are rearranged to obtain the super-resolution reconstructed frame.

[0077] like Figure 5As shown in the figure, the STSR super-resolution neural network in this case is constructed in detail. A 32-core convolution with a 1-layer convolution kernel size of 3*3 activated by the relu function is used to extract features of the input frames F(i) and F(i-2) to obtain feature maps FM(i) and FM(i-2). The obtained feature maps are feature spliced ​​to obtain the spliced ​​feature map of the input frame.

[0078] The intermediate frame F(i-1) itself does not have any information and needs to be generated from the previous and next frames. Using the concatenated feature map of the input frame mentioned above, a 32-core convolution with a 3*3 kernel size activated by the relu function is used to calculate the motion offset fields OF(i,i-1) and OF(i-2,i-1) of the frames FM(i) and FM(i-2) for the intermediate frame F(i-1), and then the offset fields OF(i,i-1) and OF(i-2,i-1) are added to the feature maps FM(i) and FM(i-2) of the input frame mentioned above for motion compensation, and the motion-compensated feature maps FM'(i,i-1) and FM'(i-2,i-1) are obtained. After that, the feature maps FM'(i,i-1) and FM'(i-2,i-1) are concatenated and then activated by the relu function. A 12-core convolution with a kernel size of 3*3 is performed to obtain the feature map of the intermediate frame F(i-1) after feature fusion. Finally, the super-resolution high-quality intermediate frame F'(i-1) is obtained by pixel rearrangement.

[0079] For the current input frame F(i), first use the concatenated feature map of the input frame mentioned above to calculate the offset field OF(i-2,i) of the feature map FM(i) by a 32-core convolution with a 1-layer convolution kernel size of 3*3 activated by the relu function, then add the feature map FM(i-2) and the offset field OF(i-2,i) to obtain the motion compensated feature map FM'(i-2), then concatenate the feature map FM(i) of the current input frame and the aforementioned motion compensated feature map FM'(i-2), finally, obtain the fused feature map of the current input frame F(i) after feature fusion through a 1-layer convolution kernel size of 3*3 activated by the relu function, and finally, obtain the super-resolution high-quality input frame F'(i) after pixel rearrangement.

[0080] Step 3: Train the network model.

[0081] The loss function uses the Huber loss function to enhance the robustness of the square error loss function to outliers. The Huber loss function expression is as follows:

[0082]

[0083] The data set constructed in step 1 is used to train the aforementioned model using the back propagation algorithm. The data set constructed in step 1 is used to train the aforementioned model using the back propagation algorithm, and iterates until the loss value L≤10-5 and the accuracy of the training set and the test set are both ≥99.9%, and the model is retained.

[0084] Step 4: Network model deployment.

[0085] After quantizing the model trained in step 3, deploy it on the hardware acceleration platform, so that the device can almost instantly upgrade the video stream specification to 1080P 60fps when playing 540P, 30fps video. This model can be applied to game rendering, video recording and other scenarios involved in the super-resolution dataset.

[0086] like Figure 6 As shown, the embodiment of the present application also provides a spatiotemporal super-resolution real-time reconstruction system, comprising:

[0087] The data set construction module is used to collect video data, downsample the resolution of the video data, and divide the downsampled video data into a training set and a test set according to a predetermined ratio, or divide the downsampled video data and the collected video data into a training set and a test set according to a predetermined ratio.

[0088] In some embodiments, the collected video data constitutes a first data set, having a first resolution and a first frame rate, and the downsampled video data constitutes a second data set, having a second resolution and a second frame rate, wherein the second resolution is 1 / 2 of the first resolution, and the second frame rate is 1 / 2 of the first frame rate. The first data set and the second data set are divided into a training set and a test set according to a predetermined ratio. The second resolution is the initial resolution, the second frame rate is the initial frame rate, the second data set is the initial data set, the first resolution is the target resolution, the first frame rate is the target frame rate, and the first data set is the target data set. Figure 2 As shown, for example, for super-resolution reconstruction with an input of 1080P, a frame rate of 30fps and an output of 2160P, a frame rate of 60fps, the first resolution is 2160P, the first frame rate is 60fps, the second resolution is 1080P, and the second frame rate is 30fps, and the first data set and the second data set are divided into 2K training sets and 2K test sets in a ratio of 7:3.

[0089] In some embodiments, the downsampled video data constitutes a third data set, having a third resolution and a third frame rate, the third resolution is 1 / 2 of the first resolution, and the third frame rate is equal to the first frame rate; the downsampled video data also constitutes a fourth data set, having a fourth resolution and a fourth frame rate, the fourth resolution is 1 / 4 of the first resolution, and the fourth frame rate is 1 / 2 of the first frame rate. The third data set and the fourth data set are divided into a training set and a test set according to a predetermined ratio. The fourth resolution is the initial resolution, the fourth frame rate is the initial frame rate, the fourth data set is the initial data set, the third resolution is the target resolution, the third frame rate is the target frame rate, and the third data set is the target data set. Figure 2 As shown, for example, for super-resolution reconstruction with an input of 540P, a frame rate of 30fps and an output of 1080P, a frame rate of 60fps, the third resolution is 1080P, the third frame rate is 60pfs, the fourth resolution is 540P, and the fourth frame rate is 30fps, and the third data set and the fourth data set are divided into a 1080P training set and a 1080P test set in a ratio of 7:3.

[0090] In some embodiments, the data proportion of the training set is greater than that of the test set. For example, the initial data set is divided into an initial training set and an initial test set at a ratio of 7:3, and the target data set is divided into a target training set and a target test set at a ratio of 7:3, the data in the target training set corresponds to the data in the initial training set, the target test set corresponds to the data in the initial test set, the target training set and the initial training set constitute the training set, and the target test set and the initial test set constitute the test set.

[0091] The network model construction module is used to extract features and splice features of the input frame to obtain a feature map of the input frame and a spliced ​​feature map of the input frame; perform offset field compensation based on the feature map of the input frame and the spliced ​​feature map of the input frame to obtain a feature map after offset field compensation; perform feature fusion based on the feature map after offset field compensation and the feature map of the input frame to obtain a fused feature map; based on the fused feature map, generate a super-resolution reconstructed image of the current input frame and a super-resolution reconstructed image of an intermediate frame between the current input frame and a frame previous to the current input frame.

[0092] In some embodiments, the network model building module includes a feature extraction module for extracting features from each frame in the input frame to obtain a feature map of the input frame.

[0093] In some embodiments, the network model construction module also includes a feature stitching module, which is used to stitch the feature maps of all input frames to obtain a stitched feature map of the input frame; it is also used to perform feature stitching based on the feature map after offset field compensation and the feature map of the input frame, specifically, it is also used to stitch all feature maps of the intermediate frames based on the input frame after motion compensation, and it is also used to stitch all feature maps of the current input frame after motion compensation based on other input frames with the feature map of the current input frame.

[0094] In some embodiments, the network model construction module also includes a motion compensation module, which is used to calculate the offset field of each input frame relative to the current intermediate frame based on the spliced ​​feature map of the input frame, and superimpose the offset field of each input frame relative to the current intermediate frame with the feature map of the corresponding input frame to obtain a feature map of the intermediate frame after motion compensation of the input frame. The motion compensation module is also used to calculate the offset field of each frame in the other input frames relative to the current input frame based on the spliced ​​feature map of the input frame, and superimpose the offset field of each frame in the other input frames relative to the current input frame with the feature map of the corresponding other input frames to obtain a feature map of the current input frame after motion compensation of the other input frames.

[0095] In some embodiments, the network model construction module also includes a feature fusion module, which is used to perform feature fusion on the image after feature splicing of the feature map based on the offset field compensation and the feature map of the input frame to obtain the fused feature map; specifically, it is used to perform feature fusion based on the convolution block on the image after splicing all feature maps of the current intermediate frame after motion compensation based on the input frame, and calculate the fused feature map of the current intermediate frame. The feature fusion module is also used to perform feature fusion based on the convolution block on the image after splicing all feature maps of the current input frame after motion compensation based on other input frames and the feature map of the current input frame, and calculate the fused feature map of the current input frame.

[0096] In some embodiments, the network model building module further includes a pixel rearrangement module for rearranging pixels of the feature map after the current intermediate frame is fused to generate a super-resolution high-quality intermediate frame. The pixel rearrangement module is also used to rearrange pixels of the feature map after the current input frame is fused to obtain a super-resolution reconstructed image of the current input frame.

[0097] In some embodiments, the network model building module further includes a pixel correction module for performing pixel correction on the pixel rearranged image, for example, using a convolution kernel without an activation function to perform pixel correction on the pixel rearranged image.

[0098] The network model training module is used to use the image data of the initial resolution and target resolution in the training set to calculate the loss value according to the preset loss function. When the loss value and the accuracy of the training set and the test set reach the preset standard, the model is retained to complete the model training.

[0099] In some embodiments, the Huber loss function is used to calculate the loss value to enhance the robustness of the square error loss function to outliers. The expression of the loss value is as follows:

[0100]

[0101] Where y is the image data of the target resolution in the training set, that is, the target output. x is the image data of the initial resolution in the training set, the f() function is the model calculation process, and f(x) is the super-resolution reconstructed image result calculated by the neural network model. δ is a custom global parameter. When the deviation value |yf(x)| is less than δ, the square error is used. When the deviation value |yf(x)| is greater than δ, the linear error is used.

[0102] In some embodiments, the network model constructed by the network model construction module is trained using a back-propagation algorithm using a dataset constructed by the dataset construction module, that is, the initial resolution (low-resolution) data of the training set is input into the model, the model calculates and outputs a predicted high-resolution image, and then the predicted high-resolution image and the target resolution image (Ground Truth) of the training set are substituted into the loss function to calculate the loss value (loss), and then the loss value is back-propagated to adjust the parameters of the model, and the above calculation process is repeated, and iterated continuously until the loss value and accuracy of the training set and the loss value and accuracy of the test set reach the preset standard, and the model training is completed.

[0103] In some embodiments, the preset criteria include: the loss value of the training set is ≤10 -5 , the accuracy of the training set is ≥ 99.9%, and the loss value of the test set is ≤ 10 -5 , the accuracy of the test set is ≥99.9%.

[0104] The network model deployment module is used to quantify the network model trained by the network model training module and deploy it to the hardware platform to achieve the target function.

[0105] Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. The present application embodiment also provides an electronic device, such as Figure 7As shown, the electronic device includes: at least one processor 701, and a memory 703 connected to the at least one processor 701 in communication. Instructions that can be executed by at least one processor 701 are stored in the memory 703. The instructions are executed by at least one processor 701. When the processor 701 executes the instructions, the driving scene reconstruction method in the above embodiment is implemented. The number of memory 703 and processor 701 can be one or more. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0106] The electronic device may also include a communication interface 705 for communicating with external devices and performing data exchange transmission. The various devices are interconnected using different buses and may be installed on a common motherboard or in other ways as needed. The processor 701 may process instructions executed in the electronic device, including instructions stored in or on a memory to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses may be used together with multiple memories and multiple memories. Similarly, multiple electronic devices may be connected, each device providing some of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0107] Optionally, in a specific implementation, if the memory 703, the processor 701 and the communication interface 705 are integrated on a chip, the memory 703, the processor 701 and the communication interface 705 can communicate with each other through an internal interface.

[0108] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0109] The embodiment of the present application provides a computer-readable storage medium (such as the memory 703 mentioned above), which stores computer instructions. When the program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0110] Optionally, the memory 703 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device of the driving scene reconstruction method, etc. In addition, the memory 703 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 703 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device of the driving scene reconstruction method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0113] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more (two or more) executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0115] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0116] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A spatiotemporal super-resolution real-time reconstruction method, characterized in that: This includes building a network model; The constructing of the network model comprises: Perform feature extraction and feature splicing on the input frame to obtain a feature map of the input frame and a spliced ​​feature map of the input frame; For the current input frame F(i), based on the spliced ​​feature map of the input frame, the offset field OF(i-2t, i) of each frame in the other input frames F(i-2t) relative to the current input frame F(i) is calculated, where t=1, ..., N; the offset field OF(i-2t, i) of each frame in the other input frames F(i-2t) relative to the current input frame F(i) and the corresponding feature map FM(i-2t) of the other input frames F(i-2t) are superimposed to obtain N feature maps FM'(i-2t, i) of the current input frame after motion compensation based on the other input frames F(i-2t); For the intermediate frame F(i-1), based on the spliced ​​feature map of the input frame, the offset field of each input frame relative to the intermediate frame F(i-1) is calculated respectively; the offset field of each input frame relative to the current intermediate frame F(i-1) and the feature map of the corresponding input frame are superimposed respectively to obtain N+1 feature maps of the current intermediate frame after motion compensation based on the input frame; After concatenating all N feature maps FM'(i-2t, i) of the current input frame after motion compensation based on other input frames F(i-2t) with the feature map FM(i) of the current input frame F(i), the fused feature map of the current input frame F(i) is calculated by feature fusion based on the convolution block; after concatenating all N+1 feature maps of the intermediate frames after motion compensation based on the input frames, the fused feature map of the intermediate frame F(i-1) is calculated by feature fusion based on the convolution block; Based on the fused feature map, a super-resolution reconstructed image is generated.

2. The spatiotemporal super-resolution real-time reconstruction method according to claim 1, characterized in that: It also includes building a data set; The constructing of the data set comprises: Collect video data, downsample the resolution of the video data, divide the downsampled video data into a training set and a test set according to a predetermined ratio, or divide the downsampled video data and the collected video data into a training set and a test set according to a predetermined ratio.

3. The spatiotemporal super-resolution real-time reconstruction method according to claim 2, characterized in that: It also includes training network models; The training network model includes: Using the image data of the initial resolution and the target resolution in the training set, the loss value is calculated according to a preset loss function. When the loss value and / or the accuracy of the training set and / or the accuracy of the test set reaches a preset standard, the training of the network model is completed.

4. A spatiotemporal super-resolution real-time reconstruction method, characterized in that: After the network model trained by the method described in claim 3 is quantified, it is deployed on a hardware platform to achieve the target function.

5. A spatiotemporal super-resolution real-time reconstruction system, characterized in that: Includes network model building modules; The network model building module includes: A feature extraction module is used to extract features from each frame in the input frame to obtain a feature map of the input frame; A feature splicing module is used to splice the feature maps of all input frames to obtain a spliced ​​feature map of the input frame; A motion compensation module, for respectively calculating the offset field of each input frame relative to the current intermediate frame based on the spliced ​​feature map of the input frame, respectively superimposing the offset field of each input frame relative to the current intermediate frame and the feature map of the corresponding input frame, to obtain a feature map of the intermediate frame after the input frame has been motion compensated; and for respectively calculating the offset field of each frame in other input frames relative to the current input frame based on the spliced ​​feature map of the input frame, respectively superimposing the offset field of each frame in other input frames relative to the current input frame and the feature map of the corresponding other input frames, to obtain a feature map of the current input frame after the other input frames have been motion compensated; The feature stitching module is also used to perform feature stitching based on the feature map after offset field compensation and the feature map of the input frame; A feature fusion module is used to perform feature fusion based on convolution blocks on an image obtained by splicing feature maps of the current intermediate frame after motion compensation based on all input frames, and calculate the fused feature map of the current intermediate frame; and to perform feature fusion based on convolution blocks on an image obtained by splicing feature maps of the current input frame after motion compensation based on all other input frames and the feature map of the current input frame, and calculate the fused feature map of the current input frame; The pixel rearrangement module is used to rearrange the pixels of the fused feature map to generate a super-resolution reconstructed image.

6. An electronic device, characterized in that: The spatiotemporal super-resolution real-time reconstruction system comprises the system described in claim 5; Alternatively, the electronic device comprises: processor; a memory communicatively coupled to the processor; The memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images