A video reconstruction method, device, electronic equipment and storage medium

Abstract

The application discloses a video reconstruction method and device, electronic equipment and storage medium. The method comprises the following steps: bicubic up-sampling an initial video to be reconstructed to obtain a first video; inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features comprise local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model; adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video. The technical scheme of the embodiment of the application can provide more comprehensive feature information for the video reconstruction task of the initial video, and further improve the quality of the target video obtained by reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a video reconstruction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In actual production and life, the hardware devices on the camera and other imaging devices are limited by their own process, such as the size and dimensions of the photosensitive pixels of the sensor, and the imaging obtained through these sensors cannot actually meet the requirements in different application environments. In addition, the transmission process of the video may also cause the degradation of the video. Therefore, how to optimize the video reconstruction technology from the software level through precise algorithms and technical means, and then improve the video quality through the video reconstruction technology, has become an important academic research that has attracted much attention. In the prior art, video reconstruction is mainly performed in the following two ways: 1. Directly regarding video reconstruction as a composite task of video super-resolution and video frame interpolation, and performing video frame interpolation and video super-resolution in sequence. 2. Training a single end-to-end model based on deep learning, which only extracts features from the input video frames once, and then performs upsampling on the extracted features in the network model in the order of time and space. The first method does not utilize the internal relationship between temporal interpolation and spatial super-resolution, and the second method is prone to focus on part of the features in the video frames, resulting in the loss of other features. SUMMARY

[0003] The present application provides a video reconstruction method, device, electronic equipment and storage medium, which can provide more comprehensive feature information for the video reconstruction task of the initial video, and thus improve the quality of the target video obtained by reconstruction.

[0004] According to an aspect of the present application, a video reconstruction method is provided, which comprises:

[0005] Performing bicubic upsampling on the initial video to be reconstructed to obtain a first video;

[0006] Inputting the initial video into a pre-determined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model;

[0007] Adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video.

[0008] According to another aspect of the present application, a video reconstruction device is provided, which comprises:

[0009] a first video determining module configured to perform bicubic up-sampling on the initial video to be reconstructed to obtain a first video;

[0010] a second video determining module configured to input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model;

[0011] a target video determining module configured to adjust the deep features in the second video and fuse the adjusted second video with the first video to obtain a target video.

[0012] According to another aspect of the present application, an electronic device is provided, which comprises:

[0013] at least one processor; and

[0014] a memory connected to the at least one processor in communication; wherein,

[0015] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the video reconstruction method according to any one of the embodiments of the present application.

[0016] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the video reconstruction method according to any one of the embodiments of the present application when executed by the processor.

[0017] The technical scheme of the embodiment of the application performs bicubic up-sampling on an initial video to be reconstructed to obtain a first video; inputs the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model; the deep features in the second video are adjusted, and a target video is obtained by fusing the adjusted second video and the first video. Through the combination of the adaptive multi-level convolution unit for extracting local details and the frequency converter unit for extracting global information, the target reconstruction model can more comprehensively understand the initial video, provide more comprehensive feature information for the video reconstruction task of the initial video, and further improve the quality of the target video obtained by reconstruction.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flow chart of a video reconstruction method provided by the embodiment one of the application;

[0021] Figure 2 is a structural schematic diagram of a target reconstruction model provided by the embodiment one of the application;

[0022] Figure 3 is a flow chart of a video reconstruction method provided by the embodiment two of the application;

[0023] Figure 4 is a structural schematic diagram of a frame interpolation module provided by the embodiment two of the application;

[0024] Figure 5 is a structural schematic diagram of an adaptive multi-level convolution unit provided by the embodiment two of the application;

[0025] Figure 6 is a structural schematic diagram of a multi-axis 3D convolution subunit provided by the embodiment two of the application;

[0026] Figure 7 is a structural schematic diagram of a fusion subunit according to Embodiment Two of the present application

[0027] Figure 8 is a structural schematic diagram of a frequency divider unit according to Embodiment Two of the present application

[0028] Figure 9 is a structural schematic diagram of a feature extraction layer of a frequency divider unit according to Embodiment Two of the present application

[0029] Figure 10 is a structural schematic diagram of a video reconstruction apparatus according to Embodiment Three of the present application

[0030] Figure 11 is a structural schematic diagram of an electronic device implementing a video reconstruction method according to Embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment One

[0034] Figure 1 A flowchart of a video reconstruction method according to Embodiment One of the present application is provided, which can be applicable to the case of reconstructing a low-resolution, low-frame-rate video into a high-resolution, high-frame-rate video. The method can be performed by a video reconstruction apparatus, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in FIG. 1, the video reconstruction method according to Embodiment One of the present application includes the following steps.Figure 1 The method comprises:

[0035] S110, bicubic up-sampling the initial video to be reconstructed to obtain a first video.

[0036] The initial video to be reconstructed can be a low-resolution and low-frame-rate video, such as an old video resource or a low-quality video obtained by a download service provided by a website. Alternatively, a video with a resolution lower than a first preset threshold and / or a frame rate lower than a second preset threshold can be used as the initial video. The first preset threshold and the second preset threshold can be set according to actual conditions, and the embodiments of the present application do not limit this. The bicubic up-sampling is an interpolation method commonly used in image and video processing, which can improve the resolution of the image or video.

[0037] In the embodiments of the present application, after determining the initial video that needs to be reconstructed, the initial video can be bicubic up-sampled by video processing software or programming to generate a first video with higher resolution. It should be noted that bicubic up-sampling can increase the size of the video file and the required storage space while improving the resolution of the video, and excessive up-sampling can cause the video to be blurred. Therefore, when bicubic up-sampling the initial video, appropriate parameters need to be selected to balance the quality and file size of the output video.

[0038] S120, inputting the initial video into a pre-determined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model.

[0039] The target reconstruction model includes a shallow feature extraction sub-model, a deep feature extraction sub-model, and a video reconstruction sub-model. The deep feature extraction sub-model has a U-shaped network architecture and includes at least two feature extraction layers. Each feature extraction layer includes an encoder module, a decoder module, and a frame interpolation module arranged at the skip connection of the U-shaped network. The encoder module includes an adaptive multi-level convolution unit and a frequency converter unit, which are respectively used to extract local details and global information in the shallow features of the initial video as deep features of the initial video. For example, Figure 2 A structural diagram of a target reconstruction model is shown.

[0040] In the embodiment of the present application, the initial video can be input into the pre-determined target reconstruction model to obtain a second video containing deep features of the initial video. Specifically, the initial video can be first input into a shallow feature extraction sub-model to extract shallow features of the initial video through the shallow feature extraction sub-model. Then, the shallow features are input into a deep feature extraction sub-model, and local details and global information in the shallow features are extracted through the adaptive multi-level convolution unit and the frequency converter unit in the deep feature extraction sub-model as deep features of the initial video. Finally, the deep features of the initial video are input into a video reconstruction sub-model, and a second video containing deep features of the initial video is reconstructed by the video reconstruction sub-model, which has higher resolution and higher frame rate. By combining the adaptive multi-level convolution unit to extract local details and the frequency converter unit to extract global information, the target reconstruction model can be more adaptive to process video data of different scales and complexities, and the target reconstruction model can comprehensively understand the entire initial video to provide more comprehensive background information for high-resolution and high-frame-rate reconstruction of video frames in the initial video.

[0041] Optionally, the process of pre-determining the target reconstruction model comprises: pre-processing a video dataset to obtain a first sample video, and performing down-sampling processing on the first sample video according to a preset sampling multiple to obtain a second sample video; inputting the second sample video into an initial reconstruction model to obtain an output result, and calculating a loss function according to the first sample video and the output result; iteratively updating parameter weights in the initial reconstruction model through the loss function until the number of iterations reaches a preset upper limit of the number of iterations to obtain the target reconstruction model.

[0042] The video dataset can be a public super-resolution training video dataset obtained from the network. The first sample video I H is a high-resolution and high-frame-rate video obtained after pre-processing the video dataset. The second sample video I L is a low-resolution and low-frame-rate video obtained by down-sampling processing the first sample video. The initial reconstruction model is a convolutional neural network model before the parameter weights in the target reconstruction model are iteratively updated.

[0043] In the embodiment of the present application, first, the video dataset obtained is pre-processed to obtain a first sample video I H . Optionally, the pre-processing of the video dataset comprises extracting a y channel from a video frame image, rotating or flipping the video frame image for data enhancement, and cropping the video frame image. Second, the first sample video I H is down-sampled according to a preset sampling multiple to obtain a second sample video I LThe preset sampling multiple can be adjusted according to actual conditions. Then, the second sample video is input into the initial reconstruction model to obtain an output result and according to the first sample video I H and the output result The loss function is calculated Finally, according to the calculated loss function, the parameter weight in the initial reconstruction model is iteratively updated through back propagation until the iteration number reaches a preset upper limit of iteration number, and a target reconstruction model is obtained.

[0044] Optionally, the formula for calculating the loss function according to the first sample video and the output result is as follows:

[0045]

[0046] wherein, is the loss function, I H is the first sample video, is the output result of the initial reconstruction model, and ε is a constant.

[0047] S130, adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video.

[0048] In the embodiment of the application, after obtaining the second video through the target reconstruction model, the deep features in the second video need to be adjusted to further optimize or enhance the deep features extracted from the initial video, so as to improve the quality of the final target video. Then, the adjusted second video can be fused with the first video through a specific algorithm to ensure that the two videos are visually consistent and coherent, so as to obtain a high-resolution, high-frame-rate target video and complete the video reconstruction of the initial video.

[0049] The technical scheme of the embodiment of the application performs bicubic up-sampling on the initial video to be reconstructed to obtain a first video; inputs the initial video into a pre-determined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model; the deep features in the second video are adjusted, and the adjusted second video is fused with the first video to obtain a target video. The technical scheme of the embodiment of the application combines the adaptive multi-level convolution unit to extract local details and the frequency converter unit to extract global information, so that the target reconstruction model can more comprehensively understand the initial video, provide more comprehensive feature information for the video reconstruction task of the initial video, and further improve the quality of the target video obtained by reconstruction.

[0050] Example 2

[0051] Figure 3 This is a flowchart of a video reconstruction method provided in Embodiment 2 of the present invention. The embodiments of the present invention are optimized based on the above embodiments. Solutions not described in detail in the embodiments of the present invention can be found in the above embodiments. Figure 3 As shown, the method includes:

[0052] S210. Perform bicubic upsampling on the initial video to be reconstructed to obtain the first video.

[0053] S220. Input the initial video into the shallow feature extraction sub-model to obtain the shallow feature map sequence of the initial video.

[0054] Here, the shallow feature map sequence refers to a video frame sequence that includes shallow features of the initial video. In this embodiment of the invention, the initial video can be first... The input is fed into the shallow feature extraction sub-model in the target reconstruction model, and the initial video is processed by a 3*3 convolutional layer in the shallow extraction sub-model. Shallow feature extraction is performed to obtain a sequence of shallow feature maps from the initial video.

[0055]

[0056] S230. Input the shallow feature map sequence into the deep feature extraction sub-model to obtain the deep feature map sequence of the initial video.

[0057] The deep feature map sequence refers to a video frame sequence that includes deep features of the initial video, namely local details and global information. In this embodiment of the invention, the shallow feature map sequence of the initial video is obtained. Then, the shallow feature map sequence can be... The input is fed into the deep feature extraction sub-model to obtain the deep feature map sequence x of the initial video. d1 It is used to restore clearer, high-resolution, high-frame-rate videos.

[0058] Optionally, the shallow feature map sequence is input into the deep feature extraction sub-model to obtain a deep feature map sequence of the initial video, including: inputting the shallow feature map sequence into an encoder module in a first feature extraction layer to obtain an initial encoding result; inputting the initial encoding result into an encoder module in a next feature extraction layer until a target encoding result of an encoder module in a last feature extraction layer is obtained; inputting the target encoding result into a decoder module in the last feature extraction layer to obtain an initial decoding result, and inputting the target encoding result into a frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result; adding the initial decoding result and the initial frame interpolation result and inputting the sum into a decoder module in a previous feature extraction layer, and inputting an encoding result of an encoder module in the previous feature extraction layer into a frame interpolation module in the previous feature extraction layer until a target decoding result of a decoder module in the first feature extraction layer and a target frame interpolation result of a frame interpolation module in the first feature extraction layer are obtained; adding the target decoding result and the target frame interpolation result to obtain the deep feature map sequence of the initial video.

[0059] Wherein, the frame interpolation module first extracts residual features from the input frames respectively, then enhances the result of another frame by cross-attention mechanism using the correlation between adjacent frames, and finally adds the two cross-attention results and obtains the intermediate frame of the two adjacent frames through the fusion unit. Exemplarily, Figure 4 A structural diagram of a frame interpolation module is shown.

[0060] In the embodiment of the application, the deep feature extraction sub-model is a U-shaped net structure, including at least two feature extraction layers, each of which includes an encoder module, a decoder module and a frame interpolation module arranged at the jump connection of the U-shaped net. Here, taking the deep feature sub-model including four feature extraction layers as an example, the extraction process of the deep feature map sequence is explained: 1, input the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain the initial encoding result x e1 , input x e1 into the encoder module in the second feature extraction layer to obtain x e2 , and so on until the target encoding result x e4 of the encoder module in the fourth feature extraction layer is obtained. e4 2, input the target encoding result x m4 into the decoder module in the fourth feature extraction layer to obtain the initial decoding result x e4 ; input the target encoding result x n4 into the frame interpolation module in the fourth feature extraction layer to obtain the initial frame interpolation result x m4 ; and add x n4The sum is x d4 , will x d4 x is obtained from the decoder module in the third feature extraction layer. m3 The encoding result x from the encoder module in the third feature extraction layer e3 The input is fed into the frame interpolation module in the third feature extraction layer to obtain x. n3 This process continues until the target decoding result x of the decoder module in the first feature extraction layer is obtained. m The interpolation result x of the target frame in the frame interpolation module of the first feature extraction layer n1 3. x m1 With x n The sum is x d1 This serves as a sequence of deep feature maps from the initial video. Through the U-shaped encoder-decoder structure and skip connections, rich spatial information can be preserved at different levels, contributing to accurate detail reconstruction and better preserving and restoring information from the initial video.

[0061] Optionally, the shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result, including: extracting local details in the shallow feature map sequence through the adaptive multi-level convolutional unit; extracting global information in the shallow feature map sequence through the frequency divider converter unit; and using the local details and the global information as the initial encoding result.

[0062] In this embodiment of the invention, local details in the shallow feature map sequence can be extracted by the adaptive multi-level convolutional unit in the encoder module, and global information in the shallow feature map sequence can be extracted by the frequency divider / converter unit in the encoder module. The local details and global information are then used as the initial encoding result. By using a processing scheme that combines local details and global information, the problems of the small receptive field of the adaptive multi-level convolutional unit and the poor processing effect of the frequency divider / converter unit on high-frequency information can be avoided. This allows for the reconstruction of high-quality video frames with fewer frames.

[0063] Optionally, extracting local details from the shallow feature map sequence using the adaptive multi-level convolutional unit includes: performing a one-dimensional temporal convolution on the shallow feature map sequence using a first multi-axis 3D convolutional subunit to obtain the temporal domain features of the shallow feature map sequence; performing a two-dimensional spatial convolution on the shallow feature map sequence using a second multi-axis 3D convolutional subunit to obtain the spatial domain features of the shallow feature map sequence; fusing the temporal domain features and spatial domain features using the fusion subunit, and extracting local details from the fused features.

[0064] The adaptive multi-level convolution unit includes two multi-axis 3D convolution subunits and a fusion subunit. The multi-axis 3D convolution subunit is used to capture different levels of abstract representation of the shallow feature map sequence and increase the expression capacity of the model. The fusion subunit is used to fuse the features and extract information from the fused features. Exemplarily, Figure 5 A structural diagram of an adaptive multi-level convolution unit is shown, Figure 6 A structural diagram of a multi-axis 3D convolution subunit is shown, Figure 7 A structural diagram of a fusion subunit is shown.

[0065] In the embodiment of the present application, the adaptive multi-level convolution unit can be used to perform pseudo 3D convolution on the shallow feature map sequence to extract local details in the shallow feature map sequence, so as to utilize the local context information of the shallow feature map sequence. Specifically, the first multi-axis 3D convolution subunit can be used to perform one-dimensional time convolution on the shallow feature map sequence to obtain the time domain features of the shallow feature map sequence, and the second multi-axis 3D convolution subunit can be used to perform two-dimensional space convolution on the shallow feature map sequence to obtain the space domain features of the shallow feature map sequence. Then, the fusion subunit can be used to fuse the time domain features and the space domain features, and extract local details from the fused features. By using the adaptive multi-level convolution unit to perform pseudo 3D convolution on the shallow feature map sequence, the short-time motion information in the shallow feature map sequence can be more accurately captured while maintaining the computational efficiency, and more detailed features can be provided for video reconstruction.

[0066] Optionally, the global information in the shallow feature map sequence is extracted by the frequency division converter unit, including: dividing the shallow feature map sequence input into the frequency division converter unit into a first subsequence and a second subsequence; extracting global information of the first subsequence by using a self-attention mechanism, and performing convolution on the second subsequence; and supplementing high-frequency information in the global information according to the convolution result.

[0067] The frequency division converter unit is used to divide the channel number of the input feature into two parts, one part is subjected to self-attention, and the other part is subjected to convolution to make up for the loss of high-frequency information in the self-attention process. Exemplarily, Figure 8 A structural diagram of a frequency division converter unit is shown, Figure 9 A structural diagram of a feature extraction layer of a frequency division converter unit is shown.

[0068] In the embodiment of the present application, the shallow feature map sequence input into the frequency division converter unit can be divided into a first sub-sequence and a second sub-sequence, then the global information of the first sub-sequence is extracted through the self-attention mechanism, and the second sub-sequence is convolved, the convolution channel is first through a depth separable convolution kernel of (5, 5, 5), and then through two parallel convolutions to extract high-frequency detail information under a large receptive field, and supplement the high-frequency information in the global information. By extracting the global information of the shallow feature map sequence through the frequency division converter unit, the global spatio-temporal information can be utilized, so that the target reconstruction model can simultaneously focus on all positions in the shallow feature map sequence without being limited by the fixed receptive field, so as to further explore the evolution trend, dynamic change and global structural features of the entire shallow feature map sequence.

[0069] S240, inputting the deep feature map sequence into the video reconstruction sub-model to generate a second video containing deep features of the initial video.

[0070] In the embodiment of the present application, the deep feature map sequence output by the deep feature extraction sub-model can be input into the video reconstruction sub-model, the height and width dimensions of the feature maps in the deep feature map sequence are restored to the height and width dimensions of the initial video, and a second video containing deep features of the initial video is generated.

[0071] S250, adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video.

[0072] The technical scheme of the embodiment of the present application performs bicubic up-sampling on the initial video to be reconstructed to obtain a first video, inputs the initial video into a shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video, inputs the shallow feature map sequence into a deep feature extraction sub-model to obtain a deep feature map sequence of the initial video, inputs the deep feature map sequence into a video reconstruction sub-model to generate a second video containing deep features of the initial video, adjusts the deep features in the second video, and fuses the adjusted second video with the first video to obtain a target video. The technical scheme of the embodiment of the present application can comprehensively understand the initial video by combining the adaptive multi-level convolution unit to extract local details and the frequency division converter unit to extract global information, provide more comprehensive feature information for the video reconstruction task of the initial video, and further improve the quality of the target video obtained by reconstruction.

[0073] Embodiment three

[0074] Figure 10 A structural schematic diagram of a video reconstruction device provided in the third embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the device includes: Figure 10

[0075] ​The first video determination module 310 is configured to perform bicubic up-sampling on the initial video to be reconstructed to obtain a first video.

[0076] The second video determination module 320 is configured to input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model.

[0077] The target video determination module 330 is configured to adjust the deep features in the second video and fuse the adjusted second video with the first video to obtain a target video.

[0078] Optionally, the second video determination module 320 comprises:

[0079] The sample video determination unit is configured to pre-process a video dataset to obtain a first sample video, and perform down-sampling processing on the first sample video according to a preset sampling multiple to obtain a second sample video.

[0080] The loss function calculation unit is configured to input the second sample video into an initial reconstruction model to obtain an output result, and calculate a loss function according to the first sample video and the output result.

[0081] The target model determination unit is configured to perform back propagation through the loss function, iteratively update parameter weights in the initial reconstruction model, and stop until the number of iterations reaches a preset upper limit of the number of iterations, to obtain a target reconstruction model.

[0082] Optionally, the target reconstruction model comprises a shallow feature extraction sub-model, a deep feature extraction sub-model, and a video reconstruction sub-model.

[0083] The second video determination module 320 comprises:

[0084] The shallow feature extraction unit is configured to input the initial video into the shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video.

[0085] The deep feature extraction unit is configured to input the shallow feature map sequence into the deep feature extraction sub-model to obtain a deep feature map sequence of the initial video.

[0086] The second video determination unit is configured to input the deep feature map sequence into the video reconstruction sub-model to generate a second video containing deep features of the initial video.

[0087] Optionally, the deep feature extraction sub-model is in a U-shaped network architecture, and includes at least two feature extraction layers, each of which includes an encoder module, a frame interpolation module, and a decoder module, and the frame interpolation module is arranged at a skip connection of the U-shaped network.

[0088] The deep feature extraction unit is specifically configured to:

[0089] input the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain an initial encoding result;

[0090] input the initial encoding result into the encoder module in the next feature extraction layer until a target encoding result of the encoder module in the last feature extraction layer is obtained;

[0091] input the target encoding result into the decoder module in the last feature extraction layer to obtain an initial decoding result, and input the target encoding result into the frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result;

[0092] add the initial decoding result and the initial frame interpolation result, input the sum into the decoder module in the previous feature extraction layer, and input an encoding result of the encoder module in the previous feature extraction layer into the frame interpolation module in the previous feature extraction layer until a target decoding result of the decoder module in the first feature extraction layer and a target frame interpolation result of the frame interpolation module in the first feature extraction layer are obtained;

[0093] add the target decoding result and the target frame interpolation result to obtain the deep feature map sequence of the initial video.

[0094] Optionally, the encoder module includes an adaptive multi-level convolution unit and a frequency division converter unit.

[0095] input the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain an initial encoding result, including:

[0096] extract local details in the shallow feature map sequence through the adaptive multi-level convolution unit;

[0097] extract global information in the shallow feature map sequence through the frequency division converter unit;

[0098] take the local details and the global information as the initial encoding result.

[0099] Optionally, the adaptive multi-level convolution unit includes two multi-axis 3D convolution sub-units and a fusion sub-unit.

[0100] The local details in the shallow feature map sequence are extracted by the adaptive multi-stage convolution unit, including:

[0101] The shallow feature map sequence is one-dimensionally time-convolved by a first multi-axis 3D convolution subunit to obtain time-domain features of the shallow feature map sequence;

[0102] The shallow feature map sequence is two-dimensionally space-convolved by a second multi-axis 3D convolution subunit to obtain space-domain features of the shallow feature map sequence;

[0103] The time-domain features and the space-domain features are fused by the fusion subunit, and local details are extracted from the features obtained after fusion.

[0104] Optionally, global information in the shallow feature map sequence is extracted by the frequency division converter unit, including:

[0105] The shallow feature map sequence input into the frequency division converter unit is divided into a first subsequence and a second subsequence;

[0106] Global information of the first subsequence is extracted by a self-attention mechanism, and the second subsequence is convolved, and high-frequency information in the global information is supplemented according to the convolution result.

[0107] The video reconstruction device provided in the embodiments of the present application can perform the video reconstruction method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0108] Embodiment four

[0109] Figure 11 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0110] As Figure 11As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0112] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the video reconstruction method.

[0113] In some embodiments, the video reconstruction method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the video reconstruction method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the video reconstruction method by any other appropriate means, such as by means of firmware.

[0114] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0115] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0116] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0118] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0119] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0120] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0121] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A method of video reconstruction, characterized by, The method comprises: double cubic upsampling the initial video to be reconstructed to obtain a first video; inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency converter unit in the target reconstruction model; adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video; wherein the target reconstruction model comprises a shallow feature extraction submodel, a deep feature extraction submodel, and a video reconstruction submodel; inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video, comprising: inputting the initial video into the shallow feature extraction submodel to obtain a shallow feature map sequence of the initial video; inputting the shallow feature map sequence into the deep feature extraction submodel to obtain a deep feature map sequence of the initial video; inputting the deep feature map sequence into the video reconstruction submodel to generate a second video containing deep features of the initial video; wherein the adaptive multi-level convolution unit comprises two multi-axis 3D convolution subunits and a fusion subunit; extracting local details in the shallow feature map sequence through the adaptive multi-level convolution unit, comprising: performing one-dimensional time convolution on the shallow feature map sequence through a first multi-axis 3D convolution subunit to obtain time domain features of the shallow feature map sequence; performing two-dimensional spatial convolution on the shallow feature map sequence through a second multi-axis 3D convolution subunit to obtain spatial domain features of the shallow feature map sequence; fusing the time domain features and the spatial domain features through the fusion subunit, and extracting local details from the fused features; extracting global information in the shallow feature map sequence through the frequency converter unit, comprising: dividing the shallow feature map sequence input into the frequency converter unit into a first subsequence and a second subsequence; extracting global information of the first subsequence through a self-attention mechanism, and performing convolution on the second subsequence, and supplementing high-frequency information in the global information according to the convolution result.

2. The method of claim 1, wherein, The process of predetermining the target reconstruction model comprises: preprocessing a video dataset to obtain a first sample video, and performing downsampling processing on the first sample video according to a preset sampling multiple to obtain a second sample video; inputting the second sample video into an initial reconstruction model to obtain an output result, and calculating a loss function according to the first sample video and the output result; iteratively updating the parameter weights in the initial reconstruction model through the loss function until the iteration number reaches a preset upper limit of iteration number, to obtain the target reconstruction model.

3. The method of claim 1, wherein, The deep feature extraction sub-model is in a U-shaped network architecture, and includes at least two feature extraction layers, each of which includes an encoder module, a frame interpolation module and a decoder module, and the frame interpolation module is arranged at a skip connection of the U-shaped network; The shallow feature map sequence is input into the deep feature extraction sub-model to obtain a deep feature map sequence of the initial video, including: The shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result; The initial encoding result is input into the encoder module in the next feature extraction layer until the target encoding result of the encoder module in the last feature extraction layer is obtained; The target encoding result is input into the decoder module in the last feature extraction layer to obtain an initial decoding result, and the target encoding result is input into the frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result; After the initial decoding result and the initial frame interpolation result are added, they are input into the decoder module in the previous feature extraction layer, and the encoding result of the encoder module in the previous feature extraction layer is input into the frame interpolation module in the previous feature extraction layer until the target decoding result of the decoder module in the first feature extraction layer and the target frame interpolation result of the frame interpolation module in the first feature extraction layer are obtained; The target decoding result and the target frame interpolation result are added to obtain the deep feature map sequence of the initial video.

4. The method of claim 3, wherein, The encoder module includes an adaptive multi-level convolution unit and a frequency divider converter unit; The shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result, including: Local details in the shallow feature map sequence are extracted through the adaptive multi-level convolution unit; Global information in the shallow feature map sequence is extracted through the frequency divider converter unit; The local details and the global information are taken as the initial encoding result.

5. A video reconstruction apparatus characterized by comprising: The device includes: A first video determination module for performing bicubic up-sampling on an initial video to be reconstructed to obtain a first video; A second video determination module for inputting the initial video into a pre-determined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained through an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained through a frequency divider converter unit in the target reconstruction model; A target video determination module for adjusting the deep features in the second video and fusing the adjusted second video with the first video to obtain a target video; The target reconstruction model includes a shallow feature extraction sub-model, a deep feature extraction sub-model and a video reconstruction sub-model; The second video determination module includes: A shallow feature extraction unit for inputting the initial video into the shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video; a deep feature extraction unit, configured to input the shallow feature map sequence into the deep feature extraction sub-model to obtain a deep feature map sequence of the initial video; a second video determination unit, configured to input the deep feature map sequence into the video reconstruction sub-model to generate a second video containing deep features of the initial video; wherein the adaptive multi-level convolution unit comprises two multi-axis 3D convolution sub-units and a fusion sub-unit; the adaptive multi-level convolution unit is configured to extract local details in the shallow feature map sequence, including: a first multi-axis 3D convolution sub-unit is configured to perform one-dimensional time convolution on the shallow feature map sequence to obtain time domain features of the shallow feature map sequence; a second multi-axis 3D convolution sub-unit is configured to perform two-dimensional space convolution on the shallow feature map sequence to obtain space domain features of the shallow feature map sequence; the fusion sub-unit is configured to fuse the time domain features and the space domain features, and extract local details from the fused features; the frequency division converter unit is configured to extract global information in the shallow feature map sequence, including: the shallow feature map sequence input into the frequency division converter unit is divided into a first sub-sequence and a second sub-sequence; a self-attention mechanism is used to extract global information of the first sub-sequence, and the second sub-sequence is convolved, and high-frequency information in the global information is supplemented according to the convolution result.

6. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the video reconstruction method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the video reconstruction method of any one of claims 1-4 when executed.

Images