Video reconstruction method and device, electronic equipment and storage medium

Abstract

The invention discloses a video reconstruction method and device, electronic equipment and a storage medium. The method comprises the following steps: performing bicubic up-sampling on an initial video to be reconstructed to obtain a first video; inputting the initial video into a predetermined target reconstruction model to obtain a second video containing the deep features of the initial video; wherein the deep features comprise local details and global information; the local details are obtained through a self-adaptive multistage convolution unit in the target reconstruction model, and the global information is obtained through a frequency division converter unit in the target reconstruction model; and adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video. According to the technical scheme provided by the embodiment of the invention, more comprehensive feature information can be provided for the video reconstruction task of the initial video, so that the quality of the reconstructed target video is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video reconstruction method, device, electronic equipment and storage medium. Background Art

[0002] In actual production and life, the hardware devices on imaging devices such as cameras are limited by their own process, such as the size and dimensions of the photosensitive pixels of the sensor. In fact, the images obtained by these sensors often cannot meet the requirements of different application environments. In addition, the video quality may also be degraded during the transmission of the video. Therefore, how to optimize the video reconstruction technology from the software level through precise algorithms and technical means, and then improve the video quality through video reconstruction technology, has become an important academic research that has attracted much attention. In the existing technology, video reconstruction is mainly carried out in the following two ways: 1. Directly regard video reconstruction as a composite task of video super-resolution and video frame interpolation, and perform video frame interpolation and video super-resolution successively. 2. Train a single end-to-end model based on deep learning. This network model only extracts features from the input video frame once, and then upsamples the extracted features in the network model in time and space order. The first method does not take advantage of the intrinsic connection between time interpolation and spatial super-resolution, and the second method tends to focus on some features in the video frame, resulting in the loss of other features. Summary of the invention

[0003] The present invention provides a video reconstruction method, device, electronic device and storage medium, which can provide more comprehensive feature information for the video reconstruction task of the initial video, thereby improving the quality of the reconstructed target video.

[0004] According to one aspect of the present invention, a video reconstruction method is provided, the method comprising:

[0005] Perform bicubic upsampling on the initial video to be reconstructed to obtain a first video;

[0006] Inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model;

[0007] The deep features in the second video are adjusted, and the adjusted second video is fused with the first video to obtain a target video.

[0008] According to another aspect of the present invention, a video reconstruction device is provided, the device comprising:

[0009] A first video determination module, used for performing bicubic upsampling on the initial video to be reconstructed to obtain a first video;

[0010] A second video determination module is used to input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-stage convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model;

[0011] The target video determination module is used to adjust the deep features in the second video and fuse the adjusted second video with the first video to obtain the target video.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video reconstruction method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video reconstruction method described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention is to perform bicubic upsampling on the initial video to be reconstructed to obtain a first video; input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-stage convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model; adjust the deep features in the second video, and fuse the adjusted second video with the first video to obtain a target video. The technical solution of the embodiment of the present invention, by combining the extraction of local details by an adaptive multi-stage convolution unit and the extraction of global information by a frequency division converter unit, can enable the target reconstruction model to understand the initial video more comprehensively, provide more comprehensive feature information for the video reconstruction task of the initial video, and thereby improve the quality of the reconstructed target video.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a video reconstruction method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a structural schematic diagram of a target reconstruction model provided according to Embodiment 1 of the present invention;

[0022] Figure 3 is a flowchart of a video reconstruction method provided according to Embodiment 2 of the present invention;

[0023] Figure 4 is a structural diagram of a frame interpolation module provided according to Embodiment 2 of the present invention;

[0024] Figure 5 is a schematic diagram of the structure of an adaptive multi-stage convolution unit provided according to Embodiment 2 of the present invention;

[0025] Figure 6 is a schematic diagram of the structure of a multi-axis 3D convolution subunit provided according to the second embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the structure of a fusion subunit provided according to the second embodiment of the present invention.

[0027] Figure 8 is a structural schematic diagram of a frequency division converter unit provided according to Embodiment 2 of the present invention;

[0028] Fig. 9 is a structural schematic diagram of a feature extraction layer of a frequency division converter unit provided according to a second embodiment of the present invention;

[0029] Fig.10 is a structural schematic diagram of a video reconstruction device provided according to Embodiment 3 of the present invention;

[0030] Fig.11 It is a schematic diagram of the structure of an electronic device for implementing the video reconstruction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 A flowchart of a video reconstruction method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of reconstructing a low-resolution, low-frame-rate video into a high-resolution, high-frame-rate video. The method can be executed by a video reconstruction device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110 , performing bicubic upsampling on the initial video to be reconstructed to obtain a first video.

[0036] The initial video to be reconstructed may be a video with low resolution and low frame rate, such as an old video resource, or a low-quality video obtained by a download service provided by a website. Optionally, a video with a resolution lower than a first preset threshold and / or a frame rate lower than a second preset threshold may be used as the initial video. The first preset threshold and the second preset threshold may be set according to actual conditions, and the embodiments of the present invention do not limit this. Bicubic upsampling is an interpolation method commonly used in image and video processing, which can improve the resolution of an image or video.

[0037] In an embodiment of the present invention, after determining the initial video that needs to be reconstructed, the initial video can be bicubic upsampled by video processing software or programming to generate a first video with higher resolution. It should be noted that bicubic upsampling increases the size of the video file and the required storage space while improving the resolution of the video, and excessive upsampling may cause the video to be blurred. Therefore, when bicubic upsampling the initial video, it is necessary to select appropriate parameters to balance the quality and file size of the output video.

[0038] S120. Input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model.

[0039] The target reconstruction model includes a shallow feature extraction submodel, a deep feature extraction submodel and a video reconstruction submodel. The deep feature extraction submodel has a U-shaped network architecture, including at least two feature extraction layers, each of which includes an encoder module, a decoder module and a frame interpolation module set at the jump connection of the U-shaped network. The encoder module includes an adaptive multi-level convolution unit and a frequency division converter unit, which are respectively used to extract local details and global information from the shallow features of the initial video as the deep features of the initial video. Exemplarily, Figure 2 A schematic structural diagram of a target reconstruction model is shown.

[0040] In an embodiment of the present invention, the initial video can be input into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video. Specifically, the initial video can be first input into a shallow feature extraction submodel, and the shallow features of the initial video can be extracted by the shallow feature extraction submodel. The shallow features are then input into a deep feature extraction submodel, and the local details and global information in the shallow features are extracted by the adaptive multi-level convolution unit and the frequency division converter unit in the deep feature extraction submodel as the deep features of the initial video. Finally, the deep features of the initial video are input into a video reconstruction submodel, and a second video containing the deep features of the initial video with higher resolution and higher frame rate is reconstructed by the video reconstruction submodel. By combining the adaptive multi-level convolution unit to extract local details and the frequency division converter unit to extract global information, the target reconstruction model can be made more adaptable to process video data of different scales and complexities. At the same time, the target reconstruction model can be made to understand the entire initial video more comprehensively, and provide more comprehensive background information for the high-resolution and high-frame-rate reconstruction of video frames in the initial video.

[0041] Optionally, the process of predetermining the target reconstruction model includes: preprocessing the video data set to obtain a first sample video, and downsampling the first sample video according to a preset sampling multiple to obtain a second sample video; inputting the second sample video into the initial reconstruction model to obtain an output result, and calculating a loss function based on the first sample video and the output result; back-propagating through the loss function, iteratively updating the parameter weights in the initial reconstruction model until the number of iterations reaches a preset upper limit of the number of iterations, thereby obtaining the target reconstruction model.

[0042] The video dataset may be a publicly available super-resolution training video dataset obtained from the Internet. H The video data set is preprocessed to obtain a high-resolution, high-frame-rate video. The second sample video I L The first sample video is downsampled to obtain a low-resolution, low-frame-rate video. The initial reconstruction model is a convolutional neural network model before the parameter weights in the target reconstruction model are iteratively updated.

[0043] In the embodiment of the present invention, first, the acquired video data set is preprocessed to obtain a first sample video I H Optionally, preprocessing the video data set includes extracting a y channel from the video frame image, rotating or flipping the video frame image to perform data enhancement, cropping the video frame image, etc. H Perform downsampling processing to obtain the second sample video I LThe preset sampling multiple can be adjusted according to the actual situation. Then, the second sample video is input into the initial reconstruction model to obtain the output result. And according to the first sample video I H And the output result Calculating the loss function Finally, back propagation can be performed based on the calculated loss function to iteratively update the parameter weights in the initial reconstruction model until the number of iterations reaches the preset upper limit of the number of iterations to obtain the target reconstruction model.

[0044] Optionally, the formula for calculating the loss function based on the first sample video and the output result is as follows:

[0045]

[0046] in, is the loss function, I H For the first sample video, is the output result of the initial reconstruction model, and ε is a constant.

[0047] S130: Adjust the deep features in the second video, and fuse the adjusted second video with the first video to obtain a target video.

[0048] In the embodiment of the present invention, after obtaining the second video through the target reconstruction model, it is necessary to adjust the deep features in the second video, further optimize or enhance the deep features extracted from the initial video, so as to improve the quality of the target video finally obtained. Then, the adjusted second video can be fused with the first video through a specific algorithm to ensure that the two videos maintain visual consistency and coherence, obtain a high-resolution, high-frame-rate target video, and complete the video reconstruction of the initial video.

[0049] The technical solution of the embodiment of the present invention is to perform bicubic upsampling on the initial video to be reconstructed to obtain a first video; input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-stage convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model; the deep features in the second video are adjusted, and the adjusted second video is fused with the first video to obtain the target video. The technical solution of the embodiment of the present invention, by combining the adaptive multi-stage convolution unit to extract local details and the frequency division converter unit to extract global information, can enable the target reconstruction model to understand the initial video more comprehensively, provide more comprehensive feature information for the video reconstruction task of the initial video, and thus improve the quality of the reconstructed target video.

[0050] Embodiment 2

[0051] Figure 3 This is a flow chart of a video reconstruction method provided by Embodiment 2 of the present invention. The embodiment of the present invention is optimized based on the above embodiment. For solutions not described in detail in the embodiment of the present invention, please refer to the above embodiment. Figure 3 As shown, the method includes:

[0052] S210 , performing bicubic upsampling on the initial video to be reconstructed to obtain a first video.

[0053] S220, inputting the initial video into the shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video.

[0054] The shallow feature map sequence refers to a video frame sequence including shallow features of the initial video. The input is sent to the shallow feature extraction sub-model in the target reconstruction model, and the initial video is extracted through the 3*3 convolutional layer in the shallow extraction sub-model. Perform shallow feature extraction to obtain a shallow feature map sequence of the initial video

[0055]

[0056] S230, inputting the shallow feature map sequence into the deep feature extraction sub-model to obtain the deep feature map sequence of the initial video.

[0057] The deep feature map sequence refers to a video frame sequence that includes the deep features of the initial video, that is, local details and global information. Afterwards, the shallow feature map sequence can be Input into the deep feature extraction sub-model to obtain the deep feature map sequence x of the initial video d1 , used to restore clearer high-resolution, high-frame rate video.

[0058] Optionally, the shallow feature map sequence is input into the deep feature extraction submodel to obtain the deep feature map sequence of the initial video, including: inputting the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain an initial encoding result; inputting the initial encoding result into the encoder module in the next feature extraction layer until the target encoding result of the encoder module in the last feature extraction layer is obtained; inputting the target encoding result into the decoder module in the last feature extraction layer to obtain an initial decoding result, and inputting the target encoding result into the frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result; after adding the initial decoding result to the initial frame interpolation result, inputting them into the decoder module in the previous feature extraction layer, and inputting the encoding result of the encoder module in the previous feature extraction layer into the frame interpolation module in the previous feature extraction layer until the target decoding result of the decoder module in the first feature extraction layer and the target frame interpolation result of the frame interpolation module in the first feature extraction layer are obtained; adding the target decoding result and the target frame interpolation result to obtain the deep feature map sequence of the initial video.

[0059] The frame interpolation module first extracts residual features from the input frames, then uses the correlation between adjacent frames to enhance the result of another frame through the cross-attention mechanism, and then adds the two cross-attention results together to obtain the intermediate frame of the two adjacent frames through the fusion unit. For example, Figure 4 A structural schematic diagram of a frame interpolation module is shown.

[0060] In the embodiment of the present invention, the deep feature extraction sub-model is a U-shaped network architecture, including at least two feature extraction layers, each of which includes an encoder module, a decoder module, and a frame interpolation module set at the jump connection of the U-shaped network. Here, the deep feature sub-model includes 4 feature extraction layers as an example to explain the extraction process of the deep feature map sequence: 1. Input the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain the initial encoding result X e1 , x e1 Input to the encoder module in the second feature extraction layer to get x e2 , and so on, until the target encoding result x of the encoder module in the fourth feature extraction layer is obtained e4 . 2. The target encoding result x e4 Input the decoder module in the fourth feature extraction layer to obtain the initial decoding result x m4 ; Encode the target result x e4 Input the frame interpolation module in the fourth feature extraction layer to obtain the initial frame interpolation result x n4 ; x m4 With x n4After adding, we get x d4 , x d4 Input the decoder module in the third feature extraction layer to get x m3 ; The encoding result x of the encoder module in the third feature extraction layer e3 Input to the frame interpolation module in the third feature extraction layer to get x n3 , and so on, until the target decoding result x of the decoder module in the first feature extraction layer is obtained m And the target frame interpolation result x of the frame interpolation module in the first feature extraction layer n1 3. Change x m1 With x n After adding, we get x d1 As a sequence of deep feature maps of the initial video, the encoder-decoder structure and the jump connection design of the U-net can retain rich spatial information at different levels, which helps to accurately restore details and better retain and restore the information of the initial video.

[0061] Optionally, the shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result, including: extracting local details in the shallow feature map sequence through the adaptive multi-stage convolution unit; extracting global information in the shallow feature map sequence through the frequency division converter unit; and using the local details and the global information as the initial encoding result.

[0062] In an embodiment of the present invention, local details in a shallow feature map sequence can be extracted by an adaptive multi-stage convolution unit in an encoder module, global information in a shallow feature map sequence can be extracted by a frequency divider unit in an encoder module, and the local details and the global information are used as initial encoding results. By using a processing scheme combining local details with global information, the problem that the adaptive multi-stage convolution unit has a small field of view and the frequency divider unit has a poor processing effect on high-frequency information can be avoided, and a better video frame can be reconstructed with a smaller number of frames.

[0063] Optionally, local details in the shallow feature map sequence are extracted by the adaptive multi-stage convolution unit, including: performing one-dimensional time convolution on the shallow feature map sequence by a first multi-axis 3D convolution subunit to obtain the time domain features of the shallow feature map sequence; performing two-dimensional spatial convolution on the shallow feature map sequence by a second multi-axis 3D convolution subunit to obtain the spatial domain features of the shallow feature map sequence; fusing the time domain features and spatial domain features by the fusion subunit, and extracting local details from the fused features.

[0064] The adaptive multi-level convolution unit includes two multi-axis 3D convolution subunits and a fusion subunit. The multi-axis 3D convolution subunit is used to capture different levels of abstract representation of shallow feature map sequences and increase the expressive power of the model. The fusion subunit is used to fuse features and extract information from the fused features. For example, Figure 5 A schematic diagram of the structure of an adaptive multi-stage convolution unit is shown. Figure 6 A schematic diagram of the structure of a multi-axis 3D convolution subunit is shown. Figure 7 A schematic diagram of the structure of a fusion subunit is shown.

[0065] In an embodiment of the present invention, pseudo-3D convolution can be performed on a shallow feature map sequence through an adaptive multi-stage convolution unit to extract local details in the shallow feature map sequence, so as to utilize the local contextual information of the shallow feature map sequence. Specifically, the shallow feature map sequence can be first subjected to one-dimensional temporal convolution through the first multi-axis 3D convolution subunit to obtain the temporal domain features of the shallow feature map sequence, and then the shallow feature map sequence can be subjected to two-dimensional spatial convolution through the second multi-axis 3D convolution subunit to obtain the spatial domain features of the shallow feature map sequence. Afterwards, the temporal domain features and the spatial domain features can be fused through the fusion subunit, and local details can be extracted from the fused features. By performing pseudo-3D convolution on the shallow feature map sequence through an adaptive multi-stage convolution unit, while maintaining computational efficiency, it is possible to more accurately capture short-term motion information in the shallow feature map sequence, providing more refined features for video reconstruction.

[0066] Optionally, the global information in the shallow feature map sequence is extracted by the frequency division converter unit, including: dividing the shallow feature map sequence input into the frequency division converter unit into a first subsequence and a second subsequence; extracting the global information of the first subsequence through a self-attention mechanism, and convolving the second subsequence, and supplementing the high-frequency information in the global information according to the convolution result.

[0067] The frequency division converter unit is used to divide the number of channels of the input feature into two parts, one part is self-attention, and the other part is convolution to compensate for the loss of high-frequency information in the self-attention process. Figure 8 shows a schematic structural diagram of a frequency division converter unit, Fig. 9 A schematic structural diagram of a feature extraction layer of a frequency division converter unit is shown.

[0068] In an embodiment of the present invention, the shallow feature map sequence input to the frequency division converter unit can be first divided into a first subsequence and a second subsequence, and then the global information of the first subsequence is extracted through a self-attention mechanism, and the second subsequence is convolved, and the convolution channel is first through a depth-separable convolution with a convolution kernel of (5, 5, 5), and then two parallel convolutions are used to extract high-frequency detail information under a large receptive field of view to supplement the high-frequency information in the global information. By extracting the global information of the shallow feature map sequence through the frequency division converter unit, the global spatiotemporal information can be utilized, so that the target reconstruction model can simultaneously focus on all positions in the shallow feature map sequence without being restricted by a fixed receptive field of view, so as to further explore the evolution trend, dynamic changes and global structural characteristics of the entire shallow feature map sequence.

[0069] S240, inputting the deep feature map sequence into the video reconstruction sub-model to generate a second video containing the deep features of the initial video.

[0070] In an embodiment of the present invention, the deep feature map sequence output by the deep feature extraction submodel can be input into the video reconstruction submodel, the height and width dimensions of the feature maps in the deep feature map sequence are restored to the height and width dimensions of the initial video, and a second video containing the deep features of the initial video is generated.

[0071] S250: adjusting the deep features in the second video, and fusing the adjusted second video with the first video to obtain a target video.

[0072] The technical solution of the embodiment of the present invention is to perform bicubic upsampling on the initial video to be reconstructed to obtain a first video; input the initial video into a shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video; input the shallow feature map sequence into a deep feature extraction sub-model to obtain a deep feature map sequence of the initial video; input the deep feature map sequence into a video reconstruction sub-model to generate a second video containing deep features of the initial video; adjust the deep features in the second video, and fuse the adjusted second video with the first video to obtain a target video. The technical solution of the embodiment of the present invention, by combining an adaptive multi-stage convolution unit to extract local details and a frequency division converter unit to extract global information, can enable the target reconstruction model to understand the initial video more comprehensively, provide more comprehensive feature information for the video reconstruction task of the initial video, and thus improve the quality of the reconstructed target video.

[0073] Embodiment 3

[0074] Fig.10 FIG. 1 is a schematic diagram of the structure of a video reconstruction device provided in Embodiment 3 of the present invention. Fig.10 As shown, the device comprises:

[0075] A first video determination module 310, configured to perform bicubic upsampling on the initial video to be reconstructed to obtain a first video;

[0076] A second video determination module 320 is used to input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-stage convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model;

[0077] The target video determination module 330 is used to adjust the deep features in the second video and fuse the adjusted second video with the first video to obtain a target video.

[0078] Optionally, the second video determination module 320 includes:

[0079] A sample video determination unit, configured to preprocess the video data set to obtain a first sample video, and downsample the first sample video according to a preset sampling multiple to obtain a second sample video;

[0080] a loss function calculation unit, configured to input the second sample video into an initial reconstruction model to obtain an output result, and calculate a loss function according to the first sample video and the output result;

[0081] The target model determination unit is used to perform back propagation through the loss function to iteratively update the parameter weights in the initial reconstruction model until the number of iterations reaches a preset upper limit of the number of iterations to obtain the target reconstruction model.

[0082] Optionally, the target reconstruction model includes a shallow feature extraction sub-model, a deep feature extraction sub-model and a video reconstruction sub-model;

[0083] The second video determination module 320 includes:

[0084] A shallow feature extraction unit, used for inputting the initial video into the shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video;

[0085] A deep feature extraction unit, used for inputting the shallow feature map sequence into the deep feature extraction sub-model to obtain a deep feature map sequence of the initial video;

[0086] The second video determination unit is used to input the deep feature map sequence into the video reconstruction sub-model to generate a second video containing the deep features of the initial video.

[0087] Optionally, the deep feature extraction submodel is in a U-shaped network architecture, including at least two feature extraction layers, each feature extraction layer includes an encoder module, a frame interpolation module and a decoder module, and the frame interpolation module is arranged at a jump connection of the U-shaped network;

[0088] The deep feature extraction unit is specifically used for:

[0089] Inputting the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain an initial encoding result;

[0090] Inputting the initial encoding result into the encoder module in the next feature extraction layer until the target encoding result of the encoder module in the last feature extraction layer is obtained;

[0091] Inputting the target encoding result into a decoder module in the last feature extraction layer to obtain an initial decoding result, and inputting the target encoding result into a frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result;

[0092] After adding the initial decoding result and the initial frame interpolation result, the result is input into the decoder module in the previous feature extraction layer, and the encoding result of the encoder module in the previous feature extraction layer is input into the frame interpolation module in the previous feature extraction layer, until the target decoding result of the decoder module in the first feature extraction layer and the target frame interpolation result of the frame interpolation module in the first feature extraction layer are obtained;

[0093] The target decoding result and the target frame interpolation result are added to obtain a deep feature map sequence of the initial video.

[0094] Optionally, the encoder module comprises an adaptive multi-stage convolution unit and a frequency division converter unit;

[0095] The shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result, including:

[0096] Extracting local details in the shallow feature map sequence through the adaptive multi-stage convolution unit;

[0097] Extracting global information from the shallow feature map sequence by the frequency division converter unit;

[0098] The local details and the global information are used as initial encoding results.

[0099] Optionally, the adaptive multi-stage convolution unit includes two multi-axis 3D convolution subunits and a fusion subunit;

[0100] Extracting local details in the shallow feature map sequence by the adaptive multi-stage convolution unit includes:

[0101] Performing one-dimensional temporal convolution on the shallow feature map sequence through a first multi-axis 3D convolution subunit to obtain time domain features of the shallow feature map sequence;

[0102] Performing a two-dimensional spatial convolution on the shallow feature map sequence through a second multi-axis 3D convolution subunit to obtain spatial domain features of the shallow feature map sequence;

[0103] The time domain features and the space domain features are fused by the fusion subunit, and local details are extracted from the fused features.

[0104] Optionally, extracting global information in the shallow feature map sequence by the frequency division converter unit comprises:

[0105] dividing the shallow feature map sequence input to the frequency division converter unit into a first subsequence and a second subsequence;

[0106] The global information of the first subsequence is extracted through a self-attention mechanism, and the second subsequence is convolved, and the high-frequency information in the global information is supplemented according to the convolution result.

[0107] The video reconstruction device provided in the embodiment of the present invention can execute the video reconstruction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] Embodiment 4

[0109] Fig.11 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0110] like Fig.11As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0112] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video reconstruction method.

[0113] In some embodiments, the video reconstruction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video reconstruction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video reconstruction method in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0119] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video reconstruction method, characterized in that: The method comprises: Perform bicubic upsampling on the initial video to be reconstructed to obtain a first video; Inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-level convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model; The deep features in the second video are adjusted, and the adjusted second video is fused with the first video to obtain a target video.

2. The method according to claim 1, characterized in that The process of predetermining the target reconstruction model includes: Preprocessing the video data set to obtain a first sample video, and downsampling the first sample video according to a preset sampling multiple to obtain a second sample video; Inputting the second sample video into an initial reconstruction model to obtain an output result, and calculating a loss function based on the first sample video and the output result; The parameter weights in the initial reconstruction model are iteratively updated by back-propagating the loss function until the number of iterations reaches a preset upper limit of the number of iterations, thereby obtaining a target reconstruction model.

3. According to the method of claim 1, the target reconstruction model includes a shallow feature extraction sub-model, a deep feature extraction sub-model and a video reconstruction sub-model; Inputting the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video, including: Inputting the initial video into the shallow feature extraction sub-model to obtain a shallow feature map sequence of the initial video; Inputting the shallow feature map sequence into the deep feature extraction sub-model to obtain the deep feature map sequence of the initial video; The deep feature map sequence is input into the video reconstruction sub-model to generate a second video containing the deep features of the initial video.

4. The method according to claim 3, characterized in that The deep feature extraction sub-model is in a U-shaped network architecture, including at least two feature extraction layers, each of which includes an encoder module, a frame interpolation module and a decoder module, and the frame interpolation module is arranged at the jump connection of the U-shaped network; Inputting the shallow feature map sequence into the deep feature extraction sub-model to obtain the deep feature map sequence of the initial video, including: Inputting the shallow feature map sequence into the encoder module in the first feature extraction layer to obtain an initial encoding result; Inputting the initial encoding result into the encoder module in the next feature extraction layer until the target encoding result of the encoder module in the last feature extraction layer is obtained; Inputting the target encoding result into a decoder module in the last feature extraction layer to obtain an initial decoding result, and inputting the target encoding result into a frame interpolation module in the last feature extraction layer to obtain an initial frame interpolation result; After adding the initial decoding result and the initial frame interpolation result, the result is input into the decoder module in the previous feature extraction layer, and the encoding result of the encoder module in the previous feature extraction layer is input into the frame interpolation module in the previous feature extraction layer, until the target decoding result of the decoder module in the first feature extraction layer and the target frame interpolation result of the frame interpolation module in the first feature extraction layer are obtained; The target decoding result and the target frame interpolation result are added to obtain a deep feature map sequence of the initial video.

5. The method according to claim 4, characterized in that The encoder module includes an adaptive multi-stage convolution unit and a frequency division converter unit; The shallow feature map sequence is input into the encoder module in the first feature extraction layer to obtain an initial encoding result, including: Extracting local details in the shallow feature map sequence through the adaptive multi-stage convolution unit; Extracting global information from the shallow feature map sequence by the frequency division converter unit; The local details and the global information are used as initial encoding results.

6. The method according to claim 5, characterized in that The adaptive multi-stage convolution unit includes two multi-axis 3D convolution subunits and a fusion subunit; Extracting local details in the shallow feature map sequence by the adaptive multi-stage convolution unit includes: Performing one-dimensional temporal convolution on the shallow feature map sequence through a first multi-axis 3D convolution subunit to obtain time domain features of the shallow feature map sequence; Performing a two-dimensional spatial convolution on the shallow feature map sequence through a second multi-axis 3D convolution subunit to obtain spatial domain features of the shallow feature map sequence; The time domain features and the space domain features are fused by the fusion subunit, and local details are extracted from the fused features.

7. The method according to claim 5, characterized in that Extracting global information from the shallow feature map sequence by the frequency division converter unit includes: dividing the shallow feature map sequence input to the frequency division converter unit into a first subsequence and a second subsequence; The global information of the first subsequence is extracted through a self-attention mechanism, and the second subsequence is convolved, and the high-frequency information in the global information is supplemented according to the convolution result.

8. A video reconstruction device, characterized in that: The device comprises: A first video determination module, used for performing bicubic upsampling on the initial video to be reconstructed to obtain a first video; A second video determination module is used to input the initial video into a predetermined target reconstruction model to obtain a second video containing deep features of the initial video; wherein the deep features include local details and global information; the local details are obtained by an adaptive multi-stage convolution unit in the target reconstruction model, and the global information is obtained by a frequency division converter unit in the target reconstruction model; The target video determination module is used to adjust the deep features in the second video and fuse the adjusted second video with the first video to obtain the target video.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video reconstruction method according to any one of claims 1 to 7 when executed.

Images