A video reconstruction method, apparatus, device and medium

Abstract

A video reconstruction method, device, equipment and medium are disclosed, the method comprising: obtaining a first video to be reconstructed; inputting the first video into a target reconstruction model trained in advance to perform video reconstruction, the target reconstruction model performing multi-level local information and global information extraction on the input first video, and performing frame rate and resolution reconstruction processing based on the extracted local information and global information; based on the output of the target reconstruction model, obtaining a second video after reconstruction, the frame rate of the second video being greater than that of the first video, and the resolution of the second video being greater than that of the first video. Through the technical scheme of the embodiment of the present application, accurate reconstruction of the video can be realized, the reconstruction accuracy and detail recovery capability of the video can be improved, the quality of the reconstructed video can be ensured, and the generalization capability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a video reconstruction method and device, equipment and medium. BACKGROUND

[0002] With the progress of science and technology, high-quality videos can provide people with a better visual experience. However, in the process of video transmission, in order to reduce the transmission bandwidth, frame extraction processing is used to reduce the video quality, and then the video quality is improved through video reconstruction.

[0003] At present, the traditional video reconstruction method usually estimates unknown pixels in the video data by interpolation to realize video reconstruction. However, the traditional video reconstruction method has low reconstruction accuracy and detail recovery ability, and cannot guarantee the quality of the reconstructed video, and has poor generalization ability. SUMMARY

[0004] The present application provides a video reconstruction method, device, equipment and medium to realize accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery ability of the video, guarantee the quality of the reconstructed video, and improve the generalization ability.

[0005] In a first aspect, the present application provides a video reconstruction method, comprising:

[0006] obtaining a first video to be reconstructed;

[0007] inputting the first video into a target reconstruction model trained in advance to perform video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and performs frame rate and resolution reconstruction processing based on the extracted local information and global information;

[0008] based on the output of the target reconstruction model, obtaining a second video reconstructed, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0009] In a second aspect, the present application further provides a video reconstruction device, comprising:

[0010] a first video acquisition module configured to acquire a first video to be reconstructed;

[0011] a video reconstruction module configured to input the first video into a target reconstruction model trained in advance, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and performs frame rate and resolution reconstruction processing based on the extracted local information and global information;

[0012] The second video acquisition module is configured to acquire a reconstructed second video based on the output of the target reconstruction model, the frame rate of the second video being greater than the frame rate of the first video, and the resolution of the second video being greater than the resolution of the first video.

[0013] In a third aspect, an electronic device is provided, and the electronic device includes at least one processor, and

[0014] a memory communicatively connected to the at least one processor; and

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the video reconstruction method provided by any of the embodiments of the present application.

[0016] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, and the computer instructions enable a processor to perform the video reconstruction method provided by any of the embodiments of the present application when the processor executes the computer instructions.

[0017] The technical solution of the embodiments of the present application acquires a first video to be reconstructed, and inputs the first video into a target reconstruction model pre-trained to perform video reconstruction, the target reconstruction model extracts multi-level local information and global information from the input first video, and performs frame rate and resolution reconstruction processing based on the extracted local information and global information, which can significantly improve the quality and viewing experience of the video, and make the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, a second video is acquired, the frame rate of the second video being greater than the frame rate of the first video, and the resolution of the second video being greater than the resolution of the first video. The target reconstruction model realizes automatic video reconstruction processing, which can realize accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery ability of the video, ensure the quality of the reconstructed video, and improve the generalization ability.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those skilled in the art without any creative effort should also fall within the protection scope of the present application.

[0020] Figure 1 is a flow chart of a video reconstruction method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a video reconstruction method according to an embodiment of the present application;

[0022] Figure 3 is a structural diagram of a target reconstruction model according to an embodiment of the present application;

[0023] Figure 4 is a structural diagram of an information extraction layer in an encoder according to an embodiment of the present application;

[0024] Figure 5 is a structural diagram of a multi-scale gated big kernel convolution unit according to an embodiment of the present application;

[0025] Figure 6 is a structural diagram of a convolution gated aggregation unit according to an embodiment of the present application;

[0026] Figure 7 is a structural diagram of an information extraction layer in a decoder according to an embodiment of the present application;

[0027] Figure 8 is a structural schematic diagram of a video reconstruction device according to an embodiment of the present application;

[0028] Figure 9 is a structural schematic diagram of an electronic device implementing a video reconstruction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those skilled in the art without any creative effort should also fall within the protection scope of the present application.

[0030] It is to be understood that the terminology "target", "current" and the like used in the specification and the claims of the application and the above-described drawings is intended to describe differentiable objects, and is not necessarily intended to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment one

[0032] Figure 1 A flowchart of a video reconstruction method is provided for the first embodiment of the application, which can be applicable in the case of video reconstruction. As shown in the figure, Figure 1 The method can be performed by a video reconstruction device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, Figure 1 The method specifically includes the following steps:

[0033] S110, obtaining a first video to be reconstructed.

[0034] The first video can refer to a video with low frame rate and low resolution. It should be noted that the low frame rate and low resolution can be determined by a preset frame rate threshold and a preset resolution threshold.

[0035] Specifically, a video can be captured using a video capture device (such as a camera), or an existing video file can be obtained through a file transmission method (such as network download, local reading, etc.), as the first video to be reconstructed.

[0036] S120, inputting the first video into a target reconstruction model trained in advance to perform video reconstruction, the target reconstruction model being configured to extract multi-level local information and global information from the input first video, and perform frame rate and resolution reconstruction processing based on the extracted local information and global information.

[0037] The target reconstruction model can be a U-shaped neural network for improving frame rate and resolution. The local information can refer to detailed features around a small area or specific pixel points in a video frame, which usually include texture, edge, color change and other detailed information. The global information can refer to overall features of the entire scene or object in a video frame, which usually include scene layout, relationship between objects, lighting conditions, overall color distribution, etc.

[0038] Specifically, the target reconstruction model performs multi-level feature extraction on the input first video. During the feature extraction process, the model focuses on both local information (such as edges, textures, etc.) and global information (such as scene layout, object motion trajectory, etc.) of the first video. After obtaining rich local and global information, the model uses this local and global information to reconstruct the frame rate and resolution of the video. For example, for frame rate reconstruction, the target reconstruction model can use methods such as time interpolation or motion estimation to increase the number of frames in the video, thereby improving the smoothness of the video; for resolution reconstruction, the model can use super-resolution reconstruction techniques such as interpolation-based, reconstruction-based, or learning-based methods to generate higher-resolution video frames. Through multi-level local and global information extraction, rich details and overall structure in the video can be captured, providing strong support for subsequent reconstruction processing, which can significantly improve the quality and viewing experience of the video, making the reconstructed video smoother and clearer.

[0039] For example, the target reconstruction model is obtained by training based on a first sample video and a second sample video; wherein the first sample video is obtained by down-sampling the frame rate and resolution of the second sample video.

[0040] The first sample video can be a video with low resolution and low frame rate obtained by down-sampling the frame rate and resolution of the second sample video, used for model training. The second sample video can be a video with high frame rate and high resolution, used as a reference or benchmark for model training.

[0041] For example, before training, the original data set should be pre-processed. First, the video frame data is cropped to a uniform size of 256x256. To perform data augmentation, the video frames are randomly rotated or flipped. Finally, the second sample video is obtained The high-resolution frame sequence is down-sampled by a factor of r to obtain the first sample video for training input

[0042] For example, the training process of the target reconstruction model includes: inputting the first sample video into the target reconstruction model to be trained for frame rate and resolution reconstruction processing, obtaining a sample reconstruction video corresponding to the first sample video; based on the sample reconstruction video and the second sample video, determining the training error, and propagating the training error back to the target reconstruction model to adjust the parameters in the target reconstruction model until the preset convergence condition is reached, and the training is completed, obtaining the target reconstruction model.

[0043] The sample reconstructed video can refer to a video obtained by performing frame rate and resolution reconstruction on the first sample video by the target reconstruction model to be trained. The training error can refer to the difference between the sample reconstructed video and the second sample video during the model training process. The preset convergence condition can refer to a condition for determining whether the training is completed during the model training process.

[0044] Specifically, the first sample video is input into the target reconstruction model to be trained. The model performs frame rate and resolution reconstruction on the input video, and generates a sample reconstructed video by using the functions of the multi-level local information and global information extraction, encoding, decoding, frame rate reconstruction, and resolution reconstruction sub-models. Through the input and reconstruction processing steps, the model can learn how to extract key information from low-quality videos and generate high-quality reconstructed videos. Based on the sample reconstructed video and the second sample video, the training error is calculated. The training error is usually determined by comparing the difference between the reconstructed video and the reference video, and can be measured by using evaluation indicators such as loss functions. The training error is back-propagated to the target reconstruction model. During the back-propagation process, the error signal is transmitted backward along the levels of the model, guiding the adjustment of the model parameters. Through the determination and back-propagation of the training error, the model can identify the deficiencies in the reconstruction process and adjust accordingly. This helps the model to gradually optimize its parameters and improve the quality of the reconstructed video. According to the error signal of the back-propagation, the parameters in the target reconstruction model are adjusted. The parameter adjustment is usually realized by using optimization algorithms such as gradient descent, aiming to minimize the training error. It is determined whether the model meets the preset convergence condition. The convergence condition can include that the training error reaches a certain threshold, the number of training reaches a preset number, etc. When the convergence condition is met, the training process is completed, and the final target reconstruction model is obtained. Through the parameter adjustment, the model can gradually optimize its reconstruction performance and improve the quality of video reconstruction. The convergence judgment ensures the stability and effectiveness of the training process, avoiding the problems of over-training or under-training.

[0045] For example, the reconstructed high-resolution video is trained with the target high-resolution video I HR (Reference video) to calculate the loss function The training is performed, and ε=0.001 during the experiment. After the training reaches the set number of iterations, the final network model weight is obtained.

[0046] S130, based on the output of the target reconstruction model, a reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0047] The second video can be a video with high frame rate and high resolution obtained by the target reconstruction model performing frame rate and resolution reconstruction on the first video.

[0048] Specifically, after the target reconstruction model completes the frame rate and resolution reconstruction processing, the reconstructed video data is output, that is, the final reconstructed second video is obtained. The reconstructed second video has a higher frame rate and resolution, thereby providing a better viewing experience and ensuring that the reconstructed video is consistent with the original video in format and encoding, facilitating subsequent playback and transmission.

[0049] The technical scheme of the embodiment of the application, by obtaining the first video to be reconstructed and inputting the first video into the target reconstruction model pre-trained to perform video reconstruction, the target reconstruction model extracts the local information and global information of the input first video in multiple levels, and performs frame rate and resolution reconstruction processing based on the extracted local information and global information, which can significantly improve the quality and viewing experience of the video, making the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, the reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video. Through the target reconstruction model, automatic video reconstruction processing is realized, which can realize accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery capability of the video, ensure the quality of the reconstructed video, and improve the generalization capability.

[0050] Embodiment two

[0051] Figure 2 A flowchart of a video reconstruction method provided by the second embodiment of the application, the present embodiment is based on the above-mentioned embodiments, the target reconstruction model is divided into a feature extraction sub-model, an encoding sub-model, a decoding sub-model, a frame rate reconstruction sub-model and a resolution reconstruction sub-model, and the step of "inputting the first video into the target reconstruction model pre-trained to perform video reconstruction" is optimized. The explanations of the same or corresponding terms as in the above-mentioned embodiments are not repeated here.

[0052] Referring to Figure 2 Another video reconstruction method provided by the present embodiment specifically includes the following steps:

[0053] S210, obtaining a first video to be reconstructed.

[0054] S220, inputting the first video into a feature extraction sub-model to perform shallow feature extraction and obtaining a shallow feature map sequence.

[0055] The feature extraction sub-model can be a neural network model for shallow feature extraction of the video. The shallow feature map sequence can be a feature map sequence composed of shallow feature maps of each frame of image in the first video.

[0056] Specifically, such as Figure 3 As shown in the figure, the first video (i.e., input data) is input into the feature extraction sub-model. The feature extraction sub-model performs shallow feature extraction on each frame of the input video through structures such as convolutional layers. The extracted shallow features are represented as feature maps and form a shallow feature map sequence. Shallow feature extraction can capture basic visual information in the video frame, such as edges and textures. The shallow feature map sequence provides a rich information foundation for subsequent processing steps.

[0057] S230: Input the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and downsampling processing to obtain a first deep feature map sequence.

[0058] The first deep feature map sequence may refer to a deep feature map sequence obtained after being processed by the encoding sub-model.

[0059] Specifically, such as Figure 3 As shown in Figure 1, a sequence of shallow feature maps is input into the encoding sub-model. The encoding sub-model further performs multi-level feature extraction and downsampling to reduce the size of the feature maps and extract higher-level local and global information. Multi-level feature extraction and downsampling help the model capture the complex structure and dynamic changes in the video. Reducing the feature map size reduces computational complexity and improves processing speed.

[0060] Exemplarily, the encoding sub-model includes a plurality of encoders connected in series, each encoder including an information extraction layer and a downsampling layer; S230 may include: inputting the shallow feature map sequence into the first encoder for local information and global information extraction and downsampling processing, and obtaining the downsampling processing result output by the first encoder; inputting the downsampling processing result output by the previous encoder into the next encoder for local information and global information extraction and downsampling processing, and obtaining the downsampling processing result output by the next encoder; and using the information extraction result output by the last encoder as the first deep feature map sequence.

[0061] The information extraction layer may refer to a layer for extracting local information and global information, and the downsampling layer may refer to a layer for reducing the spatial resolution of data (such as the width and height of an image).

[0062] Specifically, such as Figure 3As shown, the shallow feature map sequence is input into the first encoder. Inside the first encoder, the input feature map is first subjected to local information and global information extraction by the information extraction layer. The local information can include the texture, edge, and other detailed features of the image, while the global information can include the overall brightness, contrast, color distribution, and other macro features of the image, as well as the relevant information between adjacent video frames. Then, the down-sampling layer performs down-sampling processing on the extracted feature map, reducing the resolution of the feature map while retaining the key information. The first encoder outputs the down-sampling processing result, i.e., the first-level deep feature map. The down-sampling processing result output by the first encoder is input as input to the next encoder. The processing process of the subsequent encoders is similar to that of the first encoder, also including local information and global information extraction and down-sampling processing. Each encoder outputs its corresponding down-sampling processing result, i.e., a deeper level of deep feature map. After the serial processing of multiple encoders, the information extraction result output by the last encoder is taken as the first deep feature map sequence. This first deep feature map sequence contains multi-level local information and global information extracted from the shallow feature map sequence, and has undergone multiple down-sampling processing, reducing the resolution but retaining the key information. Through the serial processing of multiple encoders, the encoding sub-model can perform multi-level information extraction and down-sampling processing on the input feature map. This multi-level processing can gradually extract more abstract and advanced features, providing a more abundant information basis for subsequent video reconstruction tasks.

[0063] For example, the information extraction layer includes at least two information extraction modules connected in series; the shallow feature map sequence is input into the first encoder for local information and global information extraction and down-sampling processing to obtain the down-sampling processing result output by the first encoder, including: inputting the shallow feature map sequence into the first information extraction module for local information and global information extraction processing to obtain the information extraction result output by the first information extraction module; inputting the information extraction result output by the last information extraction module into the next information extraction module for local information and global information extraction processing to obtain the information extraction result output by the next information extraction module; inputting the information extraction result output by the last information extraction module into the down-sampling layer for down-sampling processing to obtain the down-sampling processing result output by the first encoder.

[0064] Wherein, the information extraction result can refer to the local information and global information output by the information extraction module. The down-sampling processing result can refer to the feature map with low spatial resolution after being processed by the down-sampling layer.

[0065] Specifically, the shallow feature map sequence is input into the first information extraction module. Inside the first information extraction module, the input feature map is subjected to local information and global information extraction through a series of operations. The local information can include the texture, edge and other detailed features of the image, while the global information can include the overall brightness, contrast, color distribution and other macro features of the image, as well as the relevant information between adjacent video frames. The first information extraction module outputs the extracted feature map, i.e. the first information extraction result. The feature map output by the first information extraction module is input as an input to the next information extraction module. The subsequent information extraction modules have similar processing processes as the first information extraction module, and are also responsible for extracting local information and global information. Each information extraction module outputs its corresponding extraction result, i.e. a feature map of a deeper level. After the serial processing of multiple information extraction modules, the feature map output by the last information extraction module contains multi-level local information and global information extracted from the shallow feature map sequence. The feature map output by the last information extraction module is input into the down-sampling layer. The down-sampling layer performs down-sampling processing on the input feature map, reducing the resolution of the feature map while retaining key information. The down-sampling layer outputs the processed feature map, i.e. the down-sampling processing result output by the first encoder. The serial processing of multiple information extraction modules and the introduction of the down-sampling layer enable the model to better learn the statistical characteristics of the input data, thereby improving the generalization ability of the model. This helps the model to maintain good performance when processing unseen data.

[0066] In an example, each information extraction module includes a normalization unit, an attention unit, a multi-scale gated big kernel convolution unit, and a convolution gated aggregation unit; the shallow feature map sequence is input into the first information extraction module for local information and global information extraction processing to obtain the information extraction result output by the first information extraction module, including: inputting the shallow feature map sequence into the normalization unit for standardization processing to obtain the standardized shallow feature map sequence; inputting the standardized shallow feature map sequence into the attention unit for global information extraction to obtain the global information; inputting the standardized shallow feature map sequence into the multi-scale gated big kernel convolution unit for local information extraction to obtain the local information; performing residual addition on the shallow feature map sequence, the global information and the local information, and inputting the integrated information obtained by the residual addition into the normalization unit for standardization processing to obtain the standardized integrated information; inputting the standardized integrated information into the convolution gated aggregation unit for aggregation enhancement processing, and performing residual addition on the aggregation processing result and the integrated information to obtain the information extraction result output by the first information extraction module.

[0067] The integrated information can refer to information obtained by fusing the shallow feature map sequence, the global information and the local information together through residual addition. The aggregation processing result can refer to output obtained by the convolutional gated aggregation unit after performing aggregation enhancement processing on the standardized integrated information. The normalization unit can refer to a unit for performing standardization processing on the input feature map to ensure that the distribution of data is within a reasonable range. The attention unit can refer to a unit responsible for extracting global information in the input data. It is a window-based multi-head self-attention mechanism that can capture long-term dependencies and global context information in the input data. The multi-scale gated big kernel convolutional unit is a component for extracting local information. It uses convolution kernels of different scales and gating mechanisms to extract local features from the input data. The convolutional gated aggregation unit is a component for performing aggregation enhancement processing on the features.

[0068] Specifically, as shown in FIG. 1, the input image is first input into the normalization unit 101, and the normalization unit 101 performs standardization processing on the input image to ensure that the distribution of data is within a reasonable range. The normalized image is then input into the attention unit 102, which is responsible for extracting global information in the input data. It is a window-based multi-head self-attention mechanism that can capture long-term dependencies and global context information in the input data. The output of the attention unit 102 is then input into the multi-scale gated big kernel convolutional unit 103, which is a component for extracting local information. It uses convolution kernels of different scales and gating mechanisms to extract local features from the input data. The output of the multi-scale gated big kernel convolutional unit 103 is then input into the convolutional gated aggregation unit 104, which is a component for performing aggregation enhancement processing on the features. Figure 4As shown in Figure 1, it consists of two information extraction modules with identical structures. The shallow feature map sequence is input to the normalization unit. The normalization unit normalizes the input feature maps to ensure that the data distribution is within a reasonable range, which helps ensure the stability and efficiency of subsequent processing units. It outputs a normalized shallow feature map sequence. This normalized shallow feature map sequence is input to the attention unit. The attention unit extracts global information by calculating the correlation between different positions or channels. This involves window-based multi-head self-attention and moving window-based multi-head self-attention, which can capture long-term dependencies and global contextual information in the input data. Global information is output. Simultaneously, the normalized shallow feature map sequence is input to the multi-scale gated large kernel convolution unit. This unit uses convolution kernels of different scales and a gating mechanism to extract local information from the input data. The multi-scale convolution kernels can capture features at different scales, while the gating mechanism dynamically adjusts the convolution kernel weights, improving the flexibility and accuracy of feature extraction. Local information is output. The residuals of the shallow feature map sequence, global information, and local information are summed. Residual addition is a commonly used technique that preserves some information from the input data, preventing information loss. The comprehensive information obtained from residual addition is fed back into the normalization unit for standardization to ensure data stability. The standardized comprehensive information is then output. This standardized comprehensive information is then fed into the convolutional gated aggregation unit. This unit uses convolution operations and a gating mechanism to further aggregate and enhance the input data. Convolution operations extract more refined features, while the gating mechanism dynamically adjusts feature weights to improve the robustness of feature representation. The feature extraction results are aggregated, and residual addition is performed on the aggregated results and the comprehensive information to obtain a richer feature representation. The information extraction results of the first information extraction module are then output. The multi-scale gated large kernel convolutional unit captures local features at different scales, improving the model's ability to extract local information. Residual addition and the convolutional gated aggregation unit fuse and enhance global and local information, improving the richness and robustness of the feature representation.

[0069] For example, the information extraction layer of the encoder is as follows Figure 4 As shown in the figure, each information extraction layer contains J information extraction modules. In the experiment, J can be set to 2. The main idea is to use W-MSA (window-multi-head attention) and SW-MSA (sliding window-multi-head attention) to extract global information from the video sequence, and combine it with multi-scale gated large kernel convolution units to extract local information, and then aggregate the information through the gated convolution units. The process of one information extraction module is as follows:

[0070] x2=MSA(LN(x1))+MGLK(LN(x1))+x1

[0071] x3=CGL(LN(x2))+x2

[0072] wherein, the normalization unit represents layer normalization operation, the attention unit represents multi-head attention, and there are two cases of W-MSA and SW-MSA, when the jth is W-MSA, the j+1th is SW-MSA.

[0073] x1 represents the first level input of the block, which can be the output of the feature extraction module, the output of the previous layer decoder and the output of the previous information extraction module in the module. In general, x1 is first subjected to layer normalization operation, and then the results are respectively input into the attention unit and the multi-scale gated big kernel convolution unit for processing, and then the results are added with the original input x1 in residual, to obtain x2, and then x2 is sequentially subjected to layer normalization operation and normalization unit, and then added with x2 in residual, to finally obtain the output x3 of the information extraction module. The output can be used as the input of the next information extraction module.

[0074] The attention unit specifically divides the input video frame of size into non-overlapping windows, wherein M represents the window size, M=4 in the experiment, N represents the number of video frames, H and W are the height and width of the input video frame respectively. After flattening the features in each window to generate a feature map of size , the multi-head self-attention based on the window is calculated for the self-attention in each window. Since the windows are non-overlapping, better interaction between the windows cannot be achieved. In order to further obtain global attention information, SW-MSA shifts the windows of W-MSA left and up by blocks, so that there is interaction between the window divisions of the previous layer without sliding non-overlapping, and then the self-attention in the window is calculated to obtain the global attention information.

[0075] For example, as shown in Figure 5 , in the multi-scale gated big kernel convolution unit, the feature map is first subjected to a dynamic initialization layer composed of a 1X1 convolution PWConv and an adaptive average pooling Avgpool, which is used to dynamically exclude unimportant interactions and reduce feature redundancy, so as to better focus on subsequent multi-order feature interactions.

[0076] y=Conv 1×1 (x)

[0077] z=ActLayer(y+β1×(y-Avg(y)))

[0078] wherein x represents the input feature map, Conv 1×1The 1x1 convolution layer, ActLayer, and RELU activation layers are used in this experiment. β1 is a learnable parameter with an initial value of 0 and is used for dynamic adjustment. Avg represents adaptive average pooling. The resulting dynamically initialized feature map z is used as input for subsequent gated large kernel convolution multi-order feature interaction.

[0079] z′=ActLayer(Conv 1×1 (z))×ActLayer(MLK(z))

[0080] z″=Conv 1×1 (z′)

[0081] The input dynamic initialized feature map z passes through two branches. One branch passes through a 1x1 convolution layer and an activation layer for dense linear transformation as a gating layer; the other branch passes through a multi-scale large kernel convolution layer MLK to fully extract the local information of the video image through multi-scale large kernel convolution. Specifically, First, a layer of depth-separable convolution with a convolution kernel size of 5x5 is used to extract low-order features, and then the output low-order features are divided into Among them, α1+α2+α3=1, and in the experiment, z m and z h The mid-order features z′ are output by depth-wise separable dilated convolution DwDConv1 and DwDConv2, whose convolution sum sizes are 5x5 and 7x7 respectively, and the dilation rates are 2 and 3 respectively. m ∈ and high-level features Then, z′ is calculated on the channel. m , z′ h , zl are spliced ​​together to obtain Then, it passes through a 1x1 convolution layer and an activation layer to re-aggregate the extracted multi-order features. Then, the result of the multi-order feature aggregation is multiplied by the output of the gate layer, and finally it passes through a 1x1 convolution layer for final aggregation to obtain the output.

[0082] like Figure 6 As shown, in the convolutional gated aggregation unit, first, the input After a 1x1 convolution layer, the number of channels is expanded to 2C, and we get Then separate it into channels Then p′1 is input into the depth-separable convolution layer DwConv with a convolution kernel size of 7x7 to transform and weight it, and we get Then p'1 and p'2 are multiplied, and finally pass through a 1x1 convolution layer and multiplied by a dynamic learnable parameter β2 with an initial value of 0 to obtain the output

[0083] p'1, p'2 = Split(Conv 1×1 (p))

[0084] q2 = β2 x Conv 1×1 ((DwConv 7×7 (p'1) x p'2))

[0085] S240, input the first deep feature map sequence into the frame rate reconstruction sub-model for frame rate reconstruction processing to obtain a frame rate reconstruction result.

[0086] The frame rate reconstruction result can refer to the deep feature map sequence with increased frame rate obtained after processing by the frame rate reconstruction sub-model.

[0087] Specifically, the first deep feature map sequence is input into the frame rate reconstruction sub-model. The sub-model uses techniques such as temporal interpolation, motion estimation, or optical flow to predict and insert new frames based on the feature information of existing frames. The frame rate reconstruction sub-model outputs the frame rate reconstruction result, i.e., the deep feature map sequence with increased frame rate, thereby achieving frame rate improvement. Frame rate reconstruction can improve the smoothness and dynamic performance of the video, making the video viewing experience better.

[0088] For example, as shown in FIG. 3, in the frame rate reconstruction sub-model, the first deep feature map sequence E'3 can be expanded to a 2N-1 frame mapping Figure 3

[0089]

[0090] S250, input the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model for multi-level global information extraction and up-sampling processing to obtain a second deep feature map sequence.

[0091] The second deep feature map sequence can refer to the deep feature map sequence obtained after processing by the decoding sub-model.

[0092] Specifically, the first deep feature map sequence and the frame rate reconstruction result are input into the decoding sub-model. The decoding sub-model is responsible for up-sampling the frame rate reconstructed feature map back to the original size and further extracting and integrating global information in the process. The up-sampling processing helps to restore the original size of the video frame while maintaining the quality of the reconstructed video. The extraction and integration of global information helps to improve the overall consistency and coherence of the video.

[0093] ​Exemplary, as Figure 3 shown, E'3 and Q are input into the decoder together, and the information extraction layer is obtained After another deconvolution layer (up-sampling layer), the feature map height and width are adjusted by up-sampling, and D3 and E'2 are obtained and E'2 are input into the next layer of the decoder, and so on to obtain The last layer of the decoder has only CGTB without a deconvolution layer, that is

[0094] The information extraction layer of a single decoder can be as shown in Figure 7 Similar to the encoder, each information extraction layer contains J information extraction modules, and J can be 2 in the experiment. Each information extraction module has two inputs, which are the up-sampled output Encoder Output of the corresponding layer encoder without convolution layer, that is, the above-mentioned E' i , and the output of the previous layer of the decoder, corresponding to the above-mentioned D i The D of the first decoder corresponds to the above-mentioned Q. Specifically:

[0095] Input is Q or D i , denoted as x, first layer normalization processing is performed, then window self-attention is calculated, and the output result is added to the original input x, then, layer normalization layer is entered again, and the output result is used as the Q of the second window attention.

[0096] x' = MSA (LN (x)) + x

[0097] Q = LN (x')

[0098] Next, the second input of the decoder, Encoder Output, that is, the above-mentioned E' i , first layer normalization processing is performed, and is input into the MCA multi-head self-attention layer together with Q, K, and V to calculate the cross-attention, and the output result is added to x'.

[0099] K, V = LN (E' i)

[0100] x" = MCA (Q, K, V) + x'

[0101] Next, after layer normalization processing, the output of the CGL is added to x" to obtain an output x'" of a structure similar to Transfomer, which can be used as the input of the next Transfomer.

[0102] x'" = CGL (LN (x" + x"

[0103] In the decoder, there are two cases of attention units, W-MCA and SW-MCA, when the jth is W-MCA (window multi-head attention), the j+1th is SW-MCA (sliding window multi-head attention); the CGL structure of the convolutional gated aggregation unit is also consistent with that of the encoder.

[0104] S260, input the second deep feature map sequence into the resolution reconstruction sub-model for resolution reconstruction processing to obtain a resolution reconstruction result.

[0105] The resolution reconstruction result can be a video with improved resolution or a deep feature map sequence corresponding thereto obtained after processing by the resolution reconstruction sub-model.

[0106] Specifically, the second deep feature map sequence is input into the resolution reconstruction sub-model for spatial super-resolution reconstruction, and the super-resolution reconstruction result is subjected to residual processing with a trilinear interpolation result obtained by trilinear interpolation of the input low-resolution and low-frame-rate video in the resolution reconstruction sub-model, to obtain a final resolution reconstruction result. The resolution reconstruction sub-model uses super-resolution reconstruction technology, which can significantly improve the definition and detail performance of the video, making the video more delicate and realistic.

[0107] For example, the second deep feature map sequence D0 is input into the resolution reconstruction sub-model, and convolution and pixel shuffling operations are performed for upsampling to generate a super-resolution feature map where r represents the super-resolution factor. Further, f up is added to I' (trilinear interpolation result) to obtain the final reconstructed high-resolution and high-frame-rate video frame output i.e. f up . Wherein I' is the trilinear interpolation result of the input low-resolution and low-frame-rate video I LR

[0108] S270, based on the output of the target reconstruction model, a reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0109] ​The technical scheme of the embodiment of the application is that the first video is input into the feature extraction sub-model to perform shallow feature extraction and obtain a shallow feature map sequence, which helps the model to more accurately understand the video content. The shallow feature map sequence is input into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing, and a first deep feature map sequence is obtained, thereby helping the model to capture complex structures and dynamic changes in the video, reducing the computational complexity and improving the processing speed. The first deep feature map sequence is input into the frame rate reconstruction sub-model to perform frame rate reconstruction processing, and a frame rate reconstruction result is obtained, which can improve the fluency and dynamic expressiveness of the video and make the video watching experience better. The first deep feature map sequence and the frame rate reconstruction result are input into the decoding sub-model to perform multi-level global information extraction and up-sampling processing, and a second deep feature map sequence is obtained, which improves the overall consistency and coherence of the video. The second deep feature map sequence is input into the resolution reconstruction sub-model to perform resolution reconstruction processing, and a resolution reconstruction result is obtained, which can significantly improve the definition and detail performance of the video and make the video more delicate and real. Through the cooperative work of each sub-model, the target reconstruction model can realize the simultaneous improvement of the frame rate and resolution of the video, thereby significantly improving the quality and watching experience of the video.

[0110] Embodiment three

[0111] Figure 8 A structural schematic diagram of a video reconstruction device provided by the third embodiment of the application is shown in FIG. 3. As shown in the figure, the device includes a first video acquisition module 310, a video reconstruction module 320 and a second video acquisition module 330. Figure 8

[0112] The first video acquisition module 310 is configured to acquire a first video to be reconstructed.

[0113] The video reconstruction module 320 inputs the first video into a target reconstruction model pre-trained, and the target reconstruction model performs multi-level local information and global information extraction on the input first video and performs frame rate and resolution reconstruction processing based on the extracted local information and global information.

[0114] The second video acquisition module 330 is configured to obtain a second video after reconstruction based on the output of the target reconstruction model, the frame rate of the second video being greater than the frame rate of the first video, and the resolution of the second video being greater than the resolution of the first video.

[0115] ​The technical scheme of the embodiment is characterized in that a first video to be reconstructed is acquired, and the first video is input into a target reconstruction model pre-trained to perform video reconstruction, the target reconstruction model performs multi-level local information and global information extraction on the input first video, and performs frame rate and resolution reconstruction processing based on the extracted local information and global information, which can significantly improve the quality and viewing experience of the video, and make the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, a second video after reconstruction is obtained, the frame rate of the second video is greater than that of the first video, and the resolution of the second video is greater than that of the first video. The target reconstruction model realizes automatic video reconstruction processing, can realize accurate reconstruction of the video, can improve the reconstruction accuracy and detail recovery capability of the video, ensures the quality of the reconstructed video, and improves the generalization capability.

[0116] Optionally, the target reconstruction model comprises a feature extraction sub-model, an encoding sub-model, a decoding sub-model, a frame rate reconstruction sub-model, and a resolution reconstruction sub-model; and the video reconstruction module 320 comprises:

[0117] A first feature extraction unit is configured to input the first video into the feature extraction sub-model to perform shallow feature extraction, and obtain a shallow feature map sequence;

[0118] A second feature extraction unit is configured to input the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing, and obtain a first deep feature map sequence;

[0119] A frame rate reconstruction unit is configured to input the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing, and obtain a frame rate reconstruction result;

[0120] A third feature extraction unit is configured to input the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model to perform multi-level global information extraction and up-sampling processing, and obtain a second deep feature map sequence;

[0121] A resolution reconstruction unit is configured to input the second deep feature map sequence into the resolution reconstruction sub-model to perform resolution reconstruction processing, and obtain a resolution reconstruction result.

[0122] Optionally, the encoding sub-model comprises a plurality of encoders connected in series, each of the encoders comprising an information extraction layer and a down-sampling layer; and the second feature extraction unit is specifically configured to: input the shallow feature map sequence into a first encoder to perform local information and global information extraction and down-sampling processing, and obtain a down-sampling processing result output by the first encoder; input the down-sampling processing result output by a previous encoder into a next encoder to perform local information and global information extraction and down-sampling processing, and obtain a down-sampling processing result output by the next encoder; and input the information extraction result output by a last encoder into the down-sampling layer to perform down-sampling processing, and obtain the down-sampling processing result output by the first encoder.

[0123] Optionally, the information extraction layer comprises at least two information extraction modules connected in series; and the second feature extraction unit is specifically configured to: input the shallow feature map sequence into a first information extraction module to perform local information and global information extraction processing, and obtain an information extraction result output by the first information extraction module; input the information extraction result output by a previous information extraction module into a next information extraction module to perform local information and global information extraction processing, and obtain an information extraction result output by the next information extraction module; and input the information extraction result output by a last information extraction module into the down-sampling layer to perform down-sampling processing, and obtain the down-sampling processing result output by the first encoder.

[0124] Optionally, each information extraction module comprises a normalization unit, an attention unit, a multi-scale gated big kernel convolution unit and a convolution gated aggregation unit; and the second feature extraction unit is specifically configured to: input the shallow feature map sequence into the normalization unit to perform standardization processing, and obtain a standardized shallow feature map sequence; input the standardized shallow feature map sequence into the attention unit to perform global information extraction, and obtain global information; input the standardized shallow feature map sequence into the multi-scale gated big kernel convolution unit to perform local information extraction, and obtain local information; perform residual addition on the shallow feature map sequence, the global information and the local information, input comprehensive information obtained by the residual addition into the normalization unit to perform standardization processing, and obtain standardized comprehensive information; input the standardized comprehensive information into the convolution gated aggregation unit to perform aggregation enhancement processing, and perform residual addition on the aggregation processing result and the comprehensive information, and obtain the information extraction result output by the first information extraction module.

[0125] Optionally, the target reconstruction model is obtained by training based on a first sample video and a second sample video; and the first sample video is obtained by performing frame rate and resolution down-sampling on the second sample video.

[0126] Optionally, the apparatus further comprises a model training module,

[0127] The model training module is specifically configured to: input the first sample video into a target reconstruction model to be trained for frame rate and resolution reconstruction processing, and obtain a sample reconstruction video corresponding to the first sample video; determine a training error based on the sample reconstruction video and the second sample video, and perform back propagation of the training error to the target reconstruction model to adjust parameters in the target reconstruction model until a preset convergence condition is reached, and the training is completed, and the target reconstruction model is obtained.

[0128] The video reconstruction device provided in the embodiments of the present application can perform the video reconstruction method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0129] Figure 9 A structural schematic diagram of an electronic device 12 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0130] As shown in Figure 9 The electronic device 12 is represented in the form of a general-purpose computing device. Components of the electronic device 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

[0131] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0132] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that is accessible by the electronic device 12 and includes both volatile and non-volatile media, removable and non-removable media.

[0133] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 9 Although not shown, a magnetic disk drive can also be utilized in some embodiments for reading from and writing to a removable, non-volatile magnetic media such as a "floppy disk," and an optical disk drive can be used in some embodiments for reading from and writing to a removable, non-volatile optical media such as an optical disc (e.g., CD-ROM, DVD-ROM, etc.). In such instances, each can be connected to bus 18 by one or more data media interfaces. System memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application. Figure 9 Although not shown, a magnetic disk drive can also be utilized in some embodiments for reading from and writing to a removable, non-volatile magnetic media such as a "floppy disk," and an optical disk drive can be used in some embodiments for reading from and writing to a removable, non-volatile optical media such as an optical disc (e.g., CD-ROM, DVD-ROM, etc.). In such instances, each can be connected to bus 18 by one or more data media interfaces. System memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0134] Program / utility 40 having a set (at least one) of program modules 42 can be stored in system memory 28 by way of example, such programs include an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, can include implementation of a network environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0135] Electronic device 12 can also communicate with one or more external devices 14 such as a keyboard or pointing device, a display 24, etc. and can communicate with one or more devices that enable a user to interact with electronic device 12 and / or one or more devices that enable electronic device 12 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interfaces 22. Further, electronic device 12 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet through network adapter 20. As depicted, network adapter 20 communicates with the other components of electronic device 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software modules could be utilized in conjunction with electronic device 12 such as, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0136] Processing unit 16 can execute various program applications and data processing by running programs stored in system memory 28, such as implementing a video reconstruction method step provided by embodiments of the present application, the method comprising:

[0137] obtaining a first video to be reconstructed;

[0138] inputting the first video into a target reconstruction model pre-trained to perform video reconstruction, the target reconstruction model performing multi-level local information and global information extraction on the input first video and performing frame rate and resolution reconstruction processing based on the extracted local information and global information;

[0139] obtaining a second video reconstructed based on an output of the target reconstruction model, the second video having a frame rate greater than that of the first video and a resolution greater than that of the first video.

[0140] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the video reconstruction method provided by any embodiment of the present application.

[0141] The embodiment provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the video reconstruction method provided by any embodiment of the present application, the method comprising:

[0142] obtaining a first video to be reconstructed;

[0143] inputting the first video into a target reconstruction model pre-trained to perform video reconstruction, the target reconstruction model performing multi-level local information and global information extraction on the input first video and performing frame rate and resolution reconstruction processing based on the extracted local information and global information;

[0144] obtaining a second video reconstructed based on an output of the target reconstruction model, the second video having a frame rate greater than that of the first video and a resolution greater than that of the first video.

[0145] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0146] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which computer readable program code is embodied. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer readable signal medium can also be any computer readable medium that is not a storage medium, that is capable of storing the program for use by or in connection with the instruction execution system, apparatus or device.

[0147] The program code embodied on the computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0148] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry can execute the program code. In some embodiments, multiple processors or multiple cores can be employed.

[0149] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by the computing devices, so that they can be stored in storage devices and executed by the computing devices, or they can be respectively manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the present application is not limited to any specific combination of hardware and software.

[0150] It is noted that the above merely describes the preferred embodiments of the present application and the principles of the applied technology. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and more other equivalent embodiments can be included without departing from the concept of the present application, and the scope of the present application is determined by the appended claims.

Claims

1. A method of video reconstruction, characterized by, The method comprises the following steps: acquiring a first video to be reconstructed; inputting the first video into a target reconstruction model pre-trained to perform video reconstruction, wherein the target reconstruction model is configured to perform multi-level local information and global information extraction on the input first video, and perform frame rate and resolution reconstruction processing based on the extracted local information and global information; based on the output of the target reconstruction model, obtaining a second video after reconstruction, wherein the frame rate of the second video is greater than that of the first video, and the resolution of the second video is greater than that of the first video; the target reconstruction model comprises a feature extraction sub-model, an encoding sub-model, a decoding sub-model, a frame rate reconstruction sub-model and a resolution reconstruction sub-model; the inputting the first video into the target reconstruction model pre-trained to perform video reconstruction comprises: inputting the first video into the feature extraction sub-model to perform shallow feature extraction and obtain a shallow feature map sequence; inputting the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing and obtain a first deep feature map sequence; inputting the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing and obtain a frame rate reconstruction result; inputting the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model to perform multi-level global information extraction and up-sampling processing and obtain a second deep feature map sequence; inputting the second deep feature map sequence into the resolution reconstruction sub-model to perform resolution reconstruction processing and obtain a resolution reconstruction result; the encoding sub-model comprises a plurality of encoders connected in series, and each encoder comprises an information extraction layer and a down-sampling layer; the inputting the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing and obtain a first deep feature map sequence comprises: inputting the shallow feature map sequence into a first encoder to perform local information and global information extraction and down-sampling processing and obtain a down-sampling processing result output by the first encoder; inputting the down-sampling processing result output by a previous encoder into a next encoder to perform local information and global information extraction and down-sampling processing and obtain a down-sampling processing result output by the next encoder; inputting the information extraction result output by a last encoder as the first deep feature map sequence.

2. The method of claim 1, wherein, the information extraction layer comprises at least two information extraction modules connected in series; the inputting the shallow feature map sequence into a first encoder to perform local information and global information extraction and down-sampling processing and obtain a down-sampling processing result output by the first encoder comprises: inputting the shallow feature map sequence into a first information extraction module to perform local information and global information extraction processing and obtain an information extraction result output by the first information extraction module; inputting the information extraction result output by a previous information extraction module into a next information extraction module to perform local information and global information extraction processing and obtain an information extraction result output by the next information extraction module; The information extraction result output by the last information extraction module is input into a downsampling layer for downsampling processing, and a downsampling processing result of the first encoder output is obtained.

3. The method of claim 2, wherein, Each information extraction module comprises a normalization unit, an attention unit, a multi-scale gated large kernel convolution unit, and a convolution gated aggregation unit. The sequence of shallow feature maps is input into a first information extraction module for local information and global information extraction processing, and an information extraction result output by the first information extraction module is obtained, comprising: The sequence of shallow feature maps is input into a normalization unit for standardization processing, and a sequence of standardized shallow feature maps is obtained. The sequence of standardized shallow feature maps is input into the attention unit for global information extraction, and global information is obtained. The sequence of standardized shallow feature maps is input into the multi-scale gated large kernel convolution unit for local information extraction, and local information is obtained. The sequence of shallow feature maps, the global information, and the local information are added in residual, and the integrated information obtained by the residual addition is input into the normalization unit for standardization processing, and standardized integrated information is obtained. The standardized integrated information is input into the convolution gated aggregation unit for aggregation enhancement processing, and the aggregation processing result and the integrated information are added in residual, and the information extraction result output by the first information extraction module is obtained.

4. The method of claim 1, wherein, The target reconstruction model is obtained based on training of a first sample video and a second sample video; The first sample video is obtained by frame rate and resolution downsampling of the second sample video.

5. The method of claim 4, wherein, The training process of the target reconstruction model comprises: The first sample video is input into the target reconstruction model to be trained for frame rate and resolution reconstruction processing, and a sample reconstruction video corresponding to the first sample video is obtained. Based on the sample reconstruction video and the second sample video, a training error is determined, and the training error is back propagated to the target reconstruction model to adjust the parameters in the target reconstruction model until a preset convergence condition is reached, and the training is completed to obtain the target reconstruction model.

6. A video reconstruction apparatus characterized by comprising: Comprise: A first video acquisition module is configured to acquire a first video to be reconstructed. A video reconstruction module is configured to input the first video into a target reconstruction model pre-trained, wherein the target reconstruction model is configured to extract local information and global information in multiple levels based on the input first video, and perform frame rate and resolution reconstruction processing based on the extracted local information and global information. A second video acquisition module is configured to obtain a reconstructed second video based on an output of the target reconstruction model, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video. The target reconstruction model comprises a feature extraction sub-model, an encoding sub-model, a decoding sub-model, a frame rate reconstruction sub-model, and a resolution reconstruction sub-model; and the video reconstruction module 320 comprises: A first feature extraction unit is configured to input the first video into the feature extraction sub-model for shallow feature extraction, and obtain a sequence of shallow feature maps. The second feature extraction unit is configured to input the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing, and obtain a first deep feature map sequence. The frame rate reconstruction unit is configured to input the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing, and obtain a frame rate reconstruction result. The third feature extraction unit is configured to input the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model to perform multi-level global information extraction and up-sampling processing, and obtain a second deep feature map sequence. The resolution reconstruction unit is configured to input the second deep feature map sequence into the resolution reconstruction sub-model to perform resolution reconstruction processing, and obtain a resolution reconstruction result. The encoding sub-model includes a plurality of encoders connected in series, each of which includes an information extraction layer and a down-sampling layer; and the second feature extraction unit is specifically configured to: input the shallow feature map sequence into a first encoder to perform local information and global information extraction and down-sampling processing, and obtain a down-sampling processing result output by the first encoder; input a down-sampling processing result output by a previous encoder into a next encoder to perform local information and global information extraction and down-sampling processing, and obtain a down-sampling processing result output by the next encoder; and input an information extraction result output by a last encoder as the first deep feature map sequence.

7. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the video reconstruction method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute when the computer instructions are executed, thereby realizing the video reconstruction method of any one of claims 1-5.

Images