Video reconstruction method and device, equipment and medium

Abstract

The invention discloses a video reconstruction method, apparatus and device, and a medium. The method comprises the steps of obtaining a to-be-reconstructed first video; inputting the first video into a pre-trained target reconstruction model to perform video reconstruction, the target reconstruction model performing multi-level local information and global information extraction on the input first video, and performing frame rate and resolution reconstruction processing based on the extracted local information and global information; and obtaining a reconstructed second video based on the output of the target reconstruction model, the frame rate of the second video being greater than the frame rate of the first video, and the resolution of the second video being greater than the resolution of the first video. Through the technical scheme of the embodiment of the invention, the accurate reconstruction of the video is realized, the reconstruction precision and the detail recovery capability of the video can be improved, the quality of the reconstructed video is ensured, and the generalization capability is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video reconstruction method, device, equipment and medium. Background Art

[0002] With the advancement of technology, high-quality videos can provide people with a better visual experience. However, in the process of video transmission, in order to reduce the transmission bandwidth, frame extraction and other processing will be used to reduce the video quality, and then the video quality will be improved through video reconstruction.

[0003] At present, the traditional video reconstruction method usually estimates the unknown pixels in the video data through interpolation to achieve video reconstruction. However, the traditional video reconstruction method has low reconstruction accuracy and detail restoration ability for the video, cannot guarantee the quality of the reconstructed video, and has poor generalization ability. Summary of the invention

[0004] The present invention provides a video reconstruction method, device, equipment and medium to achieve accurate reconstruction of a video, which can improve the reconstruction accuracy and detail recovery capability of the video, ensure the quality of the reconstructed video, and improve the generalization capability.

[0005] In a first aspect, an embodiment of the present invention provides a video reconstruction method, comprising:

[0006] Obtaining a first video to be reconstructed;

[0007] Inputting the first video into a pre-trained target reconstruction model for video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information;

[0008] Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0009] In a second aspect, an embodiment of the present invention further provides a video reconstruction device, including:

[0010] A first video acquisition module, used to acquire a first video to be reconstructed;

[0011] A video reconstruction module, inputting the first video into a pre-trained target reconstruction model, wherein the target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information;

[0012] The second video acquisition module is used to obtain a reconstructed second video based on the output of the target reconstruction model, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, characterized in that the electronic device comprises: at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the video reconstruction method provided by any embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the video reconstruction method provided by any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention obtains a first video to be reconstructed, and inputs the first video into a pre-trained target reconstruction model for video reconstruction. The target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information. This can significantly improve the quality of the video and the viewing experience, making the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, a reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video. Automated video reconstruction processing is achieved through the target reconstruction model, which can achieve accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery capability of the video, ensure the quality of the reconstructed video, and improve the generalization capability.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a video reconstruction method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 is a flowchart of a video reconstruction method provided according to Embodiment 2 of the present invention;

[0022] Figure 3 is a structural diagram of a target reconstruction model involved in Embodiment 2 of the present invention;

[0023] Figure 4 is a structural diagram of an information extraction layer in an encoder according to Embodiment 2 of the present invention;

[0024] Figure 5 is a structural diagram of a multi-scale gated large kernel convolution unit involved in Embodiment 2 of the present invention;

[0025] Figure 6 is a structural diagram of a convolutional gated aggregation unit involved in Embodiment 2 of the present invention;

[0026] Figure 7 is a structural diagram of an information extraction layer in a decoder according to Embodiment 2 of the present invention;

[0027] Figure 8 is a structural schematic diagram of a video reconstruction device provided according to Embodiment 3 of the present invention;

[0028] Fig. 9 It is a schematic diagram of the structure of an electronic device for implementing the video reconstruction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a video reconstruction method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of reconstructing a video. Figure 1 As shown, the method can be performed by a video reconstruction device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method specifically comprises the following steps:

[0033] S110: Obtain a first video to be reconstructed.

[0034] The first video may refer to a video with a low frame rate and a low resolution. It should be noted that the low frame rate and the low resolution may be determined by a preset frame rate threshold and a preset resolution threshold.

[0035] Specifically, a video acquisition device (such as a camera) may be used to shoot a video, or an existing video file may be obtained through a file transfer method (such as network download, local reading, etc.) as the first video to be reconstructed.

[0036] S120, inputting the first video into a pre-trained target reconstruction model for video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information.

[0037] The target reconstruction model may refer to a U-shaped neural network for improving frame rate and resolution. Local information may refer to detailed features around a small area or a specific pixel in a video frame. These features usually include texture, edge, color change and other details. Global information may refer to the overall features of the entire scene or object in a video frame. These features usually include scene layout, relationship between objects, lighting conditions, overall color distribution, etc.

[0038] Specifically, the target reconstruction model performs multi-level feature extraction on the input first video. During the feature extraction process, the model will focus on the local information (such as edges, textures, etc.) and global information (such as scene layout, object motion trajectory, etc.) of the first video respectively. After obtaining rich local and global information, the model will use the local information and global information to reconstruct the frame rate and resolution of the video. For example, for frame rate reconstruction, the target reconstruction model can use methods such as time interpolation or motion estimation to increase the number of frames in the video, thereby improving the smoothness of the video; for resolution reconstruction, the model can use super-resolution reconstruction technology, such as interpolation-based, reconstruction-based or learning-based methods, to generate higher resolution video frames. Through multi-level local information and global information extraction, the rich details and overall structure in the video can be captured, which can provide strong support for subsequent reconstruction processing, can significantly improve the quality of the video and the viewing experience, and make the reconstructed video smoother and clearer.

[0039] Exemplarily, the target reconstruction model is obtained by training based on the first sample video and the second sample video; wherein the first sample video is obtained by downsampling the frame rate and resolution of the second sample video.

[0040] The first sample video may refer to a video with low resolution and low frame rate obtained by downsampling the frame rate and resolution of the second sample video, which is used for model training. The second sample video may refer to a video with high frame rate and high resolution, which is used as a reference or benchmark for model training.

[0041] For example, before training, the original data set should be preprocessed. First, the video frame data is cropped to make its size uniform to 256×256. In order to perform data enhancement, the video frame is randomly rotated or flipped, and finally the second sample video is obtained. The high-resolution frame sequence is downsampled by a sampling multiple of r to obtain the first sample video of the training input

[0042] Exemplarily, the training process of the target reconstruction model includes: inputting a first sample video into the target reconstruction model to be trained to perform frame rate and resolution reconstruction processing to obtain a sample reconstructed video corresponding to the first sample video; based on the sample reconstructed video and the second sample video, determining the training error, and back-propagating the training error to the target reconstruction model, adjusting the parameters in the target reconstruction model, and completing the training until a preset convergence condition is reached to obtain the target reconstruction model.

[0043] The sample reconstructed video may refer to a video obtained by reconstructing the frame rate and resolution of the first sample video by the target reconstructed video to be trained. The training error may refer to the degree of difference between the sample reconstructed video and the second sample video during the model training process. The preset convergence condition may refer to the condition used to determine whether the training is completed during the model training process.

[0044] Specifically, the first sample video is input into the target reconstruction model to be trained. The model reconstructs the frame rate and resolution of the input video, and generates a sample reconstructed video by using the functions of sub-models such as multi-level local information and global information extraction, encoding, decoding, and frame rate reconstruction and resolution reconstruction. Through the input and reconstruction processing steps, the model can learn how to extract key information from low-quality videos and generate high-quality reconstructed videos. Based on the sample reconstructed video and the second sample video, the training error is calculated. The training error is usually determined by comparing the difference between the reconstructed video and the reference video, and can be measured using evaluation indicators such as loss functions. The training error is back-propagated to the target reconstruction model. During the back-propagation process, the error signal is transmitted backward along each level of the model to guide the adjustment of the model parameters. Through the determination and back-propagation of the training error, the model can identify the deficiencies in the reconstruction process and make targeted adjustments. This helps the model gradually optimize its parameters and improve the quality of the reconstructed video. According to the back-propagated error signal, the parameters in the target reconstruction model are adjusted. Parameter adjustment is usually achieved through optimization algorithms such as gradient descent, aiming to minimize the training error. It is determined whether the model has reached the preset convergence condition. Convergence conditions may include the training error reaching a certain threshold, the number of training times reaching a preset number of times, etc. When the convergence conditions are met, the training process ends and the final target reconstruction model is obtained. Through parameter adjustment, the model can gradually optimize its reconstruction performance and improve the quality of video reconstruction. Convergence judgment ensures the stability and effectiveness of the training process and avoids the problem of over-training or under-training.

[0045] For example, the reconstructed super-resolution video With the target high resolution video I HR (See video) Calculating the loss function Training was performed, and ε = 0.001 during the experiment. When the training reached the set number of iterations, the final network model weights were obtained.

[0046] S130. Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0047] The second video may refer to a video with a high frame rate and a high resolution obtained by reconstructing the frame rate and resolution of the first video by a target reconstruction model.

[0048] Specifically, after the target reconstruction model completes the reconstruction processing of the frame rate and resolution, it will output the reconstructed video data, that is, the final reconstructed second video is obtained. The reconstructed second video has a higher frame rate and resolution, thereby providing a better viewing experience, and can ensure that the reconstructed video is consistent with the original video in format and encoding, which is convenient for subsequent playback and transmission.

[0049] The technical solution of the embodiment of the present invention obtains a first video to be reconstructed, and inputs the first video into a pre-trained target reconstruction model for video reconstruction. The target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information. This can significantly improve the quality of the video and the viewing experience, making the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, a reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video. Automated video reconstruction processing is achieved through the target reconstruction model, which can achieve accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery capability of the video, ensure the quality of the reconstructed video, and improve the generalization capability.

[0050] Embodiment 2

[0051] Figure 2 This is a flow chart of a video reconstruction method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment divides the target reconstruction model into a feature extraction sub-model, a coding sub-model, a decoding sub-model, a frame rate reconstruction sub-model and a resolution reconstruction sub-model, and optimizes the step of "inputting the first video into the pre-trained target reconstruction model for video reconstruction". The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0052] See also Figure 2 Another video reconstruction method provided in this embodiment specifically includes the following steps:

[0053] S210: Obtain a first video to be reconstructed.

[0054] S220: Input the first video into the feature extraction sub-model to perform shallow feature extraction to obtain a shallow feature map sequence.

[0055] The feature extraction sub-model may refer to a neural network model for performing shallow feature extraction on a video. The shallow feature map sequence may refer to a feature map sequence composed of shallow feature maps of each frame image in the first video.

[0056] Specifically, Figure 3 As shown, the first video (i.e., input data) is input into the feature extraction sub-model. The feature extraction sub-model performs shallow feature extraction on each frame of the input video through structures such as convolutional layers. The extracted shallow features are represented as feature maps and form a shallow feature map sequence. Shallow feature extraction can capture basic visual information in video frames, such as edges, textures, etc. The shallow feature map sequence provides a rich information basis for subsequent processing steps.

[0057] S230, inputting the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing to obtain a first deep feature map sequence.

[0058] Among them, the first deep feature map sequence may refer to a deep feature map sequence obtained after being processed by the encoding sub-model.

[0059] Specifically, Figure 3 As shown in Figure 1, the shallow feature map sequence is input into the encoding sub-model. The encoding sub-model further performs multi-level feature extraction and downsampling at the same time to reduce the size of the feature map and extract higher-level local and global information. Multi-level feature extraction and downsampling help the model capture complex structures and dynamic changes in the video. Reducing the size of the feature map can reduce computational complexity and increase processing speed.

[0060] Exemplarily, the encoding sub-model includes multiple encoders connected in series, each encoder includes an information extraction layer and a downsampling layer; S230 may include: inputting the shallow feature map sequence into the first encoder to extract local information and global information and downsampling processing, and obtaining the downsampling processing result output by the first encoder; inputting the downsampling processing result output by the previous encoder into the next encoder to extract local information and global information and downsampling processing, and obtaining the downsampling processing result output by the next encoder; using the information extraction result output by the last encoder as the first deep feature map sequence.

[0061] The information extraction layer may refer to a layer for extracting local information and global information. The downsampling layer may refer to a layer for reducing the spatial resolution of data (such as the width and height of an image).

[0062] Specifically, Figure 3As shown in FIG. 1 , the shallow feature map sequence is input to the first encoder. Inside the first encoder, the local information and global information of the input feature map are first extracted through the information extraction layer. Local information may include detailed features such as texture and edge of the image, while global information may include macro features such as overall brightness, contrast, color distribution of the image, and related information between adjacent video frames. Then, the downsampling layer downsamples the extracted feature map to reduce the resolution of the feature map while retaining key information. The first encoder outputs the downsampling result, i.e., the first-level deep feature map. The downsampling result output by the first encoder is used as input and passed to the next encoder. The processing process of the subsequent encoder is similar to that of the first encoder, which also includes the extraction of local and global information and downsampling. Each encoder will output its corresponding downsampling result, i.e., a deeper level of deep feature map. After serial processing by multiple encoders, the information extraction result output by the last encoder is used as the first deep feature map sequence. This first deep feature map sequence contains multi-level local and global information extracted from the shallow feature map sequence, and has undergone multiple downsampling processes to reduce the resolution but retain key information. Through the serial processing of multiple encoders, the encoding sub-model can perform multi-level information extraction and downsampling processing on the input feature map. This multi-level processing can gradually extract more abstract and advanced features, providing a richer information basis for subsequent video reconstruction tasks.

[0063] Exemplarily, the information extraction layer includes at least two information extraction modules connected in series; the shallow feature map sequence is input into the first encoder for local information and global information extraction and downsampling processing, and the downsampling processing result output by the first encoder is obtained, including: the shallow feature map sequence is input into the first information extraction module for local information and global information extraction processing, and the information extraction result output by the first information extraction module is obtained; the information extraction result output by the previous information extraction module is input into the next information extraction module for local information and global information extraction processing, and the information extraction result output by the next information extraction module is obtained; the information extraction result output by the last information extraction module is input into the downsampling layer for downsampling processing, and the downsampling processing result output by the first encoder is obtained.

[0064] The information extraction result may refer to the local information and global information output by the information extraction module. The downsampling processing result may refer to the feature map with low spatial resolution after being processed by the downsampling layer.

[0065] Specifically, the shallow feature map sequence is input into the first information extraction module. In the first information extraction module, local information and global information are extracted from the input feature map through a series of operations. Local information may include detailed features such as texture and edge of the image, while global information may include macro features such as overall brightness, contrast, color distribution of the image, and related information between adjacent video frames. The first information extraction module outputs the extracted feature map, that is, the first information extraction result. The feature map output by the first information extraction module is used as input and passed to the next information extraction module. The processing process of the subsequent information extraction module is similar to that of the first information extraction module, and is also responsible for extracting local information and global information. Each information extraction module will output its corresponding extraction result, that is, a deeper level feature map. After serial processing by multiple information extraction modules, the feature map output by the last information extraction module contains multi-level local information and global information extracted from the shallow feature map sequence. The feature map output by the last information extraction module is input into the downsampling layer. The downsampling layer downsamples the input feature map to reduce the resolution of the feature map while retaining key information. The downsampling layer outputs the processed feature map, that is, the downsampling processing result output by the first encoder. The serial processing of multiple information extraction modules and the introduction of downsampling layers enable the model to better learn the statistical characteristics of the input data, thereby improving the generalization ability of the model. This helps the model maintain good performance when processing unseen data.

[0066] Exemplarily, each information extraction module includes: a normalization unit, an attention unit, a multi-scale gated large kernel convolution unit and a convolution gated aggregation unit; the shallow feature map sequence is input into the first information extraction module for local information and global information extraction processing, and the information extraction result output by the first information extraction module is obtained, including: the shallow feature map sequence is input into the normalization unit for standardization processing to obtain a standardized shallow feature map sequence; the standardized shallow feature map sequence is input into the attention unit for global information extraction to obtain global information; the standardized shallow feature map sequence is input into the multi-scale gated large kernel convolution unit for local information extraction to obtain local information; the shallow feature map sequence, global information and local information are residually added, and the comprehensive information obtained by the residual addition is input into the normalization unit for standardization processing to obtain the standardized comprehensive information; the standardized comprehensive information is input into the convolution gated aggregation unit for aggregation enhancement processing, and the aggregation processing result and the comprehensive information are residually added to obtain the information extraction result output by the first information extraction module.

[0067] Among them, comprehensive information may refer to the information obtained by fusing shallow feature map sequences, global information and local information by adding residuals. The aggregation processing result may refer to the output obtained after the convolution gated aggregation unit performs aggregation enhancement processing on the standardized comprehensive information of the input. The normalization unit may refer to a unit used to standardize the input feature map to ensure that the distribution of the data is within a reasonable range. The attention unit may refer to the unit responsible for extracting global information from the input data. It is a window-based multi-head self-attention mechanism that can capture long-term dependencies and global contextual information in the input data. The multi-scale gated large kernel convolution unit is a component for extracting local information. It uses convolution kernels of different scales and gating mechanisms to extract local features of the input data. The convolution gated aggregation unit is a component for performing aggregation enhancement processing on features.

[0068] Specifically, Figure 4As shown in the figure, it contains two information extraction modules with the same structure. The shallow feature map sequence is input to the normalization unit. The normalization unit normalizes the input feature map to ensure that the data is distributed within a reasonable range, which helps the stability and efficiency of the subsequent processing unit. The standardized shallow feature map sequence is output. The standardized shallow feature map sequence is input to the attention unit. The attention unit extracts global information by calculating the correlation between different positions or channels. This involves window-based multi-head self-attention and moving window-based multi-head self-attention, which can capture long-term dependencies and global context information in the input data. Output global information. At the same time, the standardized shallow feature map sequence is input to the multi-scale gated large kernel convolution unit. This unit uses convolution kernels of different scales and gating mechanisms to extract local information from the input data. The multi-scale convolution kernel can capture features of different scales, and the gating mechanism can dynamically adjust the weights of the convolution kernel to improve the flexibility and accuracy of feature extraction. Output local information. The shallow feature map sequence, global information and local information are residually added. Residual addition is a commonly used technique that can retain some information of the input data and avoid information loss. The comprehensive information obtained by residual addition is input into the normalization unit again for standardization to ensure the stability of the data. The standardized comprehensive information is output. The standardized comprehensive information is input into the convolution gated aggregation unit. This unit uses convolution operations and gating mechanisms to further aggregate and enhance the input data. The convolution operation can extract more refined features, while the gating mechanism can dynamically adjust the weights of the features to improve the robustness of the feature representation. The feature extraction results are aggregated, and the aggregation results and the comprehensive information are residually added to obtain a richer feature representation. The information extraction result of the first information extraction module is output. The multi-scale gated large kernel convolution unit can capture local features of different scales and improve the model's ability to extract local information. The residual addition and convolution gated aggregation unit can fuse and enhance global information and local information to improve the richness and robustness of feature representation.

[0069] Exemplarily, the information extraction layer of the encoder is as follows Figure 4 As shown in the figure, each information extraction layer contains J information extraction modules, and J can be 2 in the experiment. The main idea is to use W-MSA (window-multi-head attention) and SW-MSA (sliding window-multi-head attention) to extract global information in the video sequence, and combine multi-scale gated large kernel convolution units to extract local information, and then aggregate the information through gated convolution units. The process of one information extraction module is as follows:

[0070] x2=MSA(LN(x1))+MGLK(LN(x1))+x1

[0071] x3=CGL(LN(x2))+x2

[0072] Among them, the normalization unit represents the layer normalization operation, and the attention unit represents multi-head attention. There are two cases: W-MSA and SW-MSA. When the jth one is W-MSA, the j+1th one is SW-MSA.

[0073] x1 represents the first-level input of the block, which can be the output of the feature extraction module, the output of the previous decoder, and the output of the previous information extraction module in this module. In general, x1 first undergoes layer normalization, and then the results are input into the attention unit and the multi-scale gated large kernel convolution unit for processing, and then the result is residually added to the original input x1 to obtain x2, which is then sequentially subjected to layer normalization and normalization units, and then residually added to x2 to finally obtain the output x3 of an information extraction module. This output can be used as the input of the next information extraction module.

[0074] Specifically, the attention unit is a The input video frame is split into non-overlapping windows, where M represents the window size, M=4 in the experiment, N represents the number of video frames, H and W represent the height and width of the input video frame respectively. The features in each window are flattened to generate a size of After obtaining the feature map, the self-attention in each window is calculated for the window-based multi-head self-attention. Since the windows are non-overlapping, there is no better interaction between the windows. In order to further obtain the global attention information, SW-MSA translates the image of the window divided by W-MSA to the left and upward respectively. The blocks make it possible to interact with the non-sliding and non-overlapping window divisions of the previous layer, and then calculate the self-attention within the window to obtain the global attention information.

[0075] For example, Figure 5 As shown in the figure, in the multi-scale gated large kernel convolution unit, the feature map first passes through a dynamic initialization layer composed of 1X1 convolution PWConv and adaptive average pooling Avgpool, which is used to dynamically exclude unimportant interactions and reduce feature redundancy, so as to better focus on subsequent multi-order feature interactions.

[0076] y=Conv 1×1 (x)

[0077] z=ActLayer(y+β1×(y-Avg(y)))

[0078] Where x represents the input feature map, Conv 1×1represents a 1x1 convolution layer, ActLayer represents an activation layer, and the RELU activation layer is selected in the experiment. β1 is a learnable parameter with an initial value of 0, which is used for dynamic adjustment, and Avg represents adaptive average pooling. The dynamically initialized feature map z is used to input the subsequent gated large kernel convolution multi-order feature interaction.

[0079] z′=ActLayer(Conv 1×1 (z))×ActLayer(MLK(z))

[0080] z″=Conv 1×1 (z′)

[0081] The input dynamic initialized feature map z passes through two branches. One branch passes through a 1x1 convolution layer and an activation layer for dense linear transformation as a gating layer. The other branch passes through a multi-scale large kernel convolution layer MLK to fully extract the local information of the video image through multi-scale large kernel convolution. Specifically, First, a layer of depth-separable convolution with a convolution kernel size of 5x5 is used to extract low-order features, and then the output low-order features are divided into Among them, α1+α2+α3=1, and in the experiment, z m and z h Through the depth-separable dilated convolution DwDConv1 and DwDConv2, the convolution sum size is 5x5 and 7x7 respectively, and the dilation rate is 2 and 3 respectively, the output mid-order feature z′ is obtained m ∈ and high-level features Then, z′ is calculated on the channel. m , z′ h , zl are spliced ​​to obtain Then, it passes through a 1x1 convolution layer and an activation layer to re-aggregate the extracted multi-order features. Then, the result of multi-order feature aggregation is multiplied by the output of the gate layer, and finally it passes through a 1x1 convolution layer for final aggregation to obtain the output

[0082] like Figure 6 As shown, in the convolutional gated aggregation unit, first, the input After a 1x1 convolutional layer, the number of channels is expanded to 2C, and we get Then separate it into channels Then, p′1 is input into the depth-separable convolution layer DwConv with a convolution kernel size of 7x7 to transform and weight it, and we get Then, p″1 and p′2 are multiplied, and finally passed through a 1x1 convolution layer and multiplied by the dynamic learnable parameter β2 with an initial value of 0 to obtain the output

[0083] p′1,p′2=Spilt(Conv 1×1 (p))

[0084] q2=β2×Conv 1×1 ((DwConv 7×7 (p′1)×p′2))

[0085] S240, inputting the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing to obtain a frame rate reconstruction result.

[0086] Among them, the frame rate reconstruction result may refer to a deep feature map sequence with an increased frame rate obtained after being processed by a frame rate reconstruction sub-model.

[0087] Specifically, the first deep feature map sequence is input into the frame rate reconstruction sub-model. This sub-model uses technologies such as time interpolation, motion estimation or optical flow method to predict and insert new frames based on the feature information of existing frames. The frame rate reconstruction sub-model outputs the frame rate reconstruction result, that is, the deep feature map sequence after the frame rate is increased, thereby achieving frame rate improvement. Frame rate reconstruction can improve the smoothness and dynamic expression of the video, making the video viewing experience better.

[0088] For example, Figure 3 As shown, in the frame rate reconstruction sub-model, the frame information before and after E′3 (the first deep feature map sequence) can be used to expand it into a 2N-1 frame mapping

[0089]

[0090] S250, inputting the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model for multi-level global information extraction and upsampling processing to obtain a second deep feature map sequence.

[0091] Among them, the second deep feature map sequence may refer to a deep feature map sequence obtained after being processed by the decoding sub-model.

[0092] Specifically, the first deep feature map sequence and the frame rate reconstruction result are input into the decoding sub-model. The decoding sub-model is responsible for upsampling the feature map after frame rate reconstruction back to the original size, and further extracting and integrating global information in the process. The upsampling process helps to restore the original size of the video frame while maintaining the quality of the reconstructed video. The extraction and integration of global information helps to improve the overall consistency and coherence of the video.

[0093] For example, Figure 3 As shown, E′3 and Q are input into the decoder together, and the information extraction layer is obtained. Then go through a deconvolution layer (up-sampling layer) to up-sample and adjust the feature map height and width to get Then input D3 and E′2 into the next layer of decoder, and so on to get Among them, the last layer of the decoder has only CGTB without deconvolution layer, that is,

[0094] The information extraction layer of a single decoder can be Figure 7 As shown in the figure, similar to the encoder, each information extraction layer contains J information extraction modules, and J can be 2 in the experiment. Each information extraction module has two inputs, which are the output of the encoder of the corresponding layer without upsampling by the convolutional layer, that is, the above E′ i , and the other is the output of the previous decoder, corresponding to the above D i , the D of the first decoder corresponds to the above Q. Specifically:

[0095] Input is Q or D i , denoted as x, first perform layer normalization, then calculate its window self-attention, and then add the output result to the original input x for residual addition, then enter the layer normalization layer, and output the result as the Q of the second window attention.

[0096] x′=MSA(LN(x))+x

[0097] Q = LN(x′)

[0098] Next, the second input of the decoder, Encoder Output, is E′ mentioned above. i , first perform layer normalization, and input them into the MCA multi-head self-attention layer as K, V and Q to calculate their cross attention, and add the output result to x′ for residual.

[0099] K,V=LN(E′i)

[0100] x″=MCA(Q,K,V)+x′

[0101] Then after layer normalization, it passes through the gated convolution unit CGL, and its output is added to x″ to obtain a Transfomer-like output x″′, which can be used as the input of the next Transfomer-like structure.

[0102] x″′=CGL(LN(x″+x″

[0103] In the decoder, there are two types of attention units: W-MCA and SW-MCA. When the jth one is W-MCA (window multi-head attention), the j+1th one is SW-MCA (sliding window multi-head attention); the convolutional gated aggregation unit CGL structure is also consistent with that of the encoder.

[0104] S260, inputting the second deep feature map sequence into the resolution reconstruction sub-model for resolution reconstruction processing to obtain a resolution reconstruction result.

[0105] The resolution reconstruction result may refer to a video with improved resolution obtained after being processed by a resolution reconstruction sub-model or a corresponding deep feature map sequence.

[0106] Specifically, the second deep feature map sequence is input into the resolution reconstruction sub-model for spatial super-resolution reconstruction, and the super-resolution reconstruction result is subjected to residual processing with the trilinear interpolation result obtained by trilinear interpolation based on the input low-resolution, low-frame-rate video in the resolution reconstruction sub-model to obtain the final resolution reconstruction result. The resolution reconstruction sub-model uses super-resolution reconstruction technology to significantly improve the clarity and detail of the video, making the video more delicate and realistic.

[0107] Exemplarily, the second deep feature map sequence D0 is input into the resolution reconstruction sub-model, and convolution and pixel shuffling operations are performed to perform upsampling to generate a super-resolution feature map Where r represents the super-resolution ratio. up Add it to I′ (trilinear interpolation result) to get the final reconstructed, high-resolution, high-frame-rate video frame output Right now f up Where I′ is the input low-resolution, low-frame-rate video I LR The trilinear interpolation is performed.

[0108] S270. Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0109] The technical solution of the embodiment of the present invention, by inputting the first video into the feature extraction submodel for shallow feature extraction, obtains a shallow feature map sequence, which helps the model to understand the video content more accurately. The shallow feature map sequence is input into the encoding submodel for multi-level local information and global information extraction and downsampling processing to obtain the first deep feature map sequence, which helps the model capture the complex structure and dynamic changes in the video, can reduce the computational complexity, and improve the processing speed. The first deep feature map sequence is input into the frame rate reconstruction submodel for frame rate reconstruction processing to obtain the frame rate reconstruction result, which can improve the smoothness and dynamic expression of the video and make the video viewing experience better. The first deep feature map sequence and the frame rate reconstruction result are input into the decoding submodel for multi-level global information extraction and upsampling processing to obtain the second deep feature map sequence, which improves the overall consistency and coherence of the video. The second deep feature map sequence is input into the resolution reconstruction submodel for resolution reconstruction processing to obtain the resolution reconstruction result, which can significantly improve the clarity and detail expression of the video, making the video more delicate and realistic. Through the collaborative work of various sub-models, the target reconstruction model can simultaneously improve the video frame rate and resolution, thereby significantly improving the video quality and viewing experience.

[0110] Embodiment 3

[0111] Figure 8 FIG. 1 is a schematic diagram of the structure of a video reconstruction device provided in Embodiment 3 of the present invention. Figure 8 As shown, the device includes: a first video acquisition module 310 , a video reconstruction module 320 and a second video acquisition module 330 .

[0112] The first video acquisition module 310 is used to acquire the first video to be reconstructed;

[0113] The video reconstruction module 320 inputs the first video into a pre-trained target reconstruction model, wherein the target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information;

[0114] The second video acquisition module 330 is used to obtain a reconstructed second video based on the output of the target reconstruction model, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0115] The technical solution of this embodiment obtains a first video to be reconstructed, and inputs the first video into a pre-trained target reconstruction model for video reconstruction. The target reconstruction model extracts multi-level local information and global information from the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information. This can significantly improve the quality of the video and the viewing experience, making the reconstructed video smoother and clearer. Based on the output of the target reconstruction model, a reconstructed second video is obtained, the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video. Automated video reconstruction processing is achieved through the target reconstruction model, which can achieve accurate reconstruction of the video, improve the reconstruction accuracy and detail recovery capability of the video, ensure the quality of the reconstructed video, and improve the generalization capability.

[0116] Optionally, the target reconstruction model includes: a feature extraction sub-model, a coding sub-model, a decoding sub-model, a frame rate reconstruction sub-model and a resolution reconstruction sub-model; the video reconstruction module 320 includes:

[0117] A first feature extraction unit, inputting the first video into the feature extraction sub-model to perform shallow feature extraction to obtain a shallow feature map sequence;

[0118] A second feature extraction unit is used to input the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing to obtain a first deep feature map sequence;

[0119] A frame rate reconstruction unit, used for inputting the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing to obtain a frame rate reconstruction result;

[0120] A third feature extraction unit is used to input the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model to perform multi-level global information extraction and upsampling processing to obtain a second deep feature map sequence;

[0121] A resolution reconstruction unit is used to input the second deep feature map sequence into the resolution reconstruction sub-model to perform resolution reconstruction processing to obtain a resolution reconstruction result.

[0122] Optionally, the encoding sub-model includes multiple encoders connected in series, each encoder includes an information extraction layer and a downsampling layer; the second feature extraction unit is specifically used to: input the shallow feature map sequence into the first encoder to extract local information and global information and downsampling processing, and obtain the downsampling processing result output by the first encoder; input the downsampling processing result output by the previous encoder into the next encoder to extract local information and global information and downsampling processing, and obtain the downsampling processing result output by the next encoder; use the information extraction result output by the last encoder as the first deep feature map sequence.

[0123] Optionally, the information extraction layer includes at least two information extraction modules connected in series; the second feature extraction unit is specifically used to: input the shallow feature map sequence into the first information extraction module for local information and global information extraction processing, and obtain the information extraction result output by the first information extraction module; input the information extraction result output by the previous information extraction module into the next information extraction module for local information and global information extraction processing, and obtain the information extraction result output by the next information extraction module; input the information extraction result output by the last information extraction module into the downsampling layer for downsampling processing, and obtain the downsampling processing result output by the first encoder.

[0124] Optionally, each information extraction module includes: a normalization unit, an attention unit, a multi-scale gated large kernel convolution unit and a convolution gated aggregation unit; the second feature extraction unit is specifically used to: input the shallow feature map sequence into the normalization unit for normalization processing to obtain a standardized shallow feature map sequence; input the standardized shallow feature map sequence into the attention unit for global information extraction to obtain global information; input the standardized shallow feature map sequence into the multi-scale gated large kernel convolution unit for local information extraction to obtain local information; add the residuals of the shallow feature map sequence, the global information and the local information, and input the comprehensive information obtained by the residual addition into the normalization unit for normalization processing to obtain standardized comprehensive information; input the standardized comprehensive information into the convolution gated aggregation unit for aggregation enhancement processing, and add the residuals of the aggregation processing result and the comprehensive information to obtain the information extraction result output by the first information extraction module.

[0125] Optionally, the target reconstruction model is obtained by training based on a first sample video and a second sample video; wherein the first sample video is obtained by downsampling the frame rate and resolution of the second sample video.

[0126] Optionally, the device further comprises: a model training module,

[0127] The model training module is specifically used to: input the first sample video into the target reconstruction model to be trained to reconstruct the frame rate and resolution, and obtain a sample reconstructed video corresponding to the first sample video; based on the sample reconstructed video and the second sample video, determine the training error, and back-propagate the training error to the target reconstruction model, adjust the parameters in the target reconstruction model, and terminate the training until the preset convergence condition is reached to obtain the target reconstruction model.

[0128] The video reconstruction device provided in the embodiment of the present invention can execute the video reconstruction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Fig. 9 A schematic diagram of the structure of an electronic device 12 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as desktop computers, workstations, servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Fig. 9 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0131] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0132] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0133] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig. 9 not shown, usually called a "hard drive"). Although Fig. 9 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0134] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0135] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a video reconstruction method provided by the embodiment of the present invention, the method comprising:

[0137] Obtaining a first video to be reconstructed;

[0138] Inputting the first video into a pre-trained target reconstruction model for video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information;

[0139] Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0140] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the video reconstruction method provided by any embodiment of the present invention.

[0141] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the video reconstruction method provided in any embodiment of the present invention are implemented. The method includes:

[0142] Obtaining a first video to be reconstructed;

[0143] Inputting the first video into a pre-trained target reconstruction model for video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information;

[0144] Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

[0145] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0146] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0148] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0150] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video reconstruction method, characterized in that: include: Obtaining a first video to be reconstructed; Inputting the first video into a pre-trained target reconstruction model for video reconstruction, wherein the target reconstruction model extracts multi-level local information and global information from the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information; Based on the output of the target reconstruction model, a reconstructed second video is obtained, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

2. The method according to claim 1, characterized in that The target reconstruction model includes: a feature extraction sub-model, an encoding sub-model, a decoding sub-model, a frame rate reconstruction sub-model and a resolution reconstruction sub-model; The step of inputting the first video into a pre-trained target reconstruction model for video reconstruction includes: Inputting the first video into the feature extraction sub-model to perform shallow feature extraction to obtain a shallow feature map sequence; Inputting the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and down-sampling processing to obtain a first deep feature map sequence; Inputting the first deep feature map sequence into the frame rate reconstruction sub-model to perform frame rate reconstruction processing to obtain a frame rate reconstruction result; Inputting the first deep feature map sequence and the frame rate reconstruction result into the decoding sub-model for multi-level global information extraction and upsampling processing to obtain a second deep feature map sequence; The second deep feature map sequence is input into the resolution reconstruction sub-model for resolution reconstruction processing to obtain a resolution reconstruction result.

3. The method according to claim 2, characterized in that The encoding sub-model includes a plurality of encoders connected in series, each encoder including an information extraction layer and a downsampling layer; The step of inputting the shallow feature map sequence into the encoding sub-model to perform multi-level local information and global information extraction and downsampling processing to obtain a first deep feature map sequence includes: Inputting the shallow feature map sequence into the first encoder to extract local information and global information and perform downsampling processing, and obtaining the downsampling processing result output by the first encoder; The downsampling processing result output by the previous encoder is input into the next encoder for local information and global information extraction and downsampling processing, so as to obtain the downsampling processing result output by the next encoder; The information extraction result output by the last encoder is used as the first deep feature map sequence.

4. The method according to claim 3, characterized in that The information extraction layer includes at least two information extraction modules connected in series; The step of inputting the shallow feature map sequence into the first encoder to extract local information and global information and perform downsampling processing to obtain the downsampling processing result output by the first encoder includes: Inputting the shallow feature map sequence into a first information extraction module to perform local information and global information extraction processing, and obtaining an information extraction result output by the first information extraction module; The information extraction result output by the previous information extraction module is input into the next information extraction module for local information and global information extraction processing to obtain the information extraction result output by the next information extraction module; The information extraction result output by the last information extraction module is input into the downsampling layer for downsampling processing to obtain the downsampling processing result output by the first encoder.

5. The method according to claim 4, characterized in that Each information extraction module includes: a normalization unit, an attention unit, a multi-scale gated large kernel convolution unit, and a convolutional gated aggregation unit; The step of inputting the shallow feature map sequence into the first information extraction module to perform local information and global information extraction processing, and obtaining the information extraction result output by the first information extraction module, comprises: Inputting the shallow feature map sequence into a normalization unit for normalization processing to obtain a standardized shallow feature map sequence; Inputting the standardized shallow feature map sequence into the attention unit to extract global information to obtain global information; Inputting the standardized shallow feature map sequence into the multi-scale gated large kernel convolution unit to extract local information to obtain local information; Adding residuals of the shallow feature map sequence, the global information and the local information, and inputting the comprehensive information obtained by the residual addition into the normalization unit for standardization processing to obtain the standardized comprehensive information; The standardized comprehensive information is input into the convolution gated aggregation unit for aggregation enhancement processing, and the aggregation processing result and the comprehensive information are residually added to obtain the information extraction result output by the first information extraction module.

6. The method according to claim 1, characterized in that The target reconstruction model is obtained by training based on the first sample video and the second sample video; The first sample video is obtained by downsampling the frame rate and resolution of the second sample video.

7. The method according to claim 6, characterized in that The training process of the target reconstruction model includes: Inputting the first sample video into the target reconstruction model to be trained to perform frame rate and resolution reconstruction processing, and obtaining a sample reconstructed video corresponding to the first sample video; Based on the sample reconstructed video and the second sample video, a training error is determined, and the training error is back-propagated to the target reconstruction model, and the parameters in the target reconstruction model are adjusted until the training ends when a preset convergence condition is reached to obtain the target reconstruction model.

8. A video reconstruction device, characterized in that: include: A first video acquisition module, used to acquire a first video to be reconstructed; A video reconstruction module, inputting the first video into a pre-trained target reconstruction model, wherein the target reconstruction model extracts multi-level local information and global information of the input first video, and reconstructs the frame rate and resolution based on the extracted local information and global information; The second video acquisition module is used to obtain a reconstructed second video based on the output of the target reconstruction model, wherein the frame rate of the second video is greater than the frame rate of the first video, and the resolution of the second video is greater than the resolution of the first video.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video reconstruction method according to any one of claims 1 to 7 when executed.

Images