Video processing method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides a video processing method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a first video frame sequence; the first video frame sequence is input into a video processing model, a second video frame sequence is obtained, the second video frame sequence is output by the video processing model, the resolution ratio of the second video frame sequence is higher than that of the first video frame sequence, and the video processing model comprises an encoder, a decoder and at least one one-dimensional convolution module; each of the encoder and the decoder comprises a plurality of two-dimensional convolution modules, and at least part of the two-dimensional convolution modules are connected with the one-dimensional convolution module. According to the embodiment of the invention, the two-dimensional convolution module is connected with the one-dimensional convolution module, a lightweight video processing model is realized, the video processing model can capture inter-frame dynamic association in a video frame sequence, and the problems of texture flicker, detail drift, incoherent structure and the like in a video reconstruction process are improved.

Description

Technical Field

[0001] This disclosure relates to multimedia processing technology, and more particularly to a video processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the continuous improvement of video capture equipment resolution and the massive growth of online video content, the demand for video super-resolution technology has increased significantly. However, compared with static image data, video data contains spatial distribution characteristics and temporal characteristics, making video super-resolution processing more complex. Summary of the Invention

[0003] This disclosure provides a video processing method, apparatus, device, medium, and program product that can achieve efficient, stable, and lightweight video reconstruction results.

[0004] In a first aspect, embodiments of this disclosure provide a video processing method, including:

[0005] Obtain the first video frame sequence;

[0006] The first video frame sequence is input into the video processing model to obtain the second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

[0007] Secondly, embodiments of this disclosure also provide a video processing apparatus, the apparatus comprising:

[0008] The acquisition module is used to acquire the first video frame sequence;

[0009] A processing module is used to input the first video frame sequence into a video processing model to obtain a second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

[0010] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0011] One or more processors;

[0012] Storage device for storing one or more programs.

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in any embodiment of this disclosure.

[0014] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the video processing method as described in any embodiment of this disclosure.

[0015] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the video processing method as described in any embodiment of this disclosure. Attached Figure Description

[0016] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1 This is a schematic flowchart of a video processing method provided in an embodiment of the present disclosure;

[0018] Figure 2 This is a schematic diagram of the structure of a video processing model provided in an embodiment of the present disclosure;

[0019] Figure 3 This is a flowchart illustrating a model training method in a video processing method provided in an embodiment of the present disclosure.

[0020] Figure 4 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] For ease of understanding, the terms that may appear in the embodiments of this disclosure are explained below.

[0034] The core objective of Video Super-Resolution (VSR) is to recover a video output with high resolution, clear details, and high visual quality from a low-resolution video sequence while maintaining temporal continuity.

[0035] Image super-resolution (ISR) is a method for recovering high-resolution images from low-resolution images. It typically uses two-dimensional convolutional neural networks to learn the mapping relationship from low-resolution to high-resolution images, effectively restoring image texture details. However, image super-resolution methods cannot be directly applied to video reconstruction tasks. Therefore, in video reconstruction, single-frame super-resolution methods ignore the temporal dependencies between adjacent frames, leading to inter-frame flickering, texture jitter, or detail drift in the generated video sequence during continuous playback, resulting in a lack of temporal consistency.

[0036] To model the temporal dimension of videos, a 3D convolutional neural network is introduced to achieve a joint modeling method based on the spatial and temporal features of 3D convolution. This type of method extends the temporal dimension on top of 2D convolutional kernels, enabling it to capture cross-frame motion information to some extent. However, compared to 2D convolution, 3D convolution exhibits a cubic increase in computational complexity, a massive number of model parameters, and significantly increases memory and computing power costs. In practical applications, 3D convolutional networks are often difficult to deploy on mobile devices or embedded platforms, and are prone to gradient vanishing and training instability during long-sequence training.

[0037] Video super-resolution solutions in related technologies struggle to balance computational complexity and video reconstruction quality. For example, using 3D convolution or temporal self-attention mechanisms to capture cross-frame relationships to obtain stable video details results in a huge number of parameters and high computational complexity, making it difficult to deploy in resource-constrained scenarios such as mobile devices.

[0038] In summary, the core challenge in the field of video super-resolution is how to simultaneously restore clear spatial details and stable temporal consistency with limited computational costs.

[0039] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of the present disclosure. This embodiment is applicable to video enhancement, such as video quality enhancement, camera post-processing, video communication clarification, smart terminal video reconstruction, and generative imaging systems. The method can be executed by a video processing device, which can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server.

[0040] like Figure 1 As shown, the method includes:

[0041] S110, Obtain the first video frame sequence.

[0042] The first video frame sequence represents the original video to be reconstructed. For example, the first video frame sequence can refer to a degraded video signal that has experienced a decrease in image quality or loss of detail during acquisition, encoding, transmission, storage, or playback due to equipment limitations, environmental interference, or human intervention. Degradation types include blurring, noise interference, compression distortion, reduced resolution, or color deviation. Optionally, the video frames in the first video frame sequence may have low resolution. Alternatively, at least some video frames in the first video frame sequence may have missing pixel details. Or, at least some video frames in the first video frame sequence may have blurred image quality.

[0043] In some embodiments, the first video frame sequence is a low-quality video captured in the past. Alternatively, the first video frame sequence is video captured by a low-resolution camera. Or, the first video frame is a video shared by a friend. Or, the second video frame is a low-quality meeting video, etc.

[0044] S120. Input the first video frame sequence into the video processing model to obtain the second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

[0045] The video processing model can be a video super-resolution model that embeds a one-dimensional convolutional module into a two-dimensional backbone network. For example, the two-dimensional backbone network includes a two-dimensional convolutional neural network or a two-dimensional diffusion network. The two-dimensional backbone network includes an encoder and a decoder. The encoder includes multiple downsampling layers. The decoder includes multiple upsampling layers. The downsampling layers are composed of multiple stacked two-dimensional convolutional modules. The upsampling layers are composed of multiple stacked two-dimensional convolutional modules. At least one one-dimensional convolutional module is embedded into the two-dimensional backbone network. For example, a one-dimensional convolutional module can be connected before a two-dimensional convolutional module. Alternatively, a one-dimensional convolutional module can be connected after a two-dimensional convolutional module. In some embodiments, a one-dimensional convolutional module is inserted after at least some of the multiple two-dimensional convolutional modules. It is understood that some two-dimensional convolutional modules are followed by at least one one-dimensional convolutional module, while others are not followed by a one-dimensional convolutional module. Optionally, a one-dimensional convolutional module is inserted after each two-dimensional convolutional module. This disclosure does not specifically limit the number and embedding position of the one-dimensional convolutional modules. By embedding one-dimensional convolutional modules into the backbone network, a temporal feature modeling mechanism is introduced on top of a two-dimensional spatial convolutional structure with low computational overhead, thereby effectively capturing inter-frame dynamic correlations with limited parameters. One-dimensional convolutional modules are embedded after the two-dimensional convolutional modules in each layer of the encoder and decoder. The spatial features of video frames are extracted through the two-dimensional convolutional modules, while the cross-frame temporal features are captured through the one-dimensional convolutional modules. These cross-frame temporal features represent the dynamic changes between adjacent time frames, such as motion trends and / or state transitions. The cross-frame temporal features are fused with spatial features to achieve lightweight joint spatial-temporal modeling. Through an attention mechanism and skip connections between the encoder and decoder, the fused features containing both spatial and temporal features are passed to the corresponding upsampling layer in the decoder, enabling the video processing model to generate rich spatial details while maintaining inter-frame visual stability. This model structure can learn cross-frame correlations without significantly increasing parameters, thus simultaneously improving detail recovery and temporal consistency. This disclosure provides a method for video super-resolution that combines high efficiency, high detail fidelity, and high temporal consistency by introducing an independent temporal dimension modeling channel into the backbone of a two-dimensional convolutional neural network or a two-dimensional diffusion network. This achieves low-cost cross-frame temporal feature learning and balances inference efficiency with video reconstruction quality. It solves the problems of structural complexity, temporal modeling capability, reconstruction quality, and practical deployment efficiency in related video super-resolution technologies.

[0046] In some embodiments, multiple one-dimensional convolutional modules are embedded in the backbone of a two-dimensional convolutional neural network or a two-dimensional diffusion network. By independently performing lightweight convolutional operations in the temporal dimension, the aggregation of cross-frame temporal features is achieved. This disclosure introduces independent temporal modeling channels into the backbone of a two-dimensional convolutional neural network or a two-dimensional diffusion network. At different levels of the backbone network, the cross-frame temporal features aggregated through temporal convolution are fused with two-dimensional spatial features. Combined with channel attention and skip connections, the robustness of the model in complex motion scenes is improved, enabling the reconstruction output of high-resolution video frames. This disclosure achieves explicit decoupling and implicit fusion of spatial and temporal features, effectively balancing spatial details and temporal continuity, improving the clarity, smoothness, and visual stability of the output video, and achieving efficient, stable, and lightweight video reconstruction results.

[0047] Figure 2 This is a schematic diagram of the structure of a video processing model provided in an embodiment of this disclosure. Figure 2 As shown, the video processing model 210 includes an encoder 220, a decoder 230, a one-dimensional convolutional module 270, and a reconstruction module 280. The encoder 220 includes multiple downsampling layers 240. The decoder 230 includes multiple upsampling layers 250. Each downsampling layer 240 includes multiple two-dimensional convolutional modules 260. Each upsampling layer 250 includes multiple two-dimensional convolutional modules 260. A one-dimensional convolutional module 270 is connected after each convolutional module 260. Optionally, a two-dimensional convolutional module 260 may be present within either the downsampling layer 240 or the upsampling layer 250. 270 one-dimensional convolutional modules 2D Convolution Module 260 270 one-dimensional convolutional modules ...and similar stacked structures. Skip connections are established between downsampling layer 240 and upsampling layer 250 to achieve multi-scale fusion of second feature maps through an attention mechanism and skip connections. Reconstruction module 280 is connected to the back end of decoder 230 to use the output features of decoder 230 as input features to reconstruction module 280. High-resolution video frame sequences are output through reconstruction module 280.

[0048] In some embodiments, one-dimensional convolutional modules are embedded along the time axis between the multi-layered structure of the two-dimensional backbone network. Since the two-dimensional backbone network can directly reuse existing pre-trained image super-resolution networks, it achieves cross-framework scalability and reusability, solving the problem that the network structure of existing video super-resolution models is customized for specific architectures and difficult to migrate to different network backbones or generation frameworks. Specifically, the one-dimensional convolutional modules are embedded after the two-dimensional convolutional modules. The number of channels in the one-dimensional convolutional modules is related to the number of channels in the two-dimensional convolutional modules. Each two-dimensional convolutional module is connected to each one-dimensional convolutional module, and the one-dimensional convolutional modules employ one-dimensional depthwise separable convolution. In this embodiment, the number of channels in the one-dimensional and two-dimensional convolutional modules is the same, ensuring that the convolution result of the one-dimensional convolutional module can be directly superimposed on the output result of the two-dimensional convolutional module, simplifying computational complexity. Because the one-dimensional convolution module uses one-dimensional depthwise separable convolution, it can significantly improve the efficiency of convolution operations, maintain a simple computation path, continuous output, and smooth timing during the inference stage, thereby achieving high-quality, low-latency video reconstruction effects that meet the real-time and stability requirements of application scenarios such as video post-processing, real-time preview of mobile cameras, and video conferencing enhancement.

[0049] In some embodiments, a one-dimensional convolutional module comprises at least two stacked convolutional layers. Optionally, activation layers may be included between the convolutional layers. The input features of the one-dimensional convolutional module are superimposed as residuals onto the output features of the last convolutional layer in the one-dimensional convolutional module. For example, spatial features of n consecutive video frames are extracted using a two-dimensional convolutional module to obtain a first feature map set { , ..., , ..., It is understandable that the number of input frames for the 2D convolutional module is related to the length of the sliding window. The input features of the 1D convolutional module connected to the 2D convolutional module are a first feature map set, which is used to extract cross-frame temporal features from the first feature map set. These features are then jointly modeled with the first feature map output by the 2D convolutional module to obtain a fused feature of spatial and temporal features, i.e., a second feature map. This embodiment of the present disclosure achieves joint modeling of spatial and temporal features by embedding a 1D convolutional module in a 2D backbone network, effectively solving the technical bottleneck that traditional 2D networks cannot capture cross-frame temporal correlations. The model structure of this embodiment of the present disclosure can complete temporal modeling without significantly increasing the number of parameters, enabling the model to have stronger motion perception and inter-frame consistency in video frame sequences, and improving problems such as texture flicker, detail drift, and structural inconsistency that occur during video reconstruction.

[0050] In some embodiments, the first convolutional layer in the one-dimensional convolutional module is used to process { , ..., , ..., A convolution operation along the time axis is performed to obtain the first cross-frame temporal feature. For example, a one-dimensional convolution with a kernel size of k is slid along the time axis to perform the convolution operation. Optionally, an activation operation is performed on the first cross-frame temporal feature through an activation layer to obtain the second cross-frame temporal feature. Then, a second convolutional layer is used to perform a convolution operation along the time axis on the second cross-frame temporal feature to obtain the temporal convolution result. The features of each channel in the temporal convolution result are superimposed with the features of each channel in the first feature map of each first video frame to obtain the second feature map set.

[0051] For example, inputting a first video frame sequence into a video processing model to obtain a second video frame sequence includes: determining multiple sets of first feature maps based on multiple sets of first video frames in the first video frame sequence, wherein the first feature map sets are output by the two-dimensional convolutional module; determining each set of second feature maps based on each set of first feature maps, wherein the second feature map sets are output by a one-dimensional convolutional module following the two-dimensional convolutional module, wherein the second feature maps in the second feature map sets include spatial and temporal features of the first video frames; and determining the second video frame sequence based on the multiple sets of second feature maps.

[0052] Understandably, a dynamic window sliding mechanism is used to determine multiple groups of first video frames in the first video frame sequence. Optionally, multiple first video frames within the sliding window are grouped together. The length of the sliding window is related to the performance of the processing chip.

[0053] In some embodiments, the first video frame sequence {I} is received. t-n , ..., I t , ..., I t+n The adjacent frame windows centered on the target frame are used as sliding windows. Multiple first video frames are input to the video processing model through a dynamic window sliding mechanism. In this embodiment, the dynamic frame window adapts to inputs of arbitrary video lengths. Spatial features of each video frame are extracted by an encoder. The encoder includes multiple downsampling layers. Each downsampling layer includes multiple two-dimensional convolutional modules. Optionally, the two-dimensional convolutional modules include residual structures. Convolution operations are performed on each first video frame within the sliding window using the two-dimensional convolutional modules to obtain a first feature map corresponding to the first video frame. The first feature maps corresponding to each first video frame within the sliding window constitute a first feature map set, represented as { , ..., , ..., }

[0054] Optionally, determining each second feature map set based on each first feature map set includes: performing a convolution operation on the first feature map set of each group of first video frames along the time dimension, and superimposing the time convolution result onto the first feature map set to obtain each second feature map set.

[0055] Specifically, for each group of first video frames, the two-dimensional convolution module outputs the first feature map set to the one-dimensional convolution module. The one-dimensional convolution module performs a convolution operation on the first feature map set along the time dimension, and the temporal convolution result is superimposed on the first feature map set to obtain the second feature map set corresponding to each group of first video frames.

[0056] In some embodiments, in each downsampling layer of the encoder, the first feature map set output by the preceding two-dimensional convolutional module connected to the one-dimensional convolutional module is sent to the one-dimensional convolutional module. The one-dimensional convolutional module then performs a convolution operation along the temporal dimension on the first feature map set, outputting a fused feature set that integrates spatial and temporal features, i.e., a second feature map set. The one-dimensional convolutional module outputs the second feature map set to the following two-dimensional convolutional module connected to it for further convolution. Downsampling layers are connected via feature mapping channels, and the processing result of the current downsampling layer is transmitted to the next downsampling layer, repeating the above feature extraction operation and spatiotemporal feature joint modeling operation. This process continues, achieving spatial and temporal joint modeling representation and feature dimensionality reduction through multiple intervals of two-dimensional and one-dimensional convolutional operations in the encoder. The encoder outputs encoded features to the decoder for feature reconstruction. For example, the decoder includes multiple upsampling layers. The upsampling layers include multiple two-dimensional convolutional modules. Optionally, the two-dimensional convolutional modules include residual structures. The upsampling and downsampling layers achieve multi-scale feature fusion through an attention mechanism and skip connections, effectively aggregating cross-layer spatial features and temporal context, thus improving the stability of the generated video. Optionally, decoding features are determined through multiple two-dimensional convolutional modules in the decoder and one-dimensional convolutional modules connected after each two-dimensional convolutional module. The decoding features represent high-resolution feature maps. This embodiment fuses the temporal features aggregated by the one-dimensional convolutional modules with the spatial features extracted by the two-dimensional convolutions at different levels. Combined with a channel attention mechanism and skip connection strategy, it achieves high-resolution video frame reconstruction output, resulting in output video with good detail, texture fidelity, and inter-frame smoothness. It can be widely applied in video post-processing, motion image enhancement, video communication de-sharpening, and generative video restoration.

[0057] Optionally, the video processing model further includes a reconstruction module connected to the decoder. The reconstruction module determines a second video frame sequence based on multiple sets of second feature maps, including: determining decoding features based on each set of second feature maps using the decoder; upsampling the decoding features using the reconstruction module to obtain multiple second video frames; and concatenating the multiple second video frames corresponding to each set of second feature maps in chronological order to obtain the second video frame sequence.

[0058] In some embodiments, the reconstruction module may employ a variational autoencoder decoding module. The reconstruction module upsamples the spatiotemporal fusion features to the target resolution, thereby restoring video features from the latent space to the pixel space. High-resolution video frames are then stitched together in chronological order to form a complete high-resolution video sequence, i.e., a second video sequence, ensuring inter-frame continuity and content consistency. Optionally, the reconstruction module can be used in conjunction with a diffusion model, or with a generative adversarial network, to further improve perceptual quality.

[0059] The technical solution of this disclosure involves acquiring a first video frame sequence and generating a second video frame sequence with higher resolution based on the first video frame sequence using a video processing model. Since the video processing model includes an encoder, a decoder, and a one-dimensional convolutional module, and both the encoder and decoder include multiple two-dimensional convolutional modules, the spatial features of the first video frame are extracted and the feature dimensionality is reduced through the two-dimensional convolutional modules in the encoder. The low-dimensional features generated by the encoder are upsampled by the decoder. By connecting the two-dimensional convolutional modules with the one-dimensional convolutional modules, dynamic correlation information between frames is captured, achieving joint modeling of spatial and temporal features. This solves the problem that related video super-resolution methods struggle to balance computational complexity and video reconstruction quality, achieving synergistic optimization of cross-frame temporal feature modeling and spatial detail restoration without significantly increasing computational complexity. This disclosure embodiment achieves a lightweight video processing model by connecting the two-dimensional convolutional modules with the one-dimensional convolutional modules. This video processing model can capture the dynamic correlation between frames in the video frame sequence, improving problems such as texture flicker, detail drift, and structural inconsistency that occur during video reconstruction, and achieving efficient, stable, and lightweight video reconstruction results.

[0060] Figure 3 This is a flowchart illustrating a model training method in a video processing method provided in this embodiment of the present disclosure. Based on the above embodiments, this embodiment of the present disclosure further specifies the training method for the video processing model.

[0061] like Figure 3 As shown, the method includes:

[0062] S310. Obtain a first video frame sample sequence and a second video frame sample sequence, wherein the first video frame sample sequence and the second video frame sample sequence are associated with different resolution versions of the same video.

[0063] In some embodiments, multiple high-resolution video samples are acquired. Each video sample undergoes frame extraction to obtain a second video frame sample sequence. Noise is added to the video frames in the second video frame sample sequence to obtain a first video frame sequence.

[0064] S320. Input the first video frame sample sequence into the video processing model to be trained to obtain the predicted video frame sequence, which is output by the video processing model to be trained.

[0065] The video processing model to be trained includes a two-dimensional backbone network and at least one one-dimensional convolutional module. The two-dimensional backbone network includes an encoder and a decoder. The encoder includes multiple downsampling layers. The decoder includes multiple upsampling layers. The downsampling layers are composed of multiple stacked two-dimensional convolutional modules. The upsampling layers are composed of multiple stacked two-dimensional convolutional modules. One-dimensional convolutional modules are embedded in the two-dimensional backbone network. For example, one-dimensional convolutional modules can be connected before or after the two-dimensional convolutional modules. In some embodiments, a one-dimensional convolutional module is inserted after at least some of the multiple two-dimensional convolutional modules. It is understood that some two-dimensional convolutional modules are followed by at least one one-dimensional convolutional module, while others are not followed by a one-dimensional convolutional module. Optionally, a one-dimensional convolutional module is embedded after each two-dimensional convolutional module. The initial weights of the one-dimensional convolutional modules are zero, making the initial training phase of the model equivalent to a pure two-dimensional network, ensuring that the two-dimensional feature distribution is not disrupted during initial training, and enabling stable model convergence. The number of channels in a one-dimensional convolutional module is the same as the number of channels in the two-dimensional convolutional modules to which it is connected. Each one-dimensional convolutional module comprises at least two convolutional layers, with activation layers between at least some of the convolutional layers. The kernel size of the one-dimensional convolutional module is preset. The activation function is also preset. The inputs of the one-dimensional convolutional layers are connected to the outputs via residuals.

[0066] For example, a first video frame sample sequence is obtained. A sliding window is used, with adjacent frames centered on the target frame. Multiple first video frame samples are input into the video processing model to be trained via a dynamic window sliding mechanism. Each first video frame sample within the sliding window is convolved using a two-dimensional convolution module in the downsampling layer to obtain a first feature map sample corresponding to the first video frame sample. The first feature map samples corresponding to each first video frame sample within the sliding window constitute a first feature map sample set.

[0067] The first feature map sample set, output from the preceding two-dimensional convolutional module, is fed into the one-dimensional convolutional module. This one-dimensional convolutional module then performs a convolution operation along the temporal dimension on the first feature map sample set, outputting a fused feature set that integrates spatial and temporal features, i.e., the second feature map sample set. The one-dimensional convolutional module outputs the second feature map sample set to the next connected two-dimensional convolutional module for further convolution. This process continues, using feature transfer between downsampling layers and multiple intervals of two-dimensional and one-dimensional convolution operations within each downsampling layer to achieve joint spatial and temporal modeling and feature dimensionality reduction, resulting in encoded features. The encoder inputs these encoded features into the decoder, which performs feature reconstruction. The decoder then performs convolution operations on the multi-scale spatiotemporal fusion features processed by the upsampling layers using multiple two-dimensional convolutional modules and subsequent one-dimensional convolutional modules, yielding decoded features. These decoded features represent high-resolution feature maps.

[0068] The reconstruction module upsamples the decoded features to the target resolution, restoring the video features from the latent space to the pixel space, resulting in multiple predicted video frames corresponding to each second feature map sample set. These multiple predicted video frames are then concatenated in chronological order to obtain a complete high-resolution video sequence.

[0069] S330. Based on the predicted video frame sequence and the second video frame sample sequence, determine the first loss, the second loss, and the third loss. Adjust the module parameters of the one-dimensional convolution module based on the first loss, the second loss, and the third loss to obtain the video processing model.

[0070] The first loss characterizes the spatial domain reconstruction loss. The second loss characterizes the perception loss. The third loss characterizes the optical flow error or feature smoothing loss error.

[0071] Further, the predicted video frame sequence includes predicted video frames, and the second video frame sample sequence includes second video frame samples. Based on the predicted video frame sequence and the second video frame sample sequence, a first loss, a second loss, and a third loss are determined, including: determining the first loss based on the pixel values ​​of the predicted video frames and the pixel values ​​of the second video frame samples; determining the second loss based on the feature maps of the predicted video frames and the second video frame samples; and determining the third loss based on the optical flow information of the predicted video frames and the optical flow information of the second video frame samples.

[0072] Optionally, a first loss is calculated based on the pixel values ​​of the predicted video frames corresponding to each first video frame sample and the pixel values ​​of the second video frame samples.

[0073] Optionally, the predicted video frames corresponding to each first video frame sample are input into a deep convolutional network to obtain feature maps of the predicted frames. The second video frame samples corresponding to each first video frame sample are input into a deep convolutional network to obtain feature maps of the second video frame samples. Based on the feature maps of the predicted video frames and the second video frame samples, a second loss is calculated.

[0074] Optionally, optical flow information of the predicted video frame corresponding to each first video frame sample is calculated. Optical flow information of the second video frame sample corresponding to each first video frame sample is also calculated. Based on the optical flow information of the predicted video frame and the optical flow information of the second video frame sample, a third loss is calculated.

[0075] In some embodiments, the first loss, the second loss, and the third loss are weighted to obtain the total model loss. The weights of the second and third losses are balancing coefficients, and their values ​​are preset values.

[0076] The parameters of the 2D backbone network are frozen, and the parameters of the 1D convolutional modules are adjusted using the backpropagation algorithm based on the total model loss. Since existing pre-trained 2D convolutional neural networks or 2D diffusion models are reused, there is no need to retrain large models, thus reducing training difficulty.

[0077] This embodiment of the disclosure embeds a one-dimensional convolutional module into a two-dimensional backbone network. The input and output residuals of the one-dimensional convolutional module are connected, and the initial weights of the one-dimensional convolutional module are zero. This ensures a smooth transition of the model on the pre-trained two-dimensional weights, enabling the video processing model to maintain a stable feature distribution in the early stages of training and gradually learn time-dependent features in the later stages of training. This achieves high-quality detail restoration and smooth transitions. The model structure of this embodiment not only improves the convergence speed and training stability of the video processing model but also significantly reduces computational complexity, facilitating efficient operation on mobile devices, camera chips, and real-time video processing systems.

[0078] Figure 4 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present disclosure. The device can be implemented in the form of software and / or hardware, and optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0079] like Figure 4 As shown, the device includes an acquisition module 410 and a processing module 420.

[0080] Acquisition module 410 is used to acquire the first video frame sequence;

[0081] The processing module 420 is used to input the first video frame sequence into a video processing model to obtain a second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

[0082] Optionally, the number of channels of the one-dimensional convolutional module is related to the number of channels of the two-dimensional convolutional module, each of the two-dimensional convolutional modules is connected to each of the one-dimensional convolutional modules, and the one-dimensional convolutional module adopts one-dimensional depthwise separable convolution.

[0083] Optionally, the processing module 420 is specifically used for:

[0084] Based on multiple sets of first video frames in the first video frame sequence, multiple sets of first feature maps are determined, wherein the first set of feature maps is output by the two-dimensional convolution module;

[0085] Based on each of the first feature map sets, each of the second feature map sets is determined, wherein the second feature map set is output by the one-dimensional convolution module after the two-dimensional convolution module, and the second feature map in the second feature map set includes the spatial features and temporal features of the first video frame;

[0086] The second video frame sequence is determined based on multiple sets of the second feature maps.

[0087] Optionally, determining each second feature map set based on each of the first feature map sets includes:

[0088] Perform a convolution operation along the time dimension on the first feature map set of each group of first video frames, and superimpose the temporal convolution result onto the first feature map set to obtain each second feature map set.

[0089] Optionally, the video processing model further includes a reconstruction module connected to the decoder, wherein determining the second video frame sequence based on multiple sets of second feature maps includes:

[0090] Based on each set of second feature maps, the decoder determines the decoding features, and the reconstruction module performs upsampling processing on the decoding features to obtain multiple second video frames;

[0091] By splicing the multiple second video frames corresponding to each second feature map set in chronological order, the second video frame sequence is obtained.

[0092] Optionally, the video processing model can be trained in the following manner:

[0093] Obtain a first video frame sample sequence and a second video frame sample sequence, wherein the first video frame sample sequence and the second video frame sample sequence are associated with different resolution versions of the same video.

[0094] The first video frame sample sequence is input into the video processing model to be trained to obtain the predicted video frame sequence, which is output by the video processing model to be trained.

[0095] Based on the predicted video frame sequence and the second video frame sample sequence, a first loss, a second loss, and a third loss are determined. The module parameters of the one-dimensional convolution module are adjusted based on the first loss, the second loss, and the third loss to obtain the video processing model.

[0096] Optionally, the predicted video frame sequence includes predicted video frames, the second video frame sample sequence includes second video frame samples, and determining the first loss, the second loss, and the third loss based on the predicted video frame sequence and the second video frame sample sequence includes:

[0097] The first loss is determined based on the pixel values ​​of the predicted video frame and the pixel values ​​of the second video frame sample;

[0098] The second loss is determined based on the feature map of the predicted video frame and the feature map of the second video frame sample;

[0099] The third loss is determined based on the optical flow information of the predicted video frame and the optical flow information of the second video frame sample.

[0100] The video processing apparatus provided in this disclosure can execute the video processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0101] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0102] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 5The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0103] like Figure 5 As shown, electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0104] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0105] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0106] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0107] The electronic device provided in this embodiment and the video processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0108] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the video processing method provided in the above embodiments.

[0109] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0110] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0111] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0112] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0113] Obtain the first video frame sequence;

[0114] The first video frame sequence is input into the video processing model to obtain the second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

[0115] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0121] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0122] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, comprising: Obtain the first video frame sequence; The first video frame sequence is input into the video processing model to obtain the second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

2. The method according to claim 1, wherein the number of channels of the one-dimensional convolution module is related to the number of channels of the two-dimensional convolution module, each of the two-dimensional convolution modules is connected to each of the one-dimensional convolution modules, and the one-dimensional convolution module adopts one-dimensional depth-separable convolution.

3. The method according to claim 1, wherein inputting the first video frame sequence into a video processing model to obtain the second video frame sequence comprises: Based on multiple sets of first video frames in the first video frame sequence, multiple sets of first feature maps are determined, wherein the first set of feature maps is output by the two-dimensional convolution module; Based on each of the first feature map sets, each of the second feature map sets is determined, wherein the second feature map set is output by the one-dimensional convolution module after the two-dimensional convolution module, and the second feature map in the second feature map set includes the spatial features and temporal features of the first video frame; The second video frame sequence is determined based on multiple sets of the second feature maps.

4. The method according to claim 3, wherein determining each second feature map set based on each of the first feature map sets comprises: Perform a convolution operation along the time dimension on the first feature map set of each group of first video frames, and superimpose the temporal convolution result onto the first feature map set to obtain each second feature map set.

5. The method according to claim 3, wherein the video processing model further comprises a reconstruction module connected to the decoder, and the step of determining the second video frame sequence based on a plurality of second feature map sets comprises: Based on each set of second feature maps, the decoder determines the decoding features, and the reconstruction module performs upsampling processing on the decoding features to obtain multiple second video frames. By splicing the multiple second video frames corresponding to each second feature map set in chronological order, the second video frame sequence is obtained.

6. The video processing model according to claim 1 is trained in the following manner: Obtain the first video frame sample sequence and the second video frame sample sequence, wherein, The first video frame sample sequence and the second video frame sample sequence are associated with different resolution versions of the same video. The first video frame sample sequence is input into the video processing model to be trained to obtain the predicted video frame sequence, which is output by the video processing model to be trained. Based on the predicted video frame sequence and the second video frame sample sequence, a first loss, a second loss, and a third loss are determined. The module parameters of the one-dimensional convolution module are adjusted based on the first loss, the second loss, and the third loss to obtain the video processing model.

7. The method according to claim 6, wherein the predicted video frame sequence includes predicted video frames, the second video frame sample sequence includes second video frame samples, and determining the first loss, the second loss, and the third loss based on the predicted video frame sequence and the second video frame sample sequence includes: The first loss is determined based on the pixel values ​​of the predicted video frame and the pixel values ​​of the second video frame sample; The second loss is determined based on the feature map of the predicted video frame and the feature map of the second video frame sample; The third loss is determined based on the optical flow information of the predicted video frame and the optical flow information of the second video frame sample.

8. A video processing apparatus, comprising: The acquisition module is used to acquire the first video frame sequence; A processing module is used to input the first video frame sequence into a video processing model to obtain a second video frame sequence. The second video frame sequence is output by the video processing model. The resolution of the second video frame sequence is higher than that of the first video frame sequence. The video processing model includes an encoder, a decoder, and at least one one-dimensional convolutional module. The encoder and decoder each include multiple two-dimensional convolutional modules. At least some of the multiple two-dimensional convolutional modules are connected to the one-dimensional convolutional module.

9. An electronic device, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in any one of claims 1-7.

10. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the video processing method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the video processing method as described in any one of claims 1-7.

Images