Video processing method and device, display equipment and storage medium

Abstract

The invention discloses a video processing method and device, display equipment and a storage medium, and relates to the technical field of video processing, and the video processing method comprises the steps: determining the definition level of an initial video; when the definition level is lower than the target definition level, the initial video is input into a preset video processing model, a target video with the definition level higher than that of the initial video is obtained, and the video processing model comprises a sampler, a variational auto-encoder and a video fusion device. The video definition can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, apparatus, display device and storage medium. Background Art

[0002] With the development of consumer electronics technology, users have higher and higher requirements for the clarity of video playback, especially on large-screen devices. High-definition video playback often brings users a better visual experience.

[0003] However, existing video playback services often provide videos of different resolutions based on the user's membership level when providing online playback. When the user's membership level is low, the video content has low resolution, which will cause blurry images and loss of details, seriously affecting the viewing experience. In addition, traditional video compression and transmission technologies often sacrifice video details while maintaining transmission efficiency. Especially when the network conditions are unstable, the video resolution drops more significantly, making it difficult to meet users' pursuit of high-quality visual experience.

[0004] In summary, how to improve video clarity has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] The main purpose of this application is to provide a video processing method, device, display device and storage medium, aiming to improve video clarity.

[0006] To achieve the above object, the present application proposes a video processing method, which includes:

[0007] Determining the clarity level of the initial video;

[0008] When the clarity level is lower than the target clarity level, the initial video is input into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

[0009] In one embodiment, the step of inputting the initial video into a preset video processing model to obtain a target video having a higher definition level than the initial video includes:

[0010] Inputting the initial video into a preset video processing model, performing sampling analysis on the initial video through a sampler to obtain an initial feature map;

[0011] Reconstructing the initial feature map into a target feature map by a variational autoencoder, wherein a resolution of the target feature map is higher than a resolution of the initial feature map;

[0012] The target feature map is converted into a high-resolution video frame by a video fusion device to obtain a target video with a higher definition level than the initial video.

[0013] In one embodiment, the sampler includes a temporal attention mechanism, a pixel rearrangement mechanism and a feature fusion module, and the step of sampling and analyzing the initial video through the sampler to obtain an initial feature map includes:

[0014] Extracting an initial video frame from the initial video by a sampler;

[0015] Analyzing the temporal relationship of the initial video frames through the temporal attention mechanism to obtain temporal information;

[0016] Mapping the spatial information of the initial video frame to the depth dimension through the pixel rearrangement mechanism to obtain spatial domain information;

[0017] The time series information and the spatial domain information are fused through the feature fusion module to obtain an initial feature map including the time series information and the spatial domain information.

[0018] In one embodiment, the temporal attention mechanism includes a global branch and a local branch, and the step of analyzing the temporal relationship of the initial video frame by the temporal attention mechanism to obtain the temporal information includes:

[0019] Generate a video-adaptive dynamic convolution kernel based on the global time series data of the initial video frame through the global branch to aggregate the global time series data to obtain global information;

[0020] Learning the local structural information of the initial video frame through the local branch, generating an importance weight that is sensitive to the temporal position, and obtaining local information;

[0021] The global information and the local information are combined to obtain timing information.

[0022] In one embodiment, the pixel rearrangement mechanism includes an SPD layer (Space to depth layer), an ESE layer (Effective SE layer, a convolutional neural network structure for image classification) and a non-strided convolutional layer, and the step of mapping the spatial information of the initial video frame to the depth dimension through the pixel rearrangement mechanism to obtain spatial domain information includes:

[0023] Rearranging the pixel space blocks of the initial video frame through the SPD layer to obtain a first feature map;

[0024] Performing weighted processing on the first feature map through the ESE layer to obtain a second feature map;

[0025] The second feature map is convolved by the non-step convolution layer to obtain spatial domain information.

[0026] In one embodiment, the variational autoencoder includes an encoder, a fine-tuning control module and a decoder, and the step of reconstructing the initial feature map into a target feature map by the variational autoencoder includes:

[0027] Mapping the initial feature map into a continuous latent variable by the encoder;

[0028] Performing noise processing and denoising training on the latent variables through the fine-tuning control module to obtain high-definition latent variables;

[0029] The decoder performs a reconstruction operation on the high-definition latent variable to obtain a target feature map.

[0030] In one embodiment, the step of converting the target feature map into a high-resolution video frame by a video fusion device to obtain a target video with a higher definition level than the initial video includes:

[0031] Converting the target feature map into a high-resolution video frame through a video fusion device;

[0032] The display parameters of the high-resolution video frame are adjusted to obtain a target video having a higher definition level than the initial video.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a video processing device, the video processing device comprising:

[0034] A level determination module, used to determine the clarity level of the initial video;

[0035] A video processing module is used to input the initial video into a preset video processing model to obtain a target video with a higher clarity level than the initial video when the clarity level is lower than the target clarity level, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

[0036] In addition, to achieve the above objectives, the present application also proposes a display device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method as described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0039] The present application proposes a video processing method, which determines the clarity level of an initial video; then, when the clarity level is lower than a target clarity level, the initial video is input into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fuser.

[0040] In summary, this application introduces a video processing model including a sampler, a variational autoencoder and a video fusion device, which can improve the clarity level of the played video without increasing the transmission burden. It not only solves the problem of low video clarity for users with low membership levels, but also can maintain a high video quality when network conditions are poor, thereby effectively satisfying users' pursuit of high-quality visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the video processing method of the present application;

[0044] Figure 2 A schematic diagram of the structure of a pixel rearrangement mechanism provided in Embodiment 2 of the video processing method of the present application;

[0045] Figure 3 A schematic diagram of a video processing flow provided for Embodiment 2 of the video processing method of the present application;

[0046] Figure 4 This is a schematic diagram of the module structure of the video processing device according to an embodiment of the present application;

[0047] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the video processing method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] With the development of consumer electronics technology, users have higher and higher requirements for the clarity of video playback, especially on large-screen devices. High-definition video playback often brings users a better visual experience.

[0052] However, existing video playback services often provide videos of different resolutions based on the user's membership level when providing online playback. When the user's membership level is low, the video content has low resolution, which will cause blurry images and loss of details, seriously affecting the viewing experience. In addition, traditional video compression and transmission technologies often sacrifice video details while maintaining transmission efficiency. Especially when the network conditions are unstable, the video resolution drops more significantly, making it difficult to meet users' pursuit of high-quality visual experience.

[0053] In summary, how to improve video clarity has become a technical problem that needs to be solved urgently in this field. The main solution of the embodiment of the present application is: determine the clarity level of the initial video; when the clarity level is lower than the target clarity level, input the initial video into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

[0054] The video processing method proposed in the embodiment of the present application, by introducing a video processing model including a sampler, a variational autoencoder and a video fuser, can improve the clarity level of the played video without increasing the transmission burden. It not only solves the problem of low video clarity for users with low membership levels, but also can maintain a high video quality when network conditions are poor, thereby effectively satisfying users' pursuit of high-quality visual experience.

[0055] It should be noted that the executor of this embodiment can be a computing service device with data processing, network communication, screen display and program running functions, such as tablet computers, personal computers, mobile phones, smart TVs, smart screens and other electronic devices with display functions. The following takes the display device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides a video processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the video processing method of the present application.

[0057] In this embodiment, the video processing method includes steps S10 to S20:

[0058] Step S10, determining the definition level of the initial video;

[0059] The clarity level of the video to be played (i.e., the initial video) can be determined. For example, the clarity options set by the current video playback platform when providing online playback services, such as 480P, 720P, 1080P or 4K, can be directly obtained, and the clarity option currently selected by the user can be used as the clarity level; or, the resolution, color depth, contrast, sharpness and other dimensions of the video frames in the initial video can be analyzed to obtain a clarity level that can accurately reflect the video quality level.

[0060] Step S20, when the clarity level is lower than the target clarity level, the initial video is input into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

[0061] After determining the clarity level of the initial video, the clarity level is compared with the target clarity level. If the clarity level of the initial video is lower than the target clarity level, the initial video is input into a pre-trained video processing model, which consists of a sampler, a variational autoencoder, and a video fusion model. The modules in the model work together to extract useful information from the initial video and generate a target video with a higher clarity level than the initial video through algorithm processing.

[0062] It should be noted that the target clarity level is the highest clarity that can be supported by the display device and the current video playback platform. The clarity level of the target video is higher than the clarity level of the initial video. In practical applications, the clarity level of the target video is usually set to one level higher than the clarity level of the initial video to avoid the situation where the video processing model cannot generate a high-definition video with optimal instructions due to insufficient video details in the initial video.

[0063] At the same time, the process of improving video clarity can be performed locally on the display device without relying on network transmission, ensuring the stability and reliability of video processing, thereby avoiding the adverse effects of network conditions on video quality.

[0064] In a feasible embodiment, step S20 may include steps S201 to S203:

[0065] Step S201, inputting the initial video into a preset video processing model, and performing sampling analysis on the initial video through a sampler to obtain an initial feature map;

[0066] The initial video is input into a pre-built and trained video processing model, which has a built-in sampler, such as an Upscaler sampler, which is responsible for sampling and analyzing the input initial video to extract basic visual features of the initial video and generate an initial feature map containing these features.

[0067] Step S202, reconstructing the initial feature map into a target feature map through a variational autoencoder, wherein the resolution of the target feature map is higher than the resolution of the initial feature map;

[0068] After obtaining the initial feature map, the video processing model uses the built-in variational autoencoder (VAE) to further process it. The core of the variational autoencoder is that it introduces continuous latent variable distribution and noise, so that the encoding space is no longer an isolated discrete point, but a continuous probability distribution. This design greatly improves the smooth transition ability between the encoder and decoder, thereby generating high-quality images. Through the encoding and decoding process of the variational autoencoder, the initial feature map is reconstructed into a target feature map with higher resolution.

[0069] For example, taking a 480P movie clip as the initial video, its initial feature map is input into the variational autoencoder. The variational autoencoder first encodes the initial feature map and maps it to the latent space. In the latent space, the variational autoencoder denoises and optimizes the encoded features by introducing noise and denoising training mechanism. Finally, the variational autoencoder decodes the optimized features and generates a target feature map with a resolution increased to 720P.

[0070] Step S203, converting the target feature map into a high-resolution video frame through a video fusion device to obtain a target video with a higher definition level than the initial video.

[0071] After obtaining the target feature map, the video processing model uses a video fusion device to convert the target feature map into a high-resolution video frame to obtain a target video with a higher clarity level than the initial video. The video fusion device uses a deep learning framework and frame-by-frame processing technology to map the feature vectors in the target feature map back to high-resolution video frames.

[0072] In summary, in the embodiments of the present application, by introducing a video processing model including a sampler, a variational autoencoder and a video fusion device, the clarity level of the played video can be improved without increasing the transmission burden. This not only solves the problem of low video clarity for users with low membership levels, but also maintains a high video quality when network conditions are poor, thereby effectively satisfying users' pursuit of high-quality visual experience.

[0073] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the sampler includes a temporal attention mechanism, a pixel rearrangement mechanism and a feature fusion module, and step S201 may include steps S2011 to S2014:

[0074] Step S2011, extracting an initial video frame from the initial video through a sampler;

[0075] The sampler extracts continuous initial video frames from the input initial video. These initial video frames contain the original information of the initial video content, such as color, brightness, texture, etc. The sampler can ensure that the extracted initial video frames can accurately reflect the content of the initial video through precise time control.

[0076] Step S2012, analyzing the temporal relationship of the initial video frames through a temporal attention mechanism to obtain temporal information;

[0077] After extracting the initial video frames, the sampler further processes these initial video frames using the temporal attention mechanism, where the temporal attention mechanism can analyze the temporal relationship between the initial video frames, thereby extracting key timing information to understand the dynamic changes of the video content.

[0078] Specifically, the temporal attention mechanism focuses on the temporal continuity between video frames, as well as the changing patterns and trends between them. By calculating the similarities and differences between video frames, the temporal attention mechanism can identify the key frames and key events in the initial video, thereby extracting important timing information.

[0079] In a feasible embodiment, the temporal attention mechanism includes a global branch and a local branch, and step S2012 may include steps a10 to a30:

[0080] Step a10, generating a video-adaptive dynamic convolution kernel based on the global time series data of the initial video frame through a global branch to aggregate the global time series data to obtain global information;

[0081] The global branch in the temporal attention mechanism is responsible for processing the global temporal data of the initial video frame. Specifically, the global branch receives a series of continuous initial video frames and generates video-adaptive dynamic convolution kernels based on these initial video frames. These dynamic convolution kernels are flexible and adaptive, and can be adjusted and optimized according to the dynamic changes of the video content.

[0082] After generating the dynamic convolution kernels, the global branch will use these convolution kernels to aggregate the global time series data. By calculating the similarities and differences between different video frames, the global branch can extract the global information in the initial video, such as overall motion trends, scene changes, etc.

[0083] Step a20, learning the local structural information of the initial video frame through the local branch, generating the importance weight sensitive to the temporal position, and obtaining the local information;

[0084] The local branch in the temporal attention mechanism focuses on the local structural information of the initial video frame, and carefully analyzes and processes the initial video frame to extract importance weights that are sensitive to temporal position. The importance weights reflect the contribution of different regions in the initial video frame to the temporal relationship, which helps to identify key frames and key events in the video.

[0085] Specifically, the local branch uses deep learning algorithm techniques, such as convolutional neural networks, to extract features and learn representations of the initial video frames. By calculating the similarities and correlations between different features, the local branch can generate importance weights that are sensitive to temporal positions and obtain feature maps containing local information. These feature maps reflect the local structures and features of different regions in the video frames.

[0086] Step a30, combining global information and local information to obtain timing information.

[0087] After obtaining global information and local information, the temporal attention mechanism will utilize the complementarity and correlation between global information and local information to perform weighted processing and fusion on them. By calculating the similarity and complementarity between different information, the temporal attention mechanism can generate a temporal information containing global information and local information.

[0088] Step S2013, mapping the spatial information of the initial video frame to the depth dimension through a pixel rearrangement mechanism to obtain spatial domain information;

[0089] The sampler also maps the spatial information of the initial video frame using a pixel rearrangement mechanism, wherein the pixel rearrangement mechanism is an algorithm that can change the order of pixel arrangement, and it can map the spatial information in the initial video frame to the depth dimension, thereby obtaining spatial domain information.

[0090] Specifically, the pixel rearrangement mechanism groups and rearranges the pixels in the initial video frame to form new feature maps. These feature maps contain the spatial information in the initial video frame, such as edges, textures, and shapes. Through the processing of the pixel rearrangement mechanism, the sampler can present the spatial information in the video frame more clearly.

[0091] In a feasible embodiment, the pixel rearrangement mechanism includes an SPD layer, an ESE layer, and a non-strided convolution layer, and step S2013 may include steps b10 to b30:

[0092] Step b10, rearranging the pixel space blocks of the initial video frame through the SPD layer to obtain a first feature map;

[0093] It should be noted that the pixel rearrangement mechanism can rearrange and combine pixels in the video frame to extract richer spatial information. The pixel rearrangement mechanism includes the SPD layer, the ESE layer, and the non-strided convolution layer. The layers in the pixel rearrangement mechanism work together to efficiently process the initial video frame and generate a feature map containing rich spatial information.

[0094] The SPD layer receives a series of continuous initial video frames and divides them into multiple small pixel space blocks. Then, the SPD layer rearranges and combines these pixel space blocks according to certain rules or algorithms to form a new feature map, namely the first feature map. During the rearrangement process, the SPD layer considers the spatial relationship and adjacency between pixels to ensure that the rearranged feature map can retain the important information and structural features in the original video frame.

[0095] Step b20, performing weighted processing on the first feature map through the ESE layer to obtain a second feature map;

[0096] After obtaining the first feature map, the ESE layer will further perform weighted processing on it to obtain the second feature map. The ESE layer is an enhanced spatial coding layer that can calculate the importance weight of each pixel and perform weighted processing on the pixels according to these weights. Through weighted processing, the ESE layer can highlight the key information and features in the first feature map, while suppressing unimportant information and noise, thereby obtaining a clearer and more accurate second feature map.

[0097] Step b30, performing convolution processing on the second feature map through a non-step convolution layer to obtain spatial domain information.

[0098] The non-stride convolution layer performs convolution on the second feature map to obtain spatial domain information. The non-stride convolution layer is a special convolution layer that does not use the step parameter in the convolution process, thereby retaining more spatial information and detail features.

[0099] Specifically, the non-stride convolution layer uses multiple convolution kernels (or filters) to perform convolution operations on the second feature map to extract richer spatial features and texture information. Through convolution processing, a feature map containing spatial domain information is obtained.

[0100] For example, in a feasible implementation manner, the structural diagram of the pixel rearrangement mechanism is as follows: Figure 2 As shown in the figure, the pixel rearrangement mechanism includes the SPD layer (i.e., the Space to depth layer in the figure), the ESE layer (i.e., the Effective SE layer in the figure) and the non-strided convolution layer (i.e., the Non-strided Conv layer in the figure). The input initial video frame is first processed by the SPD layer, and the pixel space blocks of the initial video frame are rearranged to reduce the spatial resolution and increase the number of channels. Then, the ESE layer adaptively mines the relationship between channels, pays attention to useful channel information, suppresses useless channel information, and improves the expression ability of features. Finally, it enters the non-strided convolution layer to extract features layer by layer from the processed feature map to retain more image details and spatial information. The specific processing details are as follows:

[0101] SPD layer: For the input initial video frame X∈(S, S, C), a slicing operation with a step size of 2 is performed to obtain several pixel blocks. The pixel blocks are rearranged to obtain four sub-images, each of which has a size of (S / 2, S / 2, C). The four sub-images are connected along the channel dimension to finally obtain the first feature map, whose size is (S / 2, S / 2, 4C).

[0102] ESE layer: Performs an attention operation on the first feature map of the input to further explore the relationship between channels, and uses the obtained weight vector to assign weights to the feature map to obtain the second feature map. This operation does not change the size of the feature map.

[0103] Non-stride convolution layer: Use a convolution operation with a stride of 1 to process the second feature map, so that the network can reduce the number of channels of the feature map without losing detail information and generate rich feature representations.

[0104] Step S2014: The time series information and the spatial domain information are fused through a feature fusion module to obtain an initial feature map including the time series information and the spatial domain information.

[0105] The sampler uses the feature fusion module to merge and integrate the temporal information and the spatial domain information to generate a comprehensive feature map containing temporal information and spatial domain information, namely the initial feature map.

[0106] Therefore, the sampler can efficiently process the initial video by integrating the temporal attention mechanism, the pixel rearrangement mechanism and the feature fusion module to generate an initial feature map containing both temporal and spatial domain information.

[0107] In a feasible embodiment, the variational autoencoder includes an encoder, a fine-tuning control module and a decoder, and step S202 may include steps S2021 to S2023:

[0108] Step S2021, mapping the initial feature map into a continuous latent variable through an encoder;

[0109] It should be noted that the Variational Autoencoder (VAE) is a generative model that includes an encoder, a fine-tuning control module, and a decoder. It can learn effective representations of data and generate high-quality data samples.

[0110] The encoder maps the initial feature map of the sampler input into a continuous latent variable space, which is low-dimensional and continuous and can capture the latent structure and distribution characteristics of the input data.

[0111] Specifically, the encoder converts the initial feature map into a mean vector and a standard deviation vector through a series of nonlinear transformations (such as convolutional layers, fully connected layers, etc.) and activation functions (such as ReLU, Sigmoid, etc.). These two vectors together define the Gaussian distribution of the latent variable. Then, by sampling from this distribution, a specific latent variable is obtained, which represents the potential representation of the input data (i.e., the initial feature map).

[0112] Step S2022, performing noise processing and denoising training on the latent variables through a fine-tuning control module to obtain high-definition latent variables;

[0113] After obtaining the latent variables, the fine-tuning control module further improves the quality of the latent variables by introducing noise and performing denoising training, so that it can more accurately reflect the latent structure of the input data.

[0114] During the noise processing stage, the fine-tuning control module adds noise to the latent variables to simulate the uncertainty of real-world data. Then, during the denoising training stage, the fine-tuning control module recovers the original and clear information from the latent variables contaminated by noise. This process actually refines and optimizes the latent variables to make them more robust and representative.

[0115] Through multiple iterative training, the fine-tuning control module can learn how to effectively remove noise while retaining the key information in the latent variables, ultimately obtaining high-definition latent variables.

[0116] Step S2023, reconstructing the high-definition latent variables through the decoder to obtain a target feature map.

[0117] The decoder is the inverse process of the encoder, which transforms the latent variables back to the feature map space through a series of nonlinear transformations and activation functions.

[0118] Specifically, the decoder receives high-definition latent variables as input, and gradually converts the high-definition latent variables into target feature maps that match the size and number of channels of the original input feature map through a series of upsampling layers, convolutional layers, and fully connected layers. During the decoding process, the decoder uses the learned latent structure and feature information to restore the details and texture of the input data as much as possible.

[0119] In a feasible embodiment, step S203 may include steps S2031 to S2032:

[0120] Step S2031, converting the target feature map into a high-resolution video frame through a video fuser;

[0121] The video fuser will first analyze the key information and feature points in the target feature map, including edges, textures, colors, etc., and then use this information, combined with upsampling, super-resolution reconstruction and other technologies in the deep learning algorithm, to gradually enlarge the feature map to the target resolution. In this process, the video fuser will also perform fine smoothing on the enlarged image to reduce possible adverse effects such as aliasing and blur. Finally, after processing by the video fuser, a video frame with improved resolution and rich details is obtained. This video frame not only retains the key information in the initial video, but also has a significant improvement in visual effects.

[0122] Step S2032, adjusting display parameters of the high-resolution video frame to obtain a target video with a higher definition level than the initial video.

[0123] The video blender can further adjust the display parameters of the high-resolution video frames to improve their overall clarity level, wherein the display parameters include but are not limited to brightness, contrast, saturation, sharpness and other parameters that affect the visual effect of video playback.

[0124] In summary, in this embodiment, by introducing a pixel rearrangement mechanism and a time domain attention mechanism in the sampler, the initial video frame can be efficiently processed, and an initial feature map containing spatial domain information and time domain information can be extracted. Then, the initial feature map is subjected to noise processing and denoising training using a variational autoencoder to obtain a target feature map with improved clarity. Finally, the target feature map is converted into a high-resolution video frame through a video fuser. Through the feature fusion and reconstruction process, the video fuser can retain the key information in the initial video and improve the resolution of the video to enhance the visual effect.

[0125] In summary, through the collaborative work of the above-mentioned layers (sampler, variational autoencoder and video fusion), this embodiment can effectively convert low-definition video into high-definition video, while significantly reducing the demand for server and network bandwidth, providing users with high-quality video viewing experience.

[0126] For example, to help understand the implementation process of the video processing method obtained by combining this embodiment with the above embodiments, please refer to Figure 3 , Figure 3 A brief flow chart of a video processing method is provided, specifically:

[0127] The low-definition video to be played (i.e., the initial video) is input into a preset video processing model, which includes a sampler, a variational autoencoder and a video fusion model. The initial video is sampled and analyzed by the sampler using a time-domain attention mechanism and a pixel rearrangement mechanism to obtain an initial feature map; then, the initial feature map is reconstructed into a target feature map with higher resolution through a variational autoencoder, wherein the variational autoencoder includes an encoder (i.e., the VAE encoder in the figure), a fine-tuning control module (i.e., the LatentEncoder, the noise input part and the denoising network UNet in the figure) and a decoder (i.e., the VAE decoder in the figure); then, the target feature map is converted into a high-resolution video frame through the video fusion model, and finally a target video with a higher clarity level than the initial video, i.e., a high-definition video, is obtained and output.

[0128] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0129] The present application also provides a video processing device. Figure 4 , the video processing device comprises:

[0130] A level determination module 10, used to determine the clarity level of the initial video;

[0131] The video processing module 20 is used to input the initial video into a preset video processing model to obtain a target video with a higher clarity level than the initial video when the clarity level is lower than the target clarity level, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

[0132] Optionally, the video processing module 20 is further configured to:

[0133] Input the initial video into a preset video processing model, perform sampling analysis on the initial video through a sampler, and obtain an initial feature map;

[0134] Reconstructing the initial feature map into a target feature map through a variational autoencoder, wherein the resolution of the target feature map is higher than the resolution of the initial feature map;

[0135] The target feature map is converted into a high-resolution video frame through a video fusion device to obtain a target video with a higher definition level than the initial video.

[0136] Optionally, the sampler includes a temporal attention mechanism, a pixel rearrangement mechanism and a feature fusion module, and the video processing module 20 is further used for:

[0137] Extracting an initial video frame from the initial video by a sampler;

[0138] Analyze the temporal relationship of the initial video frames through the temporal attention mechanism to obtain the temporal information;

[0139] The spatial information of the initial video frame is mapped to the depth dimension through a pixel rearrangement mechanism to obtain spatial domain information;

[0140] The temporal information and the spatial domain information are fused through the feature fusion module to obtain an initial feature map containing temporal information and spatial domain information.

[0141] Optionally, the temporal attention mechanism includes a global branch and a local branch, and the video processing module 20 is further used for:

[0142] Generate a video-adaptive dynamic convolution kernel based on the global time series data of the initial video frame through the global branch to aggregate the global time series data to obtain global information;

[0143] The local structure information of the initial video frame is learned through the local branch, and the importance weights that are sensitive to the temporal position are generated to obtain the local information;

[0144] Combining global information and local information to obtain timing information.

[0145] Optionally, the pixel rearrangement mechanism includes an SPD layer, an ESE layer and a non-strided convolution layer, and the video processing module 20 is further used for:

[0146] Rearrange the pixel space blocks of the initial video frame through the SPD layer to obtain a first feature map;

[0147] The first feature map is weighted by the ESE layer to obtain a second feature map;

[0148] The second feature map is convolved through a non-step convolutional layer to obtain spatial domain information.

[0149] Optionally, the variational autoencoder includes an encoder, a fine-tuning control module and a decoder, and the video processing module 20 is further used for:

[0150] Mapping the initial feature map into a continuous latent variable through an encoder;

[0151] The latent variables are processed by noise processing and denoising training through fine-tuning control modules to obtain high-definition latent variables;

[0152] The decoder reconstructs the high-definition latent variables to obtain the target feature map.

[0153] Optionally, the video processing module 20 is further configured to:

[0154] Convert the target feature map into a high-resolution video frame through a video fuser;

[0155] The display parameters of the high-resolution video frames are adjusted to obtain a target video having a higher definition level than the initial video.

[0156] The video processing device provided in the embodiment of the present application adopts the video processing method in the above embodiment to improve the video clarity. Compared with the prior art, the beneficial effects of the video processing device provided in the embodiment of the present application are the same as the beneficial effects of the video processing method provided in the above embodiment, and other technical features in the video processing device are the same as the features disclosed in the video processing method in the above embodiment, which will not be described in detail here.

[0157] An embodiment of the present application provides a display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video processing method in the above-mentioned embodiment 1.

[0158] Reference below Figure 5 , which shows a schematic diagram of the structure of a display device suitable for implementing an embodiment of the present application. The display device in the embodiment of the present application may include a mobile terminal and a fixed terminal of an MCU. Figure 5 The display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0159] like Figure 5As shown, the display device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the display device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / generation (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; a generation device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the display device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a display device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0160] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0161] The display device provided in the embodiment of the present application adopts the video processing method in the above embodiment to improve the video clarity. Compared with the prior art, the beneficial effects of the display device provided in the embodiment of the present application are the same as the beneficial effects of the video processing method provided in the above embodiment, and other technical features in the display device are the same as the features disclosed in the video processing method in the previous embodiment, which will not be repeated here.

[0162] It should be understood that the various parts disclosed in the embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0163] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0164] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the video processing method in the above embodiment.

[0165] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0166] The computer-readable storage medium may be included in the display device; or may exist independently without being assembled into the display device.

[0167] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the display device, the display device: determines the clarity level of the initial video; when the clarity level is lower than the target clarity level, inputs the initial video into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fuser.

[0168] The computer program code for performing the operation of the embodiment of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN: Local Area Network) or a wide area network (WAN: Wide Area Network), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0169] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0171] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video processing method, and can improve the video clarity. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the video processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0172] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video processing method when executed by a processor.

[0173] The computer program product provided by the present application can improve the video clarity. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, which will not be described in detail here.

[0174] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video processing method, characterized in that: The video processing method comprises: Determining the clarity level of the initial video; When the clarity level is lower than the target clarity level, the initial video is input into a preset video processing model to obtain a target video with a higher clarity level than the initial video, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

2. The video processing method according to claim 1, characterized in that: The step of inputting the initial video into a preset video processing model to obtain a target video having a higher definition level than the initial video comprises: Inputting the initial video into a preset video processing model, performing sampling analysis on the initial video through a sampler to obtain an initial feature map; Reconstructing the initial feature map into a target feature map by a variational autoencoder, wherein a resolution of the target feature map is higher than a resolution of the initial feature map; The target feature map is converted into a high-resolution video frame by a video fusion device to obtain a target video with a higher definition level than the initial video.

3. The video processing method according to claim 2, characterized in that: The sampler includes a temporal attention mechanism, a pixel rearrangement mechanism and a feature fusion module. The step of sampling and analyzing the initial video through the sampler to obtain an initial feature map includes: Extracting an initial video frame from the initial video by a sampler; Analyzing the temporal relationship of the initial video frames through the temporal attention mechanism to obtain temporal information; Mapping the spatial information of the initial video frame to the depth dimension through the pixel rearrangement mechanism to obtain spatial domain information; The time series information and the spatial domain information are fused through the feature fusion module to obtain an initial feature map including the time series information and the spatial domain information.

4. The video processing method according to claim 3, characterized in that: The temporal attention mechanism includes a global branch and a local branch. The step of analyzing the temporal relationship of the initial video frame by the temporal attention mechanism to obtain the temporal information includes: Generate a video-adaptive dynamic convolution kernel based on the global time series data of the initial video frame through the global branch to aggregate the global time series data to obtain global information; Learning the local structural information of the initial video frame through the local branch, generating an importance weight that is sensitive to the temporal position, and obtaining local information; The global information and the local information are combined to obtain timing information.

5. The video processing method according to claim 3, characterized in that: The pixel rearrangement mechanism includes an SPD layer, an ESE layer, and a non-strided convolution layer. The step of mapping the spatial information of the initial video frame to the depth dimension through the pixel rearrangement mechanism to obtain spatial domain information includes: Rearranging the pixel space blocks of the initial video frame through the SPD layer to obtain a first feature map; Performing weighted processing on the first feature map through the ESE layer to obtain a second feature map; The second feature map is convolved by the non-step convolution layer to obtain spatial domain information.

6. The video processing method according to claim 2, characterized in that: The variational autoencoder includes an encoder, a fine-tuning control module and a decoder, and the step of reconstructing the initial feature map into a target feature map by the variational autoencoder includes: Mapping the initial feature map into a continuous latent variable by the encoder; Performing noise processing and denoising training on the latent variables through the fine-tuning control module to obtain high-definition latent variables; The decoder performs a reconstruction operation on the high-definition latent variable to obtain a target feature map.

7. The video processing method according to claim 2, characterized in that: The step of converting the target feature map into a high-resolution video frame by a video fusion device to obtain a target video with a higher definition level than the initial video includes: Converting the target feature map into a high-resolution video frame through a video fusion device; The display parameters of the high-resolution video frame are adjusted to obtain a target video having a higher definition level than the initial video.

8. A video processing device, characterized in that: The video processing device comprises: A level determination module, used to determine the clarity level of the initial video; A video processing module is used to input the initial video into a preset video processing model to obtain a target video with a higher clarity level than the initial video when the clarity level is lower than the target clarity level, wherein the video processing model includes a sampler, a variational autoencoder and a video fusion device.

9. A display device, characterized in that: The display device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images