Video processing method, device and storage medium

Abstract

The present application provides a video processing method, device and storage medium. The method encodes and decodes the original video to obtain a mixed-resolution video, which includes a first-resolution frame and a second-resolution frame corresponding to the key frame, and also includes a third-resolution frame corresponding to the non-key frame. The resolution of the first-resolution frame is higher than the resolution of the second-resolution frame and the resolution of the third-resolution frame; based on the first-resolution frame and the second-resolution frame, the third-resolution frame corresponding to the non-key frame is amplified and the amplified video is output. The amplified video includes the first-resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame. The above scheme only needs to encode and decode the non-key frame to obtain a low-resolution frame, and then amplify the low-resolution frame according to the information of the key frame to obtain an amplified high-resolution video with less bit rate, thereby reducing the bandwidth pressure of video transmission.

Description

Technical Field

[0001] The present application relates to video processing technology, for example, to a video processing method, device and storage medium. Background Art

[0002] In today's information age, video information accounts for an increasingly high proportion, and the amount of video data is increasing. Video transmission and compression are particularly important. Although codec technology continues to advance, in areas with poor conditions, such as mountainous areas or other harsh environments, insufficient base stations and insufficient broadband resources may result in video information loss and long transmission delays. If the entire video is significantly compressed, the decoded video will have low resolution, poor subjective and objective quality, and poor amplification, seriously affecting the viewing experience. Currently, there is a lack of effective video processing methods that can reduce the resources occupied by video data transmission while still decoding high-quality video in harsh environments, with insufficient resources or limited traffic. Summary of the Invention

[0003] The present application provides a video processing method, device and storage medium, which only needs to encode and decode non-key frames to obtain low-resolution frames, and obtain amplified high-resolution video with less bit rate, thereby reducing the bandwidth pressure of video transmission.

[0004] The present invention provides a video processing method, including:

[0005] A mixed-resolution video is obtained by encoding and decoding an original video, wherein the mixed-resolution video includes a first-resolution frame and a second-resolution frame corresponding to a key frame in the original video, and also includes a third-resolution frame corresponding to a non-key frame in the original video, wherein a resolution of the first-resolution frame is higher than a resolution of the second-resolution frame and a resolution of the third-resolution frame;

[0006] According to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed resolution video is amplified, and the amplified video is output, wherein the amplified video includes the first resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame.

[0007] The present application also provides a device, including:

[0008] one or more processors;

[0009] a storage device for storing one or more programs;

[0010] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned video processing method.

[0011] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above-mentioned video processing method is implemented.

[0012] The embodiments of the present application provide a video processing method, device and storage medium. The method encodes and decodes the original video to obtain a mixed-resolution video, which includes a first-resolution frame and a second-resolution frame corresponding to the key frame, and also includes a third-resolution frame corresponding to the non-key frame. The resolution of the first-resolution frame is higher than the resolution of the second-resolution frame and the resolution of the third-resolution frame; based on the first-resolution frame and the second-resolution frame, the third-resolution frame corresponding to the non-key frame is amplified and the amplified video is output. The amplified video includes the first-resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame. The above scheme only needs to encode and decode the non-key frame to obtain a low-resolution frame, and then amplify the low-resolution frame according to the information of the key frame to obtain an amplified high-resolution video with less bit rate, thereby reducing the bandwidth pressure of video transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a video processing method provided by an embodiment;

[0014] Figure 2 A schematic diagram of a mixed-resolution video provided by an embodiment;

[0015] Figure 3 A flowchart of a video processing method provided by another embodiment;

[0016] Figure 4 A schematic diagram of a video amplification network according to an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of a motion estimation subnetwork provided by an embodiment;

[0018] Figure 6 A schematic diagram of a motion compensation subnetwork provided by an embodiment;

[0019] Figure 7 A schematic diagram of an amplification sub-network provided by an embodiment;

[0020] Figure 8 A schematic structural diagram of a video processing device provided by an embodiment;

[0021] Figure 9 A schematic structural diagram of a video processing system provided by an embodiment;

[0022] Figure 10 A schematic diagram of the hardware structure of a device provided in one embodiment. DETAILED DESCRIPTION

[0023] The present application is described below in conjunction with the accompanying drawings and embodiments. It will be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application. It should be noted that, unless there is a conflict, the embodiments and features within the embodiments of the present application may be combined with each other in any manner. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present application, not all structures.

[0024] In an embodiment of the present application, a video processing method is provided. The method only needs to encode and decode non-key frames to obtain low-resolution frames, and then amplify the low-resolution frames with reference to the information of the key frames, thereby obtaining amplified high-resolution video with less bit rate, reducing the bandwidth pressure of video transmission, and improving the subjective and objective quality of video amplification.

[0025] Figure 1 A flowchart of a video processing method provided by an embodiment is shown in FIG. Figure 1 As shown, the method provided in this embodiment includes step 110 and step 120.

[0026] In step 110, a mixed-resolution video is obtained by encoding and decoding the original video, wherein the mixed-resolution video includes a first resolution frame and a second resolution frame corresponding to the key frames in the original video, and also includes a third resolution frame corresponding to the non-key frames in the original video, wherein the resolution of the first resolution frame is higher than the resolution of the second resolution frame and the resolution of the third resolution frame.

[0027] In this embodiment, the original video refers to the high-resolution video to be processed or transmitted. The original video first undergoes encoding and decoding to obtain a mixed-resolution video. Compared to the original video, the resolution of some frames has been reduced, achieving video data compression and completing transmission with fewer resources. The original video includes key frames and non-key frames. Key frames include scene transition frames, frames selected from similar frames, and intra-coded frames (I frames) within a Group of Pictures (GOP). Key frames carry key information of the video frames and can be used to summarize the scene events of the video. All frames other than key frames are non-key frames.

[0028] The video codec can be used to process the key frames and non-key frames in the original video differently based on the Scaled Video Coding (SVC) technology, and the video can be divided into multiple resolutions to obtain a mixed-resolution video. SVC is a hybrid codec framework that combines the spatial redundancy within the video frame with the temporal redundancy between frames to construct different coding blocks to form the entire codec protocol. Specifically, for key frames, the video codec obtains a first resolution frame (high-resolution frame) and a second resolution frame (low-resolution frame); for non-key frames, the third resolution frame (low-resolution frame) is obtained by downsampling, where the second resolution is the same as the third resolution, thus obtaining a mixed-resolution video.

[0029] Figure 2 FIG. 1 is a schematic diagram of a mixed resolution video provided by an embodiment. Figure 2 As shown in the figure, larger boxes represent keyframes, and smaller boxes represent non-keyframes. The mixed-resolution video generated by the encoding and decoding process contains high-resolution frames for keyframes and low-resolution frames for non-keyframes. The high-resolution frames of keyframes provide a basis for upscaling the low-resolution frames of non-keyframes.

[0030] In step 120, the third resolution frame corresponding to the non-key frame in the mixed resolution video is amplified according to the first resolution frame and the second resolution frame corresponding to the key frame, and the amplified video is output, wherein the amplified video includes the first resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame.

[0031] In this embodiment, the codec can be used to obtain a mixed-resolution video based on a spatial hierarchical strategy (using different spatial resolutions for compression), which can reduce the number of bits in video transmission, reduce bandwidth pressure, and save resources; the mixed-resolution video can then be processed by video amplification to amplify the non-key frames with reduced resolution and regain high-resolution frames, thereby obtaining a complete high-resolution video and acquiring complete high-quality video information. In this embodiment, the third-resolution frame of the non-key frame is amplified based on the intra-frame reference mode of the bidirectional reference of the hybrid codec framework, that is, each non-key frame in the middle is amplified with reference to the information of the key frames before and after the non-key frame, and the high-resolution frame after the amplification of the third-resolution frame is the target frame.

[0032] The video processing method of this embodiment obtains mixed-resolution video based on SVC, wherein, for non-key frames, only low-resolution frames need to be encoded and decoded, and then the low-resolution frames are amplified with reference to the information of the key frames, so as to obtain amplified high-resolution video with less bit rate, reducing the bandwidth pressure of video transmission and improving the subjective and objective quality of video amplification.

[0033] Figure 3 This is a flowchart of a video processing method provided by another embodiment. In this embodiment, the encoding and decoding process of the original video and the video amplification process are described in detail. The video encoding and decoding process uses the residual map to implement SVC of the original video, and the video amplification process uses a trained neural network. The combination of the two ensures the efficiency of video transmission and the quality of amplification. Figure 3 As shown, the method provided in this embodiment includes steps 210-250.

[0034] In step 210, key frames in the original video are detected.

[0035] Before encoding and decoding the high-resolution original video, the key frame is first detected. In this embodiment, the first frame (I frame) in the GOP in the H265 protocol is used as the key frame. The key frame is encoded and decoded to obtain the first resolution frame, which has a higher resolution and a very small encoding and decoding loss. Therefore, it can be used as a reference frame in the video amplification process to provide a basis for video amplification.

[0036] In step 220, a residual image of the key frame and a second-resolution frame are generated, and the non-key frame is downsampled to obtain a third-resolution frame.

[0037] For key frames, it is necessary to obtain first-resolution frames and second-resolution frames after encoding and decoding. For the first-resolution frames, the key frames can be downsampled to second-resolution frames, and then upsampled and the upsampled frames can be subtracted from the original video frames to obtain a residual map. The residual map can clearly reflect the changed areas and non-changed areas between the downsampled and upsampled images (or the key frames after splitting and splicing) and the original video key frames, so that the pixel value change range is smaller than before, and a smaller change range can be used for quantization, thereby reducing the number of transmitted bits; for the second-resolution frames, the lower-resolution second-resolution frames corresponding to the key frames can be obtained through downsampling, image splitting and other methods; and for non-key frames, compression downsampling is directly performed to obtain lower-resolution third-resolution frames.

[0038] In step 230, video encoding and decoding is performed on the residual image, the second resolution frame, and the third resolution frame to obtain a mixed-resolution video.

[0039] The residual image, the second resolution frame, and the third resolution frame are encoded to obtain a transmittable bit stream, and then the bit stream is decoded to obtain a mixed-resolution video.

[0040] In step 240, for each non-key frame, the third resolution frame of the non-key frame (denoted as (I t LR), the third resolution frame before the non-key frame before the non-key frame is enlarged (denoted as I t-1 LR ) and the enlarged high-resolution frame (denoted as I t-1 SR ) and the first resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame (respectively represented as I k1 SR , I k2 SR ) and the corresponding second resolution frame (respectively represented as (I k1 LR , I k2 LR ) inputs the video amplification network.

[0041] In this embodiment, a video upscaling network (e.g., a trained recurrent neural network) can be used to upscale a non-key frame based on the intra-frame reference mode of the bidirectional reference in the hybrid codec framework, referring to the relationship and changes between the high-resolution frame and the low-resolution frame (first resolution frame and second resolution frame) of the key frame before and after each non-key frame, combined with the upscaling result of the previous non-key frame. In this case, when upscaling a non-key frame, the video upscaling network has a total of seven inputs:

[0042] The third resolution frame (I t LR );

[0043] The first resolution frame (I k1 SR ) and the corresponding second resolution frame (I k1 LR );

[0044] The first resolution frame (I k2 SR ) and the second resolution frame (I k2 LR );

[0045] The third resolution frame (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ).

[0046] In step 250, the video zoom network is used to zoom in on the third resolution frame (i.e., I t-1 LR ) and the enlarged high-resolution frame (i.e., It-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (i.e., I k1 SR , I k2 SR ) and the corresponding second resolution frame (ie I k1 LR , I k2 LR ), the third resolution frame of the non-key frame (i.e., I t LR ) to amplify the non-key frame to obtain the amplified target frame corresponding to the non-key frame.

[0047] The video magnification network of this embodiment can be a trained recurrent neural network. Its operating principle is as follows: by inputting the first-resolution frame (high-resolution frame) and second-resolution frame (low-resolution frame) corresponding to two adjacent key frames as reference frames for the third-resolution frame (low-resolution frame) of the entire intermediate non-key frame, and also inputting the third-resolution frame before magnification and the high-resolution frame after magnification of the previous non-key frame, the recurrent neural network can magnify the third-resolution frame of the current non-key frame based on the spatiotemporal relationship between each reference frame and the current frame. The current non-key frame references reference frames in two directions. In the event of a scene change, the information of the forward and backward low-resolution and high-resolution frames can be fully utilized to ensure the magnification effect of the current non-key frame.

[0048] In one embodiment, in step 220, generating the residual image of the key frame and the second resolution frame includes:

[0049] Step 221: down-sample the key frame to obtain a second resolution frame of the key frame.

[0050] In this embodiment, the key frame is downsampled to obtain a second resolution frame (low resolution frame) of the key frame. The downsampling may adopt a bilinear or bicubic method, and the decoding process adopts the same method to upsample the key frame.

[0051] Step 222: up-sample the second resolution frame of the key frame to obtain a temporary frame of the first resolution frame of the key frame, and subtract the temporary frame of the first resolution frame from the key frame to obtain a residual map of the key frame.

[0052] In this embodiment, the second resolution frame of the key frame is upsampled to obtain a temporary frame of the first resolution frame of the key frame (high resolution frame), and the temporary frame of the first resolution frame is subtracted from the real key frame (Ground Truth Frame) in the original video to obtain a residual map. The residual map can clearly reflect the change area and the non-change area between the first resolution frame and the key frame of the original video, reduce the processing amount of encoding and decoding, and save encoding and decoding and transmission resources. A residual code stream can be obtained by encoding the residual map. In the decoding process, the second resolution frame (low resolution frame) of the key frame is first obtained by decoding, and then the enlarged image (recorded as the fourth resolution frame) is obtained by the upsampling method. The residual map is obtained by decoding the residual code stream, and then the enlarged image is added to the residual map to obtain the first resolution frame of the key frame. In this embodiment, the encoding and decoding of the residual map needs to be performed at high resolution. When designing the chip, the clocks for decoding high-resolution frames and decoding low-resolution frames need to be set separately.

[0053] In one embodiment, step 230 specifically includes:

[0054] Step 231: Encode the residual image, the second resolution frame, and the third resolution frame through an encoder to obtain a residual bitstream, a bitstream of the second resolution frame, and a bitstream of the third resolution frame;

[0055] Step 232: Decode the residual bitstream, the bitstream of the second resolution frame, and the bitstream of the third resolution frame using a decoder to obtain the residual image, the second resolution frame, and the third resolution frame, respectively.

[0056] Step 233: up-sample the second resolution frame to obtain an amplified fourth resolution frame, and add the fourth resolution frame to the residual map to obtain the first resolution frame of the key frame.

[0057] In this embodiment, during the video encoding and decoding process, the residual image is first encoded to obtain a residual bitstream. The second-resolution and third-resolution frames are also encoded (compressed) and transmitted. During the decoding process, the second-resolution frame (low-resolution frame) of the key frame is first decoded. Then, an upsampling method is used to obtain an enlarged image (referred to as the fourth-resolution frame). The residual bitstream is decoded to obtain a residual image. This enlarged image is then added to the residual image to obtain the first-resolution frame of the key frame. In this embodiment, the encoding and decoding of the residual image must be performed at high resolution. Therefore, when designing the chip, the clocks for decoding the high-resolution and low-resolution frames must be set separately.

[0058] In one embodiment, step 110 specifically includes:

[0059] Step 111: Detect key frames in the original video;

[0060] Step 112: Divide the key frame into a plurality of sub-images, wherein the resolution of each sub-image is the same as the resolution of the second-resolution frame, and downsample the non-key frame to obtain a third-resolution frame;

[0061] Step 113: Input each of the sub-images and the non-key frame at the third resolution into a video codec, wherein the video codec includes an encoder and a decoder;

[0062] Step 114: Encode each of the sub-images and the third-resolution frame of the non-key frame by the encoder to obtain a code stream corresponding to each of the sub-images and the non-key frame;

[0063] Step 115: splicing the sub-images according to the bitstreams of the sub-images through the decoder to obtain the first resolution frame of the key frame, and decoding the bitstream of the third resolution frame to obtain the third resolution frame.

[0064] In this embodiment, downsampling and encoding / decoding are used for non-key frames to obtain third-resolution frames, while downsampling is used for key frames to obtain second-resolution frames. In addition, the key frames are split into multiple sub-images, which are then spliced ​​together to obtain first-resolution frames. That is, the key frames in the high-resolution original video are split into several sub-images of equal size, and the sub-images are then spliced ​​into a high-resolution image through the codec. The resulting image is the first-resolution frame of the key frame, which has a higher resolution than the resolution of the key frames in the original video. Both the splitting method and the upsampling method can be used to obtain first-resolution frames for key frames. In this embodiment, the splitting method is mainly suitable for video processing scenarios where timeliness requirements are not high. In cases where the super-resolution method requires a relatively large magnification, if the splitting method is used, more sub-images need to be split, resulting in a longer decoding process and relatively large delay.

[0065] In one embodiment, the video upscaling network includes a motion estimation (ME) sub-network, a motion compensation (MC) sub-network, and an upscaling sub-network.

[0066] In this embodiment, the video magnification network includes three sub-networks: a motion estimation sub-network, a motion compensation sub-network, and an amplification sub-network. During the amplification process, the estimation of motion vectors of low-resolution frames and motion compensation of high-resolution frames are also considered, which can obtain more complete information of the reference frame and improve the quality of video amplification.

[0067] In this embodiment, step 250 specifically includes:

[0068] Step 251: Based on the third resolution frame (I t LR ), through the motion estimation sub-network, respectively amplify the third resolution frame (I t-1 LR ) and the second resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 LR , I k2 LR ) for motion estimation.

[0069] In the inter-frame prediction process, since there is a certain correlation between the scenes in adjacent frames, the non-key frame can be divided into several blocks or macroblocks, and the position of each block or macroblock in the adjacent frame can be searched out, and the relative offset of the spatial position between the two can be obtained, that is, the motion vector. Through vector motion estimation, the inter-frame redundancy can be removed, which greatly reduces the number of bits of video transmission.

[0070] In this embodiment, for each non-key frame, the low-resolution frame I of the non-key frame is estimated. t LR The motion vectors between the other three low-resolution frames, i.e., the estimated I t LR with I t-1 LR , I t LR with I k1 LR , I t LR with I k2 LR The motion vector between them.

[0071] Step 252: Based on the high-resolution frame (I t-1 SR ) and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ), motion compensation is performed on each frame after motion estimation and the upsampled motion vector to obtain an intermediate high-resolution feature map, and the intermediate high-resolution feature map is converted into a low-resolution feature map through a recombination operation.

[0072] In this embodiment, for each non-keyframe, motion compensation is performed on the motion-estimated frames and the upsampled motion vectors to obtain an intermediate high-resolution feature map. This intermediate high-resolution feature map is then converted into a low-resolution feature map through a reassembly operation. Motion estimation provides a rough estimate of the motion vector, while motion compensation refines the estimate, further clarifying the offset between frames and providing more efficient input to the amplification subnetwork.

[0073] Step 253: amplify the third resolution frame according to the low-resolution feature map through the amplification subnetwork to obtain an amplified target frame corresponding to the non-key frame.

[0074] The video processing method of this embodiment uses the temporal and spatial relationship between frames to amplify the third resolution frame, so that the amplified video does not jitter and the visual effect is more coherent.

[0075] Figure 4 The schematic diagram of the principle of a video magnification network provided by an embodiment. In this embodiment, the video magnification network uses a convolutional recurrent neural network to magnify low-resolution frames in mixed-resolution videos. The network structure of the video magnification network is as follows: Figure 4 As shown, it includes three parts: motion estimation sub-network, motion compensation sub-network and amplification sub-network. The amplification sub-network can be a super resolution network (SRNet). There are three motion estimation sub-networks, which are used to estimate I t LR with I t-1 LR , I t LR with I k1 LR , I t LR with I k2 LR The motion vectors between the three motion estimation sub-networks are shared. LR Represents the low-resolution frame in the mixed-resolution video after encoding and decoding, I SR Represents a high-resolution frame in a mixed-resolution video, for example, I k1 LR and I k2 LR are low-resolution frames (second-resolution frames) corresponding to the previous key frame and the next key frame, respectively. k1 SR and I k2 SR is the high-resolution frame (first resolution frame) corresponding to the previous key frame and the next key frame, I t LRIndicates the low-resolution frame (third resolution frame) of the current non-key frame that needs to be enlarged, I t-1 LR and I t-1 SR They are the low-resolution frame of the previous non-key frame and the high-resolution frame after enlargement.

[0076] In one embodiment, the video zooming process may further include the following operations:

[0077] “↑” indicates upsampling operation;

[0078] Pixel shift (Warp) operation, which means moving the high-resolution pixel to the position of the current pixel plus the motion vector;

[0079] The Shuffle operation represents the reorganization operation of the spatial feature map into the channel feature map.

[0080] Figure 5 FIG. 1 is a schematic diagram of a motion estimation subnetwork provided by an embodiment. Figure 5 As shown, the motion estimation subnetwork may include a concatenation layer (Concat) for concatenating two or more feature maps on the channel; an activation layer, such as a leaky rectified linear unit (LRLU) excitation layer and a hyperbolic Tanh (HT) excitation layer; a maximum pooling layer (Max Pool) and a bilinear transformation layer.

[0081] Figure 6 FIG. 1 is a schematic diagram of a motion compensation subnetwork provided by an embodiment. Figure 6 As shown in the figure, the motion estimation subnetwork includes a cascade layer (Concat), which is used to splice two or more feature maps on the channel; a convolution layer (Conv), where the number of channels of the convolution kernel can be 64, 128, 256, etc.; a residual block (Residual Block); a downsampling layer (denoted as "↓") and an upsampling layer (denoted as "↑").

[0082] Figure 7 FIG. 1 is a schematic diagram of an amplifying sub-network provided in an embodiment. Figure 7 As shown in the figure, the motion estimation subnetwork includes a convolutional layer (Conv), an excitation layer (ReLU), and a deconvolution (Conv Transpose) layer.

[0083] In one embodiment, step 252 specifically includes:

[0084] Step 2521: Calculate the motion vector MV corresponding to each frame after motion estimation. t-1LR , MV k1 LR and MV k2 LR Upsample to get MV t-1 SR , MV k1 SR and MV k2 SR ; Among them, MV t-1 LR , MV k1 LR and MV k2 LR They are the third resolution frame (I t-1 LR ), the second resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 LR , I k2 LR ) are respectively associated with the amplified motion vectors of the third resolution frame of the non-key frame after motion estimation;

[0085] Step 2522: Based on MV t-1 SR 、MV k1 SR and MV k2 SR , through the pixel shift operation, the high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2 SR ) to perform pixel shifting, and obtain the pixel shifted I t-1->t SR , I k1->t SR and I k2->t SR ;

[0086] Step 2523: I t-1->t SR , I k1->t SR , I k2->t SR 、MV t-1 SR , MV k1 SR and MV k2 SRThe compensation sub-network is input, and the intermediate high-resolution feature map is obtained after motion compensation of the compensation sub-network.

[0087] like Figure 4 As shown, through the motion estimation sub-network, based on I t LR , respectively for three low-resolution frames other than the low-resolution frame of the current non-key frame (I k1 LR , I k2 LR , I t-1 LR ) respectively perform motion estimation and obtain the corresponding motion vector MV t-1 LR , MV k1 LR and MV k2 LR ; For each motion vector MV estimated by motion t-1 LR , MV k1 LR and MV k2 LR Upsample separately to get the enlarged MV t-1 SR , MV k1 SR and MV k2 SR ; Then use MV t-1 SR , MV k1 SR and MV k2 SR , through pixel shift operation, I t-1 SR , I k1 SR , I k2 SR Perform pixel shift to obtain the pixel-shifted I t-1->t SR , I k1->t SR and I k2->t SR ; will I t-1->t SR , I k1->t SR , I k2->t SR , MV t-1 SR , MV k1 SR and MV k2 SRThe compensation subnetwork is input and motion-compensated by the compensation subnetwork to obtain the intermediate high-resolution feature map. Based on this, the amplification subnetwork amplifies the third-resolution frames of non-keyframes according to the intermediate high-resolution feature map to obtain the amplified target frames corresponding to the non-keyframes. This process is repeated for each non-keyframe between two adjacent keyframes to obtain the amplified target frame. Ultimately, the third-resolution frames of all non-keyframes are amplified to obtain high-quality super-resolution video.

[0088] In one embodiment, if the non-key frame is the first non-key frame after the adjacent previous key frame, the second resolution frame of the previous key frame is used as the third resolution frame before the previous non-key frame of the non-key frame is enlarged, and the first resolution frame of the previous key frame is used as the high resolution frame after the previous non-key frame of the non-key frame is enlarged.

[0089] In this embodiment, if the non-key frame to be enlarged is the first non-key frame after the adjacent previous key frame, that is, there is no other non-key frame between the non-key frame and the adjacent previous key frame, in this case, there is no low-resolution frame (I2) of the previous non-key frame that can be used as a reference frame within the interval between the two key frames where the non-key frame is located. t-1 LR ), nor does there exist a corresponding enlarged high-resolution frame (I t-1 SR ), the low-resolution frame (I t-1 LR ) is replaced by the low-resolution frame of the previous key frame (the second resolution frame, I k1 LR ), and the high-resolution frame of the previous non-key frame (I t-1 SR ) is replaced by the high-resolution frame of the previous key frame (the first resolution frame, I k1 SR ), so that in the absence of a previous non-key frame, the previous key frame is used as a reference frame to provide a complete basis for the magnification algorithm and ensure the feasibility of the algorithm.

[0090] In one embodiment, before upscaling the third resolution frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame, the method further includes:

[0091] Step 100: Based on the sample video, the video magnification network is trained according to the first loss function and the second loss function; wherein the first loss function is determined according to the mean square error between the original high-resolution frame of the non-key frame between two adjacent key frames in the original video and the original high-resolution frame after downsampling and then magnified by the video magnification network; the second loss function is determined according to the I loss function between the original high-resolution frame of the non-key frame, the third-resolution frame of the non-key frame and the third-resolution frame of the previous non-key frame after motion estimation, motion vector upsampling, pixel shifting, etc. t-1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent previous key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent next key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k2->t SR The mean square error is determined.

[0092] The video upscaling network is pre-trained and has good upscaling capabilities. In this embodiment, high-resolution videos of different resolutions can be obtained from a database. Mixed-resolution videos obtained by encoding and decoding each high-resolution video are used as sample videos for training the video upscaling network. The sample videos should contain sufficient content and features.

[0093] In this embodiment, the video magnification network is trained according to the first loss function and the second loss function, wherein the second loss function is mainly used to train a subnetwork (motion estimation subnetwork) in the video magnification network. The motion estimation subnetwork can be trained separately and the second loss function can be constructed in a targeted manner.

[0094] The first loss function is, for example: Among them, G represents the sum of two adjacent key frames and the middle non-key frame. It can be seen that the first loss function is independent of the two key frames.

[0095] The second loss function is, for example:

[0096] Loss ME =MSE(I t HR ,W b (I k1 SR ,f t→k1 ))+MSE(I t HR ,W b (I k1 SR ,f t→k2))+MSE(I t HR ,W b (I t-1 SR ,f t→t-1 )), where I t HR Indicates the original high-resolution frame of the non-key frame to be enlarged, I t-1 SR represents the high-resolution frame after the previous non-key frame is enlarged, f t→t-1 Represents the motion vector between the non-key frame and the previous non-key frame, f t→k1 Represents the motion vector between the non-key frame and the previous key frame, f t→k2 Represents the motion vector between the non-key frame and the next key frame, W b (Backward Warp) represents a reverse pixel warp operation. The principle of the video upscaling network upscaling mixed-resolution videos can be found in the above embodiment. When the first loss function and the second loss function are both stable near a certain value, the video upscaling network training is complete. At this point, the video upscaling network has reliable upscaling capabilities and can be applied to upscaling low-resolution frames in mixed-resolution videos corresponding to other videos.

[0097] In this embodiment, the sample videos selected are authentic, valid, and diverse. The training data for the amplification network generated by inputting the sample videos into the video codec is also authentic and reliable, rather than artificially generated. By treating the video codec as a whole, the training data for the video amplification network is obtained from the decoding end, which is authentic and valid. The trained video amplification network is also more targeted, thus ensuring effective training, improving the amplification capability of the video amplification network, and enhancing the quality of the amplified videos.

[0098] In one embodiment, the video upscaling network includes a motion estimation subnetwork, a motion compensation subnetwork, and an upscaling subnetwork; step 100 includes: training the video upscaling network according to the first loss function, and training the motion estimation subnetwork according to the second loss function until the first loss function and the second loss function meet a set condition. The set condition means that the first loss function and the second loss function are respectively stable near a certain value, for example, the fluctuation range of the values ​​of the first loss function and the second loss function is less than a certain threshold.

[0099] In one embodiment, before training the video upscaling network according to the first loss function and the second loss function, the method further includes: Step 101: preprocessing the sample video to obtain a sample video of a preset resolution.

[0100] In some embodiments, part of the content of the acquired high-resolution video can be cut out to make it conform to a preset resolution (for example, 256*256), and then input into a video codec, and a mixed-resolution code stream is obtained after encoding, and a mixed-resolution frame is obtained after decoding.

[0101] The video processing method of this embodiment utilizes residual maps for encoding and decoding, requiring only the changed regions of key frames to be processed, thus reducing the encoding and decoding bitrate of key frames. During video magnification, bidirectional key frames are used as reference frames. When the scene changes, information from both forward and backward low-resolution and high-resolution frames can be fully utilized to ensure the magnification of the current non-key frame. Downsampling is performed on key frames to obtain second-resolution frames, reducing latency and improving timeliness. During the magnification process, motion vector estimation and motion compensation of low-resolution frames are also considered, enabling more comprehensive information from reference frames to be obtained, thereby improving the quality of video magnification. Vector motion estimation can remove inter-frame redundancy, significantly reducing the number of bits required for video transmission. Furthermore, the training data for the video magnification network is obtained from the decoding end, ensuring authenticity and effectiveness. The trained video magnification network is also more targeted, thus ensuring effective training, improving the magnification capability of the video magnification network, and enhancing the quality of the magnified video.

[0102] An embodiment of the present application also provides a video processing device. Figure 8 FIG. 1 is a structural diagram of a video processing device provided by an embodiment. Figure 8 As shown, the video processing device includes: a coding module 310 and an amplification module 320.

[0103] The encoding and decoding module 310 is configured to obtain a mixed-resolution video by encoding and decoding the original video, wherein the mixed-resolution video includes a first-resolution frame and a second-resolution frame corresponding to a key frame in the original video, and also includes a third-resolution frame corresponding to a non-key frame in the original video, wherein the resolution of the first-resolution frame is higher than the resolution of the second-resolution frame and the resolution of the third-resolution frame;

[0104] The magnification module 320 is configured to magnify the third resolution frame corresponding to the non-key frame in the mixed resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame, and output the magnified video, wherein the magnified video includes the first resolution frame corresponding to the key frame and the magnified target frame corresponding to the non-key frame.

[0105] The video processing device of this embodiment obtains a mixed-resolution video by encoding and decoding the original video; amplifies the third-resolution frame corresponding to the non-key frame in the mixed-resolution video according to the first-resolution frame and the second-resolution frame corresponding to the key frame. For the non-key frame, only low-resolution frames need to be encoded and decoded, and then the low-resolution frame is amplified according to the key frame to obtain an amplified high-resolution video with a smaller bit rate, thereby reducing the bandwidth pressure of video transmission and improving the subjective and objective quality of video amplification.

[0106] In one embodiment, the encoding and decoding module 310 includes:

[0107] A detection unit, configured to detect key frames in the original video;

[0108] a frame processing unit configured to generate a residual map of the key frame and a second resolution frame, and downsample the non-key frame to obtain a third resolution frame;

[0109] The encoding and decoding unit is configured to perform video encoding and decoding on the residual image, the second resolution frame, and the third resolution frame to obtain a mixed-resolution video.

[0110] In one embodiment, generating the residual map of the key frame and the second resolution frame includes:

[0111] Downsampling the key frame to obtain a second resolution frame of the key frame;

[0112] The key frame is upsampled to obtain a temporary frame of the key frame at a first resolution, and the temporary frame of the key frame at the first resolution is subtracted from the key frame to obtain a residual map of the key frame.

[0113] In one embodiment, the encoding and decoding unit is specifically configured to:

[0114] Obtaining an encoder to encode the residual image, the second resolution frame, and the third resolution frame to obtain a residual bitstream, a bitstream of the second resolution frame, and a bitstream of the third resolution frame;

[0115] Decoding the residual bitstream, the bitstream of the second resolution frame, and the bitstream of the third resolution frame by the decoder to obtain the residual image, the second resolution frame, and the third resolution frame, respectively;

[0116] The second-resolution frame is up-sampled to obtain an amplified fourth-resolution frame, and the fourth-resolution frame is added to the residual map to obtain the first-resolution frame of the key frame.

[0117] In one embodiment, the encoding and decoding module 310 includes:

[0118] A detection unit, configured to detect key frames in the original video;

[0119] a frame processing unit configured to equally divide the key frame into a plurality of sub-images, wherein the resolution of each sub-image is the same as the resolution of the second-resolution frame, and downsample the non-key frame to obtain a third-resolution frame;

[0120] an encoding unit configured to input each of the sub-images and the third-resolution frame of the non-key frame into a video codec, the video codec comprising an encoder and a decoder; and encode each of the sub-images and the third-resolution frame of the non-key frame by the encoder to obtain a code stream corresponding to each of the sub-images and the non-key frame;

[0121] The decoding unit is configured to splice the sub-images according to the code streams of the sub-images through the decoder to obtain the first resolution frame of the key frame, and decode the code stream of the third resolution frame to obtain the third resolution frame.

[0122] In one embodiment, the amplification module 320 includes:

[0123] The input unit is configured to, for each non-key frame, respectively convert the third resolution frame (I t LR ), the third resolution frame before the non-key frame before the non-key frame is enlarged (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ), and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ) and the corresponding second resolution frame (I k1 LR , I k2 LR ) Input video upscaling network;

[0124] The amplification unit is configured to amplify the third resolution frame (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2 SR ) and the corresponding second resolution frame (Ik1 LR , I k2 LR ), the third resolution frame (I t LR ) to amplify the non-key frame to obtain the amplified target frame corresponding to the non-key frame.

[0125] In one embodiment, the video upscaling network includes a motion estimation subnetwork, a motion compensation subnetwork, and an upscaling subnetwork;

[0126] Amplification unit, specifically set to:

[0127] The third resolution frame (I t LR ), through the motion estimation sub-network, respectively amplify the third resolution frame (I t-1 LR ) and the second resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 LR , I k2 LR ) perform motion estimation;

[0128] According to the high-resolution frame (I t-1 SR ) and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ), performing motion compensation on each frame after motion estimation and the upsampled motion vector to obtain an intermediate high-resolution feature map, and converting the intermediate high-resolution feature map into a low-resolution feature map through a recombination operation;

[0129] The third resolution frame is amplified by the amplification subnetwork according to the low resolution feature map to obtain an amplified target frame corresponding to the non-key frame.

[0130] In one embodiment, the high-resolution frame (I t-1 SR ) and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ), motion compensation is performed on each frame after motion estimation and the upsampled motion vector to obtain an intermediate high-resolution feature map, including:

[0131] The motion vector MV corresponding to each frame after motion estimation is t-1 LR , MV k1 LR and MV k2 LR Upsample to get MV t-1 SR , MV k1 SR and MV k2 SR ; Among them, MV t-1 LR , MV k1 LR and MV k2 LR They are the third resolution frame (I t-1 LR ), a second resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 LR , I k2 LR ) are respectively associated with the amplified motion vectors of the third resolution frame of the non-key frame after motion estimation;

[0132] Based on MV t-1 SR , MV k1 SR and MV k2 SR , through the pixel shift operation, the high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2 SR ) to perform pixel shifting, and obtain the pixel shifted I t-1->t SR , I k1->t SR and I k2->t SR ;

[0133] Will I t-1->t SR , I k1->t SR , I k2->t SR , MV t-1 SR , MV k1 SR and MV k2SR The compensation sub-network is input, and the intermediate high-resolution feature map is obtained after motion compensation of the compensation sub-network.

[0134] In one embodiment, if the non-key frame is the first non-key frame after the adjacent previous key frame, the second resolution frame of the previous key frame is used as the third resolution frame before the previous non-key frame of the non-key frame is enlarged, and the first resolution frame of the previous key frame is used as the high resolution frame after the previous non-key frame of the non-key frame is enlarged.

[0135] In one embodiment, it further includes:

[0136] a training module configured to train the video upscaling network based on a sample video according to a first loss function and a second loss function before upscaling the third resolution frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame;

[0137] The first loss function is determined based on the mean square error between an original high-resolution frame of a non-key frame between two adjacent key frames in the original video and the original high-resolution frame that is downsampled and then amplified by a video upscaling network;

[0138] The second loss function is based on the original high-resolution frame of the non-key frame, the third-resolution frame of the non-key frame and the third-resolution frame of the previous non-key frame after motion estimation, motion vector upsampling, pixel shifting, and I t-1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent previous key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent next key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k2->t SR The mean square error is determined.

[0139] In one embodiment, the video upscaling network includes a motion estimation subnetwork, a motion compensation subnetwork, and an upscaling subnetwork;

[0140] The step of training the video magnification network according to the first loss function and the second loss function includes:

[0141] The video magnification network is trained according to the first loss function, and the motion estimation subnetwork is trained according to the second loss function until the first loss function and the second loss function meet a set condition.

[0142] In one embodiment, it further includes:

[0143] The preprocessing module is configured to preprocess the sample video before training the video upscaling network according to the first loss function and the second loss function to obtain a sample video of a preset resolution.

[0144] The video processing device proposed in this embodiment and the video processing method proposed in the above embodiment belong to the same inventive concept. Technical details not fully described in this embodiment can be referred to any of the above embodiments, and this embodiment has the same beneficial effects as executing the video processing method.

[0145] An embodiment of the present application also provides a video processing system. Figure 9 FIG. 1 is a structural diagram of a video processing system provided by an embodiment. Figure 9 As shown, the system includes: a video codec 410, configured to process an original video to obtain a mixed-resolution video, wherein the mixed-resolution video includes a first-resolution frame and a second-resolution frame corresponding to a key frame in the original video, and further includes a third-resolution frame corresponding to a non-key frame in the original video, wherein the resolution of the first-resolution frame is higher than the resolution of the second-resolution frame and the resolution of the third-resolution frame;

[0146] The video magnification network 420 is configured to magnify the third resolution frame corresponding to the non-key frame in the mixed-resolution video based on the first resolution frame and the second resolution frame corresponding to the key frame, and output the magnified video, wherein the magnified video includes the first resolution frame corresponding to the key frame and the magnified target frame corresponding to the non-key frame.

[0147] In one embodiment, the video codec 410 is specifically configured to:

[0148] Detecting key frames in the original video;

[0149] generating a residual image of the key frame and a second-resolution frame, and downsampling the non-key frame to obtain a third-resolution frame;

[0150] After video encoding and decoding is performed on the residual image, the second resolution frame, and the third resolution frame, a mixed resolution video is obtained.

[0151] In one embodiment, generating the residual map of the key frame and the second resolution frame includes:

[0152] Downsampling the key frame to obtain a second resolution frame of the key frame;

[0153] The key frame is upsampled to obtain a temporary frame of the key frame at a first resolution, and the temporary frame of the key frame at the first resolution is subtracted from the key frame to obtain a residual map of the key frame.

[0154] In one embodiment, after performing video encoding and decoding on the residual image, the second resolution frame, and the third resolution frame, a mixed-resolution video is obtained, including:

[0155] Encoding the residual image, the second resolution frame, and the third resolution frame by the encoder to obtain a residual bitstream, a bitstream of the second resolution frame, and a bitstream of the third resolution frame;

[0156] Decoding the residual bitstream, the bitstream of the second resolution frame, and the bitstream of the third resolution frame by the decoder to obtain the residual image, the second resolution frame, and the third resolution frame, respectively;

[0157] The second-resolution frame is up-sampled to obtain an amplified fourth-resolution frame, and the fourth-resolution frame is added to the residual map to obtain the first-resolution frame of the key frame.

[0158] In one embodiment, the video amplification network 420 is specifically configured to:

[0159] For each non-key frame, the third resolution frame (I t LR ), the third resolution frame before the non-key frame before the non-key frame is enlarged (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ), and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ) and the corresponding second resolution frame (I k1 LR , I k2 LR ) Input video upscaling network;

[0160] The amplification unit is configured to amplify the third resolution frame (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2 SR ) and the corresponding second resolution frame (I k1LR , I k2 LR ), the third resolution frame (I t LR ) to amplify the non-key frame to obtain the amplified target frame corresponding to the non-key frame.

[0161] In one embodiment, the video upscaling network includes a motion estimation subnetwork, a motion compensation subnetwork, and an upscaling subnetwork;

[0162] The third resolution frame (I t-1 LR ) and the enlarged high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2 SR ) and the corresponding second resolution frame (I k1 LR , I k2 LR ), the third resolution frame (I t LR ) is amplified to obtain an amplified target frame corresponding to the non-key frame, including a third resolution frame (I t LR ), through the motion estimation sub-network, respectively amplify the third resolution frame (I t-1 LR ) and the second resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 LR , I k2 LR ) perform motion estimation;

[0163] According to the high-resolution frame (I t-1 SR ) and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ), performing motion compensation on each frame after motion estimation and the upsampled motion vector to obtain an intermediate high-resolution feature map, and converting the intermediate high-resolution feature map into a low-resolution feature map through a recombination operation;

[0164] The third resolution frame is amplified by the amplification subnetwork according to the low resolution feature map to obtain an amplified target frame corresponding to the non-key frame.

[0165] In one embodiment, the high-resolution frame (I t-1 SR ) and the first resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 SR , I k2 SR ), motion compensation is performed on each frame after motion estimation and the upsampled motion vector to obtain an intermediate high-resolution feature map, including:

[0166] The motion vector MV corresponding to each frame after motion estimation is t-1 LR , MV k1 LR and MV k2 LR Upsample to get MV t-1 SR , MV k1 SR and MV k2 SR ; Among them, MV t-1 LR , MV k1 LR and MV k2 LR They are the third resolution frame (I t-1 LR ) and the second resolution frame (I corresponding to the previous key frame and the next key frame adjacent to the non-key frame k1 LR , I k2 LR ) are respectively associated with the amplified motion vectors of the third resolution frame of the non-key frame after motion estimation;

[0167] Based on MV t-1 SR , MV k1 SR and MV k2 SR , through the pixel offset Warp operation, the high-resolution frame (I t-1 SR ), the first resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame (I k1 SR , I k2SR ) to perform pixel shifting, and obtain the pixel shifted I t-1->t SR , I k1->t SR and I k2->t SR ;

[0168] Will I t-1->t SR , I k1->t SR , I k2->t SR , MV t-1 SR , MV k1 SR and MV k2 SR The compensation sub-network is input, and the intermediate high-resolution feature map is obtained after motion compensation of the compensation sub-network.

[0169] In one embodiment, if the non-key frame is the first non-key frame after the adjacent previous key frame, the second resolution frame of the previous key frame is used as the third resolution frame before the previous non-key frame of the non-key frame is enlarged, and the first resolution frame of the previous key frame is used as the high resolution frame after the previous non-key frame of the non-key frame is enlarged.

[0170] In one embodiment, before upscaling the third resolution frame in the mixed-resolution video based on the first resolution frame and the second resolution frame corresponding to the key frame, the video upscaling network is trained based on a sample video according to a first loss function and a second loss function;

[0171] The first loss function is determined based on the mean square error between an original high-resolution frame of a non-key frame between two adjacent key frames in the original video and the original high-resolution frame that is downsampled and then amplified by a video upscaling network;

[0172] The second loss function is based on the original high-resolution frame of the non-key frame, the third-resolution frame of the non-key frame and the third-resolution frame of the previous non-key frame after motion estimation, motion vector upsampling, pixel shifting, and I t-1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent previous key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent next key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k2->t SR The mean square error is determined.

[0173] In one embodiment, the video upscaling network 420 includes a motion estimation sub-network, a motion compensation sub-network, and an upscaling sub-network;

[0174] The step of training the video magnification network according to the first loss function and the second loss function includes:

[0175] The video magnification network is trained according to the first loss function, and the motion estimation subnetwork is trained according to the second loss function until the first loss function and the second loss function meet a set condition.

[0176] In one embodiment, before training the video upscaling network according to the first loss function and the second loss function, the sample video is preprocessed to obtain a sample video of a preset resolution.

[0177] The video processing system in this embodiment and the video processing method proposed in the above embodiment belong to the same inventive concept. Technical details not fully described in this embodiment can be referred to any of the above embodiments, and this embodiment has the same beneficial effects as executing the video processing method.

[0178] The embodiment of the present application further provides a device. The video processing method can be executed by a video processing device, which can be implemented in software and / or hardware and integrated into the device.

[0179] Figure 10 This is a schematic diagram of the hardware structure of a device provided in one embodiment. Figure 10 As shown, a device provided in this embodiment includes: a processor 510 and a storage device 520. The processor in the device may be one or more, Figure 10 Taking a processor 510 as an example, the processor 510 and the storage device 520 in the device can be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0180] The one or more programs are executed by the one or more processors 510, so that the one or more processors implement the video processing method described in any of the above embodiments. The method includes: obtaining a mixed-resolution video by encoding and decoding the original video, the mixed-resolution video including a first resolution frame and a second resolution frame corresponding to the key frame in the original video, and also including a third resolution frame corresponding to the non-key frame in the original video, wherein the resolution of the first resolution frame is higher than the resolution of the second resolution frame and the resolution of the third resolution frame; according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed-resolution video is amplified, and the amplified video is output, wherein the amplified video includes the first resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame.

[0181] The storage device 520 in the device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the video processing method in the embodiment of the present invention (for example, the attached Figure 8 The modules in the video processing device shown include: a codec module 310 and an amplification module 320). The processor 510 executes the software programs, instructions, and modules stored in the storage device 520 to perform various functional applications and data processing of the device, that is, to implement the video processing method in the above method embodiment.

[0182] The storage device 520 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and applications required for at least one function; the data storage area can store data created according to the use of the device, etc. (such as the mixed resolution video, third resolution frame, etc. in the above embodiment). In addition, the storage device 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 520 may further include a memory remotely located relative to the processor 510, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] Furthermore, when one or more programs included in the above-mentioned device are executed by the one or more processors 510, the following operations are implemented: a mixed-resolution video is obtained by encoding and decoding the original video, wherein the mixed-resolution video includes a first resolution frame and a second resolution frame corresponding to the key frame in the original video, and also includes a third resolution frame corresponding to the non-key frame in the original video, wherein the resolution of the first resolution frame is higher than the resolution of the second resolution frame and the resolution of the third resolution frame; according to the first resolution frame and the second resolution frame corresponding to the key frame, the third resolution frame corresponding to the non-key frame in the mixed-resolution video is amplified, and the amplified video is output, wherein the amplified video includes the first resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame.

[0184] The device proposed in this embodiment and the video processing method proposed in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to any of the above embodiments, and this embodiment has the same beneficial effects as executing the video processing method.

[0185] An embodiment of the present application also provides a storage medium containing computer-executable instructions, which are used to perform a video processing method when executed by a computer processor. The method includes: encoding and decoding an original video to obtain a mixed-resolution video, wherein the mixed-resolution video includes a first-resolution frame and a second-resolution frame corresponding to a key frame in the original video, and also includes a third-resolution frame corresponding to a non-key frame in the original video, wherein the resolution of the first-resolution frame is higher than the resolution of the second-resolution frame and the resolution of the third-resolution frame; based on the first-resolution frame and the second-resolution frame corresponding to the key frame, the third-resolution frame corresponding to the non-key frame in the mixed-resolution video is amplified, and the amplified video is output, wherein the amplified video includes the first-resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame.

[0186] Through the above description of the implementation methods, those skilled in the art can understand that the present application can be implemented with the help of software and general hardware, or can be implemented by hardware. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the method described in any embodiment of the present application.

[0187] The above description is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application.

[0188] The block diagram of any logical flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logical circuits, modules and functions, or can represent a combination of program steps and logical circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory device and system (digital versatile disc DVD or CD optical disc) etc. Computer-readable media can include non-transient storage media. The data processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FPGA) and a processor based on a multi-core processor architecture.

[0189] The above description of exemplary embodiments of the present application has been provided by way of exemplary and non-limiting examples. However, various modifications and adaptations to the above embodiments will be apparent to those skilled in the art, when considered in conjunction with the accompanying drawings and claims, without departing from the scope of the present invention. Therefore, the proper scope of the present invention will be determined by reference to the claims.

Claims

1. A video processing method, characterized in that: include: A mixed-resolution video is obtained by encoding and decoding an original video, wherein the mixed-resolution video includes a first-resolution frame and a second-resolution frame corresponding to a key frame in the original video, and also includes a third-resolution frame corresponding to a non-key frame in the original video, wherein a resolution of the first-resolution frame is higher than a resolution of the second-resolution frame and a resolution of the third-resolution frame; amplifying a third resolution frame corresponding to a non-key frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame, and outputting an amplified video, wherein the amplified video includes the first resolution frame corresponding to the key frame and the amplified target frame corresponding to the non-key frame; The step of amplifying the third resolution frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame includes: For each non-key frame, the third resolution frame of the non-key frame, the third resolution frame of the previous non-key frame before being amplified and the high resolution frame after being amplified, and the first resolution frame and the second resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame are respectively input into the video amplification network; The video magnification network amplifies the third resolution frame of the non-key frame according to the third resolution frame before amplification and the high resolution frame after amplification of the previous non-key frame of the non-key frame, and the first resolution frame and the corresponding second resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame, to obtain an amplified target frame corresponding to the non-key frame.

2. The method according to claim 1, characterized in that The method of encoding and decoding the original video to obtain a mixed-resolution video includes: Detecting key frames in the original video; generating a residual image of the key frame and a second-resolution frame, and downsampling the non-key frame to obtain a third-resolution frame; After video encoding and decoding is performed on the residual image, the second resolution frame, and the third resolution frame, a mixed resolution video is obtained.

3. The method according to claim 2, characterized in that Generating a residual map of the key frame and a second resolution frame, comprising: Downsampling the key frame to obtain a second resolution frame of the key frame; The key frame is upsampled to obtain a temporary frame of the key frame at a first resolution, and the temporary frame of the key frame at the first resolution is subtracted from the key frame to obtain a residual map of the key frame.

4. The method according to claim 2, characterized in that Performing video encoding and decoding on the residual image, the second resolution frame, and the third resolution frame to obtain a mixed-resolution video, including: Encoding the residual image, the second resolution frame, and the third resolution frame through an encoder to obtain a residual bitstream, a bitstream of the second resolution frame, and a bitstream of the third resolution frame; Decoding the residual bitstream, the bitstream of the second resolution frame, and the bitstream of the third resolution frame by a decoder to obtain the residual image, the second resolution frame, and the third resolution frame, respectively; The second-resolution frame is up-sampled to obtain an amplified fourth-resolution frame, and the fourth-resolution frame is added to the residual map to obtain the first-resolution frame of the key frame.

5. The method according to claim 1, wherein The method of encoding and decoding the original video to obtain a mixed-resolution video includes: Detecting key frames in the original video; Divide the key frame into a plurality of sub-images, each of which has the same resolution as the second-resolution frame, and downsample the non-key frame to obtain a third-resolution frame; Inputting each of the sub-images and the third resolution frame of the non-key frame into a video codec, wherein the video codec includes an encoder and a decoder; Encoding each of the sub-images and the third resolution frame of the non-key frame by the encoder to obtain a code stream corresponding to each of the sub-images and the non-key frame; The decoder splices the sub-images according to the bit streams of the sub-images to obtain the first resolution frame of the key frame, and decodes the bit stream of the third resolution frame to obtain the third resolution frame.

6. The method according to claim 1, characterized in that The video magnification network includes a motion estimation subnetwork, a motion compensation subnetwork and an amplification subnetwork; The video upscaling network upscaling the third resolution frame of the non-key frame according to the third resolution frame before upscaling and the high resolution frame after upscaling of a previous non-key frame of the non-key frame, and the first resolution frame and the corresponding second resolution frame corresponding to the previous key frame and the next key frame adjacent to the non-key frame, to obtain an upscaling target frame corresponding to the non-key frame, including: Based on the third-resolution frame of the non-key frame, performing motion estimation on the third-resolution frame of the non-key frame before being amplified, and the second-resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, respectively, through the motion estimation subnetwork; Performing motion compensation on each frame after motion estimation and the upsampled motion vector based on the high-resolution frame after the previous non-key frame is enlarged and the first-resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, respectively, to obtain an intermediate high-resolution feature map, and converting the intermediate high-resolution feature map into a low-resolution feature map through a recombination operation; The third resolution frame is amplified by the amplification subnetwork according to the low resolution feature map to obtain an amplified target frame corresponding to the non-key frame.

7. The method according to claim 6, characterized in that Perform motion compensation on each frame after motion estimation and the upsampled motion vectors based on the high-resolution frame after the previous non-key frame is enlarged and the first-resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, respectively, to obtain an intermediate high-resolution feature map, including: The motion vector MV corresponding to each frame after motion estimation is t-1 LR , MV k1 LR and MV k2 LR Upsample to get MV t-1 SR , MV k1 SR and MV k2 SR , among which, MV t-1 LR , MV k1 LR and MV k2 LR The amplified motion vectors corresponding to the third resolution frame before the previous non-key frame is amplified, the second resolution frames corresponding to the previous key frame and the next key frame adjacent to the non-key frame, and the third resolution frame of the non-key frame after motion estimation are respectively obtained; Based on MV t-1 SR , MV k1 SR and MV k2 SR By pixel shifting operation, pixel shifting is performed on the high-resolution frame after the previous non-key frame is magnified, the previous key frame adjacent to the non-key frame, and the first resolution frame corresponding to the next key frame, respectively, to obtain the pixel-shifted I t-1->t SR , I k1->t SR and I k2->t SR ; Will I t-1->t SR , I k1->t SR , I k2->t SR , MV t-1 SR , MV k1 SR and MV k2 SR The compensation sub-network is input, and the intermediate high-resolution feature map is obtained after motion compensation of the compensation sub-network.

8. The method according to claim 1, characterized in that If the non-key frame is the first non-key frame after the adjacent previous key frame, the second resolution frame of the previous key frame is used as the third resolution frame before the previous non-key frame of the non-key frame is enlarged, and the first resolution frame of the previous key frame is used as the high-resolution frame after the previous non-key frame of the non-key frame is enlarged.

9. The method according to claim 7, characterized in that Before amplifying the third resolution frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame, the method further includes: Based on the sample video, training a video upscaling network according to a first loss function and a second loss function, wherein the video upscaling network is used to upscale the third resolution frame in the mixed-resolution video according to the first resolution frame and the second resolution frame corresponding to the key frame; The first loss function is determined based on the mean square error between an original high-resolution frame of a non-key frame between two adjacent key frames in the original video and a frame of the original high-resolution frame that is downsampled and then amplified by a video upscaling network; The second loss function is based on the original high-resolution frame of the non-key frame, the third-resolution frame of the non-key frame and the third-resolution frame of the previous non-key frame after motion estimation, motion vector upsampling, pixel shifting, and I t-1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent previous key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k1->t SR The third resolution frame of the non-key frame and the second resolution frame of the adjacent next key frame are subjected to motion estimation, motion vector upsampling, and pixel shifting. k2->t SR The mean square error is determined.

10. The method according to claim 9, characterized in that The video magnification network includes a motion estimation subnetwork, a motion compensation subnetwork and an amplification subnetwork; The step of training the video magnification network according to the first loss function and the second loss function includes: The video magnification network is trained according to the first loss function, and the motion estimation subnetwork is trained according to the second loss function until the first loss function and the second loss function meet a set condition.

11. The method according to claim 9, characterized in that Before training the video upscaling network according to the first loss function and the second loss function, the method further includes: The sample video is preprocessed to obtain a sample video with a preset resolution.

12. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video processing method according to any one of claims 1 to 11 is implemented.

Images