Video processing methods, apparatus, storage media and electronic devices

Abstract

This application discloses a video processing method, apparatus, storage medium, and electronic device, belonging to the field of image processing technology. The method includes: acquiring an original video; downsampling each frame of the original video to obtain a low-resolution image; sparsely sampling a preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at preset intervals; and encoding the low-resolution image and the sparse image to obtain a compressed video. Therefore, this application's method of downsampling and sparse sampling the original video to obtain a compressed video can both preserve high-frequency information in the video and achieve high video compression, solving the problem that compressed videos generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a video processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In video-related applications, video compression is often required, followed by the storage and transmission of the compressed video data stream. However, when storage space is limited or network signal is poor, video compression at extremely low rates is often necessary. Because compression results in the loss of some original information, the decoded video typically exhibits visible noise and compression artifacts, making it difficult to meet the viewing needs of actual users.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a video processing method, apparatus, storage medium, and electronic device to at least solve the technical problem that compressed videos generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression.

[0005] According to one aspect of the embodiments of this application, a video processing method is provided, comprising: acquiring an original video; downsampling each frame of the original video to obtain a low-resolution image; sparsely sampling a preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval; and encoding the low-resolution image and the sparse image to obtain a compressed video.

[0006] Optionally, sparse sampling of a preset image in the original video to obtain a sparse image includes: dividing the preset image into blocks to obtain multiple image blocks; determining the amount of high-frequency information contained in each image block; and selecting at least one image block from the multiple image blocks based on the amount of high-frequency information to generate a sparse image.

[0007] Optionally, each image block can be processed using a total variational regularization function to obtain the amount of high-frequency information contained in each image block.

[0008] Optionally, generating a sparse image by selecting at least one image patch from multiple image patches based on high-frequency information includes: sorting multiple image patches in descending order of high-frequency information; obtaining at least one image patch that is ranked first to generate a sparse image.

[0009] Optionally, after encoding the low-resolution image and the sparse image to obtain a compressed video, the method further includes: decoding the compressed video to obtain a sparse image and a low-resolution image; upsampling the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and fusing the reconstructed image and the sparse image to generate a target video.

[0010] Optionally, upsampling the low-resolution image to obtain the reconstructed image includes: processing the low-resolution image using a super-resolution model to obtain the reconstructed image, wherein the super-resolution model is trained using the original training image and the compressed training image, and the compressed training image is obtained by downsampling and encoding the original training image.

[0011] Optionally, fusing information between the reconstructed image and the sparse image to generate the target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0012] Optionally, matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block includes: determining a first position of each first image block in the reconstructed image and a second position of each second image block in the sparse image; obtaining a second image block corresponding to the second position within a preset range around the first position to obtain at least one target image block; and matching each first image block with at least one target image block to obtain a matching image block.

[0013] Optionally, processing each first image block and the matching image block using an information fusion model to obtain a fused image block includes: extracting features from each first image block and the matching image block using the feature extraction layer in the information fusion model to obtain a first feature of each first image block and a second feature of the matching image block; concatenating the first feature and the second feature to obtain a concatenated feature; and processing the concatenated feature using the information fusion layer in the information fusion model to obtain a fused image block.

[0014] According to another aspect of the embodiments of this application, a video processing method is also provided, comprising: acquiring a compressed video; decoding the compressed video to obtain a sparse image and a low-resolution image, wherein the low-resolution image is an image obtained by downsampling each frame of the original video, and the sparse image is an image obtained by sparsely sampling a preset image in the original video; upsampling the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and fusing the reconstructed image and the sparse image to generate a target video.

[0015] Optionally, upsampling the low-resolution image to obtain the reconstructed image includes: processing the low-resolution image using a super-resolution model to obtain the reconstructed image, wherein the super-resolution model is trained using the original training image and the compressed training image, and the compressed training image is obtained by downsampling and encoding the original training image.

[0016] Optionally, fusing information between the reconstructed image and the sparse image to generate the target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0017] According to another aspect of the embodiments of this application, a video processing apparatus is also provided, comprising: an acquisition module for acquiring an original video; a downsampling module for downsampling each frame of the original video to obtain a low-resolution image; a sparse sampling module for sparsely sampling a preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval; and an encoding module for encoding the low-resolution image and the sparse image to obtain a compressed video.

[0018] According to another aspect of the embodiments of this application, a video processing apparatus is also provided, comprising: an acquisition module for acquiring compressed video; a decoding module for decoding the compressed video to obtain a sparse image and a low-resolution image, wherein the low-resolution image is an image obtained by downsampling each frame of the original video, and the sparse image is an image obtained by sparsely sampling a preset image in the original video; an upsampling module for upsampling the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and a fusion module for fusing the reconstructed image and the sparse image to generate a target video.

[0019] According to another aspect of the embodiments of this application, a computer storage medium is also provided, which stores a plurality of instructions adapted for loading by a processor and executing the video processing method described above.

[0020] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the video processing method described above.

[0021] In this embodiment, the original video is acquired, and each frame of the original video is downsampled to obtain a low-resolution image. A preset image in the original video is then sparsely sampled to obtain a sparse image. The preset image is obtained by sampling the original video at preset intervals. The low-resolution image and the sparse image are then encoded to obtain a compressed video. It is noteworthy that because the compressed video contains sparse images, and these sparse images are obtained by sparsely sampling the preset images from the original video, the compressed video retains only the high-frequency information of the original video and does not contain other redundant information. This achieves the technical effect of obtaining a high-quality compressed video, thereby solving the technical problem that compressed videos generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of a video processing method according to Embodiment 1 of this application;

[0024] Figure 2 This is a flowchart of a video processing method according to Embodiment 2 of this application;

[0025] Figure 3 This is a schematic diagram of an optional sparse sampling scheme according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an optional video encoding scheme according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an optional video decoding scheme according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of an optional information fusion model according to an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of a video processing apparatus according to Embodiment 3 of this application;

[0030] Figure 8 This is a schematic diagram of a video processing apparatus according to Embodiment 4 of this application;

[0031] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. Additionally, in the description of this application, unless otherwise stated, "a plurality of" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0035] To reduce video compression rates while preserving as much original information as possible, existing technologies typically employ video downsampling and super-resolution algorithms for video encoding and decoding. Existing solutions first downsample the original high-resolution video to obtain a low-resolution video, then use an encoder to compress the video into a binary bitstream to reduce video size, thereby reducing storage space and transmission costs. During video decoding, existing solutions use a decoder to decode the binary bitstream into a low-resolution video, and then use a super-resolution algorithm to upsample the decoded low-resolution video to reconstruct the original resolution video data. However, downsampling the entire original video leads to the loss of high-frequency information, and super-resolution algorithms often struggle to recover this high-frequency information effectively, resulting in poor video stream quality from existing solutions. Since video is temporal data, high-frequency information repeats in adjacent frames. Therefore, hybrid resolution coding techniques that retain original frames at intervals can reduce information redundancy while preserving high-frequency information. However, this method also retains a large amount of easily reconstructable repetitive low-frequency information, resulting in inadequate removal of video redundancy and thus limiting the video compression rate.

[0036] In summary, the above solutions have the following problems: (1) Traditional video coding compression algorithms lose a lot of original video information when compressing at low bitrates, resulting in a lot of noise and compression artifacts in the decoded video, leading to poor video quality; (2) The technical solution using downsampling and super-resolution algorithms can reconstruct the original high-resolution video well under low bitrate compression conditions, but because downsampling operations will lose high-frequency information, the quality of the reconstructed video has an upper limit; (3) The technical solution of saving the original high-resolution video frames at intervals can retain the high-frequency information of the original video, but it will also cause some information redundancy in the original video, which will limit the video compression rate.

[0037] To address the aforementioned issues, this application proposes a method of obtaining compressed video by downsampling and sparse sampling of the original video, thereby achieving the technical effect of obtaining high-quality compressed video and solving the technical problem that compressed video generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression.

[0038] Example 1

[0039] According to one aspect of the embodiments of this application, a video processing method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 1 This is a flowchart of a video processing method according to Embodiment 1 of this application, as follows: Figure 1 As shown, the method includes the following steps:

[0041] Step S102: Obtain the original video.

[0042] The original video in the above steps can be a high-resolution video that needs to be compressed.

[0043] The above steps can be used to obtain all video frames of the original video.

[0044] Step S104: Downsample each frame of the original video to obtain a low-resolution image.

[0045] In the steps described above, each frame of the original video can be downsampled using linear interpolation or DCT coefficient truncation to obtain a low-resolution image, but this is not the only method. For example, bilinear interpolation can be used to downsample the original video frames by a factor of 2, reducing the resolution of the original video from 720p to 320p.

[0046] Step S106: Sparsely sample the preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval.

[0047] The preset interval in the above steps can be manually set according to the requirements of video quality and compression rate, such as capturing a preset image every 20 frames, but it is not limited to this. The preset image can be an important video frame in the original video frame, i.e., a key frame, which contains high-frequency information required for image restoration. The embodiments of this application can use existing technology to obtain the preset image by sampling the original video at a preset interval, and then perform sparse sampling on the preset image to obtain a sparse image, which will not be elaborated here.

[0048] Step S108: Encode the low-resolution image and the sparse image to obtain the compressed video.

[0049] In the above steps, after obtaining the low-resolution image and the sparse image respectively, a video encoder can be selected according to the actual compression ratio requirements, such as an H.264 video encoder or an AVC video encoder, but not limited to these. The low-resolution video frames and the sparse image are encoded into a binary bitstream by the video encoder and then stored or transmitted over the network.

[0050] In this embodiment, the original video is acquired, and each frame of the original video is downsampled to obtain a low-resolution image. A preset image in the original video is then sparsely sampled to obtain a sparse image. The preset image is obtained by sampling the original video at preset intervals. The low-resolution image and the sparse image are then encoded to obtain a compressed video. It is noteworthy that because the compressed video contains sparse images, and these sparse images are obtained by sparsely sampling the preset images from the original video, the compressed video retains only the high-frequency information of the original video and does not contain other redundant information. This achieves the technical effect of obtaining a high-quality compressed video, thereby solving the technical problem that compressed videos generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression.

[0051] Optionally, sparse sampling of a preset image in the original video to obtain a sparse image includes: dividing the preset image into blocks to obtain multiple image blocks; determining the amount of high-frequency information contained in each image block; and selecting at least one image block from the multiple image blocks based on the amount of high-frequency information to generate a sparse image.

[0052] Optionally, each image block can be processed using a total variational regularization function to obtain the amount of high-frequency information contained in each image block.

[0053] Optionally, generating a sparse image by selecting at least one image patch from multiple image patches based on high-frequency information includes: sorting multiple image patches in descending order of high-frequency information; obtaining at least one image patch that is ranked first to generate a sparse image.

[0054] In the above steps, every few video frames (e.g., every 20 frames), important video frames, i.e., key frames, in the original video can be selected using existing technology and sparsely sampled to preserve the high-frequency information of the key frames. Embodiments of this application can use a block-based sparse sampling scheme, such as... Figure 3 As shown, the original patch is an image block obtained by dividing the original video frame into blocks, and the saved patch blocks are the image blocks that make up the sparse image after filtering. For example, the video frame can be divided into image blocks of a fixed size (such as 8×8 or 16×16), then the amount of high-frequency information in each image block can be calculated, and then the image blocks with more high-frequency information can be retained. The formula for this sparse sampling is as follows:

[0055] Score i =TV(P) i ), i∈{1,...,n} (1)

[0056] S={P i |i∈topK(Sorce i , K), i∈{1,...,n}} (2)

[0057] Each video frame can be divided into n image blocks, P i Let i represent the i-th image block i∈{1,...,n}, TV(·) represent the total variational regularization function, which is used to evaluate how much high-frequency information each image block contains, and TopK(·,K) represent selecting the K image blocks with the most high-frequency information.

[0058] In the above steps, the amount of high-frequency information contained in each image block can be obtained according to formula (1). According to formula (2), multiple image blocks can be sorted in descending order of high-frequency information to obtain at least one image block at the top of the sort and generate a sparse image.

[0059] In one alternative embodiment, such as Figure 4 As shown, the steps for video compression include: acquiring all video frames of the original video, performing video downsampling and interval sparse sampling on them respectively to obtain low-resolution video frames and a sparse graph of the original frames, encoding them to obtain a compressed video stream.

[0060] Optionally, after encoding the low-resolution image and the sparse image to obtain a compressed video, the method further includes: decoding the compressed video to obtain a sparse image and a low-resolution image; upsampling the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and fusing the reconstructed image and the sparse image to generate a target video.

[0061] The above steps involve decoding the compressed video using the decoder corresponding to the encoder, such as the H.264 video decoder or the AVC video decoder, but not limited to these.

[0062] In one alternative embodiment, such as Figure 5 As shown, the compressed video stream is decoded using a decoder to obtain the original video frame sparse image and low-resolution video frame. The low-resolution video frame is input into the super-resolution model for upsampling to obtain the reconstructed high-resolution video frame with the same resolution as the original video. The reconstructed high-resolution video frame is then fused with the sparse image using an information fusion model to generate the target video.

[0063] Optionally, upsampling the low-resolution image to obtain the reconstructed image includes: processing the low-resolution image using a super-resolution model to obtain the reconstructed image, wherein the super-resolution model is trained using the original training image and the compressed training image, and the compressed training image is obtained by downsampling and encoding the original training image.

[0064] The super-resolution model in the above steps can be trained using REDS data, but is not limited to this. This embodiment of the application can use the existing publicly available REDS dataset as the training dataset. This video dataset contains 269 video segments, each with 10 frames. Additionally, four video segments are selected from the REDS dataset as validation data. The super-resolution model can be obtained by repeatedly training the low-resolution data (after compression and decoding of the video) as input and the original high-resolution video as output. The reconstructed image is obtained by upsampling the low-resolution image using the trained super-resolution model.

[0065] It should be noted that, since the input data in this embodiment is encoded, compressed, and then decoded, compression artifacts and noise exist in the data. Therefore, the trained model can simultaneously recover both the downsampling loss and the encoding / compression loss. The target loss function for training is the absolute mean error, and its calculation method is as follows:

[0066]

[0067] Where N is the number of training sample pairs, x i and y i Let || be the low-resolution video frame and the high-resolution video frame in the i-th sample pair, || be the L1 norm in the || pair, and SR(·) represents the super-resolution model used, which takes a low-resolution video frame as input and outputs its corresponding high-resolution video frame.

[0068] Optionally, fusing information between the reconstructed image and the sparse image to generate the target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0069] In one optional embodiment, after upsampling the low-resolution image using a trained super-resolution model to obtain a reconstructed image, the video frames in the reconstructed image, i.e., the reconstructed video frames (i.e., high-resolution video frames), are segmented. For example, the video frames can be divided into image blocks of a fixed size (e.g., 8×8 or 16×16) to obtain multiple first image blocks. The cosine similarity between each first image block in the sparse image and the corresponding target image block in the reconstructed video frame is calculated, and the most similar image block, i.e., the matching image block, is found. Then, an information fusion model is used to extract the features of each image block and the matching image block in the sparse image and concatenate them to obtain a fused image block. Multiple fused image blocks are combined to form the target video.

[0070] Optionally, matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block includes: determining a first position of each first image block in the reconstructed image and a second position of each second image block in the sparse image; obtaining a second image block corresponding to the second position within a preset range around the first position to obtain at least one target image block; and matching each first image block with at least one target image block to obtain a matching image block.

[0071] The preset range in the above steps can be a position near an image block in the reconstructed image. For example, if the reconstructed image is divided into 5×5 image blocks, and the first position of the first image block is in the 2nd row and 2nd column, the preset range can be above (1st row and 2nd column), below (3rd row and 2nd column), to the left (2nd row and 1st column), or to the right (2nd row and 3rd column) of the first position, but is not limited to these. If the second position of the second image block is within the preset range of the first position, then the second image block is the target image block, and there can be multiple second image blocks.

[0072] Optionally, processing each first image block and the matching image block using an information fusion model to obtain a fused image block includes: extracting features from each first image block and the matching image block using the feature extraction layer in the information fusion model to obtain a first feature of each first image block and a second feature of the matching image block; concatenating the first feature and the second feature to obtain a concatenated feature; and processing the concatenated feature using the information fusion layer in the information fusion model to obtain a fused image block.

[0073] In one alternative embodiment, a lightweight information fusion network is used for each group of similar regions, with the model structure as follows: Figure 6 As shown, Conv is a convolutional neural network, and each input image patch p in the sparse image is a convolutional neural network. i and image patches in reconstructed images Then a convolutional neural network (with weights) is used. The features of the two image patches, namely the first feature and the second feature, are extracted separately. These features are then concatenated along the channel dimension. Finally, a convolutional neural network is used to fuse the information from the concatenated features (with weights of 1 / 2). After processing, it is finally passed through a convolutional neural network (with weights of ). The module outputs fused image patches, and finally combines all the fused image patches into the final video frame. The training of this module also uses the mean absolute error, calculated as follows:

[0074]

[0075] Where, p iIt is an image patch stored in a sparse graph, and It is to reconstruct high-resolution video frames and p i The most similar image patch, F(·) represents the information fusion network, and n represents the total number of image patches retained in the sparse graph.

[0076] Example 2

[0077] According to one aspect of the embodiments of this application, a video processing method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0078] Figure 2 This is a flowchart of a video processing method according to Embodiment 2 of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0079] Step S202: Obtain the compressed video.

[0080] In the above steps, the compressed video stream, i.e. the encoded binary bitstream, is obtained.

[0081] Step S204: Decode the compressed video to obtain a sparse image and a low-resolution image. The low-resolution image is obtained by downsampling each frame of the original video, and the sparse image is obtained by sparsely sampling a preset image in the original video.

[0082] In the above steps, the decoder corresponding to the encoder is used to decode the compressed video to obtain sparse images and low-resolution images, such as H264 video decoder, AVC video decoder, but not limited to these.

[0083] Step S206: Upsample the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video.

[0084] In the steps described above, a super-resolution algorithm can be used to upsample the low-resolution image, but this is not the only method used. The resulting reconstructed image is a high-resolution video frame, with the same resolution as the original video. For example, by upsampling the low-resolution image, the resolution of the compressed low-resolution video can be restored from 320p to the original video's resolution of 720p.

[0085] Step S208: Information fusion is performed on the reconstructed image and the sparse image to generate the target video.

[0086] In the above steps, an information fusion model can be used to fuse the reconstructed image and the sparse image to generate the target video. For example, by using an information fusion model, the high-frequency information stored in the sparse image can be fused with the reconstructed 720p video frames to obtain higher quality 720p video frames.

[0087] In this embodiment, a compressed video is acquired, decoded to obtain a sparse image and a low-resolution image. The low-resolution image is obtained by downsampling each frame of the original video, and the sparse image is obtained by sparsely sampling a preset image from the original video. The low-resolution image is then upsampled to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as that of the original video. The reconstructed image and the sparse image are then fused to generate the target video. It is noteworthy that by using an information fusion algorithm to further enhance the visual effect of the reconstructed high-resolution video frames using high-frequency information in the sparse image, a higher quality video can be obtained.

[0088] Optionally, upsampling the low-resolution image to obtain the reconstructed image includes: processing the low-resolution image using a super-resolution model to obtain the reconstructed image, wherein the super-resolution model is trained using the original training image and the compressed training image, and the compressed training image is obtained by downsampling and encoding the original training image.

[0089] The super-resolution model in the above steps can be trained using REDS data, but is not limited to this. This embodiment of the application can use the existing publicly available REDS dataset as the training dataset. This video dataset contains 269 video clips, each with 100 frames. In addition, four video clips are selected from the REDS dataset as validation data. The super-resolution model can be obtained by repeatedly training the low-resolution data (after compression and decoding of the video) as input and the original high-resolution video as output. The reconstructed image is obtained by upsampling the low-resolution image using the trained super-resolution model.

[0090] It should be noted that, since the input data in this embodiment is encoded, compressed, and then decoded, compression artifacts and noise exist in the data. Therefore, the trained model can simultaneously recover both the downsampling loss and the encoding / compression loss. The target loss function for training is the absolute mean error, and its calculation method is as follows:

[0091]

[0092] Where N is the number of training sample pairs, x i and y iLet be the low-resolution video frame and the high-resolution video frame in the i-th sample pair, ||·||1 is the L1 norm, and SR(·) represents the super-resolution model used, which takes a low-resolution video frame as input and outputs its corresponding high-resolution video frame.

[0093] Optionally, fusing information between the reconstructed image and the sparse image to generate the target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0094] In one optional embodiment, after upsampling the low-resolution image using a trained super-resolution model to obtain a reconstructed image, the video frames in the reconstructed image, i.e., the reconstructed video frames (i.e., high-resolution video frames), are segmented. For example, the video frames can be divided into image blocks of a fixed size (e.g., 8×8 or 16×16) to obtain multiple first image blocks. The cosine similarity between each first image block in the sparse image and multiple second image blocks in the corresponding reconstructed video frame is calculated, and the most similar image block, i.e., the matching image block, is found. Then, an information fusion model is used to extract the features of each image block and the matching image block in the sparse image and concatenate them to obtain a fused image block. Multiple fused image blocks are combined into the target video.

[0095] Example 3

[0096] According to another aspect of the embodiments of this application, a video processing apparatus is also provided, such as... Figure 7 As shown, it includes:

[0097] Module 72 is used to acquire the original video.

[0098] The downsampling module 74 is used to downsample each frame of the original video to obtain a low-resolution image.

[0099] The sparse sampling module 76 is used to perform sparse sampling on a preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval.

[0100] Encoding module 78 is used to encode low-resolution and sparse images to obtain compressed video.

[0101] Optionally, the sparse sampling module includes: a first segmentation unit for segmenting a preset image into multiple image blocks; a determination unit for determining the amount of high-frequency information contained in each image block; and a filtering unit for filtering at least one image block from the multiple image blocks based on the amount of high-frequency information to generate a sparse image.

[0102] Optionally, the determining unit is also used to process each image block using a total variational regularization function to obtain the amount of high-frequency information contained in each image block.

[0103] Optionally, the filtering unit includes: a sorting subunit for sorting multiple image blocks in descending order of high-frequency information content; and a generation subunit for obtaining at least one image block that is ranked first to generate a sparse image.

[0104] Optionally, the video processing apparatus further includes: a decoding module for decoding the compressed video to obtain a sparse image and a low-resolution image; an upsampling module for upsampling the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and a fusion module for fusing the reconstructed image and the sparse image to generate a target video.

[0105] Optionally, the upsampling module also uses a super-resolution model to process the low-resolution image to obtain a reconstructed image. The super-resolution model is trained using the original training image and the compressed training image, which is obtained by downsampling and encoding the original training image.

[0106] Optionally, the fusion module includes: a second segmentation unit for segmenting the reconstructed image into multiple first image blocks; a matching unit for matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; a fusion unit for processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and a generation unit for merging the multiple fused image blocks to generate a target video.

[0107] Optionally, the matching unit includes: a determining subunit, used to determine a first position of each first image block in the reconstructed image and a second position of each second image block in the sparse image; an obtaining subunit, used to obtain a second image block corresponding to the second position within a preset range around the first position, to obtain at least one target image block; and a matching subunit, used to match each first image block with at least one target image block to obtain a matched image block.

[0108] Optionally, the fusion unit includes: an extraction subunit, used to extract features from each first image block and the matching image block using the feature extraction layer in the information fusion model, to obtain a first feature of each first image block and a second feature of the matching image block; a stitching subunit, used to stitch the first feature and the second feature together to obtain stitched features; and a fusion subunit, used to process the stitched features using the information fusion layer in the information fusion model to obtain a fused image block.

[0109] Example 4

[0110] According to another aspect of the embodiments of this application, a video processing apparatus is also provided, such as... Figure 8 As shown, it includes:

[0111] Module 82 is used to acquire compressed video.

[0112] The decoding module 84 is used to decode the compressed video to obtain sparse images and low-resolution images. The low-resolution images are obtained by downsampling each frame of the original video, and the sparse images are obtained by sparsely sampling preset images in the original video.

[0113] The upsampling module 86 is used to upsample the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video.

[0114] The fusion module 88 is used to fuse information from the reconstructed image and the sparse image to generate the target video.

[0115] Optionally, the upsampling module is also used to process the low-resolution image using a super-resolution model to obtain a reconstructed image. The super-resolution model is trained using the original training image and the compressed training image, which is obtained by downsampling and encoding the original training image.

[0116] Optionally, the fusion module includes: a segmentation unit for segmenting the reconstructed image into multiple first image blocks; a matching unit for matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; a fusion unit for processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and a generation unit for merging the multiple fused image blocks to generate a target video.

[0117] Example 5

[0118] According to another aspect of the embodiments of this application, a computer storage medium is also provided, which stores a plurality of instructions adapted to be loaded and executed by a processor as described above. Figures 1-2 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 1-2 The specific details of the illustrated embodiments will not be elaborated here.

[0119] The storage medium can be located in an electronic device.

[0120] Example 6

[0121] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the video processing method described above.

[0122] like Figure 9 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0123] The communication bus 1002 is used to realize the connection and communication between these components.

[0124] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0125] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0126] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.

[0127] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 9 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and operating applications for electronic devices.

[0128] exist Figure 9 In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 1001 can be used to call the electronic device's operating application stored in the memory 1005 and specifically perform the following operations:

[0129] Acquire the original video; downsample each frame of the original video to obtain a low-resolution image; sparsely sample a preset image from the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval; encode the low-resolution image and the sparse image to obtain a compressed video.

[0130] In one embodiment, the operating system of the electronic device is the Android system. In the Android system, the processor 1001 further performs the following steps:

[0131] The process of sparsely sampling a preset image in the original video to obtain a sparse image includes: dividing the preset image into blocks to obtain multiple image blocks; determining the amount of high-frequency information contained in each image block; and selecting at least one image block from the multiple image blocks based on the amount of high-frequency information to generate a sparse image.

[0132] In one embodiment, the processor 1001 further performs the following steps:

[0133] Each image block is processed using a total variational regularization function to obtain the amount of high-frequency information contained in each image block.

[0134] In one embodiment, the processor 1001 further performs the following steps:

[0135] The process of generating a sparse image by selecting at least one image patch from multiple image patches based on high-frequency information includes: sorting multiple image patches in descending order of high-frequency information; obtaining at least one image patch that is ranked first to generate a sparse image.

[0136] In one embodiment, the processor 1001 further performs the following steps:

[0137] After encoding the low-resolution image and the sparse image to obtain the compressed video, the method further includes: decoding the compressed video to obtain the sparse image and the low-resolution image; upsampling the low-resolution image to obtain the reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; and fusing the reconstructed image and the sparse image to generate the target video.

[0138] In one embodiment, the processor 1001 further performs the following steps:

[0139] Upsampling a low-resolution image to obtain a reconstructed image involves processing the low-resolution image using a super-resolution model to obtain the reconstructed image. The super-resolution model is trained using the original training image and a compressed training image. The compressed training image is obtained by downsampling and encoding the original training image.

[0140] In one embodiment, the processor 1001 further performs the following steps:

[0141] Optionally, fusing information between the reconstructed image and the sparse image to generate the target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and the matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0142] In one embodiment, the processor 1001 further performs the following steps:

[0143] Matching each first image block with multiple second image blocks in a sparse image to obtain a matching image block corresponding to each first image block includes: determining a first position of each first image block in the reconstructed image and a second position of each second image block in the sparse image; obtaining a second image block corresponding to the second position within a preset range around the first position to obtain at least one target image block; and matching each first image block with at least one target image block to obtain a matching image block.

[0144] In one embodiment, the processor 1001 further performs the following steps:

[0145] The information fusion model is used to process each first image block and the matching image block to obtain a fused image block. This process includes: using the feature extraction layer in the information fusion model to extract features from each first image block and the matching image block to obtain a first feature of each first image block and a second feature of the matching image block; concatenating the first feature and the second feature to obtain a concatenated feature; and processing the concatenated feature using the information fusion layer in the information fusion model to obtain the fused image block.

[0146] In this embodiment, the provided electronic device can acquire an original video via a processor, downsample each frame of the original video to obtain a low-resolution image, and sparsely sample a preset image from the original video to obtain a sparse image. The preset image is obtained by sampling the original video at preset intervals. The low-resolution image and the sparse image are then encoded to obtain a compressed video. By downsampling and sparsely sampling the original video to obtain the compressed video, the technical effect of obtaining a high-quality compressed video is achieved, thereby solving the technical problem that compressed videos generated by existing technologies cannot simultaneously satisfy low bitrate compression and high-quality compression.

[0147] In one embodiment, the operating system of the electronic device is the Android system. In the Android system, the processor 1001 further performs the following steps:

[0148] Acquire compressed video; decode compressed video to obtain sparse image and low-resolution image, wherein the low-resolution image is obtained by downsampling each frame of the original video, and the sparse image is obtained by sparsely sampling preset images in the original video; upsample the low-resolution image to obtain reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; fuse the reconstructed image and sparse image to generate target video.

[0149] In one embodiment, the processor 1001 further performs the following steps:

[0150] Upsampling a low-resolution image to obtain a reconstructed image involves processing the low-resolution image using a super-resolution model to obtain the reconstructed image. The super-resolution model is trained using the original training image and a compressed training image. The compressed training image is obtained by downsampling and encoding the original training image.

[0151] In one embodiment, the processor 1001 further performs the following steps:

[0152] The process of fusing information from reconstructed images and sparse images to generate a target video includes: dividing the reconstructed image into blocks to obtain multiple first image blocks; matching each first image block with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; processing each first image block and matching image block using an information fusion model to obtain a fused image block; and merging multiple fused image blocks to generate the target video.

[0153] In this embodiment, the provided electronic device can acquire compressed video via a processor, decode the compressed video to obtain sparse images and low-resolution images. The low-resolution images are obtained by downsampling each frame of the original video, and the sparse images are obtained by sparsely sampling preset images from the original video. The low-resolution images are then upsampled to obtain reconstructed images, where the resolution of the reconstructed images is the same as that of the original video. The reconstructed images and sparse images are then fused to generate the target video. It is noteworthy that by using an information fusion algorithm to further enhance the visual effect of the reconstructed high-resolution video frames using high-frequency information in the sparse images, a higher quality video can be obtained.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0160] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0162] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video processing method, characterized in that, include: Obtain the original video; Each frame of the original video is downsampled to obtain a low-resolution image; A sparse image is obtained by sparse sampling a preset image in the original video, wherein the preset image is obtained by sampling the original video at a preset interval; The low-resolution image and the sparse image are encoded to obtain a compressed video.

2. The method according to claim 1, characterized in that, Sparse sampling is performed on preset images in the original video to obtain sparse images, including: The preset image is divided into blocks to obtain multiple image blocks; Determine the amount of high-frequency information contained in each image patch; Based on the high-frequency information, at least one image block is selected from the plurality of image blocks to generate the sparse image.

3. The method according to claim 2, characterized in that, Each image block is processed using a total variational regularization function to obtain the amount of high-frequency information contained in each image block.

4. The method according to claim 2, characterized in that, Generating the sparse image by selecting at least one image patch from the plurality of image patches based on the high-frequency information content includes: The multiple image blocks are sorted in descending order of the amount of high-frequency information. Obtain the first or most sorted image patch to generate the sparse image.

5. The method according to any one of claims 1 to 4, characterized in that, After encoding the low-resolution image and the sparse image to obtain a compressed video, the method further includes: The compressed video is decoded to obtain the sparse image and the low-resolution image; The low-resolution image is upsampled to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; The reconstructed image and the sparse image are fused to generate the target video.

6. The method according to claim 5, characterized in that, Upsampling the low-resolution image to obtain the reconstructed image includes: The low-resolution image is processed using a super-resolution model to obtain the reconstructed image. The super-resolution model is trained using the original training image and a compressed training image. The compressed training image is obtained by downsampling and encoding the original training image.

7. The method according to claim 5, characterized in that, The process of fusing information from the reconstructed image and the sparse image to generate the target video includes: The reconstructed image is divided into blocks to obtain multiple first image blocks; Each first image block is matched with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; Each first image block and the matching image block are processed using an information fusion model to obtain a fused image block; Multiple fused image blocks are merged to generate the target video.

8. The method according to claim 7, characterized in that, Each first image block is matched with multiple second image blocks in the sparse image to obtain the matching image block corresponding to each first image block, including: Determine the first position of each first image block in the reconstructed image, and the second position of each second image block in the sparse image; Obtain a second image block corresponding to a second position located within a preset range around the first position to obtain at least one target image block; Each first image block is matched with the at least one target image block to obtain the matched image block.

9. The method according to claim 7, characterized in that, The information fusion model is used to process each first image block and the matching image block to obtain a fused image block, including: The feature extraction layer in the information fusion model is used to extract features from each first image block and the matching image block respectively, to obtain the first feature of each first image block and the second feature of the matching image block; The first feature and the second feature are concatenated to obtain the concatenated feature; The stitching features are processed using the information fusion layer in the information fusion model to obtain the fused image block.

10. A video processing method, characterized in that, include: Get compressed video; The compressed video is decoded to obtain a sparse image and a low-resolution image, wherein the low-resolution image is an image obtained by downsampling each frame of the original video, and the sparse image is an image obtained by sparsely sampling a preset image in the original video; The low-resolution image is upsampled to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; The reconstructed image and the sparse image are fused to generate the target video.

11. The method according to claim 10, characterized in that, Upsampling the low-resolution image to obtain the reconstructed image includes: The low-resolution image is processed using a super-resolution model to obtain the reconstructed image. The super-resolution model is trained using the original training image and a compressed training image. The compressed training image is obtained by downsampling and encoding the original training image.

12. The method according to claim 10, characterized in that, The process of fusing information from the reconstructed image and the sparse image to generate the target video includes: The reconstructed image is divided into blocks to obtain multiple first image blocks; Each first image block is matched with multiple second image blocks in the sparse image to obtain a matching image block corresponding to each first image block; Each first image block and the matching image block are processed using an information fusion model to obtain a fused image block; Multiple fused image blocks are merged to generate the target video.

13. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire the original video. The downsampling module is used to downsample each frame of the original video to obtain a low-resolution image; A sparse sampling module is used to perform sparse sampling on a preset image in the original video to obtain a sparse image, wherein the preset image is obtained by sampling the original video at a preset interval; The encoding module is used to encode the low-resolution image and the sparse image to obtain a compressed video.

14. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire compressed videos; The decoding module is used to decode the compressed video to obtain sparse images and low-resolution images, wherein the low-resolution images are obtained by downsampling each frame of the original video, and the sparse images are obtained by sparsely sampling a preset image in the original video. An upsampling module is used to upsample the low-resolution image to obtain a reconstructed image, wherein the resolution of the reconstructed image is the same as the resolution of the original video; The fusion module is used to fuse the information of the reconstructed image and the sparse image to generate the target video.

15. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1 to 12.

16. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps as claimed in any one of claims 1 to 12.

Images