Video processing method and apparatus

Abstract

The present disclosure discloses a video processing method and device, and relates to the technical field of video processing. The video processing method comprises: performing feature coding on a target frame of a video and adjacent frames of the target frame to obtain feature maps of the target frame and the adjacent frames; for each pixel in the feature map of the target frame, determining a pixel in the feature map of the adjacent frame that has the highest correlation degree with the pixel; constructing a first reference feature map according to the pixel in the feature map of the adjacent frame that has the highest correlation degree with the pixel; and performing super-resolution reconstruction according to the feature map of the target frame and the first reference feature map. The present disclosure can improve the video super-resolution processing efficiency, reduce the requirement of video super-resolution processing on computing resources, and improve the quality of the video reconstructed by super-resolution.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of video processing, and in particular, to a video processing method and device. BACKGROUND

[0002] Super-resolution, i.e. improving the resolution of the original image by hardware or software method, is a process of obtaining a high-resolution image from a series of low-resolution images. The core idea of super-resolution reconstruction is to exchange time bandwidth (obtaining a sequence of multiple images of the same scene) for spatial resolution, realizing the conversion of time resolution to spatial resolution.

[0003] In the related art, super-resolution processing can be performed by a convolutional neural network to restore image details in a low-resolution video. This kind of method mainly fills in the missing details in the low-resolution video frame by fusing the local details of adjacent frames, which has the following shortcomings: first, the related art needs to calculate the image correspondence between video frames by relying on optical flow method or the like before fusing the layout details of adjacent frames, which will result in poor calculation accuracy in the case of large motion in the video, high requirement for calculation resources, slow speed and other problems; second, since the adjacent frames that can be referred to are also low-resolution, the contribution to enhancing the clarity of the key frame is small. SUMMARY

[0004] One of the technical problems to be solved by the present disclosure is to provide a video processing method and device, which can improve the video super-resolution processing efficiency, reduce the requirement of video super-resolution processing for calculation resources, and improve the quality of the video reconstructed by super-resolution.

[0005] According to an aspect of the present disclosure, a video processing method is provided, which includes: performing feature encoding on a target frame and its adjacent frames of a video to obtain feature maps of the target frame and its adjacent frames; determining, for each pixel in the feature map of the target frame, a pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel; constructing a first reference feature map according to the pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel; and performing super-resolution reconstruction according to the feature map of the target frame and the first reference feature map.

[0006] In some embodiments, performing super-resolution reconstruction according to the feature map of the target frame and the first reference feature map includes: performing super-resolution reconstruction according to the feature map of the target frame, the first reference feature map, and a second reference feature map, wherein the second reference feature map fuses general pixel features.

[0007] In some embodiments, determining the pixel in the feature map of the adjacent frame that has the highest correlation with the pixel comprises: determining the correlation of the pixel with its neighborhood pixels in the feature map of the adjacent frame; and selecting the pixel with the highest correlation from the neighborhood pixels according to the correlations.

[0008] In some embodiments, constructing the first reference feature map according to the pixel in the feature map of the adjacent frame that has the highest correlation with the pixel comprises: determining a first reference feature corresponding to the pixel according to the feature of the pixel in the feature map of the adjacent frame that has the highest correlation with the pixel and the corresponding correlation; and constructing the first reference feature map according to the first reference feature corresponding to the pixel.

[0009] In some embodiments, the video processing method further comprises: determining the correlation of a pixel in the feature map of the target frame with a plurality of general pixel features; performing fusion processing on the plurality of general pixel features according to the correlations to obtain a second reference feature corresponding to the pixel; and constructing the second reference feature map according to the second reference feature corresponding to the pixel.

[0010] In some embodiments, performing fusion processing on the plurality of general pixel features according to the correlations to obtain the second reference feature corresponding to the pixel comprises: normalizing the correlations; and performing weighted average operation on the plurality of general pixel features by taking the normalized correlations as weights, and taking the result of the weighted average operation as the second reference feature corresponding to the pixel.

[0011] In some embodiments, performing super-resolution reconstruction according to the feature map of the target frame, the first reference feature map and the second reference feature map comprises: inputting the feature map of the target frame, the first reference feature map and the second reference feature map into a feature decoding network model to output a first reconstructed image; and superimposing the first reconstructed image and a feature map obtained by performing bilinear interpolation on the feature map of the target frame to obtain a second reconstructed image.

[0012] In some embodiments, the network model used for feature encoding comprises a plurality of ResNet modules.

[0013] In some embodiments, the feature decoding network model comprises a plurality of ResNet modules and an up-sampling module.

[0014] According to another aspect of the present disclosure, a video processing apparatus is also provided, which comprises: a feature encoding module configured to encode features of a target frame and its neighboring frames of a video to obtain feature maps of the target frame and its neighboring frames; a first constructing module configured to, for each pixel in the feature map of the target frame, determine a pixel in the feature map of the neighboring frame that has the highest correlation with the pixel, and construct a first reference feature map according to the pixel in the feature map of the neighboring frame that has the highest correlation with the pixel; and a reconstructing module configured to perform super-resolution reconstruction according to the feature map of the target frame and the first reference feature map.

[0015] In some embodiments, the reconstructing module is configured to perform super-resolution reconstruction according to the feature map of the target frame, the first reference feature map and a second reference feature map, wherein the second reference feature map fuses general pixel features.

[0016] In some embodiments, the first constructing module is configured to determine the correlation of the pixel with its neighborhood pixels in the feature map of the neighboring frame, and select a pixel with the highest correlation from the neighborhood pixels according to the correlation.

[0017] In some embodiments, the first constructing module is configured to determine a first reference feature corresponding to the pixel according to the feature of the pixel in the feature map of the neighboring frame that has the highest correlation with the pixel and the corresponding correlation, and construct the first reference feature map according to the first reference feature corresponding to the pixel.

[0018] In some embodiments, the video processing apparatus further comprises a second constructing module configured to determine the correlation of a pixel in the feature map of the target frame with a plurality of general pixel features, fuse the plurality of general pixel features according to the correlation to obtain a second reference feature corresponding to the pixel, and construct the second reference feature map according to the second reference feature corresponding to the pixel.

[0019] In some embodiments, the second constructing module is configured to normalize the correlation, perform weighted average operation on the plurality of general pixel features by taking the normalized correlation as a weight, and take the result of the weighted average operation as the second reference feature corresponding to the pixel.

[0020] In some embodiments, the second constructing module is configured to input the feature map of the target frame, the first reference feature map and a second reference feature map into a feature decoding network model to output a first reconstructed image, and superimpose the first reconstructed image and a feature map obtained by performing bilinear interpolation on the feature map of the target frame to obtain a second reconstructed image.

[0021] In some embodiments, the feature encoding network model comprises a plurality of ResNet modules.

[0022] In some embodiments, the feature decoding network model comprises a plurality of ResNet modules and an up-sampling module.

[0023] According to another aspect of the present disclosure, a video processing apparatus is also provided, comprising a memory and a processor coupled to the memory, the processor being configured to perform the video processing method as described above based on instructions stored in the memory.

[0024] According to another aspect of the present disclosure, a computer readable storage medium having computer program instructions stored thereon is also provided, the instructions being executable by a processor to implement the video processing method as described above.

[0025] Compared with the related art, in the embodiments of the present disclosure, when fusing the image details of the adjacent frames, a cross-frame attention mechanism is introduced to realize the matching of the adjacent frames and the target frame at the pixel level, which is embodied in determining the pixel with the maximum pixel correlation degree in the feature map of the adjacent frame and the target frame, and constructing a first reference feature map according to the pixel with the maximum pixel correlation degree in the feature map of the adjacent frame and the target frame. In this way, the present disclosure does not need to explicitly calculate the pixel correspondence between the video frames based on the optical flow method and the like, thereby solving the problems of poor calculation accuracy and high requirement for computing resources and slow speed caused by relying on the optical flow method and the like to calculate the image correspondence between the video frames in the related art, which helps to improve the video super-resolution processing efficiency, reduce the requirement for computing resources of video super-resolution processing, and improve the quality of the video reconstructed by super-resolution.

[0026] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0028] The present disclosure can be understood moreappreciably with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0029] Figure 1 Flowchart of some embodiments of the video processing method of the present disclosure.

[0030] Figure 2Flowchart for some embodiments of the video processing method of the present disclosure.

[0031] Figure 3 Flowchart for some embodiments of the constructing a second reference feature map of the present disclosure.

[0032] Figure 4 Comparison chart of video frame processing effects between embodiments of the present disclosure and related art.

[0033] Figure 5 Structural diagram of some embodiments of the video processing device of the present disclosure.

[0034] Figure 6 Structural diagram of some other embodiments of the video processing device of the present disclosure.

[0035] Figure 7a Structural diagram of some embodiments of the feature encoding network model of the present disclosure.

[0036] Figure 7b Structural diagram of some embodiments of the feature decoding network model of the present disclosure.

[0037] Figure 8 Structural diagram of some other embodiments of the video processing device of the present disclosure.

[0038] Figure 9 Structural diagram of some other embodiments of the video processing device of the present disclosure. DETAILED DESCRIPTION

[0039] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless specifically stated otherwise.

[0040] It should be understood that the sizes of the various portions shown in the drawings are not necessarily drawn to scale for ease of description.

[0041] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the present disclosure or its application or uses.

[0042] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the present disclosure.

[0043] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0044] It should be noted that like reference numerals and letters refer to like items throughout the several views, and that once an item is defined in one view, it should not have to be discussed further in subsequent views.

[0045] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the specific embodiments and in conjunction with the accompanying drawings.

[0046] Figure 1 Flowchart of some embodiments of the video processing method of the present disclosure.

[0047] In step 110, the target frame and its adjacent frames of the video are feature-encoded to obtain the feature maps of the target frame and its adjacent frames.

[0048] In some embodiments, the key frames in the video are taken as the target frames, and one or more video frames adjacent to the key frames are taken as the adjacent frames. For example, the video is processed by using a sliding window, and each time the key frame to be processed by super-resolution and its adjacent first 3 frames and last 3 frames are taken, and the process is performed according to the flowchart shown in FIG. 2 to obtain the super-resolution reconstruction image of the key frame.

[0049] In some embodiments, the key frames in the video are taken as the target frames, and one or more video frames adjacent to the key frames are taken as the adjacent frames. For example, the video is processed by using a sliding window, and each time the key frame to be processed by super-resolution and its adjacent first 3 frames and last 3 frames are taken, and the process is performed according to the flowchart shown in FIG. 2 to obtain the super-resolution reconstruction image of the key frame. Figure 1

[0050] In step 110, the target frame and its adjacent frames are feature-encoded by using a feature encoding network model to obtain the feature map of the target frame and the feature maps of the adjacent frames. The feature dimension of the feature map obtained by encoding is greater than the feature dimension of the original video frame. For example, in a specific example, a three-channel RGB video frame is input into the feature encoding network model to obtain a 128-channel feature map.

[0051] For example, the feature encoding network model is a network model composed of multiple layers of residual network (ResNet) modules, such as a network model composed of 5 layers of ResNet modules. Those skilled in the art can understand that, without affecting the implementation of the present disclosure, the feature encoding network used in the present disclosure can also use other network model structures, such as an autoencoder or a residual dense network (RDN, full name: Residual Dense Network), etc., in addition to the network model composed of 5 layers of ResNet modules.

[0052] ​In the embodiments of the present disclosure, the feature coding network model is configured to code features of the target frame and its adjacent frames, so that the feature dimension is increased, richer image detail information is extracted, and the image super-resolution processing effect is improved.

[0053] In step 120, for each pixel in the feature map of the target frame, a pixel in the feature map of the adjacent frame having the greatest correlation with the pixel is determined.

[0054] In some embodiments, for each pixel in the feature map of the target frame, the correlation of the pixel with its neighborhood pixels in the feature map of the adjacent frame is determined, and the pixel having the greatest correlation is selected from the neighborhood pixels according to the correlation. In these embodiments, by only calculating the correlation of the pixel in the target frame with its neighborhood pixels in the adjacent frame, the calculation amount is greatly reduced compared with calculating the pixel correlation of the target frame and the adjacent frame one by one in other embodiments.

[0055] In the embodiments, the neighborhood pixels of the pixel in the feature map of the adjacent frame refer to the pixels in the feature map of the adjacent frame that are adjacent to the pixel in the target frame in spatial position. For example, a 9*9 pixel region centered on pixel a is taken as the neighborhood of the pixel a, and then the neighborhood pixels of the pixel a on a certain adjacent frame are 9*9.

[0056] In the embodiments, the correlation of the pixels describes the similarity between the pixels. For example, the correlation of two pixels can be calculated according to the following manner: the inner product of the feature vector of the pixel a and the feature vector of the pixel b is taken as the correlation of the pixel a and the pixel b.

[0057] For example, it is assumed that the target frame is the 4th frame in a video, and the adjacent frames of the target frame are the 1st, 2nd, 3rd, 5th, 6th and 7th frames in the video. For the pixel a in the feature map of the 4th frame, the correlation of the pixel a with 9*9 neighborhood pixels in the feature maps of the 6 adjacent frames is calculated, and then 6*9*9 correlation values can be obtained. According to the correlation, the pixel having the greatest correlation with the pixel a is selected from the 6*9*9 neighborhood pixels. The above process is performed for each pixel in the feature map of the target frame, so that the pixel having the greatest correlation with each pixel in the feature map of the target frame can be found from the feature maps of the 6 adjacent frames.

[0058] In step 130, a first reference feature map is constructed according to the pixel having the greatest correlation with the pixel in the feature map of the adjacent frame.

[0059] In some embodiments, for each pixel in the feature map of the target frame, the first reference feature corresponding to the pixel is determined according to the feature of the pixel having the greatest correlation with the pixel selected from the feature map of the adjacent frame and the corresponding correlation. The first reference feature map is constructed according to the first reference feature corresponding to each pixel in the feature map of the target frame.

[0060] In one optional implementation, for each pixel in the feature map of the target frame, the product of the features of the pixels with the highest correlation selected from the feature maps of adjacent frames and their corresponding correlations is used as the first reference feature corresponding to that pixel. In this way, the first reference features corresponding to each pixel in the feature map of the target frame can be obtained, and thus, the first reference feature map can be obtained. The pixel features in the first reference feature map are the first reference features corresponding to the pixels in the feature map of the target frame.

[0061] Those skilled in the art will understand that using the pixel feature with the highest relevance and its corresponding product as the first reference feature is merely an example. Other specific implementations that determine the first reference feature based on the pixel feature with the highest relevance and its corresponding product may also be used without affecting the implementation of the present invention.

[0062] In this embodiment, when fusing image details from adjacent frames, a cross-frame attention mechanism is introduced in steps 120 and 130 to achieve pixel-level matching between adjacent frames and the target frame. This eliminates the need for explicit calculation of pixel correspondences between video frames based on optical flow or similar methods. This solves the problems of poor calculation accuracy, high computational resource requirements, and slow speed caused by relying on optical flow methods to calculate image correspondences between video frames in related technologies. It helps improve the efficiency of video super-resolution processing, reduces the computational resource requirements of video super-resolution processing, and improves the quality of the super-resolution reconstructed video.

[0063] In step 140, super-resolution reconstruction is performed based on the feature map of the target frame and the first reference feature map.

[0064] In some embodiments, the feature map of the target frame and the first reference feature map are input into the feature decoding network model, and the image output by the model is used as the super-resolution reconstructed image, i.e., the high-resolution image corresponding to the target frame. The decoding network model fuses the feature map of the target frame and the first reference feature map, and remaps the fused image from a high-dimensional feature space to a low-dimensional feature space to obtain the high-resolution reconstructed image. For example, in a specific example, a 128-channel target frame feature map and a 128-channel first reference feature map are input into the feature decoding network model to obtain a high-resolution three-channel RGB image.

[0065] For example, the feature decoding network model is a network model composed of multi-layer ResNet modules and upsampling modules, such as a network model composed of 40 layers of ResNet modules and upsampling modules. In some embodiments, the detailed composition of the ResNet module is as follows: Figure 7a As shown, the detailed components of the upsampling module are as follows: Figure 7bAs shown. Among them, the ResNet module includes a convolution layer, a ReLu activation layer, and a convolution layer. The up-sampling module includes a convolution layer, a Pixel Shuffle, and a Leaky ReLu processing layer, and a Leaky ReLu processing layer. The ResNet module is responsible for fusing the feature map of the target frame and the first reference feature map to obtain a fused high-dimensional low-resolution image; the up-sampling module is responsible for recombining the high-dimensional low-resolution image features into low-dimensional high-resolution image features, such as recombining 128-channel low-resolution features into 3-channel high-resolution RGB image features. This process uses the industry-recognized Pixel Shuffle operation. Pixel Shuffle is an up-sampling method that can effectively enlarge the reduced feature map. This method can replace the interpolation or deconvolution method to achieve up-sampling. Leaky ReLu and ReLu are two commonly used activation functions in deep learning.

[0066] Those skilled in the art can understand that, without affecting the implementation of the present disclosure, the feature decoding network used in the present disclosure can use other network model structures in addition to the network model composed of 40-layer ResNet modules and up-sampling modules, such as RDN, etc.

[0067] In some embodiments, the feature map of the target frame and the first reference feature map are input into the feature decoding network model to output a first reconstructed image; the feature map obtained by performing bilinear interpolation processing on the feature map of the target frame is superimposed with the first reconstructed image to obtain a second reconstructed image, and the second reconstructed image is taken as the high-resolution image corresponding to the reconstructed target frame. In these embodiments, after obtaining the high-resolution reconstructed image through the feature decoding network model, the resolution of the reconstructed image can be further improved by superimposing it with the feature map obtained by performing bilinear interpolation on the feature map of the target frame.

[0068] In the embodiments of the present disclosure, the super-resolution processing of the target frame of the video is realized through the above steps. Compared with related technologies, not only can the video super-resolution processing efficiency be improved, the requirement of video super-resolution processing on computing resources can be reduced, but also the quality of the super-resolution reconstructed video can be improved.

[0069] Figure 2 Flowchart of another embodiment of the video processing method of the present disclosure.

[0070] In step 210, the target frame and its adjacent frames of the video are feature encoded to obtain the feature map of the target frame and its adjacent frames.

[0071] In this embodiment, a feature coding network model is used to code the target frame and its adjacent frames to obtain a feature map of the target frame and feature maps of the adjacent frames. The feature dimension of the obtained feature map is greater than the feature dimension of the original video frame. For example, in a specific example, a three-channel RGB video frame is input into the feature coding network model to obtain a 128-channel feature map.

[0072] In this embodiment, the feature coding network model is used to code the target frame and its adjacent frames, which can realize feature dimension increase, extract richer image detail information, and further help improve the effect of image super-resolution processing.

[0073] In step 220, for each pixel in the feature map of the target frame, a pixel in the feature map of the adjacent frame having the greatest correlation with the pixel is determined.

[0074] In this embodiment, for each pixel in the feature map of the target frame, the correlation of the pixel with its neighborhood pixels in the feature map of the adjacent frame is determined, and the pixel having the greatest correlation is selected from the neighborhood pixels according to the correlation. In this embodiment, by only calculating the correlation of the pixel in the target frame with its neighborhood pixels in the adjacent frame, the calculation amount is greatly reduced compared with calculating the pixel correlation of the target frame and the adjacent frame one by one.

[0075] In step 230, a first reference feature map is constructed according to the pixel having the greatest correlation in the feature map of the adjacent frame.

[0076] In this embodiment, for each pixel in the feature map of the target frame, the product of the feature of the pixel having the greatest correlation selected from the feature map of the adjacent frame and the corresponding correlation is taken as the first reference feature corresponding to the pixel. In this way, the first reference features corresponding to the pixels in the feature map of the target frame are obtained, and further, the first reference feature map can be obtained accordingly. The pixel feature in the first reference feature map is the first reference feature corresponding to the pixel in the feature map of the target frame.

[0077] In step 240, a second reference feature map is constructed. The second reference feature map fuses the general pixel features.

[0078] The general pixel features are image detail information learned from known high-resolution video images and corresponding low-resolution images. In some embodiments, the general pixel features are stored in a two-dimensional matrix. For example, a two-dimensional matrix of 256*128 is used, which is composed of 256 feature vectors of 128 channels. Each feature vector stores the most representative general pixel features learned from known high-resolution video images and corresponding low-resolution images in the training process. In these embodiments, the second reference feature map is constructed according to the two-dimensional matrix.

[0079] In step 250, super-resolution reconstruction is performed according to the feature map of the target frame, the first reference feature map, and the second reference feature map.

[0080] In some embodiments, the feature map of the target frame, the first reference feature map, and the second reference feature map are input into a feature decoding network model, and the image output by the model is taken as the super-resolution reconstructed image. The decoding network model is used to fuse the feature map of the target frame, the first reference feature map, and the second reference feature map, and map the fused image from a high-dimensional feature space to a low-dimensional feature space to obtain a high-resolution reconstructed image. For example, in a specific example, the 128-channel feature map of the target frame, the 128-channel first reference feature map, and the 128-channel second reference feature map are input into the feature decoding network model to obtain a high-resolution three-channel RGB image.

[0081] In other embodiments, the feature map of the target frame, the first reference feature map, and the second reference feature map are input into a feature decoding network model to output a first reconstructed image; a feature map obtained by performing bilinear interpolation on the feature map of the target frame is superimposed on the first reconstructed image to obtain a second reconstructed image, and the second reconstructed image is taken as the final super-resolution reconstructed image. In these embodiments, after obtaining a high-resolution reconstructed image by the feature decoding network model, the image resolution can be further improved by superimposing it on the feature map obtained by performing bilinear interpolation on the feature map of the target frame.

[0082] In specific implementations, the method of the embodiments of the present disclosure is verified to have good performance in general small displacement video super-resolution processing based on the academically recognized data sets Vimeo90K-Test and Vid4. At the same time, the method of the embodiments of the present disclosure is verified to also have good performance in large displacement video super-resolution processing based on the Parkour data set. When measuring the visual difference between the super-resolution result and the ground truth high-resolution video, the quantitative standard is the academically recognized peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0083] In the embodiments of the present disclosure, the above steps are used to achieve super-resolution processing of the target frame of the video. Compared with the related art, the cross-frame attention mechanism is introduced when fusing the image details of the adjacent frames to achieve the matching of the adjacent frames and the target frame at the pixel level, so that the present disclosure does not need to explicitly calculate the pixel correspondence between the video frames based on the optical flow method or the like, thereby solving the problems of poor calculation accuracy and high requirement for computing resources and slow speed caused by relying on the optical flow method or the like to calculate the image correspondence between the video frames in the related art, which helps to improve the video super-resolution processing efficiency, reduce the requirement for computing resources of the video super-resolution processing, and improve the quality of the video reconstructed by super-resolution.

[0084] Figure 3 The flowchart for constructing some embodiments of the second reference feature map of the present disclosure is shown. The following will be described in detail in combination with Figure 3 The step 240 in the above embodiment is described in detail.

[0085] In step 241, the correlation between the pixels in the feature map of the target frame and the plurality of general pixel features is determined.

[0086] In some embodiments, the general pixel features are stored in the form of a two-dimensional matrix composed of a plurality of feature vectors, each of which stores the most representative general pixel features learned from the known high-resolution video images and their corresponding low-resolution images in the training process. For each pixel in the feature map of the target frame, the correlation between the pixel and each feature vector in the two-dimensional matrix is calculated.

[0087] In some embodiments, the above two-dimensional matrix is obtained by pre-training. In the pre-training process, the two-dimensional matrix gradually converges from the initial random value to the final result. In the video super-resolution processing process, the above two-dimensional matrix is fixed. By learning some image features in general videos in the pre-training stage, the two-dimensional matrix is formed, so that the two-dimensional matrix can be referred to when performing super-resolution processing on the current video, thereby improving the quality of video super-resolution processing.

[0088] In step 242, the plurality of general pixel features are fused according to the correlation to obtain the second reference feature corresponding to the pixel in the feature map of the target frame.

[0089] In some embodiments, the correlation degree obtained in step 241 is normalized; the normalized correlation degree is used as a weight to perform a weighted average operation on the plurality of general pixel features, and the result of the weighted average operation is taken as the second reference feature corresponding to the pixel.

[0090] For example, a two-dimensional matrix of 256*128 is used, which is composed of 256 128-channel feature vectors, each of which stores the most representative general pixel features learned from the known high-resolution video images and the corresponding low-resolution images in the training process. For each pixel in the feature map of the target frame, the correlation between it and the 256 feature vectors in the two-dimensional matrix is calculated to obtain 256 correlation values. After normalizing the 256 correlation values, the normalized correlation values are used as weights to perform a weighted average operation on the 256 feature vectors, and the result of the weighted average operation is taken as the second reference feature corresponding to the pixel.

[0091] In step 243, a second reference feature map is constructed according to the second reference features corresponding to the pixels in the feature map of the target frame.

[0092] In the second reference feature map, the pixel features are the second reference features corresponding to the pixels in the feature map of the target frame.

[0093] In the embodiments of the present disclosure, the second reference feature map is well constructed through the above steps. When performing super-resolution reconstruction on the target frame, by introducing the first reference feature map and the second reference feature map reconstructed through the above steps, the image details of the general pixels are also fused when performing super-resolution reconstruction on the target frame, so that the image details can be increased to a certain extent, and the quality of the video reconstructed by super-resolution is improved.

[0094] Figure 4 The following is a comparison chart of the video frame processing effect of the embodiments of the present disclosure and related technologies. As shown in the chart, the video frame processing effect of the embodiments of the present disclosure is better than that of the related technologies. Figure 4As shown, the image comprises three rows and four columns. From left to right, the first column is an example video frame, and the second to fourth columns are magnified images of the rectangular regions selected in the example video frames. Specifically, the second column shows the result of bicubic interpolation magnification of the low-resolution video frame, the third column shows the result of processing the low-resolution video frame using the video processing method of this embodiment, and the last column shows the result of processing the low-resolution video frame using a leading super-resolution processing method. In the first row, because optical flow methods are inaccurate in calculating repetitive local images (such as guitar strings), using leading super-resolution methods can easily produce erroneous textures. This disclosure, however, can better recover repetitive textures because it does not require calculating pixel correspondences between video frames. In the second row, the image obtained using leading super-resolution methods is not clear enough because many details of the image are permanently lost in the low-resolution video, making it impossible to obtain useful information solely from other frames of the same video. This disclosure, however, preserves image detail information summarized from other videos during training, thus possessing a certain ability to enhance image detail. The third row shows an example of a moving video, in which objects are displaced significantly. Current industry-leading methods generally cannot recover high-resolution details in such videos, but this disclosure has been tested and proven to maintain excellent super-resolution processing performance even in such videos with large displacements.

[0095] Figure 5 These are schematic diagrams illustrating the structure of some embodiments of the video processing apparatus disclosed herein. Figure 5 As shown, the video processing apparatus in this embodiment includes: a feature encoding module 510, a first construction module 520, and a reconstruction module 530.

[0096] The feature encoding module 510 is configured to perform feature encoding on the target frame and its neighboring frames of the video to obtain feature maps of the target frame and its neighboring frames.

[0097] The target frame is the video frame to be super-resolution processed, and the adjacent frames are one or more video frames adjacent to the video frame to be super-resolution processed. For example, the three frames before and the three frames after the target frame can be taken as adjacent frames, or the two frames before the target frame can be taken as adjacent frames, or the four frames after the target frame can be taken as adjacent frames, and so on.

[0098] In some embodiments, the feature encoding module 510 uses a feature encoding network model to perform feature encoding on the target frame and its neighboring frames to obtain feature maps of the target frame and the neighboring frames. The feature dimension of the encoded feature map is larger than the feature dimension of the original video frame. For example, in a specific example, a three-channel RGB video frame is input into the feature encoding network model, resulting in a 128-channel feature map.

[0099] Exemplarily, the feature encoding network model is a network model composed of multiple layers of ResNet (Residual Network) modules, such as a network model composed of 5 layers of ResNet modules. Those skilled in the art can understand that, without affecting the implementation of the present disclosure, the feature encoding network used in the present disclosure can also use other network model structures, such as an autoencoder or a residual dense network (RDN, full name: Residual Dense Network), etc., in addition to the network model composed of 5 layers of ResNet modules.

[0100] In the embodiments of the present disclosure, the feature encoding module encodes the target frame and its adjacent frames, which can realize feature dimensionality increase, extract more abundant image detail information, and further help improve the effect of image super-resolution processing.

[0101] The first constructing module 520 is configured to, for each pixel in the feature map of the target frame, determine a pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel, and construct a first reference feature map according to the pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel.

[0102] In some embodiments, for each pixel in the feature map of the target frame, the first constructing module 520 determines the correlation of the pixel with its neighborhood pixels in the feature map of the adjacent frame, and selects the pixel with the greatest correlation from the neighborhood pixels according to the correlation. In these embodiments, the first constructing module 520 greatly reduces the amount of calculation by only calculating the correlation of the pixel in the target frame with its neighborhood pixels in the adjacent frame, compared with calculating the pixel correlation of the target frame and the adjacent frame one by one in other embodiments.

[0103] In some embodiments, for each pixel in the feature map of the target frame, the first constructing module 520 determines the first reference feature corresponding to the pixel according to the feature of the pixel with the greatest correlation selected from the feature map of the adjacent frame and the corresponding correlation, and constructs the first reference feature map according to the first reference feature corresponding to each pixel in the feature map of the target frame.

[0104] In an optional implementation, for each pixel in the feature map of the target frame, the first constructing module 520 takes the product of the feature of the pixel with the greatest correlation selected from the feature map of the adjacent frame and the corresponding correlation as the first reference feature corresponding to the pixel. In this way, the first reference feature corresponding to each pixel in the feature map of the target frame can be obtained, and further, the first reference feature map can be obtained. The pixel feature in the first reference feature map is the first reference feature corresponding to the pixel in the feature map of the target frame.

[0105] Those skilled in the art can understand that the product of the pixel feature with the largest correlation degree and the corresponding correlation degree is taken as the first reference feature, which is only used for example. Without affecting the implementation of the present application, the first construction module 520 can also use other specific implementations of determining the first reference feature according to the pixel feature with the largest correlation degree and the corresponding correlation degree.

[0106] In the embodiments of the present disclosure, the pixel correlation degrees between the target frame and the adjacent frame feature maps are calculated by the first construction module, and the first feature map is constructed according to the pixel with the largest correlation degree selected from the adjacent frame feature map, so that the present disclosure does not need to calculate the pixel correspondence relationship between the video frames based on the optical flow method and the like, thereby solving the problems of poor calculation accuracy and high requirement for computing resources and slow speed caused by relying on the optical flow method and the like to calculate the image correspondence relationship between the video frames in the related art, and helping to improve the video super-resolution processing efficiency, reduce the requirement for computing resources of the video super-resolution processing, and improve the quality of the video reconstructed by super-resolution.

[0107] The reconstruction module 530 is configured to perform super-resolution reconstruction according to the feature map of the target frame and the first reference feature map.

[0108] In some embodiments, the reconstruction module 530 inputs the feature map of the target frame and the first reference feature map into a feature decoding network model, and takes the image output by the model as the super-resolution reconstructed image, i.e., the high-resolution image corresponding to the finally reconstructed target frame. The decoding network model is used to fuse the feature map of the target frame and the first reference feature map, and map the fused image from a high-dimensional feature space to a low-dimensional feature space to obtain a high-resolution reconstructed image. For example, in a specific example, the 128-channel target frame feature map and the 128-channel first reference feature map are input into the feature decoding network model to obtain a high-resolution three-channel RGB image.

[0109] For example, the feature decoding network model is a network model composed of a multi-layer ResNet module and an up-sampling module, such as a network model composed of a 40-layer ResNet module and an up-sampling module. Those skilled in the art can understand that without affecting the implementation of the present disclosure, the feature decoding network used in the present disclosure can also use other network model structures besides the network model composed of a 40-layer ResNet module and an up-sampling module, such as RDN and the like.

[0110] In some embodiments, the reconstruction module 530 inputs the feature map of the target frame and the first reference feature map into a feature decoding network model to output a first reconstructed image; and superimposes a feature map obtained by performing bilinear interpolation processing on the feature map of the target frame on the first reconstructed image to obtain a second reconstructed image, and takes the second reconstructed image as a high-resolution image corresponding to the target frame reconstructed finally. In these embodiments, after obtaining the high-resolution reconstructed image through the feature decoding network model, the resolution of the reconstructed image can be further improved by superimposing the feature map obtained by performing bilinear interpolation on the feature map of the target frame.

[0111] In the embodiments of the present disclosure, the above device realizes super-resolution processing of a target frame of a video. Compared with related technologies, the video super-resolution processing efficiency can be improved, the requirement of the video super-resolution processing on computing resources can be reduced, and the quality of the video reconstructed by super-resolution can be improved.

[0112] Figure 6 FIG. 6 is a structural schematic diagram of another embodiment of the video processing device of the present disclosure. As shown in FIG. 6, the video processing device in the embodiments of the present disclosure includes a feature encoding module 610, a first construction module 620, a second construction module 630, and a reconstruction module 640. Figure 6

[0113] The feature encoding module 610 is configured to perform feature encoding on a target frame and its adjacent frames of a video to obtain feature maps of the target frame and its adjacent frames.

[0114] In this embodiment, the feature encoding module 610 performs feature encoding on the target frame and its adjacent frames by using a feature encoding network model to obtain the feature map of the target frame and the feature maps of the adjacent frames. The feature dimension of the obtained feature map is greater than the feature dimension of the original video frame. For example, in a specific example, a three-channel RGB video frame is input into the feature encoding network model to obtain a 128-channel feature map.

[0115] In this embodiment, by performing feature encoding on the target frame and its adjacent frames through the feature encoding module 610, feature dimension can be increased, richer image detail information can be extracted, and the effect of image super-resolution processing can be improved.

[0116] The first construction module 620 is configured to determine, for each pixel in the feature map of the target frame, a pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel, and construct a first reference feature map according to the pixel in the feature map of the adjacent frame that has the greatest correlation with the pixel.

[0117] ​In some embodiments, for each pixel in the feature map of the target frame, the first constructing module 620 determines the correlation of the pixel with its neighborhood pixels in the feature map of the adjacent frame, and selects the pixel with the highest correlation from the neighborhood pixels according to the correlation. By only calculating the correlation of the pixel in the target frame with its neighborhood pixels in the adjacent frame, the calculation amount is greatly reduced compared with calculating the pixel correlation of the target frame and the adjacent frame one by one.

[0118] In some embodiments, for each pixel in the feature map of the target frame, the first constructing module 620 selects the pixel with the highest correlation from the neighborhood pixels in the feature map of the adjacent frame, and the product of the feature of the pixel and the corresponding correlation as the first reference feature corresponding to the pixel. In this way, the first reference feature corresponding to each pixel in the feature map of the target frame can be obtained, and then the first reference feature map can be obtained accordingly. The pixel feature in the first reference feature map is the first reference feature corresponding to the pixel in the feature map of the target frame.

[0119] The second constructing module 630 is configured to construct a second reference feature map. The second reference feature map fuses the general pixel features.

[0120] The general pixel features are image detail information learned from high-resolution videos. In some embodiments, the general pixel features are stored in the form of a two-dimensional matrix. For example, a two-dimensional matrix of 256*128 is used, which is composed of 256 feature vectors of 128 channels. Each feature vector stores the most representative general pixel features learned from high-resolution videos in the training process. In these embodiments, the second reference feature map is constructed according to the two-dimensional matrix.

[0121] In some embodiments, the second constructing module 630 determines the correlation of the pixel in the feature map of the target frame with a plurality of general pixel features, fuses the plurality of general pixel features according to the correlation to obtain the second reference feature corresponding to the pixel, and constructs the second reference feature map according to the second reference feature corresponding to the pixel.

[0122] In some embodiments, the second constructing module 630 normalizes the obtained correlation degrees, performs a weighted average operation on the plurality of general pixel features by using the normalized correlation degrees as weights, and takes the result of the weighted average operation as the second reference feature corresponding to the pixel in the feature map of the target frame. For example, a two-dimensional matrix of 256*128 is used, which is composed of 256 feature vectors of 128 channels, and each feature vector stores the most representative general pixel feature learned from the known high-resolution video image and the corresponding low-resolution image in the training process. For each pixel in the feature map of the target frame, the correlation degrees between the pixel and the 256 feature vectors in the two-dimensional matrix are calculated to obtain 256 correlation values. After normalizing the 256 correlation values, the normalized correlation values are used as weights to perform a weighted average operation on the 256 feature vectors, and the result of the weighted average operation is taken as the second reference feature corresponding to the pixel.

[0123] The reconstruction module 640 is configured to perform super-resolution reconstruction according to the feature map of the target frame, the first reference feature map, and the second reference feature map.

[0124] In some embodiments, the reconstruction module 640 inputs the feature map of the target frame, the first reference feature map, and the second reference feature map into a feature decoding network model, and takes the image output by the model as the super-resolution reconstructed image. The decoding network model is used to fuse the feature map of the target frame, the first reference feature map, and the second reference feature map, and to map the fused image from a high-dimensional feature space to a low-dimensional feature space to obtain a high-resolution reconstructed image. For example, in a specific example, the 128-channel target frame feature map, the 128-channel first reference feature map, and the 128-channel second reference feature map are input into the feature decoding network model to obtain a high-resolution three-channel RGB image.

[0125] In other embodiments, the reconstruction module 640 inputs the feature map of the target frame, the first reference feature map, and the second reference feature map into a feature decoding network model to output a first reconstructed image, superimposes a feature map obtained by performing bilinear interpolation on the feature map of the target frame on the first reconstructed image to obtain a second reconstructed image, and takes the second reconstructed image as the final super-resolution reconstructed image. In these embodiments, after obtaining a high-resolution reconstructed image by the feature decoding network model, the image resolution can be further improved by superimposing the high-resolution reconstructed image on the feature map obtained by performing bilinear interpolation on the feature map of the target frame.

[0126] Optionally, the embodiments of the present disclosure further comprise: training each module in the video processing device before performing super-resolution processing on the current video based on the video processing device. Generally, the training process can be performed in an end-to-end manner, and the entire network training process is as follows: first, disconnect the second construction module, and only train the other three modules, with the loss function being the L1 distance between the output high-resolution image and the correctly labeled high-resolution sample image; next, fix the other three modules, and only train the two-dimensional matrix used by the second construction module, with the loss function being the L1 distance between the input and output of the network model used to train the two-dimensional matrix; then, train all modules in the video processing device, with the loss function being the L1 distance between the output high-resolution image and the correctly labeled high-resolution sample image.

[0127] In the embodiments of the present disclosure, the above device realizes super-resolution processing on the target frame of the video. Compared with the related art, by introducing a cross-frame attention mechanism when fusing the image details of adjacent frames, the matching between the adjacent frames and the target frame at the pixel level is realized, so that the present disclosure does not need to explicitly calculate the pixel correspondence between the video frames based on the optical flow method and the like, thereby solving the problems of poor calculation accuracy and high requirement for computing resources and slow speed caused by relying on the optical flow method and the like to calculate the image correspondence between the video frames in the related art, which helps to improve the video super-resolution processing efficiency, reduce the requirement for computing resources of video super-resolution processing, and improve the quality of the video reconstructed by super-resolution. Further, in the embodiments of the present disclosure, the super-resolution reconstruction is performed according to the feature map of the target frame, the first reference feature map, and the second reference feature map, so that not only the image details of adjacent frames are fused, but also the image details of general pixels are fused when performing super-resolution reconstruction on the target frame, thereby being able to increase the image details to a certain extent and improve the quality of the video reconstructed by super-resolution.

[0128] Figure 8 The structure diagram of another embodiment of the video processing device of the present disclosure. The device includes a memory 810 and a processor 820, wherein: the memory 810 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiment. The processor 820 is coupled to the memory 810 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 820 is used to execute the instructions stored in the memory. Figures 1-4 The instructions in the corresponding embodiment. The processor 820 is coupled to the memory 810 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 820 is used to execute the instructions stored in the memory.

[0129] In some embodiments, the video processing device can also be implemented as a system-on-chip (SoC) as shown in FIG. 8B. In this case, the video processing device can include a plurality of processors 820, a memory 810, and a bus 830. The bus 830 is used to connect the plurality of processors 820 and the memory 810, and can be implemented as a system bus, a point-to-point connection, a multi-drop bus, or any other suitable connection. The plurality of processors 820 can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The plurality of processors 820 is used to execute the instructions stored in the memory 810. Figure 9As shown, the apparatus 900 includes a memory 910 and a processor 920. The processor 920 is coupled to the memory 910 through a BUS 930. The apparatus 900 can also be connected to an external storage device 950 through a storage interface 940 to invoke external data, and can also be connected to a network or another computer system (not shown) through a network interface 960, which will not be described in detail here.

[0130] In this embodiment, the data instruction is stored in the memory, and the above instruction is processed by the processor, so that the video processing efficiency can be improved.

[0131] In some other embodiments, a computer readable storage medium has computer program instructions stored thereon, which, when executed by a processor, implement the steps of the method in the corresponding embodiment. Those skilled in the art should understand that the embodiments of the disclosure can be provided as a method, an apparatus, or a computer program product. Therefore, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. Figures 1-3 The steps of the method in the corresponding embodiment. Those skilled in the art should understand that the embodiments of the disclosure can be provided as a method, an apparatus, or a computer program product. Therefore, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the disclosure can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0132] The disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flow Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0133] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the functions specified in the flow Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0134] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide the function of implementing the processes specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 Figure 1 one block or multiple blocks.

[0135] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0136] Although some specific embodiments of the present disclosure have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A video processing method, comprising: Feature encoding is performed on the target frame and its neighboring frames in the video to obtain the feature map of the target frame and its neighboring frames; For each pixel in the feature map of the target frame, determine the pixel in the feature map of the adjacent frame that has the highest correlation with the pixel; A first reference feature map is constructed based on the pixel with the highest correlation to the pixel in the feature map of the adjacent frames. Super-resolution reconstruction is performed based on the feature map of the target frame, the first reference feature map, and the second reference feature map. The second reference feature map incorporates general pixel features, which are image detail information learned from other videos during the training phase.

2. The video processing method according to claim 1, wherein, The pixels with the highest correlation to the pixel in the feature map of the adjacent frames include: Determine the correlation between the pixel and its neighboring pixels in the feature map of the adjacent frame; Based on the correlation, the pixel with the highest correlation is selected from the neighboring pixels.

3. The video processing method according to claim 1, wherein, Constructing a first reference feature map based on the pixel with the highest correlation to the pixel in the feature map of the adjacent frames includes: Based on the features of the pixel with the highest correlation to the pixel in the feature map of the adjacent frame, and the corresponding correlation, the first reference feature corresponding to the pixel is determined; A first reference feature map is constructed based on the first reference feature corresponding to the pixel.

4. The video processing method according to claim 1, further comprising: Determine the correlation between pixels in the feature map of the target frame and multiple common pixel features; The multiple general pixel features are fused according to the correlation to obtain the second reference feature corresponding to the pixel; The second reference feature map is constructed based on the second reference feature corresponding to the pixel.

5. The video processing method according to claim 4, wherein, The multiple general pixel features are fused according to the aforementioned correlation to obtain the second reference feature corresponding to the pixel, including: The correlation is normalized; The normalized relevance is used as a weight to perform a weighted average operation on the multiple general pixel features, and the result of the weighted average operation is used as the second reference feature corresponding to the pixel.

6. The video processing method according to claim 1, wherein, Super-resolution reconstruction based on the feature map of the target frame, the first reference feature map, and the second reference feature map includes: The feature map of the target frame, the first reference feature map, and the second reference feature map are input into the feature decoding network model to output the first reconstructed image; The first reconstructed image and the feature map obtained by bilinear interpolation of the feature map of the target frame are superimposed to obtain the second reconstructed image.

7. The video processing method according to claim 1, wherein, The network model used for feature encoding includes a multi-layer ResNet module.

8. The video processing method according to claim 6, wherein, The feature decoding network model includes a multi-layer ResNet module and an upsampling module.

9. A video processing apparatus, comprising: The feature encoding module is configured to perform feature encoding on the target frame and its neighboring frames of the video to obtain feature maps of the target frame and its neighboring frames. The first building module is configured as For each pixel in the feature map of the target frame, determine the pixel in the feature map of the adjacent frame that has the highest correlation with the target pixel, and A first reference feature map is constructed based on the pixel with the highest correlation to the pixel in the feature map of the adjacent frames. as well as The reconstruction module is configured to perform super-resolution reconstruction based on the feature map of the target frame, the first reference feature map, and the second reference feature map, wherein the second reference feature map incorporates general pixel features, which are image detail information learned from other videos during the training phase.

10. The video processing apparatus according to claim 9, wherein, The first building module is configured as follows: Determine the correlation between the pixel and its neighboring pixels in the feature map of the adjacent frame; Based on the correlation, the pixel with the highest correlation is selected from the neighboring pixels.

11. The video processing apparatus according to claim 9, wherein, The first building module is configured as follows: Based on the features of the pixel with the highest correlation to the pixel in the feature map of the adjacent frame, and the corresponding correlation, the first reference feature corresponding to the pixel is determined; A first reference feature map is constructed based on the first reference feature corresponding to the pixel.

12. A video processing apparatus, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the video processing method as described in any one of claims 1 to 8 based on instructions stored in the memory.

13. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the video processing method according to any one of claims 1 to 8.

Images