Video super-resolution method, device, electronic device, and storage medium

By calculating the similarity between hidden layer features and image features in the video super-resolution network and adjusting the hidden layer features, the problem that the previous moment features in the cyclic convolutional neural network has no reference significance for the next moment, and the video super-resolution is improved Rate processing effect.

CN114549327BActive Publication Date: 2024-11-01BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210270777.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-11-01
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

In video super-resolution processing, the cyclic convolutional neural network has no reference significance for the next moment, resulting in poor video super-resolution processing, especially when the degree of information correlation between adjacent frames is small. The characteristics of the moment may become the result of noise impact processing.

Method used

By obtaining the hidden layer features of the current frame image, extracting the image features of the next frame image, calculating the feature similarity between the two, and adjusting the hidden layer features according to the similarity to suppress irrelevant information and paying attention to related information, thereby improving the video Super resolution effect.

Benefits of technology

By adjusting the feature similarity, irrelevant information between the current frame and the next frame is suppressed, and relevant information is paid attention to, which helps the network make full use of the timing information of multiple frames in the video, improving the video super-resolution processing effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549327B_ABST
    Figure CN114549327B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a video super-resolution method, device, electronic device, storage medium and computer program product, the method comprising: obtaining hidden layer features corresponding to the current frame image, the hidden layer features are features output by the hidden layer network in the video super-resolution network when processing the current frame image, extracting image features corresponding to the next frame image in the video of the current frame image, determining feature similarity between the hidden layer features and the image features, the feature similarity is used to characterize the similarity between features corresponding to the same pixel position in the hidden layer features and the image features, adjusting the hidden layer features according to the feature similarity, obtaining adjusted hidden layer features, and the adjusted hidden layer features are used for the video super-resolution network to perform super-resolution processing on the video. The method can suppress irrelevant information between adjacent frames and focus on relevant information, which helps the network to fully utilize the temporal information of multiple frames in the video to complete the video processing task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video super-resolution method, device, electronic device, storage medium and computer program product. Background Art

[0002] In the process of using convolutional neural networks to process videos for super-resolution, it is necessary to use the time sequence information between frames in the video. By transmitting features at different moments, the neural network can establish long-distance dependencies between features in time sequence, which helps the final task goal. Since the recurrent convolutional neural network uses the features of the previous moment as the input of the next moment, when the information correlation between two adjacent moments is small, the features of the previous moment have no reference significance for the next moment, and may even become noise that affects the processing results of the next moment.

[0003] Therefore, there is a problem in the related art that the video super-resolution processing effect is poor. Summary of the invention

[0004] The present disclosure provides a video super-resolution method, device, electronic device, storage medium and computer program product to at least solve the problem of poor video super-resolution processing effect in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video super-resolution method is provided, comprising:

[0006] Obtaining hidden layer features corresponding to the current frame image; the hidden layer features are features output by a hidden layer network in a video super-resolution network processing the current frame image; the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs;

[0007] Extracting image features corresponding to the next frame image of the current frame image in the video, and determining feature similarity between the hidden layer features and the image features; the feature similarity is used to characterize the similarity between features corresponding to the same pixel position in the hidden layer features and the image features;

[0008] The hidden layer features are adjusted according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used by the video super-resolution network to perform super-resolution processing on the video.

[0009] In a possible implementation, determining the feature similarity between the hidden layer feature and the image feature includes:

[0010] Obtaining pixel features corresponding to each pixel point in the image features;

[0011] According to the pixel features corresponding to the pixels, the features to be compared corresponding to the pixels are determined in the hidden layer features;

[0012] Determine the similarity comparison result corresponding to each pixel point according to the pixel feature and the feature to be compared;

[0013] The similarity comparison result corresponding to each of the pixel points is used as the feature similarity.

[0014] In a possible implementation manner, the image feature has multiple feature planes, and obtaining a pixel feature corresponding to each pixel point in the image feature includes:

[0015] Taking any pixel point in the image feature as a target pixel point;

[0016] Obtaining pixel sub-features corresponding to the target pixel point on each feature plane of the image feature;

[0017] The pixel sub-features are rearranged to obtain the pixel feature corresponding to the target pixel.

[0018] In a possible implementation, the hidden layer feature has a plurality of feature planes, and determining the feature to be compared corresponding to each pixel point in the hidden layer feature according to the pixel feature corresponding to each pixel point includes:

[0019] Determine a reference pixel point in the hidden layer feature; the pixel position of the reference pixel point is the same as the pixel position of the target pixel point;

[0020] Determining the features of the area to be compared in each feature plane of the hidden layer features according to the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel;

[0021] The features of the area to be compared are used as the features to be compared corresponding to the target pixel points.

[0022] In a possible implementation, determining the similarity comparison result corresponding to each pixel point according to the pixel feature and the feature to be compared includes:

[0023] Using the pixel features of each pixel as a convolution kernel;

[0024] The convolution kernel is used to perform a convolution operation on the features to be compared corresponding to each of the pixel points to obtain a convolution value as a similarity comparison result corresponding to each of the pixel points.

[0025] In a possible implementation, adjusting the hidden layer features according to the feature similarity to obtain adjusted hidden layer features includes:

[0026] Inputting the feature similarity into a preset activation function to obtain a feature adjustment weight for the hidden layer feature; a value corresponding to the feature adjustment weight is positively correlated with the feature similarity;

[0027] The hidden layer features are weighted using the feature adjustment weights to obtain the adjusted hidden layer features.

[0028] In a possible implementation, extracting image features corresponding to the next frame image of the current frame image in the video includes:

[0029] Obtain the next frame image of the current frame image in the video;

[0030] The next frame image is input into a preset image feature extraction network to obtain image features corresponding to the next frame image.

[0031] According to a second aspect of an embodiment of the present disclosure, a video super-resolution device is provided, including:

[0032] A feature acquisition unit is configured to acquire hidden layer features corresponding to the current frame image; the hidden layer features are features output by a hidden layer network in a video super-resolution network when processing the current frame image; the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs;

[0033] A similarity determination unit is configured to extract image features corresponding to the next frame image of the current frame image in the video, and determine feature similarity between the hidden layer features and the image features; the feature similarity is used to characterize the similarity between features corresponding to the same pixel position in the hidden layer features and the image features;

[0034] The feature adjustment unit is configured to adjust the hidden layer features according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used by the video super-resolution network to perform super-resolution processing on the video.

[0035] In a possible implementation, the similarity determination unit is specifically configured to execute the acquisition of pixel features corresponding to each pixel in the image features; determine the features to be compared corresponding to each pixel in the hidden layer features according to the pixel features corresponding to each pixel; determine the similarity comparison results corresponding to each pixel according to the pixel features and the features to be compared; and use the similarity comparison results corresponding to each pixel as the feature similarity.

[0036] In one possible implementation, the image feature has multiple feature planes, and the similarity determination unit is specifically configured to execute the steps of taking any pixel point in the image feature as a target pixel point; obtaining pixel sub-features corresponding to the target pixel point in each feature plane of the image feature; and rearranging each of the pixel sub-features to obtain a pixel feature corresponding to the target pixel point.

[0037] In one possible implementation, the hidden layer feature has multiple feature planes, and the similarity determination unit is specifically configured to determine a reference pixel in the hidden layer feature; the pixel position of the reference pixel is the same as the pixel position of the target pixel; based on the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel, the feature of the area to be compared is determined in each feature plane of the hidden layer feature; and the feature of the area to be compared is used as the feature to be compared corresponding to the target pixel.

[0038] In one possible implementation, the similarity determination unit is specifically configured to execute using the pixel features of each of the pixel points as a convolution kernel; using the convolution kernel to perform a convolution operation on the features to be compared corresponding to each of the pixel points, and obtaining a convolution value as a similarity comparison result corresponding to each of the pixel points.

[0039] In one possible implementation, the feature adjustment unit is specifically configured to execute inputting the feature similarity into a preset activation function to obtain a feature adjustment weight for the hidden layer feature; the value corresponding to the feature adjustment weight is positively correlated with the feature similarity; and the hidden layer feature is weightedly processed using the feature adjustment weight to obtain the adjusted hidden layer feature.

[0040] In a possible implementation, the similarity determination unit is specifically configured to execute the acquisition of the next frame image of the current frame image in the video; input the next frame image into a preset image feature extraction network to obtain image features corresponding to the next frame image.

[0041] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0042] processor;

[0043] a memory for storing instructions executable by the processor;

[0044] The processor is configured to execute the instructions to implement the video super-resolution method as described in any one of the above items.

[0045] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video super-resolution method as described in any one of the above items.

[0046] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the video super-resolution method as described in any one of the above items.

[0047] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0048] The scheme disclosed in the present invention obtains the hidden layer features corresponding to the current frame image, the hidden layer features are the features output by the hidden layer network in the video super-resolution network when processing the current frame image, the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs, and then extracts the image features corresponding to the next frame image in the video of the current frame image, determines the feature similarity between the hidden layer features and the image features, and the feature similarity is used to characterize the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features, and then adjusts the hidden layer features according to the feature similarity to obtain the adjusted hidden layer features, and the adjusted hidden layer features are used for the video super-resolution network to perform super-resolution processing on the video. In this way, the hidden layer features corresponding to the current frame image can be adjusted based on the feature similarity obtained by comparing the features between the current frame image and the next frame image, so as to suppress irrelevant information between the current frame and the next frame, and focus on the relevant information between the current frame and the next frame, which helps the network to fully utilize the temporal information of multiple frames in the video to complete the video processing task, and improves the video super-resolution effect.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0051] Figure 1 The figure is a flowchart of a video super-resolution method according to an exemplary embodiment.

[0052] Figure 2 The figure is a schematic diagram showing a process of comparing feature similarity according to an exemplary embodiment.

[0053] Figure 3a The figure is a schematic diagram showing a video processing effect according to an exemplary embodiment.

[0054] Figure 3b is a schematic diagram showing another video processing effect according to an exemplary embodiment.

[0055] Figure 4 The figure is a flowchart of another video super-resolution method according to an exemplary embodiment.

[0056] Figure 5 The figure is a block diagram of a video super-resolution device according to an exemplary embodiment.

[0057] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment.

[0058] Figure 7 is a block diagram of another electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure.

[0061] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0062] Figure 1 This is a flow chart of a video super-resolution method according to an exemplary embodiment. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps.

[0063] In step S110, the hidden layer features corresponding to the current frame image are obtained;

[0064] Among them, the hidden layer features can be features output by the hidden layer network in the video super-resolution network processing the current frame image, and the video super-resolution network can be a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs.

[0065] As an example, a circular convolutional network (i.e., a video super-resolution network) can be used to perform video super-resolution tasks. Taking the current frame image in the video as an example, the current frame image, the previous frame image, the previous frame hidden layer features, and the super-resolution result corresponding to the previous frame image can be input respectively, and the input objects can be spliced ​​in the feature dimension. Then, the spliced ​​features can be input into the hidden layer network in the video super-resolution network for processing, and then the hidden layer network can output the hidden layer features corresponding to the current frame image.

[0066] In practical applications, after processing the current frame image through the hidden layer network in the video super-resolution network, the hidden layer features corresponding to the current frame image output by the hidden layer network can be obtained, so as to further use the hidden layer features to perform similarity comparison with the image features corresponding to the next frame image.

[0067] Specifically, a dynamic filter based on feature similarity comparison can be constructed to filter the hidden layer features corresponding to the acquired current frame image to obtain filtered hidden layer features, such as a filtered hidden layer feature matrix, as adjusted hidden layer features.

[0068] In one example, after the splicing features obtained based on the current frame image are input into the hidden layer network in the video super-resolution network, they can be processed through n-1 convolutional layers of size 3x3 and activation function ReLU (Rectified Linear Units), and then divided into two branches at the nth layer. One branch can use a 3x3 convolutional layer and activation function ReLU to generate hidden layer features, and the other branch can use a 3x3 convolutional layer to generate super-resolution features. The generated hidden layer features can be used as input for the next frame image processing, and the super-resolution features can be obtained by 4 times pixel rearrangement to obtain super-resolution results. It can be seen that the hidden layer features corresponding to the image at time t are accumulated from the hidden layer features corresponding to times 1 to t-1, so that long-distance temporal dependencies can be established using hidden layer features.

[0069] In another example, due to the shortcomings of the hidden layer feature accumulation mechanism, the hidden layer features of the previous moment will be passed to the next moment without reservation. If there is a sudden large movement or scene switching between adjacent frames of the video, the image content of the previous moment will change drastically relative to the image content of the next moment. In this case, the hidden layer features corresponding to the image of the previous moment have no reference significance for the image of the next moment, and may even be a kind of noise. For example, if the scene is a cat in the image of the previous moment and a car in the image of the next moment, then the texture and detail features of the cat have a low correlation with the car. If the features of the cat and the image of the car are spliced ​​together and input into the network, they will have no reference significance for the feature extraction of the car. It is necessary to suppress irrelevant information between adjacent frames and pay attention to relevant information between adjacent frames. At the same time, for the video super-resolution task, in order to try to ensure the original features of the image, it is necessary to process the image features without compressing them too much.

[0070] In step S120, the image features corresponding to the next frame image of the current frame image in the video are extracted to determine the feature similarity between the hidden layer features and the image features; the feature similarity is used to characterize the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features;

[0071] In a specific implementation, the next frame image of the current frame image in the video can be obtained by acquiring the next frame image for feature extraction, and the image features corresponding to the next frame image can be obtained. Then, the hidden layer features and the image features can be used for similarity comparison. By comparing the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features, the feature similarity between the hidden layer features and the image features can be determined.

[0072] For example, Figure 2 As shown, the size of the input image (i.e., the next frame image) can be HxWx3, and a convolutional layer of size 3x3 and an activation function (such as Figure 2 The activation function 1) extracts features from the input image and obtains image features as shown in the cube, whose size is HxWxK 2 , K is a hyperparameter, which can be used to characterize the size of the filter radius of dynamic filtering; by comparing the similarity between the extracted image features and the hidden layer features corresponding to the current frame image (whose size is HxWxC), the similarity matrix (i.e., feature similarity) can be obtained.

[0073] In an optional embodiment, considering the balance between performance and speed, the value of the hyperparameter K can be a smaller value, such as K=3, to achieve better computing efficiency and accuracy. A larger K value may reduce computing efficiency. The value of the hyperparameter K can also be adjusted according to different application scenarios.

[0074] In step S130, the hidden layer features are adjusted according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used for the video super-resolution network to perform super-resolution processing on the video.

[0075] After obtaining the feature similarity, the hidden layer features can be adjusted according to the feature similarity to obtain the adjusted hidden layer features, so that the video super-resolution network can perform super-resolution processing on the video based on the adjusted hidden layer features. For example, the adjusted hidden layer features can be used as the hidden layer features to be input, so as to be input into the hidden layer network when the hidden layer network processes the next frame of image.

[0076] In one example, if Figure 2 As shown, the value corresponding to the similarity matrix can represent the correlation between the hidden layer features corresponding to the current frame image and the image features corresponding to the next frame image. By applying the Sigmoid activation function (such as Figure 2 Activation function 2), the output of the activation function can be multiplied by the hidden layer features corresponding to the current frame image, and then the filtered hidden layer feature matrix (i.e., the adjusted hidden layer features) can be obtained.

[0077] In another example, Figure 2 The dynamic filter based on feature similarity comparison shown in FIG. 1 can be used as an insertion-shaped module, such as a hidden layer adaptive module for video super-resolution tasks, which can be inserted at the hidden layer feature input of any recurrent convolutional neural network to filter the features. By suppressing irrelevant information between the current frame and the next frame and focusing on relevant information between the current frame and the next frame, this module can greatly help the network make full use of the temporal information of multiple frames in the video, such as Figure 3a and 3b As shown, the last column is the processing result of this embodiment. It can be seen that better video super-resolution results are achieved in texture recovery, such as more texture details are produced in the reconstruction of buildings, and richer textures are also produced in the reconstruction of trees. Therefore, by conveniently inserting the module into any circular convolution algorithm network, it can help the algorithm network to obtain better performance.

[0078] In an optional embodiment, the method of this embodiment can also be used for video classification processing tasks, such as by obtaining the hidden layer features corresponding to the current frame image, the hidden layer features are the features output by the hidden layer network in the video classification processing network when processing the current frame image, the video classification processing network is a convolutional neural network that classifies the video to which the current frame image belongs, and then extracting the image features corresponding to the next frame image of the current frame image in the video, determining the feature similarity between the hidden layer features and the image features, the feature similarity is used to characterize the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features, and then adjusting the hidden layer features according to the feature similarity to obtain the adjusted hidden layer features, and the adjusted hidden layer features are used for the video classification processing network to classify the video; it can also be used for other video processing tasks based on recurrent convolutional neural networks, which are not specifically limited in this embodiment.

[0079] In the above-mentioned video super-resolution method, by obtaining the hidden layer features corresponding to the current frame image, the hidden layer features are the features output by the hidden layer network in the video super-resolution network when processing the current frame image, and the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs, and then extracting the image features corresponding to the next frame image in the video of the current frame image, determining the feature similarity between the hidden layer features and the image features, and the feature similarity is used to characterize the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features, and then adjusting the hidden layer features according to the feature similarity to obtain the adjusted hidden layer features, and the adjusted hidden layer features are used for the video super-resolution network to perform super-resolution processing on the video. In this way, the hidden layer features corresponding to the current frame image can be adjusted based on the feature similarity obtained by comparing the features between the current frame image and the next frame image, so that the irrelevant information between the current frame and the next frame can be suppressed, and the relevant information between the current frame and the next frame can be focused on, which helps the network to fully utilize the temporal information of multiple frames in the video to complete the video processing task and improve the video super-resolution effect.

[0080] In an exemplary embodiment, determining the feature similarity between the hidden layer features and the image features includes: obtaining pixel features corresponding to each pixel in the image features; determining the features to be compared corresponding to each pixel in the hidden layer features according to the pixel features corresponding to each pixel; determining the similarity comparison results corresponding to each pixel according to the pixel features and the features to be compared; and using the similarity comparison results corresponding to each pixel as the feature similarity.

[0081] In practical applications, such as Figure 2 As shown, the pixel features corresponding to each pixel in the image feature can be obtained, and then the pixel features corresponding to each pixel can be obtained, such as the 1x1xK 2, which can obtain KxKx1 features through spatial rearrangement, determine the features to be compared corresponding to each pixel in the hidden layer features, such as the features to be compared can be obtained in the hidden layer features HxWxC according to the rearranged KxKx1 features and the pixel point A' position corresponding to the pixel point A in the hidden layer features, and then the similarity comparison results corresponding to each pixel can be determined according to the pixel features and the features to be compared, such as the similarity comparison results of the pixel point A in the similarity matrix, and the similarity comparison results corresponding to each pixel point are used as feature similarity, such as the similarity matrix; thereby, the feature similarity between the hidden layer features and the image features can be obtained by comparing the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features.

[0082] The technical solution of this embodiment obtains the pixel features corresponding to each pixel in the image features, and then determines the features to be compared corresponding to each pixel in the hidden layer features according to the pixel features corresponding to each pixel, and then determines the similarity comparison results corresponding to each pixel according to the pixel features and the features to be compared, and uses the similarity comparison results corresponding to each pixel as the feature similarity. The image features can be decomposed into the features corresponding to each pixel for similarity comparison, which can ensure the original features of the image while obtaining the feature similarity between adjacent frames in the video, thereby improving the video super-resolution effect.

[0083] In an exemplary embodiment, an image feature may have multiple feature planes, and obtaining pixel features corresponding to each pixel point in the image feature includes: taking any pixel point in the image feature as a target pixel point; obtaining pixel sub-features corresponding to the target pixel point in each feature plane of the image feature; and rearranging each pixel sub-feature to obtain the pixel feature corresponding to the target pixel point.

[0084] In the specific implementation, Figure 2 As shown, any pixel point on the HxW feature plane in the image feature can be selected as the target pixel point, such as pixel point A. Since the image feature is a cube feature, the pixel point A on the HxWxK plane can be obtained. 2 The pixel sub-features corresponding to each feature plane in the , such as the dimension of pixel point A can be 1x1xK 2 , and then the pixel sub-features can be spatially rearranged to obtain the pixel feature corresponding to the target pixel, such as the feature of KxKx1, which can be the dynamic filter kernel of pixel A.

[0085] The technical solution of this embodiment is to use any pixel point in the image feature as the target pixel point, then obtain the pixel sub-features corresponding to the target pixel point in each feature plane of the image feature, and then rearrange each pixel sub-feature to obtain the pixel feature corresponding to the target pixel point. The image feature can be disassembled to obtain the pixel feature corresponding to each pixel point, thereby providing data support for subsequent processing.

[0086] In an exemplary embodiment, the hidden layer feature may have multiple feature planes, and the features to be compared corresponding to each pixel point are determined in the hidden layer feature according to the pixel features corresponding to each pixel point, including: determining a reference pixel point in the hidden layer feature; the pixel position of the reference pixel point is the same as the pixel position of the target pixel point; determining the features of the area to be compared in each feature plane of the hidden layer feature according to the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel point; and using the features of the area to be compared as the features to be compared corresponding to the target pixel point.

[0087] In one example, if Figure 2 As shown, for pixel point A (i.e., target pixel point) in the image feature, pixel point A' (i.e., reference pixel point) with the same pixel position as pixel point A can be determined in the hidden layer feature, and then the corresponding feature to be compared can be obtained in the hidden layer feature HxWxC based on the dynamic filter kernel of pixel point A and the position of pixel point A'. For example, since the hidden layer feature HxWxC has C feature planes, the dynamic filter kernel of pixel point A is shared for each feature plane in the hidden layer feature, and C values ​​(i.e., the feature of the area to be compared) can also be generated for pixel point A' in the hidden layer feature.

[0088] The technical solution of this embodiment determines the reference pixel points in the hidden layer features, and then determines the features of the area to be compared in each feature plane of the hidden layer features according to the number of pixel sub-features in the pixel features and the pixel position of the reference pixel points, and then uses the features of the area to be compared as the features to be compared corresponding to the target pixel points. It can determine the corresponding features to be compared in the hidden layer features based on the pixel features obtained by decomposing the image features, thereby providing data support for subsequent processing.

[0089] In an exemplary embodiment, a similarity comparison result corresponding to each pixel point is determined based on pixel features and features to be compared, including: using the pixel features of each pixel point as a convolution kernel; using the convolution kernel to perform a convolution operation on the features to be compared corresponding to each pixel point, and obtaining a convolution value as the similarity comparison result corresponding to each pixel point.

[0090] In one example, if Figure 2As shown, taking pixel A in the image feature as an example, the dynamic filter kernel (i.e., convolution kernel) of pixel A can be convolved with pixel A' at the same position in the hidden layer feature, and then the convolution value obtained by the convolution operation can be recorded in the similarity matrix. The convolution value can represent the degree of similarity between the features corresponding to the same pixel position in the hidden layer feature and the image feature, such as the size of the similarity. Therefore, by applying the dynamic filter kernel of all pixel points in the image feature to the corresponding pixel position of the hidden layer feature, the similarity matrix can be obtained.

[0091] The technical solution of this embodiment uses the pixel features of each pixel as a convolution kernel, and then uses the convolution kernel to perform a convolution operation on the features to be compared corresponding to each pixel to obtain a convolution value as the similarity comparison result corresponding to each pixel. This can accurately obtain the similarity comparison result between features, which helps to further suppress irrelevant information and focus on relevant information.

[0092] In an exemplary embodiment, hidden layer features are adjusted according to feature similarity to obtain adjusted hidden layer features, including: inputting feature similarity into a preset activation function to obtain feature adjustment weights for hidden layer features; a value corresponding to the feature adjustment weight is positively correlated with the feature similarity; and weighting the hidden layer features using the feature adjustment weight to obtain adjusted hidden layer features.

[0093] After obtaining the feature similarity, in order to suppress irrelevant information and focus on relevant information, the small values ​​in the feature similarity (i.e., the representation of small similarity) can be further reduced, and the large values ​​(i.e., the representation of large similarity) can be further expanded. By applying the Sigmoid activation function (such as Figure 2 Activation function 2) in the activation function, the output result of the activation function (i.e., the feature adjustment weight) can be multiplied by the hidden layer feature corresponding to the current frame image, and then the filtered hidden layer feature matrix (i.e., the adjusted hidden layer feature) can be obtained.

[0094] In one example, the image features extracted from the next frame image are decomposed into multiple dynamic filter kernels, and then the dynamic filter kernels are compared with the hidden layer features corresponding to the current frame image for similarity. The activation function can be used to suppress values ​​with small similarity and enhance values ​​with large similarity, and then it can be applied to the hidden layer features corresponding to the current frame image for filtering to obtain a filtered hidden layer feature matrix.

[0095] The technical solution of this embodiment obtains feature adjustment weights for hidden layer features by inputting feature similarity into a preset activation function, and then uses the feature adjustment weights to perform weighted processing on the hidden layer features to obtain adjusted hidden layer features, which can suppress irrelevant information between the current frame and the next frame, and focus on relevant information between the current frame and the next frame, and help the network to fully utilize the timing information of multiple frames in the video to complete the video processing task.

[0096] In an exemplary embodiment, extracting image features corresponding to the next frame image of the current frame image in the video includes: obtaining the next frame image of the current frame image in the video; inputting the next frame image into a preset image feature extraction network to obtain image features corresponding to the next frame image.

[0097] In the specific implementation, Figure 2 As shown, the size of the next frame image can be HxWx3, and a convolution layer with a size of 3x3 and an activation function (such as Figure 2 The activation function 1) extracts features from the next frame image and obtains image features as shown in the cube, whose size is HxWxK 2 , K is a hyperparameter, which can be used to characterize the filter radius of dynamic filtering.

[0098] The technical solution of this embodiment, by obtaining the next frame image of the current frame image in the video, can input the next frame image into a preset image feature extraction network to obtain the image features corresponding to the next frame image, and can extract the image features of the next frame image, providing data support for subsequent processing.

[0099] Figure 4 is a flowchart of another video super-resolution method according to an exemplary embodiment. Figure 4 As shown, the method is used in a computer device such as a terminal, and includes the following steps.

[0100] In step S410, the hidden layer features corresponding to the current frame image are obtained; the hidden layer features are the features output by the hidden layer network in the video super-resolution network when processing the current frame image. In step S420, the image features corresponding to the next frame image in the video of the current frame image are extracted. In step S430, the pixel features corresponding to each pixel in the image features are obtained. In step S440, the features to be compared corresponding to each pixel are determined in the hidden layer features according to the pixel features corresponding to each pixel. In step S450, the similarity comparison results corresponding to each pixel are determined according to the pixel features and the features to be compared, and the similarity comparison results corresponding to each pixel are used as feature similarity. In step S460, the feature similarity is input into the preset activation function to obtain the feature adjustment weight for the hidden layer feature; the value corresponding to the feature adjustment weight is positively correlated with the feature similarity. In step S470, the hidden layer features are weighted by using the feature adjustment weight to obtain the adjusted hidden layer features; the adjusted hidden layer features are used for the video super-resolution network to perform super-resolution processing on the video. It should be noted that the specific definition of the above steps can refer to the specific definition of a video super-resolution method above, which will not be repeated here.

[0101] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0103] Based on the same inventive concept, an embodiment of the present disclosure further provides a video super-resolution device for implementing the above-mentioned video super-resolution method.

[0104] Figure 5 FIG. 1 is a block diagram of a video super-resolution device according to an exemplary embodiment. Figure 5 , the device comprises:

[0105] The feature acquisition unit 501 is configured to acquire hidden layer features corresponding to the current frame image; the hidden layer features are features output by the hidden layer network in the video super-resolution network when processing the current frame image; the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs;

[0106] The similarity determination unit 502 is configured to extract the image features corresponding to the next frame image of the current frame image in the video, and determine the feature similarity between the hidden layer features and the image features; the feature similarity is used to represent the similarity between the features corresponding to the same pixel position in the hidden layer features and the image features;

[0107] The feature adjustment unit 503 is configured to adjust the hidden layer features according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used by the video super-resolution network to perform super-resolution processing on the video.

[0108] In a possible implementation, the similarity determination unit 502 is specifically configured to execute the steps of obtaining pixel features corresponding to each pixel in the image features; determining features to be compared corresponding to each pixel in the hidden layer features according to the pixel features corresponding to each pixel; determining similarity comparison results corresponding to each pixel according to the pixel features and the features to be compared; and using the similarity comparison results corresponding to each pixel as the feature similarity.

[0109] In one possible implementation, the image feature has multiple feature planes, and the similarity determination unit 502 is specifically configured to execute the steps of taking any pixel point in the image feature as a target pixel point; obtaining pixel sub-features corresponding to the target pixel point in each feature plane of the image feature; and rearranging each pixel sub-feature to obtain a pixel feature corresponding to the target pixel point.

[0110] In one possible implementation, the hidden layer feature has multiple feature planes, and the similarity determination unit 502 is specifically configured to determine a reference pixel in the hidden layer feature; the pixel position of the reference pixel is the same as the pixel position of the target pixel; based on the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel, the feature of the area to be compared is determined in each feature plane of the hidden layer feature; and the feature of the area to be compared is used as the feature to be compared corresponding to the target pixel.

[0111] In one possible implementation, the similarity determination unit 502 is specifically configured to execute using the pixel features of each of the pixel points as a convolution kernel; using the convolution kernel to perform a convolution operation on the features to be compared corresponding to each of the pixel points, and obtaining a convolution value as a similarity comparison result corresponding to each of the pixel points.

[0112] In one possible implementation, the feature adjustment unit 503 is specifically configured to execute inputting the feature similarity into a preset activation function to obtain a feature adjustment weight for the hidden layer feature; the value corresponding to the feature adjustment weight is positively correlated with the feature similarity; and the hidden layer feature is weighted using the feature adjustment weight to obtain the adjusted hidden layer feature.

[0113] In a possible implementation, the similarity determination unit 502 is specifically configured to execute the acquisition of the next frame image of the current frame image in the video; input the next frame image into a preset image feature extraction network to obtain image features corresponding to the next frame image.

[0114] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0115] Each module in the above video super-resolution device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0116] Figure 6 1 is a block diagram of an electronic device 600 for implementing a video super-resolution method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0117] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0118] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0119] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0120] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0121] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0122] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting an audio signal.

[0123] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0124] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or the electronic device 600 component, the presence or absence of contact between the user and the electronic device 600, the device 600 orientation or acceleration / deceleration and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0125] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0126] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0127] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0128] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 620 of the electronic device 600 to complete the above method.

[0129] Figure 7 is a block diagram of an electronic device 700 for implementing another video super-resolution method according to an exemplary embodiment. For example, the electronic device 700 may be a server. Figure 7 , the electronic device 700 includes a processing component 720, which further includes one or more processors, and a memory resource represented by a memory 722 for storing instructions that can be executed by the processing component 720, such as an application. The application stored in the memory 722 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 720 is configured to execute the instructions to perform the above method.

[0130] The electronic device 700 may further include a power supply component 724 configured to perform power management of the electronic device 700, a wired or wireless network interface 726 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 728. The electronic device 700 may operate based on an operating system stored in the memory 722, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0131] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 722 including instructions, and the instructions can be executed by a processor of the electronic device 700 to perform the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0132] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by a processor of the electronic device 700 to complete the above method.

[0133] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video super-resolution method, characterized in that: The method comprises: Obtaining hidden layer features corresponding to the current frame image; the hidden layer features are features output by a hidden layer network in a video super-resolution network processing the current frame image; the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs; Extracting image features corresponding to the next frame image of the current frame image in the video, and determining feature similarity between the hidden layer features and the image features; the feature similarity is used to characterize the similarity between features corresponding to the same pixel position in the hidden layer features and the image features; The hidden layer features are adjusted according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used by the video super-resolution network to perform super-resolution processing on the video.

2. The method according to claim 1, characterized in that: The determining of the feature similarity between the hidden layer feature and the image feature comprises: Obtaining pixel features corresponding to each pixel point in the image features; According to the pixel features corresponding to the pixels, the features to be compared corresponding to the pixels are determined in the hidden layer features; Determine the similarity comparison result corresponding to each pixel point according to the pixel feature and the feature to be compared; The similarity comparison result corresponding to each of the pixel points is used as the feature similarity.

3. The method according to claim 2, characterized in that The image feature has multiple feature planes, and obtaining the pixel feature corresponding to each pixel point in the image feature includes: Taking any pixel point in the image feature as a target pixel point; Obtaining pixel sub-features corresponding to the target pixel point on each feature plane of the image feature; The pixel sub-features are rearranged to obtain the pixel feature corresponding to the target pixel.

4. The method according to claim 3, characterized in that The hidden layer feature has a plurality of feature planes, and determining the to-be-compared feature corresponding to each pixel point in the hidden layer feature according to the pixel feature corresponding to each pixel point includes: Determine a reference pixel point in the hidden layer feature; the pixel position of the reference pixel point is the same as the pixel position of the target pixel point; Determining the features of the area to be compared in each feature plane of the hidden layer features according to the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel; The features of the area to be compared are used as the features to be compared corresponding to the target pixel points.

5. The method according to claim 2, characterized in that: Determining the similarity comparison result corresponding to each pixel point according to the pixel feature and the feature to be compared includes: Using the pixel features of each pixel as a convolution kernel; The convolution kernel is used to perform a convolution operation on the features to be compared corresponding to each of the pixel points to obtain a convolution value as a similarity comparison result corresponding to each of the pixel points.

6. The method according to claim 1, characterized in that The step of adjusting the hidden layer features according to the feature similarity to obtain adjusted hidden layer features includes: Inputting the feature similarity into a preset activation function to obtain a feature adjustment weight for the hidden layer feature; a value corresponding to the feature adjustment weight is positively correlated with the feature similarity; The hidden layer features are weighted using the feature adjustment weights to obtain the adjusted hidden layer features.

7. The method according to any one of claims 1 to 6, characterized in that: The extracting the image feature corresponding to the next frame image of the current frame image in the video includes: Obtain the next frame image of the current frame image in the video; The next frame image is input into a preset image feature extraction network to obtain image features corresponding to the next frame image.

8. A video super-resolution device, characterized in that: The device comprises: A feature acquisition unit is configured to acquire hidden layer features corresponding to the current frame image; the hidden layer features are features output by a hidden layer network in a video super-resolution network when processing the current frame image; the video super-resolution network is a convolutional neural network that performs super-resolution processing on the video to which the current frame image belongs; A similarity determination unit is configured to extract image features corresponding to the next frame image of the current frame image in the video, and determine feature similarity between the hidden layer features and the image features; the feature similarity is used to characterize the similarity between features corresponding to the same pixel position in the hidden layer features and the image features; The feature adjustment unit is configured to adjust the hidden layer features according to the feature similarity to obtain adjusted hidden layer features; the adjusted hidden layer features are used by the video super-resolution network to perform super-resolution processing on the video.

9. The device according to claim 8, characterized in that The similarity determination unit is specifically configured to execute the acquisition of pixel features corresponding to each pixel in the image features; determine the features to be compared corresponding to each pixel in the hidden layer features according to the pixel features corresponding to each pixel; determine the similarity comparison results corresponding to each pixel according to the pixel features and the features to be compared; and use the similarity comparison results corresponding to each pixel as the feature similarity.

10. The device according to claim 9, characterized in that The image feature has multiple feature planes, and the similarity determination unit is specifically configured to execute the execution of taking any pixel point in the image feature as a target pixel point; obtaining the pixel sub-features corresponding to the target pixel point in each feature plane of the image feature; and rearranging each of the pixel sub-features to obtain the pixel feature corresponding to the target pixel point.

11. The device according to claim 10, characterized in that The hidden layer feature has multiple feature planes, and the similarity determination unit is specifically configured to determine the reference pixel point in the hidden layer feature; the pixel position of the reference pixel point is the same as the pixel position of the target pixel point; according to the number of pixel sub-features in the pixel feature and the pixel position of the reference pixel point, the feature of the area to be compared is determined in each feature plane of the hidden layer feature; and the feature of the area to be compared is used as the feature to be compared corresponding to the target pixel point.

12. The device according to claim 9, characterized in that The similarity determination unit is specifically configured to execute a convolution operation using the pixel features of each of the pixel points as a convolution kernel; using the convolution kernel to perform a convolution operation on the features to be compared corresponding to each of the pixel points to obtain a convolution value as a similarity comparison result corresponding to each of the pixel points.

13. The device according to claim 8, characterized in that The feature adjustment unit is specifically configured to input the feature similarity into a preset activation function to obtain a feature adjustment weight for the hidden layer feature; The value corresponding to the feature adjustment weight is positively correlated with the feature similarity; the hidden layer feature is weighted by using the feature adjustment weight to obtain the adjusted hidden layer feature.

14. The device according to any one of claims 8 to 13, characterized in that The similarity determination unit is specifically configured to execute the acquisition of the next frame image of the current frame image in the video; input the next frame image into a preset image feature extraction network to obtain the image features corresponding to the next frame image.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video super-resolution method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video super-resolution method as described in any one of claims 1 to 7.

17. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the video super-resolution method as described in any one of claims 1 to 7.