Video super-resolution processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for super-resolution processing of a video, the method comprising: inputting a video to be processed into a super-resolution network; using the super-resolution network to use the first hidden layer features of the first video frame before the current video frame, the second hidden layer features of the second video frame after the current video frame, and the current video frame as input parameters to obtain the current hidden layer features of the current video frame; determining a weight matrix; obtaining the super-resolution features of the current video frame based on the weight matrix, the first hidden layer features, the current hidden layer features, and the second hidden layer features; and obtaining a super-resolution video frame based on the super-resolution features. According to the technical solution of the present disclosure, it is possible to retain feature information related to the current video frame based on the weight matrix on the basis of fully utilizing the temporal information of the field, extract information that is more helpful to the current video frame from the temporal information of the field, thereby improving the super-resolution accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for super-resolution processing of a video. Background Art

[0002] Super-resolution is the mapping relationship between low-resolution images or videos and high-resolution images through algorithmic modeling. With the development of deep learning, convolutional neural networks have begun to be used in video super-resolution tasks due to their huge fitting capabilities.

[0003] In the related art, a bidirectional circular convolutional network (such as BasicVSR, GOVSR) can be used to process the super-resolution task of the video. The bidirectional circular convolutional network includes two unidirectional circular convolutional networks. These two unidirectional circular convolutional networks process the input video in forward and reverse forms respectively. For the video frame at the current moment, the hidden layer features of the previous moment and the next moment of the current moment are used to obtain the hidden layer features of the current moment. However, since the video frames of the previous moment and the next moment may have low quality, the super-resolution accuracy of the video is affected. Summary of the invention

[0004] The present disclosure provides a video super-resolution processing method, device, electronic device, computer-readable storage medium and computer program product to at least solve the problem in the related art that the video frames at the previous moment and the next moment may have low quality, thereby affecting the super-resolution precision video difference of the video. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for super-resolution processing of a video is provided, comprising:

[0006] Inputting the video to be processed into a super-resolution network, wherein the super-resolution network is a bidirectional recurrent convolutional network;

[0007] For a current video frame in the video to be processed, using the super-resolution network to take a first hidden layer feature of a first video frame before the current video frame, a second hidden layer feature of a second video frame after the current video frame, and the current video frame as input parameters, to obtain a current hidden layer feature of the current video frame;

[0008] Determine a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0009] Obtaining a super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0010] A super-resolution video frame of the current video frame is obtained according to the super-resolution feature.

[0011] In one embodiment, determining a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature includes:

[0012] Concatenate the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature to obtain a first intermediate feature;

[0013] Performing feature extraction on the first intermediate feature to obtain a second intermediate feature, wherein a feature dimension of the second intermediate feature is the sum of the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0014] The second intermediate feature is normalized from the feature dimension to obtain the weight matrix.

[0015] In one embodiment, extracting the first intermediate feature to obtain the second intermediate feature includes:

[0016] Inputting the first intermediate feature into a feature extraction network for feature extraction;

[0017] The first intermediate feature after feature extraction is subjected to convolution processing to obtain the second intermediate feature.

[0018] In one of the embodiments, the feature extraction network comprises at least one layer of residual module, and the residual module comprises a first convolutional layer, a linear rectification layer, and a second convolutional layer connected in sequence;

[0019] The step of inputting the first intermediate feature into a feature extraction network for feature extraction includes:

[0020] Performing skip connection processing on the first intermediate features;

[0021] Processing the first intermediate feature through the first convolution layer, the linear rectification layer, and the second convolution layer in the residual module;

[0022] The first intermediate feature after the jump connection processing and the first intermediate feature after the residual module processing are fused to obtain the first intermediate feature after feature extraction.

[0023] In one embodiment, obtaining the super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature includes:

[0024] Obtaining the product of the weight matrix and the first hidden layer feature as a new first hidden layer feature;

[0025] Obtaining the product of the weight matrix and the current hidden layer feature as a new current hidden layer feature;

[0026] Obtaining the product of the weight matrix and the second hidden layer feature as a new second hidden layer feature;

[0027] Acquire an initial super-resolution video frame obtained by extracting features of the current hidden layer features;

[0028] The new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame are concatenated to obtain the super-resolution feature.

[0029] In one embodiment, the first video frame includes video frames N times before the current video frame, and the second video frame includes video frames N times after the current video frame, where N is a positive integer greater than or equal to 2.

[0030] According to a second aspect of an embodiment of the present disclosure, a video super-resolution processing device is provided, comprising:

[0031] An input module is configured to input the video to be processed into a super-resolution network, wherein the super-resolution network is a bidirectional recurrent convolutional network;

[0032] A first feature generation module is configured to execute, for a current video frame in the video to be processed, using the super-resolution network to take a first hidden layer feature of a first video frame before the current video frame, a second hidden layer feature of a second video frame after the current video frame, and the current video frame as input parameters to obtain a current hidden layer feature of the current video frame;

[0033] A weight determination module, configured to determine a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0034] A second feature generation module is configured to obtain a super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0035] The super-resolution video frame generation module is configured to obtain a super-resolution video frame of the current video frame according to the super-resolution feature.

[0036] In one embodiment, the weight determination module includes:

[0037] A first concatenation unit is configured to concatenate the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature to obtain a first intermediate feature;

[0038] a feature extraction unit configured to perform feature extraction on the first intermediate feature to obtain a second intermediate feature, wherein a feature dimension of the second intermediate feature is a sum of the number of the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature;

[0039] A normalization unit is configured to perform normalization processing on the second intermediate feature from the feature dimension to obtain the weight matrix.

[0040] In one embodiment, the feature extraction unit includes:

[0041] a feature extraction subunit, configured to perform feature extraction by inputting the first intermediate feature into a feature extraction network;

[0042] The convolution subunit is configured to perform convolution processing on the first intermediate feature after feature extraction to obtain the second intermediate feature.

[0043] In one of the embodiments, the feature extraction network comprises at least one layer of residual module, and the residual module comprises a first convolutional layer, a linear rectification layer, and a second convolutional layer connected in sequence;

[0044] The feature extraction subunit is configured to perform skip connection processing on the first intermediate feature; process the first intermediate feature through the first convolution layer, the linear rectification layer, and the second convolution layer in the residual module; and fuse the first intermediate feature after the skip connection processing and the first intermediate feature after the residual module processing to obtain the first intermediate feature after feature extraction.

[0045] In one embodiment, the second feature generation module includes:

[0046] A first feature acquisition unit is configured to acquire the product of the weight matrix and the first hidden layer feature as a new first hidden layer feature;

[0047] A second feature acquisition unit is configured to acquire the product of the weight matrix and the current hidden layer feature as a new current hidden layer feature;

[0048] A third feature acquisition unit is configured to acquire the product of the weight matrix and the second hidden layer feature as a new second hidden layer feature;

[0049] A video frame acquisition unit is configured to acquire an initial super-resolution video frame obtained by extracting features from the current hidden layer features;

[0050] The second splicing unit is configured to perform splicing of the new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame to obtain the super-resolution feature.

[0051] In one embodiment, the first video frame includes video frames N times before the current video frame, and the second video frame includes video frames N times after the current video frame, where N is a positive integer greater than or equal to 2.

[0052] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0053] processor;

[0054] a memory for storing instructions executable by the processor;

[0055] The processor is configured to execute the instructions to implement the video super-resolution processing method described in any one of the embodiments of the first aspect above.

[0056] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video super-resolution processing method described in any one of the embodiments of the first aspect above.

[0057] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the video super-resolution processing method described in any one of the embodiments of the first aspect above.

[0058] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0059] The video to be processed is processed by a super-resolution network which is a bidirectional recurrent convolutional network. For the current video frame in the video to be processed, the first hidden layer feature of the first video frame before the current video frame, the second hidden layer feature of the second video frame after the current video frame, and the current video frame are used as input parameters through the super-resolution network to obtain the current hidden layer feature of the current video frame, so that the current video frame can utilize the temporal information of the domain, thereby improving the processing effect of the current video frame. By determining the weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature; according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature, the super-resolution feature of the current video frame is obtained, and on the basis of fully utilizing the temporal information of the domain, the feature information related to the current video frame can be retained based on the weight matrix, and more helpful information for the current video frame can be extracted from the temporal information of the domain, thereby further improving the super-resolution accuracy of the current video frame.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0062] Figure 1 is a schematic diagram illustrating a super-resolution processing method in related art.

[0063] Figure 2 The figure is a flowchart of a method for super-resolution processing of a video according to an exemplary embodiment.

[0064] Figure 3 The diagram is a schematic diagram of a video super-resolution processing method according to an exemplary embodiment.

[0065] Figure 4 The figure is a flowchart showing a step of generating a weight matrix according to an exemplary embodiment.

[0066] Figure 5 is a schematic structural diagram of a residual module according to an exemplary embodiment.

[0067] Figure 6 The figure is a flowchart of a method for super-resolution processing of a video according to an exemplary embodiment.

[0068] Figure 7a is a comparative schematic diagram of a super-resolution process according to an exemplary embodiment.

[0069] Figure 7b is a comparative schematic diagram of another super-resolution processing according to an exemplary embodiment.

[0070] Figure 8 The figure is a block diagram of a video super-resolution processing device according to an exemplary embodiment.

[0071] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0072] Fig.10 is a block diagram of another electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0073] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0076] In the related art, the super-resolution technology of video includes an algorithm based on a temporal sliding window and an algorithm based on a cyclic convolutional network. The algorithm based on a temporal sliding window can be implemented in the following steps. First, the input video sequence is divided into multiple subsequences, each subsequence contains M video frames, and M is a positive integer greater than 1. Then, the subsequence is input into the neural network for fusion. In an example, taking the neural network as EDVR (video restoration, deblurring, and super-resolution network based on deformable convolution) as an example, EDVR uses a 2D (two-dimensional) convolutional network. First, the input M frame subsequences are spliced ​​in the color space, and then a series of 2D convolutional layers are used to extract features from the spliced ​​subsequences to obtain super-resolution features for super-resolution. In order to cope with large inter-frame motion, EDVR adds a deformable function to the conventional 2D convolution. In another example, taking the neural network DUF (Deep Video Super-Resolution Network Using Dynamic Upsampling FiltersWithout Explicit Motion Compensation) as an example, DUF splices the input multiple video frames in the time dimension, and then uses a 3D convolutional network to extract features from the spliced ​​multiple video frames to obtain super-resolution features for super-resolution. Compared with the 2D convolution, the 3D convolution layer adds a sliding process in time series, so it can capture the complementary information between frames in time series, which is an implicit time series modeling method. The above method based on 2D convolution layers and 3D convolution layers achieves considerable super-resolution processing effects by stacking the network depth, but it also brings huge computational overhead.

[0077] Compared with the method based on time-series sliding window, the method based on recurrent convolutional network can achieve better results with less network depth by fully accumulating information in time series. The method based on recurrent convolutional network includes the method based on unidirectional recurrent convolutional network (unidirectional RNN) and the method based on bidirectional recurrent convolutional network (bidirectional RNN). Figure 1 , Figure 1 Each square in represents the hidden features of the hidden layer output. The unidirectional RNN accumulates the hidden features of different moments by passing the hidden features of the previous moment to the next moment, so that subsequent frames can use more historical accumulated features during super-resolution. However, this method also faces the problem of unequal information reception of different frames: for the first frame, no historical information can be used; while for subsequent frames, historical information accumulated over a longer period of time can be used. Moreover, the unidirectional recurrent convolutional network method needs to initialize multiple video frames in actual use, and then output the super-resolution results with better quality to the user, resulting in the super-resolution processed video being shorter than the original video and lacking real-time performance.

[0078] Continue to refer to Figure 1 , the bidirectional RNN will perform a forward propagation process and a backward propagation process on the video respectively. For each frame, the hidden layer features of the previous moment and the next moment will be used to obtain the super-resolution features of each frame. However, the bidirectional RNN in the related art only uses the timing information of the previous moment and the next moment. When there is a low quality situation between frames (such as occlusion, blur, etc.), it is easy to affect the super-resolution accuracy of the video.

[0079] In order to solve the above problems, the present disclosure provides a method for super-resolution processing of a video, which is applied to an electronic device. A trained super-resolution network is pre-deployed in the electronic device, and the super-resolution network is a bidirectional circular convolutional network. The electronic device inputs the video to be processed into the super-resolution network; for the current video frame in the video to be processed, the first hidden layer feature of the first video frame before the current video frame, the second hidden layer feature of the second video frame after the current video frame, and the current video frame are used as input parameters through the super-resolution network to obtain the current hidden layer feature of the current video frame; determine the weight matrix corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature; according to the weight matrix, the first hidden layer feature, the current hidden layer feature and the second hidden layer feature, obtain the super-resolution feature of the current video frame; according to the super-resolution feature, obtain the super-resolution video frame of the current video frame, and then obtain the super-resolution video frame corresponding to each video frame in the video to be processed, and generate a super-resolution video.

[0080] The electronic device may be a terminal or a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0081] Figure 2 is a flowchart of a method for super-resolution processing of a video according to an exemplary embodiment. Figure 2 As shown, the video super-resolution processing method is used in an electronic device and includes the following steps.

[0082] In step S210, the video to be processed is input into a super-resolution network, which is a bidirectional recurrent convolutional network.

[0083] Among them, the video to be processed is the video to be processed by super-resolution, for example, the video collected in real time by the terminal or image acquisition device; the video pre-stored in the local database or server, for example, film and television videos, videos shot and uploaded by users, etc.

[0084] The bidirectional recurrent convolutional network includes two unidirectional recurrent convolutional networks, namely the forward propagation network and the back propagation network. The forward propagation network is responsible for forward propagation of the processed video, that is, propagation from the starting frame to the ending frame, and transmitting the network output result of the past video frame to the next video frame. The back propagation network is responsible for back propagation of the processed video, that is, propagation from the ending frame to the starting frame, and transmitting the network output result of the future video frame to the next video frame.

[0085] In step S220, for the current video frame in the video to be processed, a super-resolution network is used to take the first hidden layer features of the first video frame before the current video frame, the second hidden layer features of the second video frame after the current video frame, and the current video frame as input parameters to obtain the current hidden layer features of the current video frame.

[0086] The first video frame is the N video frames before the current video frame, and the second video frame is the N video frames after the current video frame, where N is a positive integer greater than or equal to 1.

[0087] It is understandable that if N is greater than 1, the first hidden layer feature of the first video frame includes N hidden layer features corresponding to N first video frames. Similarly, the second hidden layer feature of the second video frame includes N hidden layer features corresponding to N second video frames.

[0088] It can be understood that when the forward propagation network processes the video to be processed, when the current video frame belongs to any frame from the start frame to the Nth frame, the first video frame is less than N frames. When the backward propagation network processes the video to be processed, when the current video frame belongs to any frame from the last N frames to the end frame, the second video frame is less than N frames.

[0089] Specifically, the unidirectional recurrent convolutional network includes an input layer, a hidden layer, and an output layer. After the video to be processed is input into the super-resolution network, the forward propagation network can first perform forward propagation processing on the video to be processed. For the current video frame in the video to be processed, the forward propagation network inputs the first hidden layer features corresponding to the current video frame and the first video frame into the hidden layer of the forward propagation network to obtain the forward hidden layer features corresponding to the current video frame. After the forward propagation network has processed the video to be processed, the back propagation network performs back propagation processing on the video to be processed. For the current video frame in the video to be processed, the back propagation network inputs the second hidden layer features corresponding to the current video frame and the second video frame into the hidden layer of the back propagation network to obtain the back hidden layer features corresponding to the current video frame. After the back propagation network has processed the video to be processed, the forward hidden layer features and the back hidden layer features corresponding to the current video frame are fused to obtain the current hidden layer features corresponding to the current video frame.

[0090] In step S230, a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature is determined.

[0091] Among them, the weight matrix includes weights corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature. The weight corresponding to the hidden layer feature of each video frame reflects the degree of influence of each video frame on the current video frame and / or the quality of the video frame. In one example, if the weight corresponding to the hidden layer feature of a certain video frame is high, it can be said that the quality of the video frame is high, there is no occlusion, noise, etc., or the occlusion, noise, etc. are relatively mild. In another example, if the weight corresponding to the hidden layer feature of a certain video frame is high, it can be said that the video frame has a high degree of similarity with the current video frame.

[0092] In one example, a weight generation network may be deployed in an electronic device. The weight generation network may be used to detect the image quality of the first video frame, the current video frame, and the second video frame. In a specific implementation, after obtaining the current hidden layer feature corresponding to the current video frame, the electronic device inputs the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature into the weight generation network, detects the image quality of the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature through the weight generation network, and outputs a weight matrix.

[0093] In another example, the weight generation network can be used to detect the correlation between the first video frame and the second video frame and the current video frame. The weight generation network detects the correlation between the first hidden layer feature and the current hidden layer feature, and the correlation between the second hidden layer feature and the current hidden layer feature, and outputs a weight matrix.

[0094] In step S240, the super-resolution features of the current video frame are obtained according to the weight matrix, the first hidden layer features, the current hidden layer features, and the second hidden layer features.

[0095] Specifically, the electronic device multiplies the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature with the corresponding weights in the weight matrix, respectively, and fuses the multiplied first hidden layer feature, the multiplied current hidden layer feature, and the multiplied second hidden layer feature to obtain the super-resolution feature of the current video frame.

[0096] In step S250, a super-resolution video frame of the current video frame is obtained according to the super-resolution feature.

[0097] Specifically, the electronic device inputs the super-resolution feature of the current video frame into the feature extraction network, and processes the super-resolution feature by sub-pixel convolution, deconvolution, etc. through the feature extraction network to obtain a super-resolution video frame of the current video frame. Then, a super-resolution video frame corresponding to each video frame in the video to be processed is obtained, and a super-resolution video corresponding to the video to be processed is generated.

[0098] In the above-mentioned video super-resolution processing method, a super-resolution network which is a bidirectional recurrent convolutional network is used to process the video to be processed. For the current video frame in the video to be processed, the super-resolution network uses the first hidden layer feature of the first video frame before the current video frame, the second hidden layer feature of the second video frame after the current video frame, and the current video frame as input parameters to obtain the current hidden layer feature of the current video frame, so that the current video frame can utilize the temporal information of the domain, thereby improving the processing effect of the current video frame. By determining the weight matrix corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature; according to the weight matrix, the first hidden layer feature, the current hidden layer feature and the second hidden layer feature, the super-resolution feature of the current video frame is obtained, which can retain the feature information related to the current video frame based on the weight matrix on the basis of making full use of the temporal information of the domain, and extract more helpful information for the current video frame from the temporal information of the domain, thereby further improving the super-resolution accuracy of the current video frame.

[0099] In an exemplary embodiment, since the bidirectional recurrent convolutional network in the related art uses the timing information of the previous moment and the next moment, the timing information that can be used is relatively small, and when there is occlusion, blurring, etc. between frames, errors will occur in the features of adjacent moments. Therefore, in the embodiment of the present disclosure, the first video frame may include the video frames of the previous N moments of the current video frame, and the second video frame may include the video frames of the next N moments of the current video frame, where N is a positive integer greater than or equal to 2. For the current video frame, the first hidden layer features of the first video frame adjacent to the current video frame and the second hidden layer features of the adjacent second video frame may be directly transmitted to the current video frame; the first hidden layer features of the first video frame that is not adjacent to the current video frame and the second hidden layer features of the second video frame may be transmitted to the current video frame in a jump propagation manner. Among them, the jump propagation method refers to a method of propagating features of more distant moments to the current moment in a time-series jump connection manner. Figure 3 The following diagram shows the propagation mode of hidden layer features when N is 2. Figure 3 For the current video frame t, the first hidden layer features of the adjacent t-1 (first video frame) and the second hidden layer features of t+1 (second video frame) are directly transmitted to the current video frame; the first hidden layer features of the non-adjacent t-2 (first video frame) and the second hidden layer features of t+2 (second video frame) are transmitted to the current video frame in a jump propagation manner.

[0100] In this embodiment, by propagating the hidden features of multiple fields to the current moment and enhancing the current moment, the robustness of the features at the current moment can be improved. By using a weight matrix to assign weights to the hidden features at each moment, the correlation between the hidden features in the large time series field can be improved, so that the current video frame can make full use of the information in the hidden features of N fields that is helpful for enhancing the current video frame.

[0101] In an exemplary embodiment, a method for generating a weight matrix is ​​described. Figure 4 As shown, in step S230, the weight matrix corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature is determined, which can be specifically implemented by the following steps:

[0102] In step S410, the first hidden layer feature, the current hidden layer feature and the second hidden layer feature are concatenated to obtain a first intermediate feature.

[0103] Specifically, the electronic device splices the first hidden layer feature, the current hidden layer feature and the second hidden layer feature from the feature dimension through the weight generation network to obtain the first intermediate feature. In an example, assuming that the hidden layer feature dimension corresponding to each video frame is HxWxC, where H represents the height of the video frame; W represents the width of the video frame; C represents the dimension of the hidden layer feature, if N is 2, the feature dimension of the first intermediate feature obtained by splicing is HxWx5C.

[0104] In step S420, feature extraction is performed on the first intermediate feature to obtain a second intermediate feature.

[0105] Specifically, the electronic device can use a weight generation network to extract features from the first intermediate feature to obtain a second intermediate feature. The feature dimension of the second intermediate feature is the sum of the number of the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature. In an example, if N is 2, the feature dimension of the first intermediate feature obtained by concatenation is HxWx5C, and the feature dimension of the second intermediate feature is HxWx5.

[0106] In step S430, the second intermediate feature is normalized in terms of feature dimension to obtain a weight matrix.

[0107] Specifically, the weight generation network may include a Softmax normalization layer. The electronic device normalizes the second intermediate feature from the feature dimension through the weight generation network to obtain a weight matrix. In an example, if the feature dimension of the second intermediate feature is HxWx5, the second intermediate feature is normalized in the dimension of 5.

[0108] In this embodiment, a weight generation network is used to assign weights to each point on the hidden layer feature (HxW) in the feature dimension, so that the weight of image frames with poor quality is relatively small, and the weight of image frames with good quality is relatively large. For example, when using multiple domain temporal information, if the hidden layer features of adjacent frames of the current video frame are blocked or noisy at a certain position, then the weight of the position of the adjacent frame will be relatively small, and the weight of the hidden layer features of non-adjacent frames at this position will be relatively large, thereby eliminating the errors between the hidden layer features and improving the processing accuracy of super-resolution.

[0109] In an exemplary embodiment, feature extraction is performed on the first intermediate feature to obtain the second intermediate feature, including: inputting the first intermediate feature into a feature extraction network for feature extraction; and performing convolution processing on the first intermediate feature after feature extraction to obtain the second intermediate feature.

[0110] Specifically, the weight generation network may include a feature extraction network. The feature extraction network may be, but is not limited to, any one of a residual network, a convolutional neural network, and the like. After obtaining the first intermediate feature, the first intermediate feature is input into the feature extraction network through the feature extraction network. Then, a 2D convolution layer with a 3x3 convolution kernel is used to perform convolution processing on the first intermediate feature after feature extraction to obtain a second intermediate feature. In this embodiment, by performing feature extraction and convolution processing on the first intermediate feature in sequence, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature can be better fused, so that the past and future hidden layer features can better enhance the current hidden layer feature.

[0111] In an exemplary embodiment, an implementation of a feature extraction network in a weight generation network is described. The feature extraction network includes at least one layer of residual modules. In the case of including multiple layers of residual modules, the multiple layers of residual modules can be connected in series in sequence. Figure 5 As shown, the residual module includes a first convolutional layer (Conv2D, a two-dimensional convolutional layer), a linear rectification layer (ReLU), and a second convolutional layer connected in sequence. Among them, Conv2D can be a 2D convolutional layer with a 3x3 convolution kernel. In this embodiment, the first intermediate feature is input into the feature extraction network for feature extraction, which can be specifically implemented in the following way:

[0112] After obtaining the first intermediate feature, the first intermediate feature is subjected to skip connection processing, wherein the skip connection processing refers to connecting the output results of different layers. The first intermediate feature is processed by the first convolution layer, the linear rectification layer, and the second convolution layer in the residual module. The first intermediate feature after skip connection processing and the first intermediate feature after residual module processing are fused to obtain the first intermediate feature after feature extraction. In this embodiment, by adopting a feature extraction network based on a residual module, on the one hand, the performance of the feature extraction network can be improved to obtain higher accuracy; on the other hand, the convergence speed of the network can be increased during network training, thereby improving the network training efficiency.

[0113] In an exemplary embodiment, in step S240, the super-resolution feature of the current video frame is obtained according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature, which can be specifically implemented in the following manner:

[0114] Specifically, the weight matrix can be a parameter obtained by normalizing the second intermediate feature in the feature dimension, which contains weights corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature. Therefore, the product of the weight corresponding to the first hidden layer feature in the weight matrix and the first hidden layer feature can be obtained as a new first hidden layer feature; the product of the weight corresponding to the current hidden layer feature in the weight matrix and the current hidden layer feature can be obtained as a new current hidden layer feature; the product of the weight corresponding to the second hidden layer feature in the weight matrix and the second hidden layer feature can be obtained as a new second hidden layer feature. Then, the initial super-resolution video frame is obtained by extracting the current hidden layer feature (which can be any of the sub-pixel convolution and deconvolution methods). Finally, the new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame are spliced ​​to obtain the super-resolution feature of the current video frame. In this embodiment, the optimized hidden layer feature is used to optimize the initial super-resolution video frame to obtain the super-resolution feature, so that the super-resolution feature can contain more useful information, thereby helping to improve the accuracy of super-resolution processing.

[0115] Figure 6 1 is a flowchart of a method for super-resolution processing of a video according to an exemplary embodiment. In this embodiment, the first video frame is the first 2 video frames of the current video frame, and the second video frame is the last 2 video frames of the current video frame. Figure 6 As shown, the video super-resolution processing method is used in an electronic device and includes the following steps.

[0116] In step S602, the video to be processed is input into a super-resolution network, which is a bidirectional recurrent convolutional network.

[0117] In step S604, for the current video frame in the video to be processed, the first hidden layer features of the first two frames of the current video frame, the second hidden layer features of the second two frames of the current video frame, and the current video frame are used as input parameters through the super-resolution network to obtain the current hidden layer features of the current video frame. The hidden layer feature dimension corresponding to each video frame is HxWxC, where H represents the height of the video frame; W represents the width of the video frame; and C represents the dimension of the hidden layer feature.

[0118] In step S606, feature extraction is performed on the current hidden layer features to obtain an initial super-resolution video frame.

[0119] In step S608, the first hidden layer features of the first two video frames, the current hidden layer features, and the second hidden layer features of the last two video frames are input into the weight generation network. The first hidden layer features, the current hidden layer features, and the second hidden layer features are concatenated through the weight generation network to obtain the first intermediate features. The feature dimension of the first intermediate feature is HxWx5C.

[0120] In step S610, the first intermediate feature is extracted by the feature extraction network in the weight generation network, and the first intermediate feature after feature extraction is convolved to obtain a second intermediate feature. The feature dimension of the second intermediate feature is HxWx5. The feature extraction network can be based on the residual module (refer to Figure 5 ) network, the details can refer to the above embodiment and will not be elaborated here.

[0121] In step S612, the second intermediate feature is normalized in terms of feature dimension to obtain a weight matrix.

[0122] In step S614, the product of the weight matrix and the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature is obtained to obtain a new first hidden layer feature, a new current hidden layer feature, and a new second hidden layer feature. The new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the super-resolution video frame are concatenated to obtain the super-resolution feature of the current video frame.

[0123] In step S616, feature extraction is performed on the super-resolution features of the current video frame to obtain a super-resolution video frame of the current video frame.

[0124] Figure 7a and Figure 7b The schematic diagrams of the comparison of the same image after being processed by the super-resolution method of the present invention and the interpolation upsampling method in the related art are shown respectively. Figure 7a and Figure 7b It can be seen that the super-resolution method disclosed in the present invention can restore results close to the real high resolution.

[0125] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0127] Figure 8FIG. 1 is a block diagram of a video super-resolution processing device X00 according to an exemplary embodiment. Figure 8 The device includes an input module X02, a first feature generation module X04, a weight determination module X06, a second feature generation module X08, and a super-resolution video frame generation module X10.

[0128] The input module X02 is configured to input the video to be processed into a super-resolution network, and the super-resolution network is a bidirectional recurrent convolutional network; the first feature generation module X04 is configured to execute, for the current video frame in the video to be processed, using the first hidden layer feature of the first video frame before the current video frame, the second hidden layer feature of the second video frame after the current video frame, and the current video frame as input parameters through the super-resolution network, to obtain the current hidden layer feature of the current video frame; the weight determination module X06 is configured to execute determination of the weight matrix corresponding to the first hidden layer feature, the current hidden layer feature and the second hidden layer feature; the second feature generation module X08 is configured to execute, according to the weight matrix, the first hidden layer feature, the current hidden layer feature and the second hidden layer feature, to obtain the super-resolution feature of the current video frame; the super-resolution video frame generation module X10 is configured to execute, according to the super-resolution feature, to obtain the super-resolution video frame of the current video frame.

[0129] In an exemplary embodiment, the weight determination module X06 includes: a first splicing unit, configured to perform splicing of the first hidden layer feature, the current hidden layer feature and the second hidden layer feature to obtain a first intermediate feature; a feature extraction unit, configured to perform feature extraction on the first intermediate feature to obtain a second intermediate feature, and the feature dimension of the second intermediate feature is the sum of the number of the first hidden layer feature, the current hidden layer feature and the second hidden layer feature; a normalization unit, configured to perform normalization processing on the second intermediate feature from the feature dimension to obtain a weight matrix.

[0130] In an exemplary embodiment, the feature extraction unit includes: a feature extraction subunit, configured to input the first intermediate feature into a feature extraction network for feature extraction; and a convolution subunit, configured to perform convolution processing on the first intermediate feature after feature extraction to obtain a second intermediate feature.

[0131] In an exemplary embodiment, the feature extraction network includes at least one layer of residual module, and the residual module includes a first convolutional layer, a linear rectification layer, and a second convolutional layer connected in sequence; the feature extraction subunit is configured to perform jump connection processing on the first intermediate feature; the first intermediate feature is processed by the first convolutional layer, the linear rectification layer, and the second convolutional layer in the residual module; the first intermediate feature after the jump connection processing and the first intermediate feature after the residual module processing are fused to obtain the first intermediate feature after feature extraction.

[0132] In an exemplary embodiment, the second feature generation module X08 includes: a first feature acquisition unit, configured to execute the acquisition of the product of the weight matrix and the first hidden layer feature as a new first hidden layer feature; a second feature acquisition unit, configured to execute the acquisition of the product of the weight matrix and the current hidden layer feature as a new current hidden layer feature; a third feature acquisition unit, configured to execute the acquisition of the product of the weight matrix and the second hidden layer feature as a new second hidden layer feature; a video frame acquisition unit, configured to execute the acquisition of an initial super-resolution video frame obtained by feature extraction of the current hidden layer feature; a second splicing unit, configured to execute the splicing of the new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame to obtain a super-resolution feature.

[0133] In an exemplary embodiment, the first video frame includes video frames at N moments before the current video frame, and the second video frame includes video frames at N moments after the current video frame, where N is a positive integer greater than or equal to 2.

[0134] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] Fig. 9 1 is a block diagram of an electronic device Z00 for super-resolution processing of a video according to an exemplary embodiment. For example, the electronic device Z00 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0136] Reference Fig. 9 The electronic device Z00 may include one or more of the following components: a processing component Z02, a memory Z04, a power component Z06, a multimedia component Z08, an audio component Z10, an input / output (I / O) interface Z12, a sensor component Z14, and a communication component Z16.

[0137] The processing component Z02 generally controls the overall operation of the electronic device Z00, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component Z02 may include one or more processors Z20 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component Z02 may include one or more modules to facilitate interaction between the processing component Z02 and other components. For example, the processing component Z02 may include a multimedia module to facilitate interaction between the multimedia component Z08 and the processing component Z02.

[0138] The memory Z04 is configured to store various types of data to support operations on the electronic device Z00. Examples of such data include instructions for any application or method operating on the electronic device Z00, contact data, phone book data, messages, pictures, videos, etc. The memory Z04 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0139] The power supply assembly Z06 provides power to various components of the electronic device Z00. The power supply assembly Z06 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device Z00.

[0140] The multimedia component Z08 includes a screen that provides an output interface between the electronic device Z00 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component Z08 includes a front camera and / or a rear camera. When the electronic device Z00 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0141] The audio component Z10 is configured to output and / or input audio signals. For example, the audio component Z10 includes a microphone (MIC), and when the electronic device Z00 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory Z04 or sent via the communication component Z16. In some embodiments, the audio component Z10 also includes a speaker for outputting an audio signal.

[0142] I / O interface Z12 provides an interface between processing component Z02 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0143] The sensor assembly Z14 includes one or more sensors for providing various aspects of status assessment for the electronic device Z00. For example, the sensor assembly Z14 can detect the open / closed state of the electronic device Z00, the relative positioning of the components, such as the display and keypad of the electronic device Z00, and the sensor assembly Z14 can also detect the position change of the electronic device Z00 or the components of the electronic device Z00, the presence or absence of contact between the user and the electronic device Z00, the orientation or acceleration / deceleration of the device Z00 and the temperature change of the electronic device Z00. The sensor assembly Z14 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly Z14 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly Z14 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0144] The communication component Z16 is configured to facilitate wired or wireless communication between the electronic device Z00 and other devices. The electronic device Z00 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component Z16 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component Z16 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0145] In an exemplary embodiment, the electronic device Z00 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0146] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory Z04 including instructions, and the above instructions can be executed by a processor Z20 of the electronic device Z00 to complete the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0147] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor Z20 of the electronic device Z00 to implement the above method.

[0148] Fig.10 FIG. 1 is a block diagram of another electronic device S00 for performing super-resolution processing on a video according to an exemplary embodiment. For example, the electronic device S00 may be a server. Fig.10 The electronic device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22 for storing instructions executable by the processing component S20, such as an application. The application stored in the memory S22 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0149] The electronic device S00 may further include: a power supply component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input / output (I / O) interface S28. The electronic device S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD or the like.

[0150] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory S22 including instructions, and the above instructions can be executed by a processor of the electronic device S00 to complete the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0151] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by a processor of the electronic device S00 to complete the above method.

[0152] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0153] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0154] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video super-resolution processing method, characterized in that: include: Inputting the video to be processed into a super-resolution network, wherein the super-resolution network is a bidirectional recurrent convolutional network; For a current video frame in the video to be processed, using the super-resolution network to take a first hidden layer feature of a first video frame before the current video frame, a second hidden layer feature of a second video frame after the current video frame, and the current video frame as input parameters, to obtain a current hidden layer feature of the current video frame; Determine a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature through a weight generation network, wherein the weight generation network includes a feature extraction network, the feature extraction network includes at least one layer of residual module, the residual module includes a first convolutional layer, a linear rectification layer, and a second convolutional layer connected in sequence, and the feature extraction network is used to connect the results of outputs of different layers through skip connection processing, and fuse the results after the skip connection processing and the results after the residual module processing to obtain the result after the feature extraction network performs feature extraction; Obtaining a super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature; A super-resolution video frame of the current video frame is obtained according to the super-resolution feature.

2. The video super-resolution processing method according to claim 1, characterized in that: The determining, through a weight generation network, a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature comprises: Concatenating the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature through the weight generation network to obtain a first intermediate feature; Extracting features from the first intermediate feature through the weight generation network to obtain a second intermediate feature, wherein a feature dimension of the second intermediate feature is the sum of the number of the first hidden layer features, the current hidden layer features, and the second hidden layer features; The second intermediate feature is normalized in terms of the feature dimension by the weight generation network to obtain the weight matrix.

3. The video super-resolution processing method according to claim 2, characterized in that: The extracting the first intermediate feature by the weight generation network to obtain the second intermediate feature includes: Inputting the first intermediate feature into a feature extraction network in the weight generation network for feature extraction; The first intermediate feature after feature extraction is subjected to convolution processing to obtain the second intermediate feature.

4. The video super-resolution processing method according to claim 3, characterized in that: The step of inputting the first intermediate feature into the feature extraction network in the weight generation network for feature extraction comprises: Performing skip connection processing on the first intermediate features; Processing the first intermediate feature through the first convolution layer, the linear rectification layer, and the second convolution layer in the residual module; The first intermediate feature after the jump connection processing and the first intermediate feature after the residual module processing are fused to obtain the first intermediate feature after feature extraction.

5. The video super-resolution processing method according to claim 1, characterized in that: The step of obtaining the super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature includes: Obtaining the product of the weight matrix and the first hidden layer feature as a new first hidden layer feature; Obtaining the product of the weight matrix and the current hidden layer feature as a new current hidden layer feature; Obtaining the product of the weight matrix and the second hidden layer feature as a new second hidden layer feature; Acquire an initial super-resolution video frame obtained by extracting features of the current hidden layer features; The new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame are concatenated to obtain the super-resolution feature.

6. The video super-resolution processing method according to any one of claims 1 to 5, characterized in that: The first video frame includes video frames at N moments before the current video frame, and the second video frame includes video frames at N moments after the current video frame, where N is a positive integer greater than or equal to 2.

7. A video super-resolution processing device, characterized in that: include: An input module is configured to input the video to be processed into a super-resolution network, wherein the super-resolution network is a bidirectional recurrent convolutional network; A first feature generation module is configured to execute, for a current video frame in the video to be processed, using the super-resolution network to take a first hidden layer feature of a first video frame before the current video frame, a second hidden layer feature of a second video frame after the current video frame, and the current video frame as input parameters to obtain a current hidden layer feature of the current video frame; A weight determination module is configured to determine a weight matrix corresponding to the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature through a weight generation network, wherein the weight generation network includes a feature extraction network, the feature extraction network includes at least one layer of residual module, the residual module includes a first convolutional layer, a linear rectification layer, and a second convolutional layer connected in sequence, and the feature extraction network is used to connect the results of outputs of different layers through skip connection processing, and fuse the results after the skip connection processing and the results after the residual module processing to obtain the result after the feature extraction network performs feature extraction; A second feature generation module is configured to obtain a super-resolution feature of the current video frame according to the weight matrix, the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature; The super-resolution video frame generation module is configured to obtain a super-resolution video frame of the current video frame according to the super-resolution feature.

8. The video super-resolution processing device according to claim 7, characterized in that: The weight determination module comprises: A first concatenation unit is configured to concatenate the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature through the weight generation network to obtain a first intermediate feature; a feature extraction unit configured to perform feature extraction on the first intermediate feature through the weight generation network to obtain a second intermediate feature, wherein a feature dimension of the second intermediate feature is the sum of the number of the first hidden layer feature, the current hidden layer feature, and the second hidden layer feature; A normalization unit is configured to perform normalization processing on the second intermediate feature from the feature dimension through the weight generation network to obtain the weight matrix.

9. The video super-resolution processing device according to claim 8, characterized in that: The feature extraction unit comprises: a feature extraction subunit, configured to execute feature extraction by inputting the first intermediate feature into a feature extraction network in the weight generation network; The convolution subunit is configured to perform convolution processing on the first intermediate feature after feature extraction to obtain the second intermediate feature.

10. The video super-resolution processing device according to claim 9, characterized in that: The feature extraction subunit is configured to perform skip connection processing on the first intermediate feature; process the first intermediate feature through the first convolution layer, the linear rectification layer, and the second convolution layer in the residual module; and fuse the first intermediate feature after the skip connection processing and the first intermediate feature after the residual module processing to obtain the first intermediate feature after feature extraction.

11. The video super-resolution processing device according to claim 7, characterized in that: The second feature generation module includes: A first feature acquisition unit is configured to acquire the product of the weight matrix and the first hidden layer feature as a new first hidden layer feature; A second feature acquisition unit is configured to acquire the product of the weight matrix and the current hidden layer feature as a new current hidden layer feature; A third feature acquisition unit is configured to acquire the product of the weight matrix and the second hidden layer feature as a new second hidden layer feature; A video frame acquisition unit is configured to acquire an initial super-resolution video frame obtained by extracting features from the current hidden layer features; The second splicing unit is configured to perform splicing of the new first hidden layer feature, the new current hidden layer feature, the new second hidden layer feature, and the initial super-resolution video frame to obtain the super-resolution feature.

12. The video super-resolution processing device according to any one of claims 7 to 11, characterized in that: The first video frame includes video frames at N moments before the current video frame, and the second video frame includes video frames at N moments after the current video frame, where N is a positive integer greater than or equal to 2.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video super-resolution processing method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video super-resolution processing method as described in any one of claims 1 to 6.

15. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the video super-resolution processing method as described in any one of claims 1 to 6.

Images