Video super-resolution method and device

Abstract

The present disclosure relates to a video super-resolution method and device, and the video super-resolution method includes: obtaining a video to be processed, wherein the video to be processed includes a plurality of combined frames arranged in time sequence, and each combined frame includes at least two image frames arranged in time sequence; obtaining a fusion feature of the current combined frame based on the fusion feature of the current combined frame and the previous combined frame obtained by processing; processing the current combined frame based on the fusion feature of the current combined frame to obtain a high-resolution combined frame corresponding to the current combined frame. This strategy can not only use parallel computing methods to reduce computational redundancy and reduce the overall super-resolution time of the entire video, but also implicitly learn the correlation and complementary information of the previous and next frames when super-resolving multiple frames at the same time, making full use of the timing information, increasing the temporal continuity of the super-resolution structure, and optimizing the processing efficiency while ensuring the super-resolution effect.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image and video enhancement, and in particular to a video super-resolution method and device. Background Art

[0002] Super-resolution technology is a classic research topic that aims to supplement the details and textures required when converting low-resolution images to high-resolution images. The field of super-resolution can be further divided into image super-resolution and video super-resolution. Compared with image super-resolution, the key to video super-resolution is to make full use of multi-frame information to complete the mapping from low-resolution sequences to high-resolution sequences.

[0003] Existing video super-resolution algorithms, whether based on temporal sliding windows or recurrent convolutional networks, all adopt a multi-frame-input, single-frame-output strategy, that is, multiple frames are fed into the network for fusion, and a high-resolution image is output. If this type of method is used to super-resolution the entire video sequence, each frame needs to be fed into the network for separate processing, resulting in redundant calculations, especially in long videos, which greatly limits the computational efficiency. Summary of the invention

[0004] The present disclosure provides a video super-resolution method and device to at least solve the problem of low computational efficiency in the related art, and may not solve any of the above problems.

[0005] According to a first aspect of the present disclosure, a video super-resolution method is provided, comprising: obtaining a video to be processed, wherein the video to be processed comprises a plurality of combined frames arranged in time sequence, and each combined frame comprises at least two image frames arranged in time sequence; obtaining a fusion feature of a current combined frame based on a fusion feature of a current combined frame and a previous combined frame obtained through processing; and processing the current combined frame based on the fusion feature of the current combined frame to obtain a high-resolution combined frame corresponding to the current combined frame.

[0006] Optionally, the method of obtaining the fusion features of the current combined frame based on the fusion features of the current combined frame and the previous combined frame obtained after processing includes: inputting the fusion features of the current combined frame and the previous combined frame obtained after processing into an extraction network to obtain the hidden layer features of each image frame of the current combined frame; inputting the hidden layer features of each image frame of the current combined frame into a fusion network to obtain the fusion features of the current combined frame.

[0007] Optionally, the fusion network includes a fusion sub-network, wherein the inputting of the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame comprises: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion features of the current combined frame; or the splicing features obtained by splicing the hidden layer features of each image frame of the current combined frame are input into the fusion sub-network to obtain the fusion features of the current combined frame.

[0008] Optionally, the fusion network also includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, and the fusion sub-networks are alternately cascaded with the split sub-networks, wherein the inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame also includes: inputting the fusion features of the current combined frame into the split sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; inputting the adjusted hidden layer features of each image frame of the current combined frame into the next fusion sub-network to obtain the fusion features of the current combined frame.

[0009] Optionally, the extraction network includes at least two extraction neurons, and the number of the extraction neurons is equal to and corresponds one to one to the number of image frames included in a combined frame, wherein inputting the fusion features of the current combined frame and the previous combined frame obtained after processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame includes: inputting an image frame of the current combined frame and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

[0010] Optionally, the current combined frame is processed based on the fusion features of the current combined frame to obtain a high-resolution combined frame corresponding to the current combined frame, including: inputting the fusion features of the current combined frame into super-resolution networks corresponding to each image frame of the current combined frame, respectively, to obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combining all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame.

[0011] According to a second aspect of the present disclosure, a training method for a video super-resolution model is provided, the video super-resolution model includes a feature extraction network and a super-resolution network, the training method includes: obtaining a first video sample and a second video sample with the same content, wherein the resolution of the first video sample is smaller than the resolution of the second video sample, and the first video sample and the second video sample each include a plurality of combined frames arranged in time sequence, and each combined frame includes at least two image frames arranged in time sequence; inputting a current combined frame of the first video sample and a fusion feature of a previous combined frame obtained after processing into the feature extraction network to obtain a fusion feature of the current combined frame; inputting the fusion feature of the current combined frame into the super-resolution network to obtain a high-resolution combined frame corresponding to the current combined frame; and training the feature extraction network and the super-resolution network based on the high-resolution combined frame corresponding to the current combined frame and a combined frame in the second video sample corresponding to the current combined frame.

[0012] Optionally, the feature extraction network includes an extraction network and a fusion network, wherein the step of inputting the current combined frame of the first video sample and the fused features of the previous combined frame obtained after processing into the feature extraction network to obtain the fused features of the current combined frame comprises: inputting the current combined frame of the first video sample and the fused features of the previous combined frame obtained after processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; and inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fused features of the current combined frame.

[0013] Optionally, the fusion network includes a fusion sub-network, wherein the inputting of the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame comprises: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion features of the current combined frame; or inputting the spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion sub-network to obtain the fusion features of the current combined frame.

[0014] Optionally, the fusion network also includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, and the fusion sub-networks and the split sub-networks are alternately cascaded, wherein the inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame also includes: inputting the fusion features of the current combined frame into the split sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; inputting the adjusted hidden layer features of each image frame of the current combined frame into the next fusion sub-network to obtain the fusion features of the current combined frame.

[0015] Optionally, the extraction network includes at least two extraction neurons, the number of the extraction neurons is equal to and one-to-one corresponds to the number of image frames included in a combined frame, and the inputting of the fusion features of the current combined frame of the first video sample and the previous combined frame obtained after processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame includes: inputting an image frame of the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

[0016] Optionally, the number of image frames included in the super-resolution network and a combined frame is equal to and corresponds one to one, wherein inputting the fusion features of the current combined frame into the super-resolution network to obtain a high-resolution combined frame corresponding to the current combined frame includes: inputting the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, respectively, to obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combining all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame.

[0017] According to a third aspect of the present disclosure, a video super-resolution device is provided, comprising: an acquisition unit, configured to acquire a video to be processed, wherein the video to be processed comprises a plurality of combined frames arranged in time sequence, and each combined frame comprises at least two image frames arranged in time sequence; a fusion unit, configured to obtain a fusion feature of a current combined frame based on a fusion feature of the current combined frame and a previous combined frame obtained through processing; and a super-resolution unit, configured to process the current combined frame based on the fusion feature of the current combined frame, to obtain a high-resolution combined frame corresponding to the current combined frame.

[0018] Optionally, the fusion unit is further configured to: input the fusion features of the current combined frame and the processed previous combined frame into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame.

[0019] Optionally, the fusion network includes a fusion sub-network, and the fusion unit is further configured to: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion features of the current combined frame; or inputs the spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion sub-network to obtain the fusion features of the current combined frame.

[0020] Optionally, the fusion network also includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, the fusion sub-networks and the split sub-networks are alternately cascaded, and the fusion unit is further configured to: input the fusion features of the current combined frame into the split sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; input the adjusted hidden layer features of each image frame of the current combined frame into the next fusion sub-network to obtain the fusion features of the current combined frame.

[0021] Optionally, the extraction network includes at least two extraction neurons, the number of the extraction neurons is equal to and corresponds one to one to the number of image frames included in a combined frame, and the fusion unit is further configured to: input an image frame of a current combined frame and a fusion feature of a previous combined frame obtained after processing into the corresponding extraction neuron to obtain the hidden layer features of the corresponding image frame of the current combined frame.

[0022] Optionally, the super-resolution unit is further configured to: input the fusion features of the current combined frame into the super-resolution networks corresponding to the respective image frames of the current combined frame, obtain high-resolution image frames corresponding to the respective image frames of the current combined frame, and combine all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame.

[0023] According to a fourth aspect of the present disclosure, a training device for a video super-resolution model is provided, wherein the video super-resolution model includes a feature extraction network and a super-resolution network, and the training device includes: an acquisition unit, configured to acquire a first video sample and a second video sample with the same content, wherein the resolution of the first video sample is smaller than the resolution of the second video sample, and the first video sample and the second video sample each include a plurality of combined frames arranged in time sequence, and each combined frame includes at least two image frames arranged in time sequence; a fusion unit, configured to input a current combined frame of the first video sample and a fused feature of a previous combined frame obtained after processing into the feature extraction network to obtain a fused feature of the current combined frame; a super-resolution unit, configured to input the fused feature of the current combined frame into the super-resolution network to obtain a high-resolution combined frame corresponding to the current combined frame; and a training unit, configured to train the feature extraction network and the super-resolution network based on the high-resolution combined frame corresponding to the current combined frame and a combined frame in the second video sample corresponding to the current combined frame.

[0024] Optionally, the feature extraction network includes an extraction network and a fusion network, and the fusion unit is further configured to: input the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame.

[0025] Optionally, the fusion network includes a fusion sub-network, and the fusion unit is further configured to: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion features of the current combined frame; or inputs the spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion sub-network to obtain the fusion features of the current combined frame.

[0026] Optionally, the fusion network also includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, the fusion sub-networks and the split sub-networks are alternately cascaded, and the fusion unit is further configured to: input the fusion features of the current combined frame into the split sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; input the adjusted hidden layer features of each image frame of the current combined frame into the next fusion sub-network to obtain the fusion features of the current combined frame.

[0027] Optionally, the extraction network includes at least two extraction neurons, the number of the extraction neurons is equal to and corresponds one to one to the number of image frames included in a combined frame, and the fusion unit is further configured to: input an image frame of the current combined frame of the first video sample and the fusion features of the processed previous combined frame into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

[0028] Optionally, the number of image frames included in the super-resolution network and a combined frame is equal to and corresponds one to one, and the super-resolution unit is further configured to: input the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, respectively, to obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combine all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame.

[0029] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute a video super-resolution model training method or a video super-resolution method according to the present disclosure.

[0030] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute a training method for a video super-resolution model or a video super-resolution method according to the present disclosure.

[0031] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the video super-resolution model training method or video super-resolution method according to the present disclosure.

[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0033] According to the video super-resolution method and device of the embodiments of the present disclosure, a multi-input and multi-output strategy has been developed, that is, at least two image frames are used as input at a time (that is, one combined frame is input), and then a high-resolution combined frame of the corresponding combined frame is output, that is, at least two corresponding high-resolution image frames are output. In addition, the fusion features obtained from the input combined frame can be sent to the next time state to transfer historical information to future frames. Correspondingly, this calculation will also use the fusion features of the previous time state as input. This strategy can not only use parallel operations to reduce computational redundancy and reduce the overall super-resolution time of the entire video, but also implicitly learn the correlation and complementary information of the previous and next frames when super-resolving multiple frames at the same time, making full use of the timing information, increasing the temporal continuity of the super-resolution structure, and optimizing the processing efficiency while ensuring the super-resolution effect.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0036] Figure 1 is a logical diagram showing a method based on a time series sliding window in the related art.

[0037] Figure 2 is a flowchart illustrating a method for training a video super-resolution model according to an exemplary embodiment of the present disclosure.

[0038] Figure 3 is a logical schematic diagram illustrating a video super-resolution model according to an exemplary embodiment of the present disclosure.

[0039] Figure 4 is a flowchart illustrating a video super-resolution method according to an exemplary embodiment of the present disclosure.

[0040] Figure 5 is a block diagram illustrating a training apparatus for a video super-resolution model according to an exemplary embodiment of the present disclosure.

[0041] Figure 6 is a block diagram illustrating a video super-resolution apparatus according to an exemplary embodiment of the present disclosure.

[0042] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0045] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0046] Super-resolution technology is a classic research topic, dedicated to supplementing the details and textures required when converting low-resolution images to high-resolution images. The field of super-resolution can be further divided into image super-resolution and video super-resolution. Compared with image super-resolution, the key to video super-resolution is to make full use of multi-frame information to complete the mapping from low-resolution sequences to high-resolution sequences. Existing video super-resolution algorithms are divided into time-series sliding window and circular convolution processing structures according to the video processing mode.

[0047] The temporal sliding window-based method requires dividing the original video into multiple subsequences and then super-resolving the reference frames in these subsequences separately. Figure 1 Shows the framework with a sliding window of 3, see Figure 1 , Represents the low-resolution image of the tth frame to be processed. When super-resolving the tth frame, it and its previous 3 frames and the next 3 frames will be taken as subsequences. The subsequences are sent to the video super-resolution network to obtain the high-resolution image of the tth frame. In other words, this scheme usually uses N+1 frames of images as input, among which the middle frame is used as the reference frame, that is, the frame to be super-resolved, and the remaining frames are adjacent frames, that is, the frames to supplement the information of the reference frame. The temporal sliding window method focuses on the way of information fusion, which can be roughly divided into explicit motion compensation and implicit motion compensation. The explicit method first estimates the optical flow of adjacent frames and the reference frame, and then aligns them to the reference frame to alleviate the problem of limited receptive field. Implicit motion compensation uses 3D convolution kernels or deformable convolution kernels to learn the motion relationship between the information of adjacent frames and the reference frame. Because this method is based on learning to model motion relationships, it cannot be explicitly described, so it is called implicit motion compensation. For this method, refer to Figure 1 , as the sliding window moves, when super-resolving the t+1 frame, the t+1 frame and its previous 3 frames and the next 3 frames will be taken as subsequences and sent to the video super-resolution network. At this time, it can be found that there is an overlap of t-2 frame, t-1 frame, t frame, t+1 frame, t+2 frame, and t+3 frame in the two subsequences. The overlap of these frames will introduce temporal redundancy and increase computational overhead. In addition, the sliding window is performed in time order, which means that future frames need to wait for the super-resolution of the current frame, which brings about cache problems.

[0048] Another processing structure for video super-resolution is a method based on a recurrent convolutional network. This structure usually uses the previous frame, the current frame, and the hidden features corresponding to the previous frame as input, and then inputs the super-resolved image of the current frame and the hidden features corresponding to the current frame. Among them, the hidden features corresponding to the current frame will be used as the input of the next moment. Through the means of temporal accumulation, the historical information is propagated to the future frame as supplementary information for super-resolution. This method only requires two frames as input each time, and no other additional frames are required. Compared with the method based on the temporal sliding window, although it can avoid introducing too many adjacent frames, it also has some problems. Whether it is a method based on a temporal sliding window or a method based on a recurrent convolutional network, they are all strategies of sending multiple frames into the network and outputting a single frame. This strategy cannot achieve parallel computing. If you want to super-resolve the entire video sequence, you need to send each frame into the network for separate processing, which leads to redundant calculations. The computing efficiency is particularly limited in long videos.

[0049] According to the video super-resolution algorithm of the exemplary embodiment of the present disclosure, a multi-input and multi-output strategy is developed based on a circular convolutional network, that is, at least two image frames are used as input at a time, and then at least two corresponding high-resolution images are output. In addition, the fusion features obtained from the input image frames can be sent to the next time state to transfer historical information to future frames. Correspondingly, this calculation will also use the fusion features of the previous time state as input. This strategy can not only use parallel operations to reduce computational redundancy and reduce the overall super-resolution time of the entire video, but also implicitly learn the correlation and complementary information of the previous and subsequent frames when super-resolving multiple frames at the same time, making full use of timing information and increasing the temporal continuity of the super-resolution structure. Compared with the previous methods based on timing sliding windows and circular convolutional networks, it can optimize the processing efficiency of the video while ensuring similar effects.

[0050] Next, we will refer to Figures 2 to 7 A training method and a training device for a video super-resolution model as well as a video super-resolution method and a video super-resolution device according to exemplary embodiments of the present disclosure are described in detail.

[0051] Figure 2 is a flowchart illustrating a method for training a video super-resolution model according to an exemplary embodiment of the present disclosure. Figure 3 is a logical schematic diagram showing a video super-resolution model according to an exemplary embodiment of the present disclosure. The video super-resolution model includes a feature extraction network and a super-resolution network. It should be understood that the training method of the video super-resolution model according to the exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC), or can also be implemented in a device such as a server.

[0052] Reference Figure 2 In step 201, a first video sample and a second video sample with the same content are obtained. The resolution of the first video sample is smaller than that of the second video sample, that is, the first video sample is a video to be super-resolved, and the second video sample is a learning target. The first video sample and the second video sample both include a plurality of combined frames arranged in time sequence, and a combined frame corresponds to a time state, for example Figure 3 As shown, the time step on the left is 0, and the time step on the right is 1, which respectively represent the two time states 0 and 1. Each combined frame includes at least two image frames arranged in time sequence, for example Figure 3 As shown, each combined frame includes two image frames, the combined frame with a time step of 0 includes image frames X1 and X2, and the combined frame with a time step of 1 includes image frames X3 and X4.

[0053] It can be understood that the first video sample and the second video sample are composed of multiple image frames arranged in time sequence, that is, they are composed of sequence frames, and the sequence frames need to be grouped first, and each group of image frames is regarded as a combined frame. The grouping process can be used as a pre-calculation of the video super-resolution model according to the exemplary embodiment of the present disclosure, and the first video sample and the second video sample that have been grouped are directly obtained in step 201, or it can be used as the calculation content of the video super-resolution model and implemented in step 201, and the present disclosure does not limit this.

[0054] Return to reference Figure 2 In step 202, the current combined frame of the first video sample and the fused features of the previous combined frame obtained after processing are input into the feature extraction network to obtain the fused features of the current combined frame. Specifically, the combined frames of the first video sample can be processed one by one in time sequence, and the combined frame currently being processed is the current combined frame, and the combined frame of the previous time state of the current combined frame is the previous combined frame. By performing parallel calculation on each combined frame, it is not necessary to wait for the super-resolution result of the previous combined frame, but only to wait for the fused features of the previous combined frame, which can alleviate the redundancy of the previous video super-resolution algorithm processing, and use the parallel operation method to enhance the processing flow of the recurrent convolutional network. At the same time, by inputting the fused features of the previous combined frame, the historical information can still be learned by means of time series accumulation, and the fused features can fuse the information of each image frame of the corresponding combined frame, so that the current combined frame learns the historical information with inter-frame complementary information, which helps to ensure the super-resolution effect without inputting the previous image frame, thereby taking into account the calculation speed and the super-resolution effect, and optimizing the processing efficiency.

[0055] Still Figure 3 As an example, the left side shows the process of processing the combined frame with time step 0 in the first video sample. As mentioned above, X1 and X2 are two image frames of the current combined frame. is the fusion feature of the previous combined frame. For the combined frame of the first time state of the first video sample, there is no previous combined frame. In this case, 0 can be input as the fusion feature of the previous combined frame.

[0056] Optionally, the feature extraction network includes an extraction network and a fusion network, and step 202 specifically includes: inputting the current combined frame of the first video sample and the fusion features of the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame. By first extracting the hidden layer features of each image frame of the current combined frame to obtain the information of each image frame, and then fusing the hidden layer features into the fusion features, the complementary information of each image frame can be obtained more accurately, which helps to ensure the subsequent super-resolution effect.

[0057] Optionally, refer to Figure 3 , the extraction network includes at least two extraction neurons (cells), and the number of extraction neurons is equal to and one-to-one corresponds to the number of image frames included in a combined frame. Accordingly, the operation of obtaining the hidden layer features in step 202 specifically includes: inputting an image frame of the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons, and obtaining the hidden layer features of the corresponding image frame of the current combined frame. Figure 3 , is to combine X1 and Input the cell corresponding to X1, get the hidden layer feature h1 corresponding to X1, and add X2 and Input the cell corresponding to X2 to obtain the hidden layer feature h2 corresponding to X2. By configuring the corresponding extraction neuron for each image frame in the extraction network and inputting the fusion features of the image frame and the previous combined frame into the extraction neuron, it is possible to obtain the corresponding hidden layer features for each image frame, and learn the complementary information between frames of the previous combined frame without being affected by other image frames in the current combined frame, ensuring that relatively independent and accurate hidden layer features of the image frame are obtained.

[0058] Optionally, the fused network includes fused sub-networks.

[0059] In some embodiments, the operation of obtaining the fusion feature may specifically include: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion feature of the current combined frame. Figure 3 The left part is to add h1 and h2 to get the fusion feature By directly adding and summing up the hidden layer features, it can not only play the role of integrating the hidden layer features, but also simplify the calculation, which helps to reduce the calculation load and improve the calculation speed.

[0060] In other embodiments, the operation of obtaining the fusion feature may specifically include: inputting the spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion subnetwork to obtain the fusion feature of the current combined frame. By splicing first and then fusing (for example, convolution fusion, that is, the fusion subnetwork is a convolution network), more detailed information can be extracted, which helps to improve the fusion effect and thus ensure the super-resolution effect. It can be understood that for two images with dimensions of h×w×c (h represents height, w represents width, and c represents the number of channels), after splicing in the feature dimension, the dimension becomes h×w×2c, and the dimension of the obtained fusion feature becomes h×w×c again, which is consistent with the situation where the fusion feature is obtained by adding the hidden layer features.

[0061] Optionally, the fusion network also includes at least one split subnetwork (for example, a convolutional network), and the number of fusion subnetworks is one more than the number of split subnetworks. For example, when there is one split subnetwork, there are two fusion subnetworks. The fusion subnetwork and the split subnetwork are alternately cascaded, that is, the head and tail of the fusion network are both fusion subnetworks, and the two subnetworks in the middle appear alternately. Correspondingly, the operation of obtaining the fusion feature also includes: inputting the fusion feature of the current combined frame into the split subnetwork to split the fusion feature of the current combined frame to obtain the adjusted hidden layer feature of each image frame of the current combined frame; inputting the adjusted hidden layer feature of each image frame of the current combined frame into the next fusion subnetwork to obtain the fusion feature of the current combined frame. By re-splitting the previously obtained fusion feature to obtain a new hidden layer feature, and then fusing the new hidden layer feature, and repeating this process many times, a further accurately fused fusion feature can be obtained, thereby ensuring the super-resolution effect.

[0062] Return to reference Figure 2 In step 203, the fusion features of the current combined frame are input into the super-resolution network to obtain a high-resolution combined frame corresponding to the current combined frame. By using the fusion features obtained in step 202 for super-resolution, the complementary information of adjacent frames can be implicitly learned during the super-resolution process, which makes full use of the time sequence information and helps to ensure the super-resolution effect.

[0063] It is understandable that, referring to Figure 3 , combined with the description of step 202, the fusion features of the current combined frame are not only input into the super-resolution network to obtain the corresponding high-resolution combined frame, but also sent to the next combined frame as input data for its super-resolution calculation.

[0064] Optionally, refer to Figure 3 , super-resolution networks (e.g. Figure 3 The number of image frames included in a combined frame is equal to and corresponds to the number of Conv2D, a two-dimensional convolutional network, and step 203 specifically includes: inputting the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, obtaining high-resolution image frames corresponding to each image frame of the current combined frame, and combining all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame. Figure 3, that is, input the fusion features into the Conv2D corresponding to X1 and X2 respectively, and obtain the corresponding two high-resolution image frames O1 and O2, which together constitute a high-resolution combination frame with a time step of 0. It can be understood that a super-resolution neuron can output a high-resolution image frame after one calculation, and a super-resolution network can output multiple high-resolution image frames after one calculation. These high-resolution image frames have corresponding timestamps, and are therefore regarded as outputting a high-resolution combination frame composed of all high-resolution image frames. In other words, there is no additional introduction of an operation to combine all high-resolution image frames together to form a high-resolution combination frame. By configuring a corresponding super-resolution network for each image frame and inputting the fusion features into the super-resolution network, a corresponding high-resolution image frame can be obtained for each image frame, ensuring that the super-resolution of each image frame is relatively independent.

[0065] Return to reference Figure 2 In step 204, based on the high-resolution combined frame corresponding to the current combined frame and the combined frame corresponding to the current combined frame in the second video sample, the feature extraction network and the super-resolution network are trained. As mentioned above, the first video sample and the second video sample have the same content, so the combined frames of the two and the image frames in the combined frames are in one-to-one correspondence, so that the high-resolution combined frames corresponding to each combined frame in the first video sample obtained in step 203 can also correspond to each combined frame in the second video sample. The training goal can be to make the two as close as possible, realize the training of the feature extraction network and the super-resolution network, and finally obtain a trained video super-resolution model.

[0066] It can be understood that, for the embodiment in which step 202 directly adds the hidden layer features of each image frame of the current combined frame to obtain the fused features of the current combined frame, the parameters of the fusion network in the feature extraction network are determined, so when adjusting the parameters, the parameters of the fusion network can be kept unchanged.

[0067] The video super-resolution algorithm according to the exemplary embodiment of the present disclosure was verified on the video super-resolution academic dataset Vid4, which contains 4 test scenes of long videos with different resolutions and motion modes. This dataset is also a well-known test dataset in the academic community. The evaluation this time used the academic indicators PSNR (PeakSignal to Noise Ratio) and SSIM (Structure Similarity Index Measure), which represent the pixel-level accuracy with standard high-resolution images.

[0068] Table 1 Comparison of test results between the baseline method and the disclosed method

[0069] method Foliage Walk City Calendar Baseline Method 26.58 / 0.7806 30.30 / 0.9092 27.76 / 0.8227 24.12 / 0.8160 The disclosed method 26.52 / 0.7793 30.31 / 0.9078 27.72 / 0.8213 24.14 / 0.8152

[0070] Table 1 is as follows Figure 3 The test results of the embodiment shown in FIG. 1 (i.e., each combined frame includes two image frames, and the two hidden layer features are added to obtain the fused features). The baseline method here adopts the traditional method based on the recurrent convolutional network, and its structure adopts a two-frame-in and one-frame-out scheme. Compared with this baseline method, the present disclosure Figure 3 The two-frame-in and two-frame-out solution of the illustrated embodiment optimizes the calculation speed of the super-resolution of the entire video through parallel calculation when the PSNR and SSIM effects are similar.

[0071] Figure 4 is a flow chart showing a video super-resolution method according to an exemplary embodiment of the present disclosure. It should be understood that the video super-resolution method according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC), or in a device such as a server.

[0072] Reference Figure 4 In step 401, a video to be processed is obtained. The video to be processed includes a plurality of combined frames arranged in time sequence, and a combined frame corresponds to a time state, for example Figure 3 As shown, the time step on the left is 0, and the time step on the right is 1, which respectively represent the two time states 0 and 1. Each combined frame includes at least two image frames arranged in time sequence, for example Figure 3 As shown, each combined frame includes two image frames, the combined frame with a time step of 0 includes image frames X1 and X2, and the combined frame with a time step of 1 includes image frames X3 and X4.

[0073] It is understandable that the video to be processed itself is composed of multiple image frames arranged in time sequence, that is, it is composed of sequence frames, and the sequence frames need to be grouped first, and each group of image frames is regarded as a combined frame. The grouping process can be used as a pre-calculation of the video super-resolution model according to the exemplary embodiment of the present disclosure, and the video to be processed that has been grouped is directly obtained in step 401, or it can be used as the calculation content of the video super-resolution model and implemented in step 401, and the present disclosure does not limit this.

[0074] In step 402, based on the fusion features of the current combined frame and the previous combined frame obtained through processing, the fusion features of the current combined frame are obtained. Specifically, the combined frames of the video to be processed can be processed one by one in time sequence, and the combined frame currently being processed is the current combined frame, and the combined frame of the previous time state of the current combined frame is the previous combined frame. By performing parallel calculations on each combined frame, it is not necessary to wait for the super-resolution result of the previous combined frame, but only to wait for the fusion features of the previous combined frame, which can alleviate the redundancy of the previous video super-resolution algorithm processing, and the processing flow of the recurrent convolutional network is enhanced by the parallel operation method. At the same time, by inputting the fusion features of the previous combined frame, historical information can still be learned by means of time series accumulation, and the fusion features can fuse the information of each image frame of the corresponding combined frame, so that the current combined frame learns the historical information with inter-frame complementary information, which helps to ensure the super-resolution effect without inputting the previous image frame, thereby taking into account both the calculation speed and the super-resolution effect, and optimizing the processing efficiency.

[0075] Still Figure 3 As an example, the left side shows the process of processing the combined frame with time step 0 in the video to be processed. As mentioned above, X1 and X2 are two image frames of the current combined frame. is the fusion feature of the previous combined frame. For the combined frame of the first time state of the video to be processed, there is no previous combined frame. In this case, 0 can be entered as the fusion feature of the previous combined frame.

[0076] Optionally, step 402 specifically includes: inputting the fusion features of the current combined frame and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame. By first extracting the hidden layer features of each image frame of the current combined frame to obtain the information of each image frame, and then fusing the hidden layer features into the fusion features, the complementary information of each image frame can be obtained more accurately, which helps to ensure the subsequent super-resolution effect.

[0077] Optionally, refer to Figure 3 , the extraction network includes at least two extraction neurons (cells), and the number of extraction neurons is equal to and one-to-one corresponds to the number of image frames included in a combined frame. Accordingly, the operation of obtaining the hidden layer features in step 402 specifically includes: inputting an image frame of the current combined frame and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons, and obtaining the hidden layer features of the corresponding image frame of the current combined frame. Figure 3 , is to combine X1 and Input the cell corresponding to X1, get the hidden layer feature h1 corresponding to X1, and add X2 and Input the cell corresponding to X2 to obtain the hidden layer feature h2 corresponding to X2. By configuring the corresponding extraction neuron for each image frame in the extraction network and inputting the fusion features of the image frame and the previous combined frame into the extraction neuron, it is possible to obtain the corresponding hidden layer features for each image frame, and learn the complementary information between frames of the previous combined frame without being affected by other image frames in the current combined frame, ensuring that relatively independent and accurate hidden layer features of the image frame are obtained.

[0078] Optionally, the fused network includes fused sub-networks.

[0079] In some embodiments, the operation of obtaining the fusion feature may specifically include: the fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain the fusion feature of the current combined frame. Figure 3 The left part is to add h1 and h2 to get the fusion feature By directly adding and summing up the hidden layer features, it can not only play the role of integrating the hidden layer features, but also simplify the calculation, which helps to reduce the calculation load and improve the calculation speed.

[0080] In some other embodiments, the operation of obtaining the fusion feature may specifically include: inputting the splicing feature obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion subnetwork to obtain the fusion feature of the current combined frame. By splicing first and then fusing, more detailed information can be extracted, which helps to improve the fusion effect and thus ensure the super-resolution effect.

[0081] Optionally, the fusion network also includes at least one split subnetwork, and the number of fusion subnetworks is one more than the number of split subnetworks. For example, when there is one split subnetwork, there are two fusion subnetworks. The fusion subnetwork and the split subnetwork are alternately cascaded, that is, the head and tail of the fusion network are both fusion subnetworks, and the two subnetworks in the middle appear alternately. Correspondingly, the operation of obtaining the fusion feature also includes: inputting the fusion feature of the current combined frame into the split subnetwork to split the fusion feature of the current combined frame to obtain the adjusted hidden layer feature of each image frame of the current combined frame; inputting the adjusted hidden layer feature of each image frame of the current combined frame into the next fusion subnetwork to obtain the fusion feature of the current combined frame. By re-splitting the previously obtained fusion feature to obtain a new hidden layer feature, and then fusing the new hidden layer feature, and repeating this for many times, a further accurately fused fusion feature can be obtained, thereby ensuring the super-resolution effect.

[0082] Return to reference Figure 4In step 403, the current combined frame is processed based on the fusion features of the current combined frame to obtain a high-resolution combined frame corresponding to the current combined frame. By using the fusion features obtained in step 402 for super-resolution, complementary information of adjacent frames can be implicitly learned during the super-resolution process, making full use of the time sequence information, which helps to ensure the super-resolution effect.

[0083] It is understandable that, referring to Figure 3 , combined with the description of step 402, the fusion features of the current combined frame are not only input into the super-resolution network to obtain the corresponding high-resolution combined frame, but also sent to the next combined frame as input data for its super-resolution calculation.

[0084] Optionally, refer to Figure 3 Step 403 specifically includes: inputting the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame (for example Figure 3 Conv2D, a two-dimensional convolutional network, is used to obtain high-resolution image frames corresponding to the image frames of the current combined frame, and all the high-resolution image frames are combined into a high-resolution combined frame corresponding to the current combined frame. Figure 3, that is, input the fusion features into the Conv2D corresponding to X1 and X2 respectively, and obtain the corresponding two high-resolution image frames O1 and O2, which together constitute a high-resolution combination frame with a time step of 0, and all high-resolution combination frames together constitute a high-resolution video corresponding to the video to be processed. Among them, all high-resolution combination frames are used to form a high-resolution video corresponding to the video to be processed, which is actually to arrange all high-resolution combination frames in time sequence. More specifically, all high-resolution image frames are arranged in time sequence to obtain a high-resolution video. It can be understood that a super-resolution neuron can output a high-resolution image frame after one calculation, and a super-resolution network can output multiple high-resolution image frames after one calculation. These high-resolution image frames have corresponding timestamps, and are therefore regarded as outputting a high-resolution combination frame composed of all high-resolution image frames. In other words, there is no additional operation of combining all high-resolution image frames together to form a high-resolution combination frame. In addition, each time a calculation is completed for a combined frame, a high-resolution combined frame is output. Since the high-resolution image frame has a timestamp, the high-resolution combined frame is also equivalent to having a timestamp. All high-resolution combined frames output after processing the video to be processed are arranged in time sequence to obtain the corresponding high-resolution video. In actual calculations, each high-resolution combined frame can be calculated and arranged behind the previous high-resolution combined frame calculated previously to finally obtain a high-resolution video. It can also be arranged in order after all high-resolution combined frames are calculated. The present disclosure does not limit this. By configuring a corresponding super-resolution network for each image frame and inputting the fusion features into the super-resolution network, a corresponding high-resolution image frame can be obtained for each image frame, ensuring that the super-resolution of each image frame is relatively independent.

[0085] Figure 5 is a block diagram showing a training device for a video super-resolution model according to an exemplary embodiment of the present disclosure. It should be understood that the training device for a video super-resolution model according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC) in the form of software, hardware, or a combination of software and hardware, and can also be implemented in a device such as a server.

[0086] Reference Figure 5 The video super-resolution model training device 500 includes an acquisition unit 501, a fusion unit 502, a super-resolution unit 503, and a training unit 504.

[0087] The acquisition unit 501 may acquire a first video sample and a second video sample with the same content. The resolution of the first video sample is smaller than that of the second video sample, that is, the first video sample is a video to be super-resolved, and the second video sample is a learning target. The first video sample and the second video sample each include a plurality of combined frames arranged in time sequence, and a combined frame corresponds to a time state, for example Figure 3 As shown, the time step on the left is 0, and the time step on the right is 1, which respectively represent the two time states 0 and 1. Each combined frame includes at least two image frames arranged in time sequence, for example Figure 3 As shown, each combined frame includes two image frames, the combined frame with a time step of 0 includes image frames X1 and X2, and the combined frame with a time step of 1 includes image frames X3 and X4.

[0088] It can be understood that the first video sample and the second video sample are each composed of a plurality of image frames arranged in time sequence, that is, composed of sequence frames, and the sequence frames need to be grouped first, and each group of image frames is regarded as a combined frame. The grouping process can be used as a pre-calculation of the video super-resolution model according to the exemplary embodiment of the present disclosure, and the acquisition unit 501 directly acquires the first video sample and the second video sample that have been grouped, or it can be used as the calculation content of the video super-resolution model, which is implemented by the acquisition unit 501, and the present disclosure does not limit this.

[0089] The fusion unit 502 can input the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the feature extraction network to obtain the fusion features of the current combined frame. Specifically, the combined frames of the first video sample can be processed one by one in time sequence, and the combined frame currently being processed is the current combined frame, and the combined frame of the previous time state of the current combined frame is the previous combined frame. By performing parallel calculation on each combined frame, it is not necessary to wait for the super-resolution result of the previous combined frame, but only to wait for the fusion features of the previous combined frame, which can alleviate the redundancy of the previous video super-resolution algorithm processing, and use the parallel operation method to enhance the processing flow of the recurrent convolution network. At the same time, by inputting the fusion features of the previous combined frame, historical information can still be learned by means of time series accumulation, and the fusion features can fuse the information of each image frame of the corresponding combined frame, so that the current combined frame learns the historical information with inter-frame complementary information, which helps to ensure the super-resolution effect without inputting the previous image frame, thereby taking into account the calculation speed and the super-resolution effect, and optimizing the processing efficiency.

[0090] Still Figure 3 As an example, the left side shows the process of processing the combined frame with time step 0 in the first video sample. As mentioned above, X1 and X2 are two image frames of the current combined frame. is the fusion feature of the previous combined frame. For the combined frame of the first time state of the first video sample, there is no previous combined frame. In this case, 0 can be input as the fusion feature of the previous combined frame.

[0091] Optionally, the feature extraction network includes an extraction network and a fusion network, and the fusion unit 502 can specifically perform the following operations: input the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame. By first extracting the hidden layer features of each image frame of the current combined frame to obtain the information of each image frame, and then fusing the hidden layer features into the fusion features, the complementary information of each image frame can be obtained more accurately, which helps to ensure the subsequent super-resolution effect.

[0092] Optionally, refer to Figure 3 , the extraction network includes at least two extraction neurons (cells), and the number of extraction neurons is equal to and one-to-one corresponds to the number of image frames included in a combined frame. Accordingly, in order to obtain the hidden layer features, the fusion unit 502 can input an image frame of the current combined frame of the first video sample and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame. Figure 3 , is to combine X1 and Input the cell corresponding to X1, get the hidden layer feature h1 corresponding to X1, and add X2 and Input the cell corresponding to X2 to obtain the hidden layer feature h2 corresponding to X2. By configuring the corresponding extraction neuron for each image frame in the extraction network and inputting the fusion features of the image frame and the previous combined frame into the extraction neuron, it is possible to obtain the corresponding hidden layer features for each image frame, and learn the complementary information between frames of the previous combined frame without being affected by other image frames in the current combined frame, ensuring that relatively independent and accurate hidden layer features of the image frame are obtained.

[0093] Optionally, the fused network includes fused sub-networks.

[0094] In some embodiments, in order to obtain the fusion feature, the fusion unit 502 may specifically add the hidden layer features of each image frame of the current combined frame by the fusion sub-network to obtain the fusion feature of the current combined frame. Figure 3 The left part is to add h1 and h2 to get the fusion feature By directly adding and summing up the hidden layer features, it can not only play the role of integrating the hidden layer features, but also simplify the calculation, which helps to reduce the calculation load and improve the calculation speed.

[0095] In other embodiments, to obtain the fusion feature, the fusion unit 502 may also input the spliced ​​feature obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion subnetwork to obtain the fusion feature of the current combined frame. By splicing first and then fusing, more detailed information can be extracted, which helps to improve the fusion effect and thus ensure the super-resolution effect.

[0096] Optionally, the fusion network also includes at least one split subnetwork, and the number of fusion subnetworks is one more than the number of split subnetworks. For example, when there is one split subnetwork, there are two fusion subnetworks. The fusion subnetwork and the split subnetwork are alternately cascaded, that is, the head and tail of the fusion network are both fusion subnetworks, and the two subnetworks in the middle appear alternately. Correspondingly, in order to obtain the fusion feature, the fusion unit 502 can also perform the following operations: input the fusion feature of the current combined frame into the split subnetwork to split the fusion feature of the current combined frame to obtain the adjusted hidden layer feature of each image frame of the current combined frame; input the adjusted hidden layer feature of each image frame of the current combined frame into the next fusion subnetwork to obtain the fusion feature of the current combined frame. By re-splitting the previously obtained fusion feature to obtain a new hidden layer feature, and then fusing the new hidden layer feature, repeating this process multiple times, a further accurately fused fusion feature can be obtained, thereby ensuring the super-resolution effect.

[0097] The super-resolution unit 503 can input the fusion features of the current combined frame into the super-resolution network to obtain a high-resolution combined frame corresponding to the current combined frame. By using the fusion features obtained by the fusion unit 502 to perform super-resolution, complementary information of adjacent frames can be implicitly learned during the super-resolution process, making full use of the time sequence information, which helps to ensure the super-resolution effect.

[0098] It is understandable that, referring to Figure 3 , combined with the description of the fusion unit 502, the fusion features of the current combined frame are not only input into the super-resolution network to obtain the corresponding high-resolution combined frame, but also sent to the next combined frame as input data for its super-resolution calculation.

[0099] Optionally, refer to Figure 3 , super-resolution networks (e.g. Figure 3 The number of Conv2D (two-dimensional convolutional network) and the number of image frames included in a combined frame are equal and one-to-one corresponding. Accordingly, the super-resolution unit 503 can specifically input the fusion features of the current combined frame into the super-resolution networks corresponding to the image frames of the current combined frame, obtain high-resolution image frames corresponding to the image frames of the current combined frame, and combine all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame. Figure 3, that is, input the fusion features into the Conv2D corresponding to X1 and X2 respectively, and obtain the corresponding two high-resolution image frames O1 and O2, which together constitute a high-resolution combination frame with a time step of 0. It can be understood that a super-resolution neuron can output a high-resolution image frame after one calculation, and a super-resolution network can output multiple high-resolution image frames after one calculation. These high-resolution image frames have corresponding timestamps, and are therefore regarded as outputting a high-resolution combination frame composed of all high-resolution image frames. In other words, there is no additional introduction of an operation to combine all high-resolution image frames together to form a high-resolution combination frame. By configuring a corresponding super-resolution network for each image frame and inputting the fusion features into the super-resolution network, a corresponding high-resolution image frame can be obtained for each image frame, ensuring that the super-resolution of each image frame is relatively independent.

[0100] The training unit 504 can train the feature extraction network and the super-resolution network based on the high-resolution combined frame corresponding to the current combined frame and the combined frame corresponding to the current combined frame in the second video sample. As mentioned above, the first video sample and the second video sample have the same content, so the combined frames of the two and the image frames in the combined frames are in one-to-one correspondence, so that the high-resolution combined frames corresponding to each combined frame in the first video sample obtained by the super-resolution unit 503 can also correspond to each combined frame in the second video sample. The training goal can be to make the two as close as possible to achieve the training of the feature extraction network and the super-resolution network, and finally obtain a trained video super-resolution model.

[0101] It can be understood that, for the embodiment in which the fusion unit 502 directly adds the hidden layer features of each image frame of the current combined frame to obtain the fused features of the current combined frame, the parameters in the fusion network in the feature extraction network are determined, so the parameter adjustment unit 506 only needs to keep the parameters of the fusion network unchanged.

[0102] Figure 6 is a block diagram showing a video super-resolution apparatus according to an exemplary embodiment of the present disclosure. It should be understood that the video super-resolution apparatus according to the exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC) in the form of software, hardware, or a combination of software and hardware, and can also be implemented in a device such as a server.

[0103] Reference Figure 6 The video super-resolution device 600 includes an acquisition unit 601, a fusion unit 602, and a super-resolution unit 603.

[0104] The acquisition unit 601 can acquire a video to be processed. The video to be processed includes a plurality of combined frames arranged in time sequence, and a combined frame corresponds to a time state, for example Figure 3As shown, the time step on the left is 0, and the time step on the right is 1, which respectively represent the two time states 0 and 1. Each combined frame includes at least two image frames arranged in time sequence, for example Figure 3 As shown, each combined frame includes two image frames, the combined frame with a time step of 0 includes image frames X1 and X2, and the combined frame with a time step of 1 includes image frames X3 and X4.

[0105] It is understandable that the video to be processed itself is composed of multiple image frames arranged in time sequence, that is, it is composed of sequence frames, and the sequence frames need to be grouped first, and each group of image frames is regarded as a combined frame. The grouping process can be used as a pre-calculation of the video super-resolution model according to the exemplary embodiment of the present disclosure, and the acquisition unit 601 directly acquires the video to be processed that has completed the grouping, or it can be used as the calculation content of the video super-resolution model, which is implemented by the acquisition unit 601, and the present disclosure does not limit this.

[0106] The fusion unit 602 can obtain the fusion features of the current combined frame based on the fusion features of the current combined frame and the previous combined frame obtained through processing. Specifically, the combined frames of the video to be processed can be processed one by one in time sequence, and the combined frame currently being processed is the current combined frame, and the combined frame of the previous time state of the current combined frame is the previous combined frame. By performing parallel calculations on each combined frame, it is not necessary to wait for the super-resolution result of the previous combined frame, but only to wait for the fusion features of the previous combined frame, which can alleviate the redundancy of the previous video super-resolution algorithm processing, and use the parallel operation method to enhance the processing flow of the recurrent convolutional network. At the same time, by inputting the fusion features of the previous combined frame, historical information can still be learned by means of time series accumulation, and the fusion features can fuse the information of each image frame of the corresponding combined frame, so that the current combined frame learns the historical information with inter-frame complementary information, which helps to ensure the super-resolution effect without inputting the previous image frame, thereby taking into account the calculation speed and the super-resolution effect, and optimizing the processing efficiency.

[0107] Still Figure 3 As an example, the left side shows the process of processing the combined frame with time step 0 in the video to be processed. As mentioned above, X1 and X2 are two image frames of the current combined frame. is the fusion feature of the previous combined frame. For the combined frame of the first time state of the video to be processed, there is no previous combined frame. In this case, 0 can be entered as the fusion feature of the previous combined frame.

[0108] Optionally, the fusion unit 602 may specifically perform the following operations: input the fusion features of the current combined frame and the processed previous combined frame into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame. By first extracting the hidden layer features of each image frame of the current combined frame to obtain the information of each image frame, and then fusing the hidden layer features into the fusion features, the complementary information of each image frame can be obtained more accurately, which helps to ensure the subsequent super-resolution effect.

[0109] Optionally, refer to Figure 3 , the extraction network includes at least two extraction neurons (cells), and the number of extraction neurons is equal to and one-to-one corresponds to the number of image frames included in a combined frame. Accordingly, in order to obtain the hidden layer features, the fusion unit 602 can input an image frame of the current combined frame and the fusion features of the previous combined frame obtained after processing into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame. Figure 3 , is to combine X1 and Input the cell corresponding to X1, get the hidden layer feature h1 corresponding to X1, and add X2 and Input the cell corresponding to X2 to obtain the hidden layer feature h2 corresponding to X2. By configuring the corresponding extraction neuron for each image frame in the extraction network and inputting the fusion features of the image frame and the previous combined frame into the extraction neuron, it is possible to obtain the corresponding hidden layer features for each image frame, and learn the complementary information between frames of the previous combined frame without being affected by other image frames in the current combined frame, ensuring that relatively independent and accurate hidden layer features of the image frame are obtained.

[0110] Optionally, the fused network includes fused sub-networks.

[0111] In some embodiments, to obtain the fusion feature, the fusion unit 602 may specifically add the hidden layer features of each image frame of the current combined frame by the fusion sub-network to obtain the fusion feature of the current combined frame. Figure 3 The left part is to add h1 and h2 to get the fusion feature By directly adding and summing up the hidden layer features, it can not only play the role of integrating the hidden layer features, but also simplify the calculation, which helps to reduce the calculation load and improve the calculation speed.

[0112] In other embodiments, to obtain the fusion feature, the fusion unit 602 may also input the spliced ​​feature obtained by splicing the hidden layer features of each image frame of the current combined frame into the fusion sub-network to obtain the fusion feature of the current combined frame. By splicing first and then fusing, more detailed information can be extracted, which helps to improve the fusion effect and thus ensure the super-resolution effect.

[0113] Optionally, the fusion network also includes at least one split subnetwork, and the number of fusion subnetworks is one more than the number of split subnetworks. For example, when there is one split subnetwork, there are two fusion subnetworks. The fusion subnetwork and the split subnetwork are alternately cascaded, that is, the head and tail of the fusion network are both fusion subnetworks, and the two subnetworks in the middle appear alternately. Correspondingly, in order to obtain the fusion feature, the fusion unit 602 can also perform the following operations: input the fusion feature of the current combined frame into the split subnetwork to split the fusion feature of the current combined frame to obtain the adjusted hidden layer feature of each image frame of the current combined frame; input the adjusted hidden layer feature of each image frame of the current combined frame into the next fusion subnetwork to obtain the fusion feature of the current combined frame. By re-splitting the previously obtained fusion feature to obtain a new hidden layer feature, and then fusing the new hidden layer feature, repeating this process multiple times, a further accurately fused fusion feature can be obtained, thereby ensuring the super-resolution effect.

[0114] The super-resolution unit 603 can process the current combined frame based on the fusion features of the current combined frame to obtain a high-resolution combined frame corresponding to the current combined frame. By using the fusion features obtained by the fusion unit 602 to perform super-resolution, complementary information of adjacent frames can be implicitly learned during the super-resolution process, making full use of the time sequence information, which helps to ensure the super-resolution effect.

[0115] It is understandable that, referring to Figure 3 , combined with the description of the fusion unit 602, the fusion features of the current combined frame are not only input into the super-resolution network to obtain the corresponding high-resolution combined frame, but also sent to the next combined frame as input data for its super-resolution calculation.

[0116] Optionally, refer to Figure 3 Specifically, the super-resolution unit 603 may input the fusion features of the current combined frame into the super-resolution network (e.g. Figure 3 Conv2D, a two-dimensional convolutional network, is used to obtain high-resolution image frames corresponding to each image frame of the current combined frame, and all high-resolution image frames are combined into a high-resolution combined frame corresponding to the current combined frame. Figure 3, that is, input the fusion features into the Conv2D corresponding to X1 and X2 respectively, and obtain the corresponding two high-resolution image frames O1 and O2, which together constitute a high-resolution combination frame with a time step of 0, and all high-resolution combination frames together constitute a high-resolution video corresponding to the video to be processed. Among them, all high-resolution combination frames are used to form a high-resolution video corresponding to the video to be processed, which is actually to arrange all high-resolution combination frames in time sequence. More specifically, all high-resolution image frames are arranged in time sequence to obtain a high-resolution video. It can be understood that a super-resolution neuron can output a high-resolution image frame after one calculation, and a super-resolution network can output multiple high-resolution image frames after one calculation. These high-resolution image frames have corresponding timestamps, and are therefore regarded as outputting a high-resolution combination frame composed of all high-resolution image frames. In other words, there is no additional operation of combining all high-resolution image frames together to form a high-resolution combination frame. In addition, each time a calculation is completed for a combined frame, a high-resolution combined frame is output. Since the high-resolution image frame has a timestamp, the high-resolution combined frame is also equivalent to having a timestamp. All high-resolution combined frames output after processing the video to be processed are arranged in time sequence to obtain a high-resolution video corresponding to the video to be processed. In actual calculations, each high-resolution combined frame can be calculated and arranged behind the previous high-resolution combined frame calculated previously to finally obtain a high-resolution video. It can also be arranged in order after all high-resolution combined frames are calculated. The present disclosure does not limit this. By configuring a corresponding super-resolution network for each image frame and inputting the fusion features into the super-resolution network, a corresponding high-resolution image frame can be obtained for each image frame, ensuring that the super-resolution of each image frame is relatively independent.

[0117] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0118] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 702, a training method for a video super-resolution model or a video super-resolution method according to an exemplary embodiment of the present disclosure is executed.

[0119] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0120] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0121] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0122] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.

[0123] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0124] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the video super-resolution model training method or the video super-resolution method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0125] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer instructions. When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute a video super-resolution model training method or a video super-resolution method according to an exemplary embodiment of the present disclosure.

[0126] According to the video super-resolution method and device of the exemplary embodiment of the present disclosure, a multi-input and multi-output strategy is developed based on a recurrent convolutional network, that is, at least two image frames are used as input at a time (that is, one combined frame is input), and then a high-resolution combined frame of the corresponding combined frame is output, that is, at least two corresponding high-resolution image frames are output. In addition, the fusion features are obtained from the input combined frame and can be sent to the next time state to transfer historical information to future frames. Correspondingly, this calculation will also use the fusion features of the previous time state as input. This strategy can not only use parallel operations to reduce computational redundancy and reduce the overall super-resolution time of the entire video, but also implicitly learn the correlation and complementary information of the previous and next frames when super-resolving multiple frames at the same time, making full use of the timing information, increasing the temporal continuity of the super-resolution structure, and optimizing the processing efficiency while ensuring the super-resolution effect.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video super-resolution method, characterized in that: include: Acquire a video to be processed, wherein the video to be processed includes a plurality of combined frames arranged in time sequence, and each combined frame includes at least two image frames arranged in time sequence; Input the fusion features of the current combined frame and the processed previous combined frame into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; Input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame; The fusion features of the current combined frame are respectively input into the super-resolution networks corresponding to the respective image frames of the current combined frame to obtain the high-resolution image frames respectively corresponding to the respective image frames of the current combined frame, and all the high-resolution image frames are combined into a high-resolution combined frame corresponding to the current combined frame.

2. The video super-resolution method according to claim 1, characterized in that: The fusion network includes a fusion sub-network, The step of inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame includes: The fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain a fusion feature of the current combined frame; or The spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame are input into the fusion sub-network to obtain the fusion features of the current combined frame.

3. The video super-resolution method according to claim 2, characterized in that: The fusion network further includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, and the fusion sub-networks and the split sub-networks are alternately cascaded, The step of inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame further includes: Inputting the fusion features of the current combined frame into the splitting sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; The adjusted hidden layer features of each image frame of the current combined frame are input into the next fusion sub-network to obtain the fusion features of the current combined frame.

4. The video super-resolution method according to any one of claims 1 to 3, characterized in that: The extraction network includes at least two extraction neurons, and the number of the extraction neurons is equal to and corresponds to the number of image frames included in a combined frame. The step of inputting the fusion features of the current combined frame and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame includes: An image frame of the current combined frame and the fusion features of the previous combined frame obtained through processing are input into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

5. A video super-resolution model training method, characterized in that: The video super-resolution model includes a feature extraction network and a super-resolution network, the feature extraction network includes an extraction network and a fusion network, and the training method includes: Acquire a first video sample and a second video sample with the same content, wherein the resolution of the first video sample is smaller than the resolution of the second video sample, the first video sample and the second video sample each include a plurality of combined frames arranged in time sequence, each combined frame includes at least two image frames arranged in time sequence, and the number of image frames included in the super-resolution network and a combined frame is equal and corresponds one to one; Inputting the fusion features of the current combined frame of the first video sample and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; Inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame; Inputting the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, respectively, to obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combining all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame; The feature extraction network and the super-resolution network are trained based on the high-resolution combined frame corresponding to the current combined frame and the combined frame in the second video sample corresponding to the current combined frame.

6. The training method according to claim 5, characterized in that: The fusion network includes a fusion sub-network, The step of inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame includes: The fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain a fusion feature of the current combined frame; or The spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame are input into the fusion sub-network to obtain the fusion features of the current combined frame.

7. The training method according to claim 6, characterized in that: The fusion network further includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, and the fusion sub-networks and the split sub-networks are alternately cascaded, The step of inputting the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame further includes: Inputting the fusion features of the current combined frame into the splitting sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; The adjusted hidden layer features of each image frame of the current combined frame are input into the next fusion sub-network to obtain the fusion features of the current combined frame.

8. The training method according to any one of claims 5 to 7, characterized in that: The extraction network includes at least two extraction neurons, and the number of the extraction neurons is equal to and corresponds to the number of image frames included in a combined frame. The step of inputting the fusion features of the current combined frame of the first video sample and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame includes: An image frame of the current combined frame of the first video sample and the fusion features of the previous combined frame obtained through processing are input into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

9. A video super-resolution device, characterized in that: include: An acquisition unit is configured to: acquire a video to be processed, wherein the video to be processed includes a plurality of combined frames arranged in time sequence, and each combined frame includes at least two image frames arranged in time sequence; The fusion unit is configured to: input the fusion features of the current combined frame and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; The fusion unit is further configured to: input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame; The super-resolution unit is configured to: input the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combine all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame.

10. The video super-resolution device according to claim 9, characterized in that: The fusion network includes a fusion sub-network, and the fusion unit is further configured to: The fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain a fusion feature of the current combined frame; or The spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame are input into the fusion sub-network to obtain the fusion features of the current combined frame.

11. The video super-resolution device according to claim 10, characterized in that: The fusion network further includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, the fusion sub-networks are alternately cascaded with the split sub-networks, and the fusion unit is further configured as follows: Inputting the fusion features of the current combined frame into the splitting sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; The adjusted hidden layer features of each image frame of the current combined frame are input into the next fusion sub-network to obtain the fusion features of the current combined frame.

12. The video super-resolution device according to any one of claims 9 to 11, characterized in that: The extraction network includes at least two extraction neurons, the number of the extraction neurons is equal to and corresponds to the number of image frames included in a combined frame, and the fusion unit is further configured as follows: An image frame of the current combined frame and the fusion features of the previous combined frame obtained through processing are input into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

13. A video super-resolution model training device, characterized in that: The video super-resolution model includes a feature extraction network and a super-resolution network, the feature extraction network includes an extraction network and a fusion network, and the training device includes: An acquisition unit is configured to: acquire a first video sample and a second video sample with the same content, wherein the resolution of the first video sample is smaller than the resolution of the second video sample, the first video sample and the second video sample each include a plurality of combined frames arranged in time sequence, each combined frame includes at least two image frames arranged in time sequence, and the number of image frames included in the super-resolution network and a combined frame is equal and corresponds one to one; A fusion unit is configured to: input the fusion features of the current combined frame of the first video sample and the previous combined frame obtained through processing into the extraction network to obtain the hidden layer features of each image frame of the current combined frame; The fusion unit is further configured to: input the hidden layer features of each image frame of the current combined frame into the fusion network to obtain the fusion features of the current combined frame; The super-resolution unit is configured to: input the fusion features of the current combined frame into the super-resolution network corresponding to each image frame of the current combined frame, obtain high-resolution image frames corresponding to each image frame of the current combined frame, and combine all the high-resolution image frames into a high-resolution combined frame corresponding to the current combined frame; The training unit is configured to train the feature extraction network and the super-resolution network based on the high-resolution combined frame corresponding to the current combined frame and the combined frame in the second video sample corresponding to the current combined frame.

14. The training device according to claim 13, characterized in that The fusion network includes a fusion sub-network, and the fusion unit is further configured to: The fusion sub-network adds the hidden layer features of each image frame of the current combined frame to obtain a fusion feature of the current combined frame; or The spliced ​​features obtained by splicing the hidden layer features of each image frame of the current combined frame are input into the fusion sub-network to obtain the fusion features of the current combined frame.

15. The training device according to claim 14, characterized in that The fusion network further includes at least one split sub-network, the number of the fusion sub-networks is one more than the number of the split sub-networks, the fusion sub-networks are alternately cascaded with the split sub-networks, and the fusion unit is further configured as follows: Inputting the fusion features of the current combined frame into the splitting sub-network to split the fusion features of the current combined frame to obtain the adjusted hidden layer features of each image frame of the current combined frame; The adjusted hidden layer features of each image frame of the current combined frame are input into the next fusion sub-network to obtain the fusion features of the current combined frame.

16. A training device according to any one of claims 13 to 15, characterised in that The extraction network includes at least two extraction neurons, the number of the extraction neurons is equal to and corresponds to the number of image frames included in a combined frame, and the fusion unit is further configured as follows: An image frame of the current combined frame of the first video sample and the fusion features of the previous combined frame obtained through processing are input into the corresponding extraction neurons to obtain the hidden layer features of the corresponding image frame of the current combined frame.

17. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, they prompt the at least one processor to execute the video super-resolution method as described in any one of claims 1 to 4 or the video super-resolution model training method as described in any one of claims 5 to 8.

18. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the video super-resolution method as described in any one of claims 1 to 4 or the training method of the video super-resolution model as described in any one of claims 5 to 8.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the video super-resolution method according to any one of claims 1 to 4 or the video super-resolution model training method according to any one of claims 5 to 8 is implemented.

Images