Video super-division method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video super-division method and device, equipment, a storage medium and a program product, and relates to the field of audios and videos. The method comprises the following steps: dividing each to-be-hyper-divided target video frame in a to-be-hyper-divided target video into a plurality of first reference frame subsequences with the same length, and performing frame alignment on each first reference frame in the first reference frame subsequences to obtain a first aligned frame sequence, inputting the first alignment frame sequence into a pre-trained first information accumulation model to obtain a first information accumulation feature map, and inputting the first information accumulation feature map into a pre-trained second information accumulation model to obtain a second information accumulation feature map, and inputting the second information accumulation feature map into a pre-trained super-division model to perform convolution super-division so as to obtain the super-division video frame of the target video frame, and dividing the target video to be super-divided into a plurality of first reference frame subsequences to perform short-term information accumulation, thereby improving the quality of the super-division video.

Description

Technical Field

[0001] The present application relates to the field of audio and video, and in particular to a video super-resolution method, device, equipment, storage medium and program product. Background Art

[0002] Video Super Resolution (VSR) technology refers to the use of a series of technical methods to improve the resolution of video frames, thereby significantly improving the quality of the video. Video super resolution technology can be applied to the field of video surveillance, converting low-resolution financial transaction surveillance videos into high-definition videos, allowing monitoring personnel to more clearly see the behavior of traders, the characteristics of trading items, etc.

[0003] In the prior art, the convolutional long short-term memory network combines the advantages of convolutional neural networks and long short-term memory networks, that is, convolutional neural networks are good at capturing spatial features in images, while long short-term memory networks can effectively process temporal dependencies in sequence data. Therefore, the convolutional long short-term memory network can utilize the timing information between video frames to improve the accuracy and coherence of reconstructed video frames. Since the convolutional long short-term memory network uses convolution operations at the same time, it can also effectively process the spatial information in the video frames, thereby restoring clearer image details.

[0004] However, when the super-resolved video is too long, the amount of data that the convolutional long short-term memory network needs to process increases dramatically, and the computational complexity also increases accordingly. In practical applications, computing resources are limited. Therefore, the convolutional long short-term memory network may not be able to fully process all frames due to insufficient computing resources, and thus cannot accurately capture the temporal dependencies between frames, which in turn causes artifacts in the super-resolved video. Summary of the invention

[0005] The present application provides a video super-resolution method, apparatus, device, storage medium and program product, which are used to solve the technical problem that when super-resolution of long videos is performed, the timing dependency between frames cannot be accurately captured, which leads to artifacts in the super-resolution video.

[0006] In a first aspect, the present application provides a video super-resolution method, comprising:

[0007] For each target video frame to be super-resolved in the target video to be super-resolved, determining a first reference frame sequence required for super-resolving the target video frame according to a position of the target video frame in the target video, and dividing the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length, wherein the first reference frame sequence is a subsequence of the target video including the target video frame;

[0008] Frame aligning each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence;

[0009] Inputting the first aligned frame sequence into a pre-trained first information accumulation model to perform short-term information accumulation, and obtaining a first information accumulation feature map of each first aligned frame in the first aligned frame sequence;

[0010] Inputting the first information accumulation feature map of the target first aligned frame in each of the first aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation, thereby obtaining the second information accumulation feature map of the target first aligned frame in each of the first aligned frame sequences;

[0011] The second information accumulation feature map of the target first aligned frame corresponding to the target video frame is input into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

[0012] In a second aspect, the present application provides a model training method, comprising:

[0013] For each target sample frame to be super-resolved in a sample video in a sample video set, determining a second reference frame sequence required for super-resolving the target sample frame according to a position of the target sample frame in the sample video, and dividing the second reference frame sequence of the target sample frame into a plurality of second reference frame subsequences of the same length, wherein the second reference frame sequence is a subsequence of the sample video including the target sample frame;

[0014] Performing frame alignment on each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and inputting the second aligned frame sequence into a first information accumulation model to be trained to perform short-term information accumulation to obtain a first information accumulation feature map of each second aligned frame in the second aligned frame sequence;

[0015] Inputting the first information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences, and inputting the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into a super-resolution model to be trained to perform convolution super-resolution to obtain a super-resolution video frame of the target sample frame;

[0016] A loss value is determined according to the super-resolution video frame of the target sample frame and a high-resolution video frame preset for the target sample frame, so as to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

[0017] In a third aspect, the present application provides a video super-resolution device, comprising:

[0018] a processing module, configured to determine, for each target video frame to be super-resolved in the target video to be super-resolved, a first reference frame sequence required for super-resolving the target video frame according to a position of the target video frame in the target video, and divide the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length, wherein the first reference frame sequence is a subsequence of the target video including the target video frame;

[0019] The processing module is further configured to perform frame alignment on each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence;

[0020] An input module, used for inputting the first aligned frame sequence into a pre-trained first information accumulation model to perform short-term information accumulation and obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence;

[0021] The input module is further used to input the first information accumulation feature map of the target first alignment frame in each of the first alignment frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target first alignment frame in each of the first alignment frame sequences;

[0022] The input module is also used to input the second information accumulation feature map of the target first aligned frame corresponding to the target video frame into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

[0023] In a fourth aspect, the present application provides a model training device, comprising:

[0024] a processing module, configured to determine, for each target sample frame to be super-resolved in a sample video in a sample video set, a second reference frame sequence required for super-resolving the target sample frame according to a position of the target sample frame in the sample video, and divide the second reference frame sequence of the target sample frame into a plurality of second reference frame subsequences of the same length, wherein the second reference frame sequence is a subsequence of the sample video including the target sample frame;

[0025] The processing module is further used to perform frame alignment on each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and input the second aligned frame sequence into the first information accumulation model to be trained to perform short-term information accumulation to obtain a first information accumulation feature map of each second aligned frame in the second aligned frame sequence;

[0026] An input module, used for inputting the first information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences, and inputting the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into a super-resolution model to be trained to perform convolution super-resolution to obtain a super-resolution video frame of the target sample frame;

[0027] A determination module, configured to determine a loss value according to the super-resolution video frame of the target sample frame and a high-resolution video frame preset for the target sample frame;

[0028] An adjustment module is used to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

[0029] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0030] The memory stores computer-executable instructions;

[0031] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0032] In a sixth aspect, the present application provides a server, comprising: a processor, and a memory communicatively connected to the processor;

[0033] The memory stores computer-executable instructions;

[0034] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the second aspects.

[0035] In a seventh aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspect, or the method as described in any one of the second aspect.

[0036] In an eighth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of the first aspect, or the method described in any one of the second aspect.

[0037] The present application provides a video super-resolution method, apparatus, device, storage medium and program product. For each target video frame to be super-resolution in the target video to be super-resolution, according to the position of the target video frame in the target video, a first reference frame sequence required for super-resolution of the target video frame is determined, and the first reference frame sequence of the target video frame is divided into multiple first reference frame subsequences of the same length, and then the first reference frames in the first reference frame subsequences are frame-aligned to obtain a first aligned frame sequence corresponding to the first reference frame subsequence, and the first aligned frame sequence is input into a pre-trained first information accumulation model to perform short-term information accumulation to obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence, and the first information accumulation feature map of the target first aligned frame in each first aligned frame sequence is input into a pre-trained second information accumulation model to perform short-term information accumulation to obtain a second information accumulation feature map of the target first aligned frame in each first aligned frame sequence, and finally the second information accumulation feature map of the target first aligned frame corresponding to the target video frame is input into the pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame. Compared with the prior art, when the super-resolved video is too long, the computational complexity increases accordingly, while the computing resources are limited. As a result, the convolutional long short-term memory network is unable to fully process all frames due to insufficient computing resources, and is unable to accurately capture the temporal dependencies between frames, which in turn causes artifacts in the super-resolved video. The present application divides the video into a first reference frame sequence, and then divides the first reference frame sequence into multiple first reference frame subsequences, accumulates information twice for each first reference frame subsequence, and finally super-divides it into super-resolved video frames. The present application divides the long video into a first reference frame sequence, and then divides it into multiple first reference frame subsequences, and then accumulates information for the first reference frame subsequences respectively, so as to capture the temporal dependencies between frames through information accumulation, thereby improving the quality of the final super-resolved video. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] Figure 1 A schematic diagram of a scenario of a video super-resolution method provided in this application;

[0040] Figure 2 A flowchart of a first embodiment of a video super-resolution method provided by the present application;

[0041] Figure 3 A schematic diagram of a flow chart of a second embodiment of a video super-resolution method provided by the present application;

[0042] Figure 4 A flowchart of a third embodiment of a video super-resolution method provided by the present application;

[0043] Figure 5 A flowchart of a fourth embodiment of a video super-resolution method provided by the present application;

[0044] Figure 6 A flowchart of a fifth embodiment of a video super-resolution method provided by the present application;

[0045] Figure 7 A flowchart of a first embodiment of a model training method provided in this application;

[0046] Figure 8 A flowchart of a second embodiment of a model training method provided in this application;

[0047] Fig. 9 A video super-resolution device provided in an embodiment of the present application;

[0048] Fig.10 A model training device provided in an embodiment of the present application;

[0049] Fig.11 An electronic device provided in an embodiment of the present application;

[0050] Fig.12 A server is provided in an embodiment of the present application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0054] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.

[0055] It should be noted that the video super-resolution method, device, equipment, storage medium and program product provided in the present application can be used in the audio and video field, and can also be used in any field other than audio and video. The application field of the video super-resolution method, device, equipment, storage medium and product in the present application is not limited.

[0056] In the prior art, when the video that needs to be super-resolved is too long, the amount of data that needs to be processed by the convolutional long short-term memory network begins to increase sharply, and the computational complexity also increases accordingly. However, the actual computing resources are limited. Therefore, when the video that needs to be super-resolved is too long, the convolutional long short-term memory network may not be able to fully process all frames due to insufficient computing resources, and thus cannot accurately capture the temporal dependency between frames, which in turn causes artifacts in the super-resolved video.

[0057] Based on this, the inventive concept of the present application is to provide a video super-resolution method, device, equipment, storage medium and program product, which can divide a long video into shorter videos for super-resolution during super-resolution, and apply the segmented information accumulation mechanism in human learning knowledge to the field of video super-resolution. Therefore, the present application determines the first reference frame sequence according to the position of the video frame to be super-resolution, and then divides it into multiple first reference frame sub-sequences according to the first reference frame sequence, and then inputs the multiple first reference frame sub-sequences into the information accumulation model to accumulate information, and finally inputs the feature map accumulated according to the information into the super-resolution model to obtain the super-resolution video frame of the final target video frame.

[0058] The present application provides a video super-resolution method, device, equipment, storage medium and product, which are intended to solve the above technical problems in the prior art.

[0059] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0060] Figure 1 A schematic diagram of a video super-resolution method provided in this application, such as Figure 1 As shown, the target video to be super-resolved is first grouped and divided into multiple first reference frame subsequences, and then each first reference frame subsequence is input into a deformable convolution alignment module to align adjacent frames in the first reference frame subsequence, and then respectively input into the first information accumulation model to perform the first stage of short-term information accumulation, and the first information accumulation feature map is obtained after the first information accumulation, and then the target first information accumulation feature map in each first reference frame subsequence is selected and input into the second information accumulation model, and the second stage of short-term information accumulation is performed on it, and then it is finally input into the super-resolution model to perform convolution super-resolution to obtain the final super-resolution video frame.

[0061] Figure 2 A flowchart of a video super-resolution method embodiment 1 provided in this application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0062] S201: For each target video frame to be super-resolved in the target video to be super-resolved, determine the first reference frame sequence required for super-resolving the target video frame according to the position of the target video frame in the target video, and divide the first reference frame sequence of the target video frame into multiple first reference frame subsequences of the same length, where the first reference frame sequence is a subsequence of the target video that includes the target video frame.

[0063] In an embodiment of the present application, for each target video frame to be super-resolved in the target video to be super-resolved, wherein the target video frame to be super-resolved refers to each frame in the target video, a first reference frame sequence required for super-resolving the target video frame is determined according to the position of the target video frame in the target video, and then according to the first reference frame sequence, the first reference frame sequence is divided into multiple first reference frame subsequences of the same length, wherein the first reference frame sequence is a subsequence in the target video that includes the target video frame.

[0064] For example, the target video to be super-resolved has 10 frames, It-1 represents the first frame, It-2 represents the second frame, It-3 represents the third frame, It-4 represents the fourth frame, It-5 represents the fifth frame, It-6 represents the sixth frame, It-7 represents the seventh frame, It-8 represents the eighth frame, It-9 represents the ninth frame, It-10 represents the tenth frame, if the fifth frame is the target video frame to be super-resolved, then the first reference frame sequence determined according to the position of the super-resolved target video frame is: It-1, It-2, It-3, It-4, It-5, then the first reference frame subsequence can be: (It-5, It-4, It-3), (It-4, It-3, It-2), (It-3, It-2, It-1). It should be noted that the number of frames in the first reference frame sequence can be adjusted according to actual conditions, the position of the target video frame to be super-resolved in the first reference video frame sequence can also be adjusted according to actual conditions, and the division of the first reference frame sequence can also be adjusted according to actual conditions.

[0065] S202: Frame align each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence.

[0066] In the embodiment of the present application, frame alignment is performed on each first reference frame subsequence in the divided first reference frame sequence, and then a first aligned frame sequence corresponding to the first reference frame subsequence is obtained.

[0067] S203: Input the first aligned frame sequence into a pre-trained first information accumulation model to perform short-term information accumulation, and obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence.

[0068] In an embodiment of the present application, the obtained first aligned frame sequence is input into a pre-trained first information accumulation model, which is a trained ConvLSTM model. By inputting the first aligned frame sequence into the pre-trained first information accumulation model, short-term information accumulation is performed to obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence.

[0069] For example,

[0070] in, represents the first information accumulation model, F It-5 1 、F It-4 1 、F It-3 1 、F It-2 1 、F It-11 Respectively represent the first information accumulation feature graphs of each first aligned frame in the first aligned frame sequence.

[0071] S204: Input the first information accumulation feature map of the target first aligned frame in each first aligned frame sequence into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target first aligned frame in each first aligned frame sequence.

[0072] In the embodiment of the present application, the first information accumulation feature map of the target first aligned frame in each video first aligned frame sequence is input into the pre-trained second information accumulation model, and short-term information accumulation is performed again to obtain the second information accumulation feature map of the target first aligned frame in each first aligned frame sequence. Optionally, the target first aligned frame can be determined according to its position in the first aligned frame sequence, for example, the last frame in the first information accumulation feature map of each first aligned frame is selected as the target first aligned frame, that is, F is selected. It-3 1 、F It-2 1 、F It-1 1 Input into the pre-trained second information accumulation model. where f ConvLSTM2 represents the second information accumulation model, F It-3 2 、F It-2 2 、F It-1 2 Respectively represent the second information accumulation feature maps of the target first aligned frames in each first aligned frame sequence.

[0073] S205: Input the second information accumulation feature map of the target first aligned frame corresponding to the target video frame into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

[0074] In an embodiment of the present application, the second information accumulation feature map of the target first aligned frame corresponding to the target video frame is input into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame. The super-resolution model is a sub-pixel convolution super-resolution, that is, K2 cascaded residual blocks are connected in series to form a deep network structure for learning the deep features of the second information accumulation feature map, and then the deep features of the second information accumulation feature map are up-sampled by PixelShuffle sub-pixel convolution, thereby constructing a super-resolution video frame of the target video.

[0075] It should be noted that the first information accumulation model and the second information accumulation model mentioned above are ConvLSTM models with the same structure but different parameters. The first information accumulation model, the second information accumulation model, and the super-resolution model are obtained by joint training.

[0076] In an embodiment of the present application, by jointly training the first information accumulation model, the second information accumulation model and the super-resolution model, the overfitting or underfitting problems that may occur when training each model separately can be avoided, thereby improving the generalization ability and stability of the model.

[0077] In an embodiment of the present application, for each target video frame to be super-resolved in the target video to be super-resolved, the first reference frame sequence required for super-resolving the target video frame is determined according to the position of the target video frame in the target video, and the first reference frame sequence of the target video frame is divided into multiple first reference frame subsequences of the same length, and then the first reference frames in the first reference frame subsequences are frame aligned to obtain a first aligned frame sequence corresponding to the first reference frame subsequence, and the first aligned frame sequence is input into a pre-trained first information accumulation model to perform short-term information accumulation to obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence, and the first information accumulation feature map of the target first aligned frame in each first aligned frame sequence is input into a pre-trained second information accumulation model to perform short-term information accumulation to obtain a second information accumulation feature map of the target first aligned frame in each first aligned frame sequence, and finally the second information accumulation feature map of the target first aligned frame corresponding to the target video frame is input into the pre-trained super-resolved model to perform convolution super-resolved to obtain a super-resolved video frame of the target video frame. Compared with the prior art, when the super-resolved video is too long, the computational complexity increases accordingly, while the computing resources are limited. As a result, the convolutional long short-term memory network is unable to fully process all frames due to insufficient computing resources, and is unable to accurately capture the temporal dependencies between frames, which in turn causes artifacts in the super-resolved video. The present application divides the video into a first reference frame sequence, and then divides the first reference frame sequence into multiple first reference frame subsequences, accumulates information twice for each first reference frame subsequence, and finally super-divides it into super-resolved video frames. The present application divides the long video into a first reference frame sequence, and then divides it into multiple first reference frame subsequences, and then accumulates information for the first reference frame subsequences respectively, so as to capture the temporal dependencies between frames through information accumulation, thereby improving the quality of the final super-resolved video.

[0078] Figure 3 A flowchart of a second embodiment of a video super-resolution method provided in this application is shown in FIG. Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, a specific implementation of step S202 is as follows:

[0079] S301: For each first reference frame in a first reference frame subsequence, extract L feature maps of the first reference frame respectively, where L is a positive integer greater than or equal to 1.

[0080] In an embodiment of the present application, for each first reference frame in the first reference frame subsequence, L feature maps of the first reference frame are extracted respectively, where L is a positive integer greater than or equal to 1.

[0081] For example, if the first reference frame sequence is divided according to the example in step S201, the first reference frame subsequence is: (It-5, It-4, It-3), (It-4, It-3, It-2), (It-3, It-2, It-1). Then select the first reference frame subsequence (It-5, It-4, It-3) to explain: if the three sizes of feature maps of the first reference frame are extracted, then through 6 basic residual modules, three sizes of feature maps are extracted for It-5, namely S1(It-5), S2(It-5), S3(It-5), three sizes of feature maps are extracted for It-4, namely S1(It-4), S2(It-4), S3(It-4), and three sizes of feature maps are extracted for It-3, namely S1(It-3), S2(It-3), S3(It-3), and each first reference frame subsequence will obtain 9 feature maps.

[0082] S302: For each size of the feature map of each first reference frame, perform frame alignment on the feature map to obtain an aligned feature map of the feature map.

[0083] In the embodiment of the present application, for each size feature map of each first reference frame, the feature map is frame aligned to obtain an aligned feature map of the feature map. For example, three sizes of feature maps are extracted from It-5 and frame aligned, that is, S1(It-5), S2(It-5), and S3(It-5) are frame aligned to obtain S1 ′ (It-5), S2 ′ (It-5), S3 ′ (It-5).

[0084] S303: For each first reference frame, the L alignment feature maps of the first reference frame are fused to obtain an aligned first reference frame corresponding to the first reference frame in the first alignment frame sequence.

[0085] In the embodiment of the present application, for each first reference frame, the L aligned feature maps of the first reference frame are fused to obtain the aligned first reference frame corresponding to the first reference frame in the first aligned frame sequence. For example, for It-5, three feature maps of different sizes are extracted and frame aligned to obtain S1 ′(It-5), S2 ′ (It-5), S3 ′ (It-5) is merged, that is, S1 ′ (It-5), S2 ′ (It-5), S3 ′ (It-5) The pixel values ​​in the feature map are averaged according to the corresponding positions to obtain the fused aligned first reference frame F It-5 Similarly, the aligned first reference frames (F) of each first reference subsequence (It-5, It-4, It-3), (It-4, It-3, It-2), (It-3, It-2, It-1) can be obtained. It-5 、F It-4 、F It-3 ), (F It-4 、F It-3 、F It-2 ), (F It-3 、F It-2 、F It-1 ).

[0086] In an embodiment of the present application, for each first reference frame in the first reference frame subsequence, L feature maps of the first reference frame are extracted respectively, and for each feature map of each size of each first reference frame, the feature map is frame aligned to obtain an aligned feature map of the feature map, and for each first reference frame, the L aligned feature maps of the first reference frame are fused to obtain an aligned first reference frame corresponding to the first reference frame in the first aligned frame sequence, and finally a first aligned frame sequence corresponding to the first reference frame subsequence is obtained. By aligning the frames of each first reference frame in the first reference frame subsequence, the aligned adjacent frames can provide more context information, which helps to make the super-resolution video frames obtained after subsequent information accumulation and convolution super-resolution more accurate and clear.

[0087] Figure 4 A flowchart of a third embodiment of a video super-resolution method provided in this application is shown in FIG. Figure 4 As shown in the above Figure 1 or Figure 2 Based on the embodiment shown, the specific implementation method of determining the first reference frame sequence required for the super-resolution target video frame according to the position of the target video frame in the target video in step S201 is:

[0088] S401: Acquire an original video, and according to the positional relationship between the video frames included in the first reference frame sequence and the target video frames, extend the length of the original video to obtain a target video, wherein the target video includes: the original video and an extended video, wherein the extended video includes an extended video frame located before the original video and associated with the M starting video frames of the original video, and / or an extended video frame located after the original video and associated with the N ending video frames of the original video.

[0089] In an embodiment of the present application, an original video is obtained, and then the length of the original video is extended according to the positional relationship between the video frames included in the divided first reference frame sequence and the target video frames to obtain a target video. Wherein, when the original video is extended, it can be extended before the starting frame of the original video, or after the ending frame of the original video. Therefore, the target video includes: the original video and the extended video, and the extended video includes an extended video frame located before the original video and associated with the M starting video frames of the original video, and / or, an extended video frame located after the original video and associated with the N ending video frames of the original video.

[0090] In the embodiment of the present application, due to the boundary effect, that is, the lack of adjacent frames at the beginning and end of the video, directly performing super-resolution processing on the original video may cause the quality of the super-resolution video to deteriorate. Therefore, the original video is obtained, and according to the positional relationship between the video frames included in the first reference frame sequence and the target video frame, the original video is extended forward and backward, and adjacent frames are added to the beginning and end of the video to reduce the boundary effect and improve the quality of the super-resolution video.

[0091] Preferably, according to the positional relationship between the video frames included in the first reference frame sequence and the target video frame, the original video is length-extended to obtain the target video in the following specific implementation manner:

[0092] Motion estimation is performed on M starting video frames of the original video to obtain a first motion vector, and an extended video frame located before the original video is generated based on the first motion vector and the M starting video frames. The first motion vector is a motion vector from the subsequent video frame to the previous video frame.

[0093] In an embodiment of the present application, motion estimation is performed on M starting video frames of the original video to obtain a first motion vector, where the first motion vector is a motion vector from a subsequent video frame to a previous video frame. Based on the first motion vector and the M starting video frames, an extended video frame located before the original video is generated.

[0094] For example, for an original video with 10 video frames, namely It-1, It-2, It-3, It-4, It-5, It-6, It-7, It-8, It-9, It-10, the first reference frame sequence is divided into 5 frames, and the first 4 frames of the target frame and the target frame are selected as the first reference sequence. When It-1 is selected as the target frame, there are no corresponding 4 video frames in front of it as the first reference sequence, then the first motion vector is obtained by performing motion estimation on the first 4 starting video frames of the original video, where the first motion vector refers to the pixel movement direction and distance from the rear video frame to the front video frame, and then the corresponding 4 extended video frames are generated according to the first motion vector and the first 4 starting video frames.

[0095] Perform motion estimation on the N ending video frames of the original video to obtain a second motion vector, and generate an extended video frame located after the original video based on the second motion vector and the N ending video frames, where the second motion vector is a motion vector from the previous video frame to the next video frame.

[0096] In an embodiment of the present application, motion estimation is performed on the N ending video frames of the original video to obtain a second motion vector, where the second motion vector is a motion vector from the previous video frame to the next video frame. Based on the second motion vector and the N ending video frames, an extended video frame located after the original video is generated.

[0097] For example, for an original video with 10 video frames, namely It-1, It-2, It-3, It-4, It-5, It-6, It-7, It-8, It-9, It-10, the first reference frame sequence is divided into 5 frames, and the last 4 frames of the target frame and the target frame are selected as the first reference sequence. When It-10 is selected as the target frame, there are no corresponding 4 video frames behind it as the first reference sequence, then the second motion vector is obtained by performing motion estimation on the last 4 ending video frames of the original video, where the second motion vector refers to the pixel movement direction and distance from the previous video frame to the next video frame, and then the corresponding 4 extended video frames are generated according to the second motion vector and the last 4 ending video frames.

[0098] In an embodiment of the present application, a first motion vector is obtained by performing motion estimation on M starting video frames of the original video, and then an extended video frame located before the original video is generated based on the first motion vector and the M starting video frames. Alternatively, a second motion vector is obtained by performing motion estimation on N ending frames of the original video, and then an extended video frame located after the original video is generated based on the second motion vector and the N ending video frames. Whether there is no adjacent frame in front of the original video frame and the first reference frame sequence it constitutes, or there is no adjacent frame behind the original video frame and the first reference frame sequence it constitutes, this embodiment can generate an extended video frame located before the original video and an extended video frame after the original video based on motion estimation, providing more reference frames for each frame in the original video, thereby helping to improve the quality and stability of super-resolution processing.

[0099] Figure 5 A flowchart of a fourth embodiment of a video super-resolution method provided in this application is shown in FIG. Figure 5 As shown in the above Figure 2 or Figure 3 Based on the embodiment shown, a specific implementation manner of dividing the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length in step S201 is:

[0100] S501: Calculate the matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame.

[0101] In the embodiment of the present application, when the first reference frame sequence of the target video frame is divided into multiple first reference frame subsequences of the same length, the matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame may be calculated first.

[0102] S502: Divide first reference frames whose matching degree is greater than or equal to a preset matching degree threshold into the same first reference frame subsequence.

[0103] In an embodiment of the present application, based on the calculated matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame, first reference frames whose matching degree is greater than or equal to a preset matching degree threshold are divided into the same first reference frame subsequence.

[0104] In an embodiment of the present application, by calculating the matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame, and then dividing the first reference frames whose matching degree is greater than or equal to a preset matching degree threshold into the same first reference frame subsequence, it helps to reduce the information accumulation error caused by excessive differences between frames, thereby improving the accuracy and effectiveness of information accumulation, and further improving the quality of video super-resolution.

[0105] Figure 6 A flowchart of a fifth embodiment of a video super-resolution method provided in this application is shown in FIG. Figure 6 As shown in the above Figure 5 Based on the illustrated embodiment, a specific implementation of step S501 is as follows:

[0106] S601: Calculate content similarity, time interval, and motion vector of two adjacent first reference frames in a first reference frame sequence.

[0107] In the embodiment of the present application, the content similarity, time interval, and motion vector of two adjacent first reference frames in the first reference frame sequence are calculated.

[0108] Optionally, for example: the method for calculating the content similarity of two adjacent first reference frames is to extract features from the two adjacent first reference frames, such as edges, textures or color histograms, and then use a feature matching algorithm, such as Euclidean distance or cosine similarity, to calculate the content similarity between the features of the two adjacent first reference frames.

[0109] More optionally, the time interval between two adjacent first reference frames can be obtained by reading the video frame by frame, recording the timestamp of each frame, and then calculating the difference between the timestamps of adjacent frames.

[0110] More optionally, the motion vectors of two adjacent first reference frames may be obtained by an optical flow method, that is, the displacement is estimated by comparing the changes in brightness values ​​of corresponding points in the two frames, thereby calculating the motion vector.

[0111] S602: Obtain the matching degree of the two first reference frames according to the content similarity, time interval, and motion vector of two adjacent first reference frames. The matching degree is positively correlated with the content similarity and negatively correlated with the time interval and motion vector.

[0112] In an embodiment of the present application, the matching degree of the two first reference frames is calculated based on the content similarity, time interval and motion vector of the two adjacent first reference frames. For example, weights can be set for the content similarity, time interval and motion vector according to their importance in the matching degree calculation, and the weights add up to 1. Then, the content similarity, time interval and motion vector are multiplied by their corresponding weights respectively, and finally added together to obtain the final matching degree, wherein the matching degree is positively correlated with the content similarity, and negatively correlated with the time interval and motion vector. It should be noted that, according to actual needs, the matching degree can be calculated separately using content similarity, time interval and cloud top vector, or can be calculated in pairs.

[0113] In the embodiment of the present application, by calculating the content similarity, time interval, and motion vector of two adjacent first reference frames in the first reference frame sequence, the matching degree of the two first reference frames is obtained according to the content similarity, time interval, and motion vector of the two adjacent first reference frames. The similarity of the image content can be accurately measured by calculating the content similarity, and the time interval helps to eliminate the phenomenon of significant differences in image content due to a large time span, and the matching degree of the two frames of image in terms of dynamic features can be further confirmed by comparing the motion vectors. The matching degree of the two first reference frames is obtained by the content similarity, time interval, and motion vector, thereby improving the accuracy of the matching degree between adjacent frames.

[0114] Figure 7 A flow chart of a model training method embodiment 1 provided in this application, such as Figure 7 As shown, the method comprises the following steps:

[0115] S701: For each target sample frame to be super-resolved in a sample video in a sample video set, determine a second reference frame sequence required for super-resolving the target sample frame according to the position of the target sample frame in the sample video, and divide the second reference frame sequence of the target sample frame into multiple second reference frame subsequences of the same length, where the second reference frame sequence is a subsequence in the sample video that includes the target sample frame.

[0116] In an embodiment of the present application, for each target sample frame to be super-resolved in a sample video in a sample video set, a second reference frame sequence required for super-resolving the target sample frame is determined according to the position of the target sample frame in the sample video, wherein the second reference frame sequence is a subsequence of the sample video including the target sample frame, and then the second reference frame sequence of the target sample frame is divided into multiple second reference frame subsequences of the same length.

[0117] S702: Frame align each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and input the second aligned frame sequence into the first information accumulation model to be trained to perform short-term information accumulation to obtain a first information accumulation feature map of each second aligned frame in the second aligned frame sequence.

[0118] In the embodiment of the present application, each second reference frame divided into a second reference frame subsequence is aligned respectively to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame. Then, the second aligned frame sequence is input into the first information accumulation model to be trained to perform short-term information accumulation, and finally obtain the first information accumulation feature map of each second aligned frame in the second aligned frame sequence.

[0119] S703: Input the first information accumulation feature map of the target second aligned frame in each second aligned frame sequence into the pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each second aligned frame sequence, and input the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into the super-resolution model to be trained to perform convolution super-resolution to obtain the super-resolution video frame of the target sample frame.

[0120] In an embodiment of the present application, the first information accumulation feature map of the target second aligned frame in each second aligned frame sequence is input into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each second aligned frame sequence, and the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame is input into the super-resolution model to be trained to perform convolution super-resolution to obtain the super-resolution video frame of the target sample frame.

[0121] S704: Determine a loss value according to the super-resolution video frame of the target sample frame and the high-resolution video frame preset for the target sample frame, so as to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

[0122] In an embodiment of the present application, a loss value is determined based on the obtained super-resolution video frame of the target sample frame and the preset high-resolution video frame of the target sample frame, and back propagation is used to update the parameters of the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value. When the loss value is minimized, the corresponding first information accumulation model, the second information accumulation model and the super-resolution model are the trained first information accumulation model, the second information accumulation model and the super-resolution model.

[0123] Optionally, the loss value can be calculated using the Charbonnier loss function:

[0124]

[0125] Among them, L Charbonnier is the loss value, A high-resolution video frame preset for the target sample frame, is the super-resolution video frame of the target sample frame, l is an empirical setting value, which can be set to 1×10 -3 .

[0126] In an embodiment of the present application, a sample video set is used to jointly train the first information accumulation model, the second information accumulation model and the super-resolution model, and the Charbonnier loss function is used to calculate the loss value during the training process. The first information accumulation model, the second information accumulation model and the super-resolution model corresponding to the minimum loss value are selected as the trained first information accumulation model, the second information accumulation model and the super-resolution model. This can avoid problems such as gradient disappearance or explosion during the training process, thereby accelerating the convergence speed of the model and improving the training efficiency.

[0127] Figure 8 A flow chart of a second embodiment of a model training method provided in this application, such as Figure 8 As shown in the above Figure 7 Based on the embodiment shown, the process of obtaining the sample video includes:

[0128] S801: Acquire a high-resolution video set, where the high-resolution video set includes a plurality of high-resolution videos, and the high-resolution video includes a plurality of high-resolution video frames.

[0129] In an embodiment of the present application, a high-resolution video set is first obtained, wherein the high-resolution video set includes a plurality of high-resolution videos, and the high-resolution video includes a plurality of high-resolution video frames.

[0130] Preferably, obtaining a high-resolution video set includes:

[0131] Public training videos in a public training set are preprocessed to obtain high-resolution videos, where the high-resolution videos are video clips of the public training videos, and / or, video frames in the high-resolution videos are obtained by cropping video frames in the public training videos, and / or, video frames in the high-resolution videos are obtained by converting video frames in the public training videos.

[0132] In an embodiment of the present application, a public training video in a public training set, such as Vimeo-90k, is preprocessed to obtain a high-resolution video. Since the amount of data of the public training video Vimeo-90k in the public training set is huge, directly processing the entire video requires extremely high computing resources, which exceeds the processing capacity of the processor. Therefore, the high-resolution video is preprocessed to significantly reduce the data dimension and increase the processing efficiency while maintaining the key information of the video. For example, by selecting a video clip of a public training video as a high-resolution video; or, cropping the video frames in the public training video, for example, cropping the video frames in the training video into high-resolution video frames with a length of 256 and a width of 256; or, converting the video frames in the public training video to obtain video frames in the high-resolution video, wherein the conversion operation includes but is not limited to: rotating, flipping or mirroring the video frames in the public training video, and by converting the video frames in the public training video, the model can train these converted videos so that the model can better adapt to these changes, thereby improving the robustness in practical applications. It should be noted that the preprocessing operation includes but is not limited to the above three processing methods used alone or in combination.

[0133] S802: Down-sample the high-resolution video to obtain a sample video.

[0134] In the embodiment of the present application, the obtained high-resolution video is downsampled, that is, the high-resolution video frame is downsampled to obtain a sample video. For example, the length of the high-resolution video frame in the obtained high-resolution video is 256 and the width is 256. The high-resolution video frame is reduced by 4 times through downsampling, and the length of the obtained high-resolution video frame is 64 and the width is 64. Optionally, a Gaussian blur downsampling method is selected to reduce the frame rate of the high-resolution video to simulate video data in a low frame rate environment, thereby improving the training performance of the model.

[0135] Preferably, the specific implementation method of downsampling the high-resolution video to obtain the sample video is:

[0136] The high-resolution video is downsampled and normalized to obtain a sample video.

[0137] In the embodiment of the present application, the high-resolution video is first downsampled to obtain a high-resolution video frame with a length of 64 and a width of 64, and then normalized to obtain a sample video. Optionally, the normalization process is to divide the pixel value in each frame of the high-resolution video frame by the average pixel value or the maximum pixel value in the frame, so that in the model training stage, the model can learn the image structure and texture information of the video frame faster, thereby improving the convergence speed and performance of the model.

[0138] In an embodiment of the present application, a high-resolution video set is obtained, the high-resolution video set includes multiple high-resolution videos, the high-resolution video includes multiple high-resolution video frames, and then the high-resolution video is down-sampled to obtain a sample video. By down-sampling the high-resolution video to obtain the sample video, and then using the sample video to train the model, the model can learn to extract useful information from blurry, low-resolution videos and restore high-quality videos, which helps to improve model performance.

[0139] Fig. 9 A video super-resolution device provided in an embodiment of the present application includes: a processing module 901 and an input module 902; wherein,

[0140] The processing module 901 is used to determine the first reference frame sequence required for the super-resolved target video frame for each target video frame to be super-resolved in the target video according to the position of the target video frame in the target video, and divide the first reference frame sequence of the target video frame into multiple first reference frame subsequences of the same length, and the first reference frame sequence is a subsequence of the target video that includes the target video frame; the processing module 901 is also used to perform frame alignment on each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence; the input module 902 is used to input the first aligned frame sequence into the first information sequence obtained by pre-training In the accumulation model, short-term information accumulation is performed to obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence; the input module 902 is also used to input the first information accumulation feature map of the target first aligned frame in each first aligned frame sequence into the pre-trained second information accumulation model to perform short-term information accumulation to obtain a second information accumulation feature map of the target first aligned frame in each first aligned frame sequence; the input module 902 is also used to input the second information accumulation feature map of the target first aligned frame corresponding to the target video frame into the pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

[0141] Fig.10 A model training device provided in an embodiment of the present application includes: a processing module 1001, an input module 1002, a determination module 1003 and an adjustment module 1004; wherein,

[0142] The processing module 1001 is used to determine, for each target sample frame to be super-resolved in the sample video in the sample video set, the second reference frame sequence required for super-resolving the target sample frame according to the position of the target sample frame in the sample video, and divide the second reference frame sequence of the target sample frame into multiple second reference frame subsequences of the same length, where the second reference frame sequence is a subsequence in the sample video that includes the target sample frame; the processing module 1001 is also used to perform frame alignment on each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and input the second aligned frame sequence into the first information accumulation model to be trained to perform short-term information accumulation to obtain a second aligned frame sequence of each second aligned frame in the second aligned frame sequence. An information accumulation feature map; an input module 1002, used to input the first information accumulation feature map of the target second aligned frame in each second aligned frame sequence into a pre-trained second information accumulation model to perform short-term information accumulation, and obtain the second information accumulation feature map of the target second aligned frame in each second aligned frame sequence, and input the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into the super-resolution model to be trained to perform convolution super-resolution, and obtain the super-resolution video frame of the target sample frame; a determination module 1003, used to determine the loss value according to the super-resolution video frame of the target sample frame and the high-resolution video frame preset by the target sample frame; an adjustment module 1004, used to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

[0143] Fig.11 An electronic device provided in an embodiment of the present application includes: at least one processor 1101 and a memory 1102 . The processor 1101 and the memory 1102 are connected via a bus 1103 .

[0144] The specific implementation process of the processor 1101 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0145] In the above Fig.11In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0146] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0147] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0148] Fig.12 A server provided in an embodiment of the present application includes: at least one processor 1201 and a memory 1202 . The processor 1201 and the memory 1202 are connected via a bus 1203 .

[0149] The specific implementation process of the processor 1201 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0150] In the above Fig.12In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0151] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0152] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0153] The present application also provides a computer-readable storage medium having computer execution instructions stored thereon. When a processor executes the computer execution instructions, a video super-resolution method as described above and / or a model training method are implemented.

[0154] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0155] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0156] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0160] The present application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a video super-resolution method and / or a model training method as described above.

[0161] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0162] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0163] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0164] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0165] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0166] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0169] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video super-resolution method, characterized in that: include: For each target video frame to be super-resolved in the target video to be super-resolved, determining a first reference frame sequence required for super-resolving the target video frame according to a position of the target video frame in the target video, and dividing the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length, wherein the first reference frame sequence is a subsequence of the target video including the target video frame; Frame aligning each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence; Inputting the first aligned frame sequence into a pre-trained first information accumulation model to perform short-term information accumulation, and obtaining a first information accumulation feature map of each first aligned frame in the first aligned frame sequence; Inputting the first information accumulation feature map of the target first aligned frame in each of the first aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation, thereby obtaining the second information accumulation feature map of the target first aligned frame in each of the first aligned frame sequences; The second information accumulation feature map of the target first aligned frame corresponding to the target video frame is input into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

2. The method according to claim 1, characterized in that The step of performing frame alignment on each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence includes: For each first reference frame in the first reference frame subsequence, extract L feature maps of the first reference frame respectively, where L is a positive integer greater than or equal to 1; For each feature map of the size of each of the first reference frames, perform frame alignment on the feature map to obtain an aligned feature map of the feature map; For each of the first reference frames, the L alignment feature maps of the first reference frame are fused to obtain an aligned first reference frame corresponding to the first reference frame in the first alignment frame sequence.

3. The method according to claim 1 or 2, characterized in that: The step of determining, according to a position of the target video frame in the target video, a first reference frame sequence required for super-resolution of the target video frame comprises: An original video is acquired, and according to the positional relationship between the video frames included in the first reference frame sequence and the target video frames, the length of the original video is extended to obtain the target video, wherein the target video includes: the original video and an extended video, wherein the extended video includes an extended video frame located before the original video and associated with M starting video frames of the original video, and / or an extended video frame located after the original video and associated with N ending video frames of the original video.

4. The method according to claim 3, characterized in that The step of extending the length of the original video according to the positional relationship between the video frames included in the first reference frame sequence and the target video frame to obtain the target video comprises: Performing motion estimation on M starting video frames of the original video to obtain a first motion vector, and generating an extended video frame located before the original video according to the first motion vector and the M starting video frames, wherein the first motion vector is a motion vector from a subsequent video frame to a previous video frame; And / or, motion estimation is performed on the N ending video frames of the original video to obtain a second motion vector, and based on the second motion vector and the N ending video frames, an extended video frame located after the original video is generated, wherein the second motion vector is a motion vector from the previous video frame to the next video frame.

5. The method according to claim 1 or 2, characterized in that: The step of dividing the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length comprises: Calculating the matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame; The first reference frames whose matching degree is greater than or equal to a preset matching degree threshold are divided into the same first reference frame subsequence.

6. The method according to claim 5, characterized in that The calculating the matching degree between any two adjacent first reference frames in the first reference frame sequence of the target video frame includes: Calculating content similarity, time interval, and motion vector of two adjacent first reference frames in the first reference frame sequence; According to the content similarity of the two adjacent first reference frames, the time interval, and the motion vector, the matching degree of the two first reference frames is obtained, and the matching degree is positively correlated with the content similarity and negatively correlated with the time interval and the motion vector.

7. The method according to claim 1, characterized in that The first information accumulation model and the second information accumulation model are ConvLSTM models with the same structure but different parameters. The first information accumulation model, the second information accumulation model, and the super-resolution model are obtained by joint training.

8. A model training method, characterized in that: include: For each target sample frame to be super-resolved in a sample video in a sample video set, determining a second reference frame sequence required for super-resolving the target sample frame according to a position of the target sample frame in the sample video, and dividing the second reference frame sequence of the target sample frame into a plurality of second reference frame subsequences of the same length, wherein the second reference frame sequence is a subsequence of the sample video including the target sample frame; Performing frame alignment on each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and inputting the second aligned frame sequence into a first information accumulation model to be trained to perform short-term information accumulation to obtain a first information accumulation feature map of each second aligned frame in the second aligned frame sequence; Inputting the first information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences, and inputting the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into a super-resolution model to be trained to perform convolution super-resolution to obtain a super-resolution video frame of the target sample frame; A loss value is determined according to the super-resolution video frame of the target sample frame and a high-resolution video frame preset for the target sample frame, so as to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

9. The method according to claim 8, characterized in that The process of obtaining the sample video includes: Acquire a high-resolution video set, wherein the high-resolution video set includes a plurality of high-resolution videos, and the high-resolution video includes a plurality of high-resolution video frames; The high-resolution video is downsampled to obtain the sample video.

10. The method according to claim 9, characterized in that The downsampling the high-resolution video to obtain the sample video includes: The high-resolution video is downsampled and normalized to obtain the sample video.

11. The method according to any one of claims 9 or 10, characterized in that: The step of obtaining a high-resolution video set comprises: The high-resolution video is obtained by preprocessing the public training videos in the public training set, wherein the high-resolution video is a video clip of the public training video, and / or the video frames in the high-resolution video are obtained by cropping the video frames in the public training video, and / or the video frames in the high-resolution video are obtained by converting the video frames in the public training video.

12. A video super-resolution device, characterized in that: include: a processing module, configured to determine, for each target video frame to be super-resolved in the target video to be super-resolved, a first reference frame sequence required for super-resolving the target video frame according to a position of the target video frame in the target video, and divide the first reference frame sequence of the target video frame into a plurality of first reference frame subsequences of the same length, wherein the first reference frame sequence is a subsequence of the target video including the target video frame; The processing module is further configured to perform frame alignment on each first reference frame in the first reference frame subsequence to obtain a first aligned frame sequence corresponding to the first reference frame subsequence; An input module, used for inputting the first aligned frame sequence into a pre-trained first information accumulation model to perform short-term information accumulation and obtain a first information accumulation feature map of each first aligned frame in the first aligned frame sequence; The input module is further used to input the first information accumulation feature map of the target first alignment frame in each of the first alignment frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target first alignment frame in each of the first alignment frame sequences; The input module is also used to input the second information accumulation feature map of the target first aligned frame corresponding to the target video frame into a pre-trained super-resolution model to perform convolution super-resolution to obtain a super-resolution video frame of the target video frame.

13. A model training device, characterized in that: include: a processing module, configured to determine, for each target sample frame to be super-resolved in a sample video in a sample video set, a second reference frame sequence required for super-resolving the target sample frame according to a position of the target sample frame in the sample video, and divide the second reference frame sequence of the target sample frame into a plurality of second reference frame subsequences of the same length, wherein the second reference frame sequence is a subsequence of the sample video including the target sample frame; The processing module is further used to perform frame alignment on each second reference frame in the second reference frame subsequence of the target sample frame to obtain a second aligned frame sequence corresponding to the second reference frame subsequence of the target sample frame, and input the second aligned frame sequence into the first information accumulation model to be trained to perform short-term information accumulation to obtain a first information accumulation feature map of each second aligned frame in the second aligned frame sequence; An input module, used for inputting the first information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences into a pre-trained second information accumulation model to perform short-term information accumulation to obtain the second information accumulation feature map of the target second aligned frame in each of the second aligned frame sequences, and inputting the second information accumulation feature map of the target second aligned frame corresponding to the target sample frame into a super-resolution model to be trained to perform convolution super-resolution to obtain a super-resolution video frame of the target sample frame; A determination module, configured to determine a loss value according to the super-resolution video frame of the target sample frame and a high-resolution video frame preset for the target sample frame; An adjustment module is used to adjust the first information accumulation model, the second information accumulation model and the super-resolution model according to the loss value.

14. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

15. A server, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 8 to 11.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 7, or the method according to any one of claims 8 to 11.

17. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7, or the method according to any one of claims 8 to 11.

Images