A video super-resolution method, device, apparatus and storage medium

Abstract

The application discloses a video super-resolution method, device and equipment and a storage medium. The method comprises the following steps: acquiring an original video frame sequence comprising at least two original video frames; performing a video super-resolution operation on the original video frame sequence by using a pre-trained target multi-scale video super-resolution model to output a target video frame sequence comprising target video frames; wherein the target multi-scale video super-resolution model comprises a multi-scale feature interaction network and an image reconstruction network, the multi-scale feature interaction network is used for multi-scale feature interaction and feature fusion of each original video frame, and the image reconstruction network performs image reconstruction based on a feature map output by the feature interaction network. The application constructs a multi-scale video super-resolution model, performs information interaction on feature maps of video frames in different sizes, can pay attention to feature information of different granularities, avoids using an alignment module in an existing video super-resolution model, reduces the amount of calculation, and improves the performance of the model.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and in particular to a video super-resolution method, apparatus, device, and storage medium. Background Technology

[0002] Compared to image super-resolution tasks, video super-resolution (VSR) not only requires utilizing the inherent characteristics of a single image frame for image super-resolution, but also involves aggregating information extracted from multiple highly related but misaligned frames in a video sequence. Due to the motion of objects in the video and the movement of the camera lens, there is a significant displacement between the information in different frames of the video. To better utilize the information from different frames, existing methods design dedicated alignment modules to align the information from different frames, and specific comparative experiments are conducted to demonstrate the necessity of the proposed alignment modules.

[0003] Currently, several representative methods exist. For example, in the RBPN method, multiple projection modules are used to sequentially aggregate features from multiple frames. In the BasicVSR method, the common VSR framework is summarized into four parts: Propagation, Alignment, Aggregation, and Upsampling. Bidirectional propagation is used to extract information from the entire input video for reconstruction, and optical flow is used for feature warping. The recently proposed BasicVSR++, based on BasicVSR, uses a more complex alignment module to further and better align features from different frames. Swin-Transformer combines the advantages of CNN and Transformer, showing great promise in the field of computer science. SwinIR, built based on the basic modules proposed in Swin-Transformer, has also achieved better performance than CNN in many low-level vision tasks with the same number of parameters.

[0004] Although the above methods can all achieve video super-resolution tasks, they all have various complex alignment modules in order to better fuse information between different frames. The implementation process is complicated, the amount of computation is large, and the requirements for processing equipment are high. Summary of the Invention

[0005] This invention provides a video super-resolution invention title to enable rapid super-resolution operation on video images.

[0006] According to one aspect of the present invention, a video super-resolution method is provided, the method comprising:

[0007] According to another aspect of the present invention, a video super-resolution apparatus is provided, the apparatus comprising:

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0009] At least one processor; and

[0010] A memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video super-resolution method according to any embodiment of the present invention.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video super-resolution method according to any embodiment of the present invention.

[0013] The technical solution of this invention involves acquiring an original video frame sequence containing at least two original video frames, employing a pre-trained target multi-scale video super-resolution model to perform video super-resolution on the original video frame sequence, and outputting a target video frame sequence containing the target video frame. The target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network. The multi-scale feature interaction network performs multi-scale feature interaction and feature fusion on each original video frame, and the image reconstruction network reconstructs the image based on the feature map output by the feature interaction network. This solves the problem of complex computation and high computational load caused by the use of video frame alignment methods in existing video super-resolution methods for information fusion between different frames. This invention constructs a multi-scale video super-resolution model that forms feature maps of different sizes from the video frames for information interaction, allowing focus on different fine-grained feature information, thereby reducing computational load and improving model performance.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1aThis is a flowchart of a video super-resolution method provided in Embodiment 1 of the present invention;

[0017] Figure 1b This is a schematic diagram of the VSTL layer in a video super-resolution method according to Embodiment 1 of the present invention;

[0018] Figure 1c This is a schematic diagram illustrating the principle of a video super-resolution method according to Embodiment 1 of the present invention;

[0019] Figure 2 This is a schematic diagram of the structure of a video super-resolution device according to Embodiment 2 of the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the video super-resolution method of this invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," "original," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] Figure 1a The flowchart of a video super-resolution method is provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of super-resolution processing of video images. The method can be executed by a video super-resolution device, which can be implemented in hardware and / or software and can be configured in a computer device. As shown in Figure 1, the method includes:

[0025] S110. Obtain an original video frame sequence containing at least two original video frames.

[0026] In practical applications, when super-resolution operations are required on video images, a long video segment can be cropped to form a video clip containing a certain number of video frames. In this embodiment, the set of video frames contained in each video clip before the super-resolution operation can be called an original video frame sequence. Multiple consecutive video frames within an original video frame sequence can be called original video frames.

[0027] S120. A pre-trained target multi-scale video super-resolution model is used to perform video super-resolution on the original video frame sequence, and output a target video frame sequence containing the target video frame. The target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network. The multi-scale feature interaction network is used to perform multi-scale feature interaction and feature fusion on each original video frame, and the image reconstruction network reconstructs the image based on the feature map output by the feature interaction network.

[0028] In this embodiment, a multi-scale video super-resolution model can be pre-built and trained with a large amount of training data to obtain the target multi-scale video super-resolution model. When performing super-resolution processing on the original video frame sequence, the original video frame sequence can be used as input data and input into the target multi-scale video super-resolution model. The target multi-scale video super-resolution model performs information interaction and feature fusion at different scales on each original video frame in the original video frame sequence. After reconstructing the feature map of each original video frame, the corresponding target video frame can be obtained. The target video frames are arranged in order to form the super-resolution processed target video frame sequence.

[0029] Optionally, the multi-scale feature interaction network can be composed of an initial feature interaction module, at least one multi-scale feature interaction module, and an end feature interaction module connected in series. The initial feature interaction module can be used to perform feature interaction on each original video frame. The multi-scale feature interaction module can be used to perform multi-scale feature interaction and feature fusion on all feature maps output by the initial feature interaction subnet or the previous multi-scale feature interaction module. The end feature interaction module can be used to perform feature interaction on all feature maps output by the last multi-scale feature interaction module.

[0030] In practical applications, the number of multi-scale feature interaction modules can be set according to the specific application scenario, and can be 2, 3, or 4. When there are 2 multi-scale feature interaction modules, the original video frame sequence is input into the target multi-scale video super-resolution model. The initial feature interaction module performs feature interaction on all the original video frames in the original video frame sequence, and outputs feature maps after feature interaction between each original video frame and other original video frames. After the first multi-scale feature interaction module obtains the feature maps output by the initial feature interaction module, it performs multi-scale feature interaction and feature fusion on each feature map, and then inputs it into the second multi-scale feature interaction module. The second multi-scale feature interaction module again performs multi-scale feature interaction and feature fusion on the feature maps output by the first multi-scale feature interaction module, and then inputs them into the end feature interaction module. The end feature interaction module then performs feature interaction on all the feature maps output by the second multi-scale feature interaction module.

[0031] Furthermore, the multi-scale feature interaction module may include at least two RVSTB units and one feature fusion unit; the feature map input to the multi-scale feature interaction module is downsampled according to a preset sampling frequency, the RVSTB units are used to perform feature interaction on the downsampled feature map, and after feature interaction, it is upsampled according to a preset sampling frequency; the feature fusion unit is used to perform feature fusion on all output feature maps corresponding to the same input feature map of each RVSTB unit.

[0032] In this embodiment, the RVSTB unit may include at least two VSTL layers and one convolutional layer. The VSTL layers may employ a shift window mechanism and an attention mechanism to achieve feature interaction.

[0033] In practical applications, the RVSTB (Residual Video Swing Transformer Block) unit can construct the correlation between different video frames, enabling continuous information exchange between multiple video frames. The number of RVSTB units in the multi-scale feature interaction module can be specifically set according to the actual application scenario. The number of RVSTB units can be consistent with the number of preset sampling frequencies. That is, if there are 3 preset sampling frequencies for the feature map, the number of RVSTB units in the multi-scale feature interaction module is 3; if there are 4 preset sampling frequencies for the feature map, the number of RVSTB units in the multi-scale feature interaction module is 4.

[0034] Similarly, the number of VSTL layers in an RVSTB unit can be set according to the specific application scenario. The number of VSTL layers reflects the degree of feature interaction. It can be understood that, within a certain range, the more VSTL layers there are, the deeper the information interaction between feature maps.

[0035] Specifically, a VSTL layer can consist of LayerNorm, MSA, MLP, and residual connections. This network structure can be an improvement on the standard multi-head self-attention used in the original Transformer layer, with the main differences being the local attention mechanism and the shift window mechanism. Figure 1b This is a schematic diagram of the VSTL layer structure in a video super-resolution method according to Embodiment 1 of the present invention, as shown below. Figure 1b As shown, the given input size is T×H×W×C, where T represents the number of input frames, H represents the height of each input image, W represents the width of each input image, and C represents the number of channels in each input image (the default number of channels is 3). The Video SwinTransformer can first use a single Conv3d layer to divide the input into non-overlapping N×M×M windows. For each window, its self-attention result can be calculated independently. For the features of a small window... N×M 2 ×C can represent the dimension of the vectors of all points within the small window, where the dimension of each point is C. Since there are N×M×M pixels in the small window, the dimension is N×M. 2 ×C; and combine it with three learnable mapping matrices P Q P K P V Multiplying them separately yields the corresponding query(Q), key(K), and value(V) values. The attention matrix can then be calculated.

[0036]

[0037] in, The similarity between query and key can be calculated, meaning that each point in the query and each point in the key is a similarity result; then multiplying it with V yields the new features after fusing other frames and the frame itself.

[0038] The attention operation described above will be performed in parallel h times, and then these h results will be concatenated.

[0039] An MLP layer can contain two FC layers and a GELU activation function. A LayerNorm layer is used between the MSA and MLP layers, connecting a residual connection after both the MSA and MLP operations. The entire process can be expressed by the following formula:

[0040] X = MSA(LN(X)) + X,

[0041] X = MLP(LN(X)) + X,

[0042] The operations described above are all performed within a locally divided small window. In order to enable information exchange between different small windows, a shift window mechanism can be used.

[0043] Optionally, the image reconstruction network includes a feature image reconstruction module, an interpolation image construction module, and an image fusion module. The feature image reconstruction module is used to reconstruct the features of the feature map output by the feature interaction network to form a reconstructed image. The interpolation image construction module is used to interpolate the original video frame to form an interpolated image. The image fusion module is used to fuse the reconstructed image and the interpolated image to form the target video frame.

[0044] Specifically, the feature map output by the feature interaction network is reconstructed to form a reconstructed image. At the same time, the original video frame is interpolated using an interpolation amplification method to form an interpolated image. The reconstructed image and the interpolated image corresponding to the same original video frame are then combined to obtain the super-resolution target video frame.

[0045] For example, Figure 1c This is a schematic diagram illustrating the principle of a video super-resolution method according to Embodiment 1 of the present invention. Figure 1c As shown, the acquired original video frame sequence contains 5 original video frames. The multi-scale feature interaction network in the target multi-scale video super-resolution model consists of one initial feature interaction module, three multi-scale feature interaction modules, and one terminal feature interaction module connected in series. The initial feature interaction module includes one RVSTB unit, each multi-scale feature interaction module includes three RVSTB units and one feature fusion unit, and the terminal feature interaction module includes one RVSTB unit. The preset sampling frequencies are 1x, 2x, and 4x. One RVSTB unit includes six VSTL layers and one convolutional layer, and one feature fusion unit includes one convolutional layer. The input five consecutive frames of raw video images first pass through one RVSTB module, and the output feature map passes through three multi-scale feature interaction modules. Each multi-scale feature interaction module processes three feature maps of different sizes using the RVSTB module. For the processed feature maps of the three sizes, two smaller feature maps can be transformed to the same size as the main feature map. Then, these three feature maps are concatenated, and then information is aggregated using one Conv3D module. After passing through the three multi-scale feature interaction modules, it passes through another RVSTB module for further fusion of information from different feature maps. Finally, it passes through a feature reconstruction module, which adds the output of the feature reconstruction to the input image that has been enlarged using interpolation, to obtain the final multi-frame super-resolution result.

[0046] The technical solution of this invention involves acquiring an original video frame sequence containing at least two original video frames, employing a pre-trained target multi-scale video super-resolution model to perform video super-resolution on the original video frame sequence, and outputting a target video frame sequence containing the target video frame. The target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network. The multi-scale feature interaction network performs multi-scale feature interaction and feature fusion on each original video frame, and the image reconstruction network reconstructs the image based on the feature map output by the feature interaction network. This solves the problem of complex computation and high computational load caused by the use of video frame alignment methods in existing video super-resolution methods for information fusion between different frames. This invention constructs a multi-scale video super-resolution model that forms feature maps of different sizes from the video frames for information interaction, allowing focus on different fine-grained feature information, thereby reducing computational load and improving model performance.

[0047] Based on the above scheme, the training process of the target multi-scale video super-resolution model in this embodiment may include:

[0048] A1. Obtain a training dataset containing low-resolution video frame sequences and corresponding standard high-resolution video frame sequences.

[0049] Specifically, training the model requires a large amount of training data. Each set of training data can include a low-resolution video frame sequence and a corresponding standard high-resolution video frame sequence. The low-resolution video frame sequence contains a certain number of low-resolution video frames, and the standard high-resolution video frame sequence contains a certain number of high-resolution video frames. The low-resolution video frames can be obtained by downsampling the high-resolution video frames.

[0050] A2. Input the low-resolution video frame sequence into the multi-scale video super-resolution model to be trained to obtain the actual high-resolution video frame sequence of the output.

[0051] In this embodiment, the multi-scale video super-resolution model to be trained can be pre-built. The multi-scale video super-resolution model to be trained can be composed of a multi-scale feature interaction network to be trained and an image reconstruction network to be trained. The multi-scale feature interaction network to be trained can be composed of an initial feature interaction module to be trained, at least one multi-scale feature interaction module to be trained, and an end feature interaction module to be trained connected in series. The multi-scale feature interaction module to be trained can include at least two RVSTB units to be trained and one feature fusion unit to be trained. Each RVSTB unit to be trained can include at least two VSTL layers and one convolutional layer.

[0052] Specifically, by using low-resolution video frame sequences as input data and inputting them into the pre-built multi-scale video super-resolution model to be trained, the actual high-resolution video frame sequences can be obtained as output.

[0053] A3. Obtain the fitting loss function based on the standard high-resolution video frame sequence and the actual high-resolution video frame sequence.

[0054] Specifically, since the standard high-resolution video frame sequence is an actual high-resolution video frame sequence, and the actual high-resolution video frame sequence is calculated and output by the model that has not been trained, there must be a certain error between the standard high-resolution video frame sequence and the actual high-resolution video frame sequence. Based on this error, a fitting loss function can be formed to achieve the training and parameter tuning of the multi-scale video super-resolution model to be trained.

[0055] A4. By backpropagating the training multi-scale video super-resolution model through fitting the loss function, the target multi-scale video super-resolution model is obtained.

[0056] Specifically, after obtaining the fitting loss function, the multi-scale video super-resolution model to be trained can be backpropagated through the fitting loss function to continuously adjust the behavior recognition model, and finally obtain the target multi-scale video super-resolution model.

[0057] Example 2

[0058] Figure 2 This is a schematic diagram of a video super-resolution device provided in Embodiment 2 of the present invention. Figure 2 As shown, the device includes:

[0059] The data acquisition module 210 is used to acquire a sequence of original video frames containing at least two original video frames.

[0060] The video super-resolution module 220 is used to perform video super-resolution operation on the original video frame sequence using a pre-trained target multi-scale video super-resolution model, and output a target video frame sequence containing the target video frames; wherein, the target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network, the multi-scale feature interaction network is used to perform multi-scale feature interaction and feature fusion on each of the original video frames, and the image reconstruction network reconstructs the image based on the feature map output by the feature interaction network.

[0061] Optionally, the multi-scale feature interaction network consists of an initial feature interaction module, at least one multi-scale feature interaction module, and an end feature interaction module connected in series.

[0062] The initial feature interaction module is used to perform feature interaction on each of the original video frames;

[0063] The multi-scale feature interaction module is used to perform multi-scale feature interaction and feature fusion on all feature maps output by the initial feature interaction subnet or the previous multi-scale feature interaction module.

[0064] The terminal feature interaction module is used to perform feature interaction on all feature maps output by the last multi-scale feature interaction module.

[0065] Optionally, the multi-scale feature interaction module includes at least two RVSTB units and one feature fusion unit;

[0066] The feature map input to the multi-scale feature interaction module is downsampled at a preset sampling frequency, the RVSTB unit is used to perform feature interaction on the downsampled feature map, and after feature interaction, it is upsampled at the preset sampling frequency.

[0067] The feature fusion unit is used to perform feature fusion on all output feature maps corresponding to the same input feature map of each of the RVSTB units.

[0068] Optionally, the RVSTB unit includes at least two VSTL layers and one convolutional layer.

[0069] Optionally, the VSTL layer employs a shift window mechanism and an attention mechanism to achieve feature interaction.

[0070] Optionally, the image reconstruction network includes a feature image reconstruction module, an interpolated image construction module, and an image fusion module;

[0071] The feature image reconstruction module is used to reconstruct the feature map output by the feature interaction network to form a reconstructed image.

[0072] The interpolated image construction module is used to perform image interpolation on the original video frames to form an interpolated image;

[0073] The image fusion module is used to fuse the reconstructed image and the interpolated image to form the target video frame.

[0074] Optionally, the training steps of the target multi-scale video super-resolution model include:

[0075] Obtain a training dataset containing low-resolution video frame sequences and corresponding standard high-resolution video frame sequences;

[0076] The low-resolution video frame sequence is input into the multi-scale video super-resolution model to be trained to obtain the actual high-resolution video frame sequence output.

[0077] Based on the standard high-resolution video frame sequence and the actual high-resolution video frame sequence, a fitting loss function is obtained;

[0078] The target multi-scale video super-resolution model is obtained by backpropagating the training multi-scale video super-resolution model using the fitted loss function.

[0079] The video super-resolution device provided in the embodiments of the present invention can execute the video super-resolution method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0080] Example 3

[0081] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0082] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0083] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0084] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video super-resolution methods.

[0085] In some embodiments, the video super-resolution method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video super-resolution method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video super-resolution method by any other suitable means (e.g., by means of firmware).

[0086] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0087] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0089] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0090] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0091] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0092] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video super-resolution method, characterized in that, include: Obtain a sequence of raw video frames containing at least two raw video frames; A pre-trained target multi-scale video super-resolution model is used to perform video super-resolution on the original video frame sequence, and outputs a target video frame sequence containing the target video frames; wherein, the target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network, and the multi-scale feature interaction network is used to perform multi-scale feature interaction and feature fusion on each of the original video frames. The multi-scale feature interaction network consists of an initial feature interaction module, at least one multi-scale feature interaction module, and an end feature interaction module connected in series. The multi-scale feature interaction module includes at least two RVSTB units and a feature fusion unit. The RVSTB unit includes at least two VSTL layers and a convolutional layer. The VSTL layers use a shift window mechanism and an attention mechanism to realize feature interaction. The image reconstruction network reconstructs images based on the feature maps output by the feature interaction network.

2. The method according to claim 1, characterized in that, The initial feature interaction module is used to perform feature interaction on each of the original video frames; The multi-scale feature interaction module is used to perform multi-scale feature interaction and feature fusion on all feature maps output by the initial feature interaction subnet or the previous multi-scale feature interaction module. The terminal feature interaction module is used to perform feature interaction on all feature maps output by the last multi-scale feature interaction module.

3. The method according to claim 2, characterized in that, The feature map input to the multi-scale feature interaction module is downsampled at a preset sampling frequency, the RVSTB unit is used to perform feature interaction on the downsampled feature map, and after feature interaction, it is upsampled at the preset sampling frequency. The feature fusion unit is used to perform feature fusion on all output feature maps corresponding to the same input feature map of each of the RVSTB units.

4. The method according to any one of claims 1, characterized in that, The image reconstruction network includes a feature image reconstruction module, an interpolated image construction module, and an image fusion module; The feature image reconstruction module is used to reconstruct the feature map output by the feature interaction network to form a reconstructed image. The interpolated image construction module is used to perform image interpolation on the original video frames to form an interpolated image; The image fusion module is used to fuse the reconstructed image and the interpolated image to form the target video frame.

5. The method according to any one of claims 1-4, characterized in that, The training steps of the target multi-scale video super-resolution model include: Obtain a training dataset containing low-resolution video frame sequences and corresponding standard high-resolution video frame sequences; The low-resolution video frame sequence is input into the multi-scale video super-resolution model to be trained to obtain the actual high-resolution video frame sequence output. Based on the standard high-resolution video frame sequence and the actual high-resolution video frame sequence, a fitting loss function is obtained; The target multi-scale video super-resolution model is obtained by backpropagating the training multi-scale video super-resolution model using the fitted loss function.

6. A video super-resolution device, characterized in that, include: The data acquisition module is used to acquire a sequence of raw video frames containing at least two raw video frames. The video super-resolution module is used to perform video super-resolution on the original video frame sequence using a pre-trained target multi-scale video super-resolution model, and output a target video frame sequence containing the target video frames; wherein, the target multi-scale video super-resolution model includes a multi-scale feature interaction network and an image reconstruction network, the multi-scale feature interaction network is used to perform multi-scale feature interaction and feature fusion on each of the original video frames, and the image reconstruction network reconstructs the image based on the feature map output by the feature interaction network.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the video super-resolution method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video super-resolution method according to any one of claims 1-5.

Images