Video super-resolution reconstruction method and system, electronic equipment and storage medium

Abstract

The invention provides a video super-resolution reconstruction method and system, electronic equipment and a storage medium. The method comprises the following steps: S1, acquiring a current low-resolution frame and a next low-resolution frame; s2, performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame; s3, acquiring a next high-resolution frame based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; s4, updating the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; and S5, repeating the steps S3 and S4 until super-division reconstruction of all low-resolution frames in the video stream is completed. According to the method, the information association between the continuous frames in the video stream is fully utilized, the continuity of the semantic content is kept, and the high-resolution video stream is obtained; the time domain correlation of different frames in the video stream is utilized, so that the overall calculation amount is reduced, the operation efficiency is improved, and the reconstruction quality of the video super-resolution is also improved.

Description

Technical Field

[0001] The present application belongs to the technical field of video image processing, and specifically relates to a video super-resolution reconstruction method, system, electronic device and storage medium. Background Art

[0002] Video super-resolution (SR) reconstruction refers to estimating a video sequence with a higher resolution (HR) from some low-resolution (LR) blurred video sequences. It can eliminate noise and blur caused by limited detector size and optical elements, and is an effective means to improve sequence resolution.

[0003] Currently, all existing super-resolution algorithms target single images, that is, they use a frame-by-frame processing method. Specifically, video super-resolution reconstruction algorithms usually cut the video stream into frames, perform super-resolution reconstruction on the entire image, and then connect each frame result into a video. This not only causes unnecessary computational overhead, but also cannot guarantee the semantic coherence between the previous frame and the next frame.

[0004] In addition, the super-resolution reconstruction method in the prior art only relies on a single low-resolution image, resulting in poor reconstruction effect and unable to meet the needs of actual application scenarios. Summary of the invention

[0005] The purpose of the present application is to provide a video super-resolution reconstruction method, system, electronic device and storage medium, so as to improve the quality and efficiency of video super-resolution reconstruction.

[0006] In a first aspect, the present application provides a video super-resolution reconstruction method, comprising: step S1, obtaining a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames; step S2, performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame; step S3, based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame, obtaining a next high-resolution frame corresponding to the next low-resolution frame; step S4, updating the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; step S5, repeating steps S3 and S4 until super-resolution reconstruction of all low-resolution frames in the video stream is completed.

[0007] In an implementation of the first aspect, based on the current low-resolution frame, the current high-resolution frame, and the next low-resolution frame, step S3 of acquiring a next high-resolution frame corresponding to the next low-resolution frame includes: Encoding the current low-resolution frame to obtain a feature vector of the current low-resolution frame; Encoding the current high-resolution frame to obtain a feature vector of the current high-resolution frame; Encoding the next low-resolution frame to obtain a feature vector of the next low-resolution frame; Processing the feature vector of the current low-resolution frame, the feature vector of the current high-resolution frame, and the feature vector of the next low-resolution frame based on the large language model to obtain a feature vector of the next high-resolution frame; The feature vector of the next high-resolution frame is decoded to obtain the next high-resolution frame.

[0008] In an implementation of the first aspect, the current low-resolution frame, the current high-resolution frame and the next low-resolution frame are encoded based on a VAE encoder model to obtain corresponding feature vectors; and the feature vector of the next high-resolution frame is decoded based on a VAE decoder model to obtain the next high-resolution frame.

[0009] In an implementation of the first aspect, step S4 of updating the current low-resolution frame and the current high-resolution frame includes: Replacing the current low-resolution frame with the next low-resolution frame to obtain a new current low-resolution frame; replacing the current high-resolution frame with the next high-resolution frame; The next low-resolution frame of the new current low-resolution frame is obtained from the video stream as a new next low-resolution frame.

[0010] In an implementation of the first aspect, step S1 of acquiring a current low-resolution frame and a next low-resolution frame from a video stream containing consecutive low-resolution frames includes: Obtaining position information of the current low-resolution frame in the video stream; Based on the position information, calculating the position of the next low-resolution frame; The next low-resolution frame is extracted from the video stream based on the position of the next low-resolution frame.

[0011] In an implementation of the first aspect, step S2 of performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame includes: Based on the SwinIR model, single-frame super-resolution reconstruction is performed on the current low-resolution frame to obtain a current high-resolution frame.

[0012] In an implementation of the first aspect, the method further includes: Set before the current low-resolution frame <low>character; Set before the current high-resolution frame <high>character; Set before the next low-resolution frame <low>character.

[0013] In a second aspect, the present application provides a video super-resolution reconstruction system, comprising: a video frame acquisition module, used to acquire a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames; a single-frame super-resolution reconstruction module, used to perform single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame; a high-resolution frame generation module, used to acquire a next high-resolution frame corresponding to the next low-resolution frame based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; a video frame update module, used to update the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; and a loop reconstruction module, used to repeatedly call the high-resolution frame generation module and the video frame update module until the super-resolution reconstruction of all low-resolution frames in the video stream is completed.

[0014] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs any of the methods described above.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements any of the methods described above when executed by a processor.

[0016] As described above, the video super-resolution reconstruction method, system, electronic device and storage medium described in the present application have the following beneficial effects: (1) By referring to the super-resolution reconstruction result of the current frame to obtain the next high-resolution frame, the information association between consecutive frames in the video stream is fully utilized, the coherence of the semantic content is maintained, and a more realistic and natural high-resolution video stream is obtained visually; (2) The temporal correlation between different frames in the video stream is utilized to effectively reduce the need for separate complex calculations for each frame. This not only reduces the overall amount of calculation and improves computational efficiency, but also improves the reconstruction quality of video super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a schematic structural diagram of a terminal according to an embodiment of the present application.

[0018] Figure 2 Shown is a flow chart of an embodiment of the video super-resolution reconstruction method described in this application.

[0019] Figure 3 Shown is a flow chart of another embodiment of the video super-resolution reconstruction method described in the present application.

[0020] Figure 4 Shown is a flowchart of yet another embodiment of the video super-resolution reconstruction method described in the present application.

[0021] Figure 5 FIG. 4 is a schematic diagram showing the structure of a video super-resolution reconstruction system according to an embodiment of the present application.

[0022] Figure 6 Shown is a schematic structural diagram of an electronic device described in this application in one embodiment. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0025] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0026] The following embodiments of the present application provide a video super-resolution reconstruction method, system, electronic device and storage medium, which can be applied to Figure 1 The terminal described in this application may include a mobile phone with wireless charging function, a tablet computer, a laptop computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of this application does not impose any restrictions on the specific type of the terminal.

[0027] For example, the terminal may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future evolved Non-terrestrial Network (NTN).

[0028] For example, the terminal can communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS). The technical solution in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings in the embodiment of the present application.

[0029] See also Figure 2 , which is a flowchart of an embodiment of the video super-resolution reconstruction method described in the present application.

[0030] like Figure 2 As shown, this embodiment provides a video super-resolution reconstruction method, including the following steps S1 to S5.

[0031] In step S1, a current low-resolution frame and a next low-resolution frame are obtained from a video stream containing consecutive low-resolution frames.

[0032] In step S2, single-frame super-resolution reconstruction is performed on the current low-resolution frame to obtain a current high-resolution frame.

[0033] In step S3, based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame, a next high-resolution frame corresponding to the next low-resolution frame is acquired.

[0034] In step S4, the current low-resolution frame, the current high-resolution frame and the next low-resolution frame are updated.

[0035] In step S5, steps S3 and S4 are repeated until super-resolution reconstruction of all low-resolution frames in the video stream is completed.

[0036] In one embodiment of the present application, step S1 of obtaining a current low-resolution frame and a next low-resolution frame from a video stream comprising continuous low-resolution frames includes: obtaining position information of the current low-resolution frame in the video stream; calculating the position of the next low-resolution frame based on the position information; and extracting the next low-resolution frame from the video stream based on the position of the next low-resolution frame.

[0037] Specifically, before obtaining the position information of the current low-resolution frame in the video stream, the method further includes: obtaining a video stream containing continuous low-resolution frames; and splitting the video stream into a series of continuous low-resolution video frames as video frames to be reconstructed. Each video frame to be reconstructed includes image data, position information, and timestamp or index information.

[0038] In practical applications, the first frame in the video stream is usually read as the initial current low-resolution frame. In order to obtain the next low-resolution frame, it is necessary to know the position of the current low-resolution frame in the video stream. This can be achieved by querying the timestamp or index information of the video frame. For example, if the video frame is identified by a timestamp, the order relationship between them can be determined by comparing the timestamp of the current low-resolution frame with the timestamp of the next low-resolution frame. Once the position of the current low-resolution frame is determined, the position of the next low-resolution frame can be calculated based on the position. Then, the calculated position is used to extract the next low-resolution frame from the video stream.

[0039] In one embodiment of the present application, step S2 of performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame includes: performing single-frame super-resolution reconstruction on the current low-resolution frame based on a SwinIR model to obtain a current high-resolution frame.

[0040] Specifically, SwinIR is a baseline model based on Swin Transformer, which consists of three parts: a shallow feature extraction module, a deep feature extraction module and a high-quality feature reconstruction module. The shallow feature extraction module uses a convolutional layer to extract low-frequency information in the input image; the deep feature extraction module is composed of multiple residual SwinTransformer blocks (RSTB), and each RSTB has multiple Swin Transformer layers, convolutional layers and residual connections to restore the lost high-frequency information in the input image; the high-quality feature reconstruction module achieves high-quality image reconstruction by fusing low-frequency information and high-frequency information.

[0041] In an embodiment of the present application, the current low-resolution frame may be input into a pre-trained SwinIR model, and after being processed by the SwinIR model, a current high-resolution frame corresponding to the current low-resolution frame may be obtained.

[0042] It should be noted that the SwinIR model is only used for the first frame in the video stream and not for subsequent frames. This is because there is a temporal correlation between adjacent frames in the video stream, and this correlation can be used to improve the effect of super-resolution reconstruction.

[0043] In one embodiment of the present application, based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame, step S3 of acquiring a next high-resolution frame corresponding to the next low-resolution frame includes the following steps S31 to S35.

[0044] In step S31, the current low-resolution frame is encoded to obtain a feature vector of the current low-resolution frame.

[0045] In step S32, the current high-resolution frame is encoded to obtain a feature vector of the current high-resolution frame.

[0046] In step S33, the next low-resolution frame is encoded to obtain a feature vector of the next low-resolution frame.

[0047] Variational Auto-Encoder (VAE) is a generative model used to learn the potential representation of input data. Its structure consists of two parts: VAE encoder and VAE decoder. VAE encoder and VAE decoder appear in pairs.

[0048] In one embodiment of the present application, the current low-resolution frame, the current high-resolution frame and the next low-resolution frame are encoded based on a VAE encoder model to obtain corresponding feature vectors.

[0049] In this embodiment, an independent VAE encoder is configured for each of the input current low-resolution frame, the current high-resolution frame, and the next low-resolution frame.

[0050] These independent VAE encoders are responsible for converting the input video frames into latent vectors in the latent space, which can be regarded as a compressed representation of the input video frames, forming a discrete feature vector. This discrete feature vector can capture the key features of the video frames and can be used for subsequent processing and analysis tasks.

[0051] In step S34, the feature vector of the current low-resolution frame, the feature vector of the current high-resolution frame, and the feature vector of the next low-resolution frame are processed based on the large language model to obtain a feature vector of the next high-resolution frame.

[0052] The Large Language Model (LLM) is a natural language processing model based on deep learning neural networks, with large-scale parameters and extremely high semantic understanding and generation capabilities.

[0053] In the embodiment of the present application, the large language model learns the association and dependency between various feature vectors based on the self-attention mechanism, and obtains a new feature vector through multi-layer transformation. This new feature vector integrates the information of the current low-resolution frame, the current high-resolution frame, and the next low-resolution frame, and can be regarded as a prediction of the next high-resolution frame.

[0054] In this implementation, the next high-resolution frame is obtained by referring to the super-resolution reconstruction result of the current frame, which fully utilizes the information association between consecutive frames in the video stream, maintains the coherence of the semantic content, and visually obtains a more realistic and natural high-resolution video stream.

[0055] In step S35, the feature vector of the next high-resolution frame is decoded to obtain the next high-resolution frame.

[0056] In one embodiment of the present application, the feature vector of the next high-resolution frame is decoded based on a VAE decoder model to obtain the next high-resolution frame.

[0057] VAE decoder is the inverse process of VAE encoder, which aims to convert the compressed representation in the latent space back to the original data space.

[0058] In an embodiment of the present application, a VAE decoder can be assigned to the next high-resolution feature vector output by the large language model. The next high-resolution image feature output by the large language model is also a discrete feature vector, and the main function of the VAE decoder is to decode the input next high-resolution feature vector into a high-quality image of the same size as the next low-resolution frame.

[0059] Specifically, the VAE decoder gradually restores the missing details of the low-resolution image through a series of decoding operations, such as deconvolution layers and the application of activation functions, and finally generates a high-resolution frame corresponding to the next low-resolution frame. This method not only improves the resolution of the video frame, but also enhances the temporal and spatial coherence of the video stream while maintaining image quality.

[0060] In this implementation, by combining the use of VAE encoder and VAE decoder, end-to-end conversion from low-resolution frames to high-resolution frames is effectively achieved, which significantly improves the visual quality of video content and is suitable for various applications that require high-quality video output.

[0061] In an embodiment of the present application, step S4 of updating the current low-resolution frame and the current high-resolution frame includes the following steps S41 to S43.

[0062] In step S41, the current low-resolution frame is replaced by the next low-resolution frame to obtain a new current low-resolution frame.

[0063] In step S42, the current high-resolution frame is replaced by the next high-resolution frame.

[0064] In step S43, the next low-resolution frame of the new current low-resolution frame is obtained from the video stream as a new next low-resolution frame.

[0065] In this implementation, the temporal correlation of different frames in the video stream is utilized, which effectively reduces the need for separate complex calculations for each frame. This not only reduces the overall amount of calculation and improves the computational efficiency, but also improves the reconstruction quality of video super-resolution.

[0066] The cyclic update process involved in the present application will be described in detail below in conjunction with a specific embodiment.

[0067] See also Figure 3 , which is a flowchart of another embodiment of the video super-resolution reconstruction method described in the present application.

[0068] Assume that the video stream V contains continuous low-resolution frames {L1, L2, L3, ..., LN}, where N represents the total number of low-resolution frames in the video stream V.

[0069] like Figure 3 As shown, in the first cycle, the current low-resolution frame is set to L1, so the next low-resolution frame is naturally L2. First, a single-frame super-resolution reconstruction is performed on the current low-resolution frame L1 to obtain the corresponding current high-resolution frame R1.

[0070] Then, the feature vector of the current low-resolution frame L1 is recorded as FL1, the feature vector of the current high-resolution frame R1 is recorded as FR1, and the feature vector of the next low-resolution frame L2 is recorded as FL2. The feature vector FL1 of the current low-resolution frame, the feature vector FR1 of the current high-resolution frame, and the feature vector FL2 of the next low-resolution frame are processed using the large language model (LLM) to obtain the feature vector FR2 of the next high-resolution frame. The image corresponding to the feature vector FR2 is the next high-resolution frame R2.

[0071] See also Figure 4 , which is a flowchart of another embodiment of the video super-resolution reconstruction method described in the present application.

[0072] like Figure 4 As shown, the second loop is entered (i=2 at this time), the current low-resolution frame L1 is replaced by the next low-resolution frame L2, the high-resolution frame R1 is replaced by the high-resolution image R2, and the next low-resolution frame L2 is replaced by the low-resolution frame L3. In this loop, the feature vector of the current low-resolution frame L2 is recorded as FL2, the feature vector of the current high-resolution frame R2 is recorded as FR2, and the feature vector of the next low-resolution frame L3 is recorded as FL3. The large language model (LLM) is used again to process the feature vector FL2 of the current low-resolution frame, the feature vector FR2 of the current high-resolution frame, and the feature vector FL3 of the next low-resolution frame to obtain the feature vector FR3 of the next high-resolution frame. The image corresponding to the feature vector FR3 is the next high-resolution frame R3.

[0073] By analogy, through multiple cycles, N high-resolution images can be obtained in sequence, corresponding to high-resolution frames R2 to RN. All high-resolution frames are arranged and combined in the order of timestamps to generate a complete super-resolution video stream.

[0074] In one embodiment of the present application, the video super-resolution reconstruction method provided in this embodiment further includes: setting <low>Character; set before the current high-resolution frame <high>Character; set before the next low-resolution frame <low>character.

[0075] Specifically, set 2 special characters <low>and <high>,in <low>Indicates that the frame that follows is a low-resolution frame. <high>It indicates that the frame that follows is a high-resolution frame, with the purpose of providing clear information instructions to the large language model.

[0076] For example, in the first cycle, according to " <low>Low resolution frame L1 <high>High resolution frame R1 <low>The next low-resolution frame L2" is sequentially arranged to be input into the large language model.

[0077] For example, in the second cycle, according to " <low>Low resolution frame L2 <high>High resolution frame R2 <low>The next low-resolution frame L3" is sequentially arranged to be input into the large language model.

[0078] It should be noted that the protection scope of the video super-resolution reconstruction method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0079] See also Figure 5 , which is a schematic diagram of the structure of a video super-resolution reconstruction system according to an embodiment of the present application.

[0080] like Figure 5 As shown, this embodiment provides a video super-resolution reconstruction system, including a video frame acquisition module 10, a single frame super-resolution reconstruction module 20, a high-resolution frame generation module 30, a video frame update module 40 and a cyclic reconstruction module 50.

[0081] In an embodiment of the present application, the video frame acquisition module 10 is used to acquire a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames.

[0082] In one embodiment of the present application, the single-frame super-resolution reconstruction module 20 is used to perform single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame.

[0083] In an embodiment of the present application, the high-resolution frame generation module 30 is used to obtain a next high-resolution frame corresponding to the next low-resolution frame based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame.

[0084] In an embodiment of the present application, the video frame updating module 40 is used to update the current low-resolution frame, the current high-resolution frame and the next low-resolution frame.

[0085] In one embodiment of the present application, the cyclic reconstruction module 50 is used to repeatedly call the high-resolution frame generation module and the video frame update module until the super-resolution reconstruction of all low-resolution frames in the video stream is completed.

[0086] It should be noted that the structures and principles of the video frame acquisition module 10, single frame super-resolution reconstruction module 20, high-resolution frame generation module 30, video frame update module 40 and cyclic reconstruction module 50 of the video super-resolution reconstruction system described in the embodiment of the present application correspond one by one to the steps of the above-mentioned video super-resolution reconstruction method, and will not be repeated here.

[0087] The video super-resolution reconstruction system provided in the embodiments of the present application can implement the video super-resolution reconstruction method described in the present application, but the implementation device of the video super-resolution reconstruction method described in the present application includes but is not limited to the structure of the video super-resolution reconstruction system listed in the embodiments. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0088] See also Figure 6 , which is a schematic diagram of the structure of an electronic device described in the present application in one embodiment.

[0089] like Figure 6 As shown, this embodiment provides an electronic device, including: a processor 51 and a memory 52.

[0090] In one embodiment of the present application, the memory 51 is used to store computer programs.

[0091] In one embodiment of the present application, the processor 52 is used to execute the computer program stored in the memory, so that the electronic device executes any one of the above methods.

[0092] Preferably, the processor 51 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The memory 52 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0093] This embodiment also includes one or more of a multimedia component 53 , an input / output (I / O) interface 54 , and a communication component 55 .

[0094] The multimedia component 53 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 52 or sent through the communication component 55. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 54 provides an interface between the processor 51 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 55 is used for wired or wireless communication between the timer and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 55 may include: Wi-Fi module, Bluetooth module, NFC module.

[0095] In one embodiment, the timer can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned drone network deployment and collaborative caching methods.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0097] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0098] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by the processor, any of the methods described above is implemented. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0100] The embodiment of the present application may also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0101] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0102] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0103] In summary, the video super-resolution reconstruction method, system, electronic device and storage medium described in the present application obtain the next high-resolution frame by referring to the super-resolution reconstruction result of the current frame, making full use of the information association between consecutive frames in the video stream, maintaining the continuity of the semantic content, and visually obtaining a more realistic and natural high-resolution video stream; utilizing the time domain correlation of different frames in the video stream, effectively reducing the need for separate complex calculations for each frame, which not only reduces the overall amount of calculation and improves the computing efficiency, but also improves the reconstruction quality of the video super-resolution.

[0104] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.< / low> < / high> < / low> < / low> < / high> < / low> < / high> < / low> < / high> < / low> < / low> < / high> < / low> < / low> < / high> < / low>

Claims

1. A video super-resolution reconstruction method, characterized in that: include: Step S1, obtaining a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames; Step S2, performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame; Step S3, based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame, acquiring a next high-resolution frame corresponding to the next low-resolution frame; Step S4, updating the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; Step S5, repeating steps S3 and S4 until super-resolution reconstruction of all low-resolution frames in the video stream is completed.

2. The method according to claim 1, characterized in that Based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame, step S3 of acquiring a next high-resolution frame corresponding to the next low-resolution frame comprises: Encoding the current low-resolution frame to obtain a feature vector of the current low-resolution frame; Encoding the current high-resolution frame to obtain a feature vector of the current high-resolution frame; Encoding the next low-resolution frame to obtain a feature vector of the next low-resolution frame; Processing the feature vector of the current low-resolution frame, the feature vector of the current high-resolution frame, and the feature vector of the next low-resolution frame based on the large language model to obtain a feature vector of the next high-resolution frame; The feature vector of the next high-resolution frame is decoded to obtain the next high-resolution frame.

3. The method according to claim 2, characterized in that The current low-resolution frame, the current high-resolution frame and the next low-resolution frame are encoded based on the VAE encoder model to obtain corresponding feature vectors; and the feature vector of the next high-resolution frame is decoded based on the VAE decoder model to obtain the next high-resolution frame.

4. The method according to claim 1, characterized in that: The step S4 of updating the current low-resolution frame and the current high-resolution frame comprises: Replacing the current low-resolution frame with the next low-resolution frame to obtain a new current low-resolution frame; replacing the current high-resolution frame with the next high-resolution frame; The next low-resolution frame of the new current low-resolution frame is obtained from the video stream as a new next low-resolution frame.

5. The method according to claim 1, characterized in that The step S1 of acquiring a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames comprises: Obtaining position information of the current low-resolution frame in the video stream; Based on the position information, calculating the position of the next low-resolution frame; The next low-resolution frame is extracted from the video stream based on the position of the next low-resolution frame.

6. The method according to claim 1, characterized in that The step S2 of performing single-frame super-resolution reconstruction on the current low-resolution frame to obtain the current high-resolution frame comprises: Based on the SwinIR model, single-frame super-resolution reconstruction is performed on the current low-resolution frame to obtain a current high-resolution frame.

7. The method according to claim 1, characterized in that Also includes: Set before the current low-resolution frame <low> character;< / low> Set before the current high-resolution frame <high> character;< / high> Set before the next low-resolution frame <low> character.< / low> 8. A video super-resolution reconstruction system, characterized in that: include: A video frame acquisition module, used to acquire a current low-resolution frame and a next low-resolution frame from a video stream containing continuous low-resolution frames; A single-frame super-resolution reconstruction module is used to perform single-frame super-resolution reconstruction on the current low-resolution frame to obtain a current high-resolution frame; a high-resolution frame generation module, configured to acquire a next high-resolution frame corresponding to the next low-resolution frame based on the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; A video frame updating module, configured to update the current low-resolution frame, the current high-resolution frame and the next low-resolution frame; A cyclic reconstruction module is used to repeatedly call the high-resolution frame generation module and the video frame update module until super-resolution reconstruction of all low-resolution frames in the video stream is completed.

9. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory so as to enable the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images