Blurred video spatiotemporal super-resolution method, system, device and storage medium

Abstract

The present invention discloses a fuzzy video spatiotemporal super-resolution method, system, device and storage medium, which are one-to-one corresponding schemes. In the scheme, the multi-scale features of event streams and videos are extracted by a multimodal feature extraction module, and the multi-scale features of event streams and videos for injection into a diffusion model are obtained by a multimodal data feature injection module, and finally the final clear video frame is generated by a two-stage trifurcated diffusion model; in the above scheme, the event camera can be effectively used to record the high dynamic motion of the scene, and the complex spatiotemporal combination degradation problem can be solved with the help of the powerful data distribution modeling ability of the diffusion model. The designed trifurcated diffusion model can fully extract and utilize the data priors in the event stream data and video data, so as to effectively reconstruct a clear video with high temporal resolution and high spatial resolution from the fuzzy video, and achieve the best visual effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a blurred video spatiotemporal super-resolution method, system, device and storage medium. Background Art

[0002] Blurred video spatiotemporal super-resolution is an advanced computational imaging technique that aims to improve the frame rate and spatial resolution of blurry videos captured in real scenes. In actual imaging, cameras are often limited by exposure time, resulting in low frame rates and motion blur in high-speed scenes, while sensor size limitations limit the achievable resolution. These limitations together affect the overall quality of the video. Blurred video spatiotemporal super-resolution breaks through these limitations by dealing with compound degradation problems in both spatial and temporal dimensions. It has important applications in multiple fields, including new perspective synthesis, frame rate conversion, video compression, and the creation of slow motion effects.

[0003] Current research mainly focuses on solving single or dual degradation problems, such as super-resolution, deblurring or frame interpolation, separately or in combination. However, solving all three aspects at the same time, the core problem of blurred video spatiotemporal super-resolution, remains an unsolved challenge. Moreover, existing methods usually use traditional regression models such as convolutional neural networks (CNNs), which rely heavily on data priors. In the case of insufficient data priors or dealing with complex data distributions, the results of these methods are often unsatisfactory. Blurred video spatiotemporal super-resolution provides a new way to improve video quality by comprehensively considering the degradation problems in spatial and temporal dimensions. However, there is currently no perfect solution.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a blurred video spatiotemporal super-resolution method, system, device and storage medium, which effectively overcomes the problems of insufficient data prior and weak modeling ability of traditional regression models, and through the fusion of video frame data and event data, effectively removes motion blur and realizes joint spatiotemporal super-resolution, thereby achieving optimal quantization and visual effects.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A blurred video spatiotemporal super-resolution method, comprising:

[0008] Collecting continuous video frames and corresponding asynchronous event stream signals, degrading the continuous video frames, wherein the video frames obtained by the degrading process are called blurred video frames; and processing the asynchronous event stream signals into event frames;

[0009] A blurred video spatiotemporal super-resolution network is constructed and trained. The blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are extracted respectively through the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated through the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; various types of noise are extracted from the updated multi-scale features of event frames and blurred video frames through the trifurcated decomposition stage of the trifurcated diffusion model, and the weight maps of various types of noise are estimated through the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network;

[0010] The trained blurred video spatiotemporal super-resolution network is used to process the input video frames to obtain the fused noise, and then the spatiotemporal super-resolution processed video is obtained through the denoising step of the trifurcated diffusion model.

[0011] A blurred video spatiotemporal super-resolution system, comprising:

[0012] A data acquisition and preprocessing unit, configured to acquire continuous video frames and corresponding asynchronous event stream signals, degrade the continuous video frames, wherein the video frames obtained by the degraded processing are called blurred video frames; and process the asynchronous event stream signals into event frames;

[0013] A network construction and training unit is used to construct and train a blurred video spatiotemporal super-resolution network, wherein the blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are extracted respectively by the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated by the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; various types of noise are extracted from the updated multi-scale features of event frames and blurred video frames by the trifurcated decomposition stage of the trifurcated diffusion model, and weight maps of various types of noise are estimated by the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network;

[0014] The spatiotemporal super-resolution processing unit is used to process the input video frame using the trained fuzzy video spatiotemporal super-resolution network to obtain the fused noise, and then process the video with spatiotemporal super-resolution through the denoising step of the trifurcated diffusion model.

[0015] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0018] It can be seen from the technical solution provided by the present invention that the multi-scale features of the event stream and the video are extracted by the multi-modal feature extraction module, and the multi-scale features of the event stream and the video for injecting into the diffusion model are obtained by the multi-modal data feature injection module, and finally the final clear video frame is generated by the two-stage three-pronged diffusion model; the above scheme can effectively use the event camera to record the high dynamic motion of the scene, and solve the complex time-space combination degradation problem with the help of the powerful data distribution modeling ability of the diffusion model. The designed three-pronged diffusion model can fully extract and utilize the data priors in the event stream data and video data, so as to effectively reconstruct a clear video with high temporal resolution and high spatial resolution from the blurred video, and achieve the best visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 A flowchart of a blurred video spatiotemporal super-resolution method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a fuzzy video spatiotemporal super-resolution network provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a blurred video spatiotemporal super-resolution system provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0025] First, the terms that may be used in this article are explained as follows:

[0026] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0027] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0028] The following is a detailed description of a blurred video spatiotemporal super-resolution method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed.

[0029] Embodiment 1

[0030] The embodiment of the present invention provides a fuzzy video spatiotemporal super-resolution method, such as Figure 1 As shown, it mainly includes the following steps:

[0031] Step 1: Data collection and preprocessing.

[0032] In an embodiment of the present invention, continuous video frames and corresponding asynchronous event stream signals are collected, the continuous video frames are degraded, and the video frames obtained by the degraded processing are called blurred video frames; and the asynchronous event stream signals are processed into event frames.

[0033] Step 2: Network construction and training.

[0034] In an embodiment of the present invention, a blurred video spatiotemporal super-resolution network is constructed and trained, and the blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are respectively extracted by the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated by the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; multiple types of noise are extracted from the updated multi-scale features of event frames and blurred video frames by the trifurcated decomposition stage of the trifurcated diffusion model, and weight maps of various types of noise are estimated by the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network.

[0035] Step 3: spatiotemporal super-resolution processing.

[0036] In an embodiment of the present invention, a trained blurred video spatiotemporal super-resolution network is used to process input video frames to obtain fused noise, and then a video processed with spatiotemporal super-resolution (i.e., a clear video) is obtained through a denoising step of a trifurcated diffusion model.

[0037] In the above solution provided by the embodiment of the present invention, the event camera and the diffusion model are used to simultaneously solve the spatiotemporal super-resolution task of blurred video, which can improve the performance of the studied task from two aspects: data prior and model construction. First, the event camera has the advantages of high temporal resolution and high dynamic range, and can provide motion with high temporal precision, thereby providing data prior. Secondly, the diffusion model has a strong data distribution modeling capability, which is particularly suitable for the joint spatiotemporal degradation problem of the spatiotemporal difference resolution of blurred video, thereby overcoming the deficiencies in the modeling capability of the traditional regression model, thereby bringing about the optimal de-motion blur and spatiotemporal super-resolution effect.

[0038] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0039] 1. Data collection and preprocessing.

[0040] In the embodiment of the present invention, an event camera can be used to capture a dynamic scene and obtain continuous clear video frames and asynchronous event stream signals thereof.

[0041] Specifically, a captured video frame can be recorded as , the corresponding asynchronous event stream signal is recorded as ,in, represents the kth event, and , Represents the kth event The polarity of ; Represents the kth event The moment of occurrence; and Represents the kth event The spatial coordinates of occurrence; N represents the total number of events; represents the space-time projection coordinates, is the moment in the space-time projection coordinates, x and y are the spatial coordinates in the space-time projection coordinates; is an indicative function.

[0042] After taking the average of continuous video frames, the downsampling function is used to perform degradation processing to obtain a blurred video frame B.

[0043] When the asynchronous event stream signal is processed into an event frame, it is first converted into an event accumulation graph, and then multiplied with the set conversion matrix to obtain an event frame, which is expressed as:

[0044] ;

[0045] ;

[0046] in, is the event accumulation graph, is the event frame, is the symbol of the real number set, H and W represent event frames respectively The height and width of the conversion matrix are represented by p, “2” represents positive and negative polarities, and “3” represents the number of channels. .

[0047] Exemplarily, a bicubic downsampling function may be selected.

[0048] 2. Network construction and training.

[0049] In the embodiment of the present invention, a fuzzy video spatiotemporal super-resolution network based on a diffusion model is constructed, including: a multimodal data feature extraction module, a multimodal data feature injection module, a trifurcated diffusion model consisting of a trifurcated decomposition stage and a dynamic fusion stage, such as Figure 2 As shown, the processing process of the network is shown. Figure 2 The multimodal data feature extraction module and the multimodal data feature injection module are omitted, and the trifurcated diffusion model part is mainly presented.

[0050] 1. Multimodal data feature extraction module.

[0051] In the embodiment of the present invention, the multimodal data feature extraction module can be implemented using a feature extraction backbone network, which is responsible for processing the event frame and the low-resolution blurred video frame to obtain the multi-scale features of the event frame. , multi-scale features of blurred video frames ,in, is the s-th scale feature of the event frame, is the s-th scale feature of the blurred video frame, S is the number of scales, and illustratively, S=3 can be set.

[0052] 2. Multimodal data feature injection module.

[0053] In the embodiment of the present invention, the multimodal data feature injection module includes: a single modality injection module and bimodal injection module .

[0054] (1) The single modality injection module processes the multi-scale features of the event frame and the multi-scale features of the blurred video frame separately to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame.

[0055] Specifically: The single modality injection module includes three processes: feature serialization processing, word embedding processing based on the first Transformer (transformation neural network) module, and multi-scale feature update.

[0056] During feature serialization, the multi-scale features of the event frame and the multi-scale features of the blurred video frame are serialized separately to obtain the corresponding word embedding sequence (i.e., token sequence); the processing process of the s-th scale feature is expressed as:

[0057] ;

[0058] ;

[0059] in, , For , The processed word embedding sequence; , The corresponding expression for , The linear projection layer used in processing, and Represent the average pooling operation and position encoding respectively.

[0060] Then, use the first Transformer module to , Perform word embedding processing respectively to obtain the processed word embedding sequence, which is recorded as , Exemplarily, the first Transformer module may include three layers of Transformer feature processing units.

[0061] Finally, the processed word embedding sequence is used to update the multi-scale features of the event frame and the blurred video frame. The s-th scale feature update method is expressed as:

[0062] ;

[0063] ;

[0064] in, is the s-th scale feature of the updated event frame, is the s-th scale feature of the updated blurred video frame, , The corresponding representation is updated , The linear projection layer to use.

[0065] All scale features of the event frame updated by the single modality injection module are the first final multi-scale features of the event frame, and all scale features of the blurred video frame updated by the single modality injection module are the first final multi-scale features of the blurred video frame.

[0066] (2) The dual-modal injection module comprehensively processes the multi-scale features of the event frame and the multi-scale features of the blurred video frame to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame.

[0067] The bimodal injection module includes three processes: feature serialization processing, word embedding processing based on the second Transformer module, and multi-scale feature update.

[0068] In this section, the feature serialization and subsequent updating of multi-scale features are the same as those described in the single-modality injection module. The difference lies in the word embedding processing of the second Transformer module. A newly initialized network is used in the bimodal injection module, which also contains three layers of Transformer feature processing units, but the input is a bimodal word embedding sequence, that is, the word embedding sequence corresponding to the event frame and the blurred video frame obtained by the second Transformer module for feature serialization.

[0069] Specifically: For the s-th scale feature, the word embedding sequence is obtained by feature serialization processing , is a modal word embedding sequence, when m=e, it represents the event frame mode, and when m=B, it represents the blurred video frame mode; then, the word embedding sequence is processed based on the second Transformer module to obtain the processed word embedding sequence. The processing process here is that the word embedding sequences of the two modalities are simultaneously input into the second Transformer module to obtain the processed word embedding sequence. Finally, the processed word embedding sequence is used to obtain the updated event frame and blurred video frame multi-scale features through multi-scale feature update.

[0070] All scale features of the event frame updated by the bimodal injection module are the second final multi-scale features of the event frame, and all scale features of the blurred video frame updated by the bimodal injection module are the second final multi-scale features of the blurred video frame.

[0071] The first final multi-scale features obtained in the multimodal data feature injection module and the first final multi-scale features of the blurred video frame, as well as the second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame will be applied to the trifurcated decomposition stage to extract various types of noise.

[0072] 3. Trifurcated diffusion model.

[0073] The trifurcated diffusion model is divided into two stages: trifurcated decomposition stage and dynamic fusion stage.

[0074] (1) Trifurcated decomposition stage.

[0075] Using a single-modal subnetwork in the trichotomy stage and bimodal subnetwork .

[0076] The unimodal sub-network takes the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame as input and outputs motion noise and content noise , expressed as:

[0077] ;

[0078] ;

[0079] in, represents the noisy input, It means taking the first K channels of the output, where K is a preset value. For example, K can be set to 3. is the s-th scale feature of the updated event frame output by the single modality injection module (i.e., the s-th scale feature in the first final multi-scale feature of the event frame), It is the s-th scale feature of the updated blurred video frame output by the single modality injection module (i.e., the s-th scale feature in the first final multi-scale feature of the blurred video frame).

[0080] Bimodal subnetwork The second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame, as well as motion noise and content noise are taken as input, and physical noise is output. , expressed as:

[0081] ;

[0082] in, Indicates the channel merge operation, is the s-th scale feature of the updated event frame output by the bimodal injection module (i.e., the s-th scale feature in the second final multi-scale feature of the event frame), It is the s-th scale feature of the updated blurred video frame output by the bimodal injection module (i.e., the s-th scale feature in the second final multi-scale feature of the blurred video frame).

[0083] (2) Dynamic fusion stage.

[0084] In the dynamic fusion stage, the weight map of motion noise and content noise is estimated through the unimodal sub-network, and the weight map of physical noise is estimated through the bimodal sub-network, which is expressed as:

[0085] ;

[0086] ;

[0087] ;

[0088] in, , , are the weight maps of motion noise, content noise, and physical noise, respectively. It means taking the last three channels of the output, and K+2 is the last channel. These three weight maps are further normalized by softmax (normalized exponential function) operation to obtain the normalized weight map, which is recorded as , , .

[0089] Then perform noise fusion as follows to obtain the fused noise:

[0090] ;

[0091] in, is the noise after fusion. The final clear image is obtained through the sampling formula of the trifurcated diffusion model.

[0092] 4. Online training program.

[0093] The gradient descent method is used to train the blurred video spatiotemporal super-resolution model to construct the loss function shown in the following formula: , thus obtaining the trained network:

[0094] ;

[0095] in, is the noise input The noise added in represents the mean square error.

[0096] The subsequent training process involved in this part can be implemented by referring to conventional technology, so it will not be described in detail.

[0097] Figure 2 The process shown includes T rounds of iterations, and the input in round t is , output , as the input of the next iteration; the iteration starts from t=T (i.e. the first iteration), the input at this time is the blurred video frame, and the output is As the input of the T-1th iteration (i.e. the second iteration); Figure 2 t in * Represents the timestamp of the corresponding blurred video frame; when t=1, the output obtained That is, the video frame after spatiotemporal super-resolution processing.

[0098] 3. Spatiotemporal super-resolution processing.

[0099] After completing the network training according to the scheme introduced in the previous article, the video frames are input into the trained blurred video spatiotemporal super-resolution network for processing to obtain the fused noise, and then the denoising step of the trifurcated diffusion model is used to obtain clear video frames with high temporal resolution and high spatial resolution (that is, video frames after spatiotemporal super-resolution processing).

[0100] The above solution provided by the embodiment of the present invention mainly achieves the following beneficial effects:

[0101] (1) The present invention proposes to use event cameras and diffusion models to simultaneously solve the task of spatiotemporal super-resolution of blurred videos, which can improve the performance of the studied task from two aspects: data prior and model construction. First, the event camera has the advantages of high temporal resolution and high dynamic range, and can provide motion with high temporal precision, thereby providing data prior. Secondly, the diffusion model has a strong data distribution modeling capability, which is particularly suitable for the joint spatiotemporal degradation problem of spatiotemporal difference resolution of blurred videos, thereby overcoming the shortcomings of the traditional regression model modeling capability and bringing about the optimal de-motion blur and spatiotemporal super-resolution effect.

[0102] (2) The present invention proposes a two-stage three-pronged diffusion model. The first stage is the three-pronged decomposition, which can fully mine motion priors, content priors and physical priors from event stream data, video data or both. These three data priors represent different components of diffusion model denoising, which can affect the temporal continuity, content fidelity and detail restoration of the final generated video. The second stage is the dynamic fusion stage, which can estimate different weight maps for the three data priors, so as to dynamically determine the importance of different data priors. This two-stage three-pronged diffusion model makes the entire data generation process more efficient and generates higher quality videos.

[0103] 4. Experimental verification.

[0104] In order to verify the effectiveness of the present invention, an experimental verification was carried out, in which the commonly used RealSharp-DAVIS dataset was selected for training and testing. The blurred video spatiotemporal super-resolution network provided by the present invention is trained based on the RealSharp-DAVIS dataset. The RealSharp-DAVIS dataset is divided into a training set and a test set. The entire dataset contains 59 different scenes, which are shot using a DAVIS-color-346 event camera with a resolution of 346×260.

[0105] In the experiment, four existing general methods EDVR+QVI+RBPN, UTI+BVSRpp, FMANet+EMA, and eSLNetpp are selected for comparison, among which EDVR is a video deblurring algorithm, QVI is a video interpolation algorithm, RBPN is a video super-resolution algorithm, UTI is a joint video deblurring and interpolation algorithm, BVSRpp is a video super-resolution algorithm, FMANet is a joint video deblurring and super-resolution algorithm, EMA is a video interpolation algorithm, and eSLNetpp is a blurred video spatiotemporal super-resolution algorithm. The results are shown in Table 1.

[0106] Table 1: Comparison results

[0107]

[0108] In Table 1, PSNR is peak signal-to-noise ratio, SSIM is structural similarity, LPIPS is perceptual loss, and MUSIQ is multi-scale image quality. The experimental results shown in Table 1 show that in the spatiotemporal super-resolution task of blurred video based on event camera and diffusion model, the present invention is superior to the four existing general methods, thus verifying that the present invention method can effectively solve the spatiotemporal super-resolution restoration task under blurred video.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0110] Embodiment 2

[0111] The present invention also provides a fuzzy video spatiotemporal super-resolution system, which is mainly used to implement the method provided in the above embodiment, such as Figure 3 As shown, the system mainly includes:

[0112] A data acquisition and preprocessing unit, configured to acquire continuous video frames and corresponding asynchronous event stream signals, degrade the continuous video frames, wherein the video frames obtained by the degraded processing are called blurred video frames; and process the asynchronous event stream signals into event frames;

[0113] A network construction and training unit is used to construct and train a blurred video spatiotemporal super-resolution network, wherein the blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are extracted respectively by the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated by the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; various types of noise are extracted from the updated multi-scale features of event frames and blurred video frames by the trifurcated decomposition stage of the trifurcated diffusion model, and weight maps of various types of noise are estimated by the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network;

[0114] The spatiotemporal super-resolution processing unit is used to process the input video frames using the trained blurred video spatiotemporal super-resolution network to obtain fused noise, and then obtain the video (i.e., clear video) processed by spatiotemporal super-resolution through the denoising step of the trifurcated diffusion model.

[0115] Considering that the main technical details involved in the above system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0116] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0117] Embodiment 3

[0118] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided by the aforementioned embodiments.

[0119] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0120] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0121] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0122] The output device may be a display terminal;

[0123] The memory may be a random access memory (RAM) or a non-volatile memory such as a disk memory.

[0124] Embodiment 4

[0125] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0126] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0127] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A blurred video spatiotemporal super-resolution method, characterized in that: include: Collecting continuous video frames and corresponding asynchronous event stream signals, and performing quality degradation processing on the continuous video frames, wherein the video frames obtained by the quality degradation processing are called blurred video frames; and, processing the asynchronous event stream signal into an event frame; A blurred video spatiotemporal super-resolution network is constructed and trained. The blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are extracted respectively through the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated through the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; various types of noise are extracted from the updated multi-scale features of event frames and blurred video frames through the trifurcated decomposition stage of the trifurcated diffusion model, and the weight maps of various types of noise are estimated through the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network; The trained blurred video spatiotemporal super-resolution network is used to process the input video frames to obtain the fused noise, and then the spatiotemporal super-resolution processed video is obtained through the denoising step of the trifurcated diffusion model. The method of extracting multiple types of noise from the multi-scale features of the updated event frame and the blurred video frame through the trifurcated decomposition stage of the trifurcated diffusion model includes: Using a single-modal subnetwork in the trichotomy stage and bimodal subnetwork ;in: The unimodal sub-network takes the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame as input and outputs motion noise and content noise , expressed as: ； ； in, represents the noisy input, It means taking the first K channels of the output, where K is the preset value. is the s-th scale feature in the first final multi-scale feature of the event frame, is the sth scale feature in the first final multi-scale feature of the blurred video frame, and S is the number of scales; by processing the multi-scale features of the event frame and the multi-scale features of the blurred video frame separately, the updated multi-scale features of the event frame and the blurred video frame are obtained, which are called the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame; Bimodal subnetwork The second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame, as well as motion noise and content noise are taken as input, and physical noise is output. , expressed as: ； in, Indicates the channel merge operation, is the sth scale feature in the second final multi-scale feature of the event frame, is the sth scale feature in the second final multi-scale feature of the blurred video frame; by comprehensively processing the multi-scale features of the event frame and the multi-scale features of the blurred video frame, updated multi-scale features of the event frame and the blurred video frame are obtained, which are called the second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame; The weight map of each type of noise is estimated through the dynamic fusion stage, and is fused with the corresponding type of noise, and the output fused noise includes: The weight maps of motion noise and content noise are estimated through the unimodal sub-network, and the weight map of physical noise is estimated through the bimodal sub-network, which is expressed as: ； ； ； in, , , They are the weight maps of motion noise, content noise, and physical noise, respectively. Indicates taking the last three channels of the output; normalizing the three weight maps to obtain the normalized weight map, the corresponding record is , , ; Then perform noise fusion in the following way to obtain the fused noise: ； in, is the noise after fusion.

2. The blurred video spatiotemporal super-resolution method according to claim 1, characterized in that: The continuous video frames and the corresponding asynchronous event stream signals are collected, and the continuous video frames are degraded. The video frames obtained by the degraded processing are called blurred video frames; And, processing the asynchronous event stream signal into an event frame comprises: The captured video frame is recorded as , the corresponding asynchronous event stream signal is recorded as ,in, represents the kth event, and , Represents the kth event Polarity; Represents the kth event The moment of occurrence; and Represents the kth event The spatial coordinates of occurrence; N represents the total number of events; represents the space-time projection coordinates, is the moment in the space-time projection coordinates, x and y are the spatial coordinates in the space-time projection coordinates; is the indicative function; After taking the average of continuous video frames, use the downsampling function to degrade the quality and obtain blurred video frames; When the asynchronous event stream signal is processed into an event frame, it is first converted into an event accumulation graph, and then multiplied with a set conversion matrix to obtain an event frame.

3. The blurred video spatiotemporal super-resolution method according to claim 1, characterized in that: The updating of the multi-scale features of the event frame and the blurred video frame by the multi-modal data feature injection module to obtain the updated multi-scale features of the event frame and the blurred video frame includes: The multimodal data feature injection module includes: a single modality injection module and a dual modality injection module; The single modality injection module processes the multi-scale features of the event frame and the multi-scale features of the blurred video frame separately to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame; The dual-modal injection module comprehensively processes the multi-scale features of the event frame and the multi-scale features of the blurred video frame to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame.

4. The method for spatiotemporal super-resolution of blurred video according to claim 3, characterized in that: The single modality injection module processes the multi-scale features of the event frame and the multi-scale features of the blurred video frame separately to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the first final multi-scale features of the event frame and the first final multi-scale features of the blurred video frame, including: The multi-scale features of the event frame are denoted as , the multi-scale features of the blurred video frame are recorded as ,in, is the s-th scale feature of the event frame, is the s-th scale feature of the blurred video frame, S is the number of scales; The single modality injection module includes three processes: feature serialization processing, word embedding processing based on the first Transformer module, and multi-scale feature update; Transformer is a transformation neural network; During feature serialization, the multi-scale features of the event frame and the multi-scale features of the blurred video frame are serialized separately to obtain the corresponding word embedding sequence; the processing process of the s-th scale feature is expressed as: ； ； in, , For , The processed word embedding sequence; , The corresponding expression for , The linear projection layer used in processing, and Represent the average pooling operation and position encoding respectively; Use the first Transformer module to , Perform word embedding processing respectively to obtain the processed word embedding sequence, which is recorded as , ; The processed word embedding sequence is used to update the multi-scale features; the update method of the s-th scale feature is expressed as: ； ； in, is the s-th scale feature of the updated event frame, is the s-th scale feature of the updated blurred video frame, , The corresponding representation is updated , The linear projection layer used when All scale features of the event frame updated by the single modality injection module are the first final multi-scale features of the event frame, and all scale features of the blurred video frame updated by the single modality injection module are the first final multi-scale features of the blurred video frame.

5. The blurred video spatiotemporal super-resolution method according to claim 4, characterized in that: The dual-modal injection module performs comprehensive processing on the multi-scale features of the event frame and the multi-scale features of the blurred video frame to obtain updated multi-scale features of the event frame and the blurred video frame, which are called the second final multi-scale features of the event frame and the second final multi-scale features of the blurred video frame, including: The dual-modal injection module includes three processes: feature serialization processing, word embedding processing based on the second Transformer module, and multi-scale feature updating; the feature serialization processing is the same as that of the single-modal injection module, and the word embedding sequences corresponding to the event frame and the blurred video frame are simultaneously input into the second Transformer module to obtain the processed word embedding sequence, and the processed word embedding sequence is used to obtain the updated multi-scale features of the event frame and the blurred video frame through multi-scale feature updating; All scale features of the event frame updated by the bimodal injection module are the second final multi-scale features of the event frame, and all scale features of the blurred video frame updated by the bimodal injection module are the second final multi-scale features of the blurred video frame.

6. A blurry video spatiotemporal super-resolution system, characterized in that: The method for implementing any one of claims 1 to 5 comprises: A data acquisition and preprocessing unit, configured to acquire continuous video frames and corresponding asynchronous event stream signals, degrade the continuous video frames, wherein the video frames obtained by the degraded processing are called blurred video frames; and process the asynchronous event stream signals into event frames; A network construction and training unit is used to construct and train a blurred video spatiotemporal super-resolution network, wherein the blurred video spatiotemporal super-resolution network includes: a multimodal data feature extraction module, a multimodal data feature injection module, and a trifurcated diffusion model composed of a trifurcated decomposition stage and a dynamic fusion stage; the multi-scale features of event frames and blurred video frames are extracted respectively by the multimodal data feature extraction module; the multi-scale features of event frames and blurred video frames are updated by the multimodal data feature injection module to obtain updated multi-scale features of event frames and blurred video frames; various types of noise are extracted from the updated multi-scale features of event frames and blurred video frames by the trifurcated decomposition stage of the trifurcated diffusion model, and weight maps of various types of noise are estimated by the dynamic fusion stage, and are fused with the corresponding types of noise to output the fused noise; a loss function is constructed based on the fused noise to train the blurred video spatiotemporal super-resolution network; The spatiotemporal super-resolution processing unit is used to process the input video frame using the trained fuzzy video spatiotemporal super-resolution network to obtain the fused noise, and then process the video with spatiotemporal super-resolution through the denoising step of the trifurcated diffusion model.

7. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images