A video deblurring method and computer-readable storage medium
Through the self-attention mechanism and loop embedding mechanism guided by optical flow, the problem that the video defuzzing algorithm cannot model long-distance dependencies is solved, and the better defuzzing effect and robustness are achieved, which significantly improves the video defuzzing algorithm. performance.
Patent Information
- Application Number
- CN202210311024.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-28
AI Technical Summary
The existing video defuzzing algorithm cannot effectively capture long-distance dependencies in video sequences, resulting in poor robustness and defuzzing effects. Especially due to the receptive field limitation of convolutional neural networks, it is difficult to perform long-distance features. Information fusion.
The optical flow-guided self-attention mechanism is adopted to estimate the optical flow diagram between video frames through the optical flow network and input it into the self-attention network to establish a long-distance dependency relationship, integrate the inter-frame information of the video frame, and combine the cyclic embedding mechanism to transmit time Dimensional information forms a more robust defuzzing method.
Significantly improves the video debuffering effect and robustness, can capture self-similarity between pixels in a global scope, retain the original prior information of the video frame, reduces the calculation cost and memory usage, and improves the efficiency and robustness of the algorithm. sex.
Smart Images

Figure CN114677301B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video deblurring, and in particular to a video deblurring method and a computer-readable storage medium. Background Art
[0002] As an important information carrier, video plays an important role in human life and production activities. However, due to camera shake, lens defocus, object movement and other reasons, the obtained video is often blurred, resulting in quality degradation. Video deblurring algorithm can reconstruct degraded blurred video to obtain high-quality clear video. It is a very important and basic task in computer vision and plays a very important role in video stabilization, mobile phone photography, tracking and autonomous driving tasks. Traditional video deblurring algorithms are mainly based on artificially designed prior rules and assumptions. However, these algorithms have poor expression ability and generalization, so they cannot achieve satisfactory video deblurring effects. The emerging algorithms based on convolutional neural networks can achieve better reconstruction effects. However, due to the limited receptive field of convolutional neural networks, the current algorithms cannot capture long-distance dependencies in video sequences and it is difficult to fuse information for features that are far apart, resulting in the inability to achieve ideal deblurring effects and poor robustness.
[0003] Therefore, the present invention lacks an efficient and robust video deblurring method.
[0004] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0005] In order to solve the existing problems, the present invention provides a video deblurring method and a computer-readable storage medium.
[0006] In order to solve the above problems, the technical solution adopted by the present invention is as follows:
[0007] A video deblurring method comprises the following steps: S1: obtaining a video to be processed and extracting video features of the video to be processed; S2: inputting the video to be processed into an optical flow network to estimate an optical flow map between video frames of the video to be processed; S3: inputting the video features and the optical flow map into a self-attention network to obtain a residual video; S4: adding the residual video to the video to be processed to obtain a deblurred clear video.
[0008] Preferably, inputting the video to be processed into an optical flow network to estimate the optical flow graph between video frames of the video to be processed specifically includes the following steps: taking two video frames of the video to be processed as inputs of the optical flow network, and extracting motion information between the two video frames, specifically:
[0009] O=F o (f 1 ,f 2 ),
[0010] Among them, F o represents the optical flow network, f 1 ,f 2 represents adjacent video frames, and O represents the optical flow map.
[0011] Preferably, a spatial pyramid network or a light field convolutional neural network is used as the optical flow network to perform feature extraction on the video frame to be processed, the video information of the video to be processed is mapped from the color space to the feature space, the optical flow of each pyramid level is learned in a coarse-to-fine manner, and finally an accurate optical flow map between the two video frames is calculated.
[0012] Preferably, the video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video, and the optical flow guided self-attention network uses the optical flow to guide the self-attention mechanism and fuse the information between video frames.
[0013] Preferably, the optical flow guided self-attention network uses optical flow to guide the self-attention mechanism, and the fused video frame information is identified as:
[0014]
[0015] Among them, q represents the query element, Ω represents the set of address elements, which are selected by guiding the optical flow information; A nqk Represents the similarity between the index element and the address element, calculated by the self-similarity mechanism, W n and W' n is the mapping matrix, is a learnable parameter, and the calculation formula of the self-similar mechanism is:
[0016] A nqk =softmax(q T U T Vk)
[0017] Among them, U and V are mapping matrices and are learnable parameters.
[0018] Preferably, the video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video, and the optical flow guided self-attention network uses the optical flow to guide the window attention mechanism to fuse the information between video frames.
[0019] Optionally, the video features are divided into evenly distributed windows. For each query element in the window, not only the address element corresponding to the query element is referenced, but also the address elements corresponding to other query elements in the window are referenced. The process is expressed by the following formula:
[0020] FGSW-MSA(Φ,Ψ)={FGS-MSA(q,Ψ)|q∈Φ}
[0021] Wherein, q represents a query element, Φ represents a query element set in the window, and Ψ represents an address element corresponding to the query element set.
[0022] Preferably, the self-attention network also includes: a recurrent embedding mechanism, which stores the previous video frame information in the recurrent embedding feature before calculating the optical flow to guide the self-attention network, splices it with the query element of the current frame, and then performs the attention mechanism calculation to pass the information to subsequent video frames, iterate frame by frame, and then establish long-distance dependency in the time dimension.
[0023] Preferably, in the recurrent embedding mechanism, the features of the previous video frame need to be aligned to the current frame, and then connected with the current video frame information in the channel dimension as a query element and input into the self-attention mechanism. The process can be expressed by the following formula:
[0024]
[0025]
[0026] Among them, x f-1 ,x f Represent the features of the f-1th and fth video frames respectively, F w ,F c Represents alignment and splicing operations.
[0027] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0028] The beneficial effects of the present invention are: providing a video deblurring method and a computer-readable storage medium, modeling long-distance dependencies through a self-attention mechanism guided by optical flow, capturing the self-similarity between pixels on a global scale, solving the problem that the current video deblurring algorithm cannot model global dependencies, and having better deblurring effect and robustness.
[0029] Furthermore, the present invention proposes a new mode of utilizing optical flow information. Instead of using bilinear interpolation operations to align features, motion compensation and feature fusion are organically combined to apply optical flow maps more flexibly. Therefore, the original prior information of the video frame will not be destroyed, and the self-similarity and clear texture of the features are retained. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a schematic diagram of a video deblurring method according to an embodiment of the present invention.
[0031] Figure 2 Schematic diagram of an optical flow-guided sparse self-attention module in an embodiment of the present invention.
[0032] Figure 3 Schematic diagram of a loop embedding mechanism in an embodiment of the present invention.
[0033] Figure 4 Schematic diagram of a network composed of a self-attention mechanism and a recurrent embedding mechanism in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0035] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit connection.
[0036] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0037] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0038] like Figure 1 As shown, the present invention provides a video deblurring method, comprising the following steps:
[0039] S1: Obtain a video to be processed and extract video features of the video to be processed;
[0040] S2: Inputting the video to be processed into an optical flow network to estimate an optical flow graph between video frames of the video to be processed;
[0041] S3: Inputting the video features and the optical flow map into a self-attention network to obtain a residual video;
[0042] S4: Adding the residual video to the video to be processed to obtain a deblurred clear video.
[0043] The present invention uses the self-attention mechanism to model long-distance dependencies and capture the self-similarity between pixels on a global scale, thus solving the problem that the current video deblurring algorithm cannot model global dependencies and has better deblurring effect and robustness.
[0044] The present invention proposes a new mode of utilizing optical flow information. Instead of using bilinear interpolation operation to align features, motion compensation and feature fusion are organically combined to apply optical flow maps more flexibly. Therefore, the original prior information of the video frame will not be destroyed, and the self-similarity and clear texture of the features are retained.
[0045] In addition, there are some problems with using the self-attention mechanism directly for video deblurring. On the one hand, when the original global attention mechanism is used, the computational cost is proportional to the square of the number of pixels in the video sequence, which will bring too much computational and video memory burden. In addition, the global attention mechanism will refer to all features, which can easily lead to non-convergence and over-smooth results. On the other hand, when the local attention mechanism is used, the attention mechanism is limited to the local window. When there is a large relative motion between video frames, it is easy to miss similar and clear features, and it is impossible to establish long-distance spatial dependencies. These two problems can be summarized as: lack of guidance of motion information when calculating the attention mechanism.
[0046] Optical flow estimation is an important technology for evaluating the motion information between video frames. In the early days, optical flow estimation was regarded as an energy optimization problem, which was optimized based on artificially designed energy criteria to find the balance between the data term and the regularization term. With the development of deep learning, people have proposed end-to-end learning algorithms, which use optical flow networks to directly predict the optical flow information between pictures, achieving better results and faster speeds. In recent years, optical flow networks have been widely used in the field of video deblurring to evaluate the motion information between video frames and perform motion compensation. The current optical flow-based video deblurring algorithm is mainly based on such a mode: first, the optical flow network is used to evaluate the motion information between video frames, and then the adjacent frames are warped by bilinear interpolation to align the adjacent frames to the current frame, and then the aligned adjacent frames and the current frame are connected and information is aggregated to obtain a clear picture. In this mode, the interpolation operation in the warp module will change the original prior information of the image, causing the original self-similarity and clear texture in the image to be destroyed. Moreover, the inter-frame alignment and information aggregation in this mode are separate, so it is inflexible and cannot make full use of the optical flow information. In addition, the performance of this mode is also easily affected by the performance of the optical flow estimation network. Therefore, optical flow estimation based on deep learning is an effective method to evaluate the motion information between video frames, but the robustness and reliability of the way in which the current video deblurring algorithm uses optical flow information needs to be improved. The present invention introduces optical flow estimation into the self-attention mechanism and proposes a more robust and effective way to utilize optical flow information.
[0047] In the video deblurring algorithm, establishing long-distance dependencies in the time dimension of the video can make use of more information in the video reconstruction process, significantly improving the video deblurring effect. The RNN-based video deblurring algorithm propagates feature information in the form of latent features, allowing information to be transmitted frame by frame in an iterative form, thereby establishing long-distance dependencies in the time dimension. Therefore, the present invention also introduces this iterative mechanism into the optical flow guided self-attention network, and proposes a recurrent embedding feature (Recurrent Embedding) to assist the optical flow guided self-attention network in modeling long-distance dependencies in the time dimension.
[0048] In a specific embodiment, inputting the to-be-processed video into an optical flow network to estimate an optical flow graph between video frames of the to-be-processed video specifically includes the following steps:
[0049] The two video frames of the video to be processed are used as the input of the optical flow network, and the motion information between the two video frames is extracted, specifically:
[0050] O=F o (f 1 ,f 2 ),
[0051] Among them, F o represents the optical flow network, f 1 ,f 2 represents adjacent video frames, and O represents the optical flow map.
[0052] Specifically, a spatial pyramid network or a light field convolutional neural network is used as the optical flow network to extract features of the video frame to be processed, the video information of the video to be processed is mapped from the color space to the feature space, the residual optical flow of each pyramid level is learned in a coarse to fine manner, and finally an accurate optical flow map between the two video frames is calculated.
[0053] The self-attention mechanism calculates the similarity between each element in the query element and the address element set, rescales the similarity through the normalized exponential function (Softmax), obtains the weight of each address element, and aggregates the address element information according to the weight as an information supplement to the query element. The memory usage and computational cost of the self-attention mechanism are proportional to the square of the number of elements. In the self-attention mechanism, since the selection range of address elements is global, the self-attention mechanism can model long-distance dependencies in the spatial dimension, solving the problem that the existing defuzzification algorithm cannot model long-distance dependencies.
[0054] However, the memory usage and computational cost of the self-attention mechanism are in a square relationship with the number of selected elements. If the original self-attention mechanism is used, all pixels are globally selected as address elements for reference, which will bring great computational costs in the future and is not feasible. In order to reduce the computational cost and memory usage, the present invention uses an optical flow map to guide the self-attention mechanism so that the attention mechanism focuses on features with a high degree of correspondence. For each query element in the reference frame, similarity calculations are not performed with all address elements, but only with sparsely distributed address elements specified by the optical flow. According to the calculated similarity, information aggregation is performed on these address elements specified by the optical flow to supplement the features of the query element. This significantly reduces the computational cost and memory usage of the self-attention mechanism, so that the attention is focused on features with a high degree of correspondence, making it more efficient; at the same time, since the selected address elements are elements specified by the optical flow, the self-attention mechanism can be focused on elements with a high degree of correspondence, which can ensure that the effect of the self-attention mechanism will not be affected.
[0055] When calculating the self-attention mechanism, in order to reduce the computational cost and memory usage, the present invention uses an optical flow graph to guide the self-attention mechanism so that the attention mechanism focuses on features with a high degree of correspondence. For each query element (Query Element) in the reference frame, similarity calculations are not performed with all address elements (Key Element), but only with address elements specified by sparse optical flows. Based on the calculated similarity, information aggregation is performed on the address elements specified by these optical flows to supplement the features of the query elements. This significantly reduces the computational cost and memory usage of the self-attention mechanism, allowing attention to be focused on features with a high degree of correspondence, making it more efficient.
[0056] The present invention uses optical flow information to guide the selection of address elements. Each query element only performs self-similarity calculations with a number of sparsely distributed and highly correlated address elements, which can focus attention on content with a high degree of correspondence. The self-attention mechanism proposed in the present invention has fast calculation speed and occupies little memory.
[0057] The optical flow-guided sparse attention mechanism (FGS-MSA) guides the fusion of video frame information:
[0058] In a specific embodiment, the video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video, and the optical flow guided self-attention network uses the optical flow to guide the self-attention mechanism to fuse the information between video frames.
[0059] The optical flow guided self-attention network uses optical flow to guide the self-attention mechanism to fuse the information between video frames as follows:
[0060]
[0061] Among them, q represents the query element, Ω represents the set of address elements, which are selected by guiding the optical flow information; A nqk Represents the similarity between the index element and the address element, calculated by the self-similarity mechanism, W n and W' n is the mapping matrix, is a learnable parameter, and the calculation formula of the self-similar mechanism is:
[0062] A nqk =softmax(q T U T Vk)
[0063] Among them, U and V are mapping matrices and are learnable parameters.
[0064] In the optical flow-guided sparse attention mechanism, for each query element (QueryElement) in the current frame, only one address element (Key Element) specified by the optical flow in each adjacent frame will be referenced. When the optical flow estimation is inaccurate, the performance of this mechanism will be greatly affected. Therefore, the present invention further proposes an optical flow-guided window sparse attention mechanism. The feature map is divided into evenly distributed non-overlapping windows. For each query element in the window, not only the address element specified by the optical flow corresponding to the query element is referenced, but also the address elements corresponding to other query elements in the window. In this case, when the corresponding optical flow information of a query element in the window is inaccurate, the performance of the mechanism will not be greatly affected, so it has better robustness.
[0065] In another specific embodiment, the video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video, and the optical flow guided self-attention network uses the optical flow to guide the window attention mechanism to fuse the information between video frames.
[0066] The video features are divided into evenly distributed windows. For each query element in the window, not only the address element corresponding to the query element is referenced, but also the address elements corresponding to other query elements in the window are referenced. This process is expressed by the following formula:
[0067] FGSW-MSA(Φ,Ψ)={FGS-MSA(q,Ψ)|q∈Φ}
[0068] Wherein, q represents a query element, Φ represents a query element set in the window, and Ψ represents an address element corresponding to the query element set.
[0069] The optical flow-guided window sparse self-attention mechanism proposed in the present invention uses an optical flow graph to guide the self-attention mechanism. Each query element only refers to a number of address elements specified by the optical flow, which solves the problems of excessive video memory and lack of motion information guidance in ordinary attention mechanisms. Compared with the current mainstream self-attention mechanism, the sparse self-attention mechanism proposed in the present invention has the advantages of fast calculation speed and small memory usage, and can focus attention on content with a high degree of correspondence, thereby achieving sharper results.
[0070] The present invention proposes a sparse self-attention mechanism guided by optical flow, which uses an optical flow network to estimate the motion relationship between frames, so as to guide the self-attention mechanism to select highly correlated and sparsely distributed address elements and perform self-similarity calculations, thereby efficiently capturing long-distance dependencies in the spatial dimension.
[0071] like Figure 2 As shown, it is a schematic diagram of a sparse self-attention module guided by optical flow of the present invention. The module includes an optical flow estimation module and a self-attention module. When the self-attention mechanism is guided by optical flow information, the address element is sampled on the adjacent frame according to the optical flow map estimated in step S2, and the address element and the query element are input into the self-attention mechanism for calculation. In order to prevent the information loss caused by the indexing process of the address element, the address element is indexed by the nearest neighbor interpolation method. The coordinates of the query element are added to the optical flow bias to obtain the address element position, and then the nearest neighbor interpolation is used to index the address element closest to the position as a reference for the query element, and the self-attention calculation is performed. The result of the self-attention calculation is used as the information supplement of the current address element and added.
[0072] Optionally, a recurrent embedding mechanism can be added to supplement the optical flow-guided sparse attention mechanism to enhance the modeling capability of long-distance dependencies in the temporal dimension.
[0073] The loop iteration mechanism propagates information in the form of latent features, and propagates information frame by frame through frame-by-frame iteration, thereby establishing long-distance dependencies in the time dimension.
[0074] Specifically, the self-attention network of the present invention also includes:
[0075] A recurrent embedding mechanism stores the information of the previous video frame in the recurrent embedding feature before calculating the optical flow to guide the self-attention network, and concatenates it with the query element of the current frame. When performing the attention mechanism calculation, the information is passed to subsequent video frames, iterating frame by frame, thereby establishing long-distance dependency in the time dimension.
[0076] In the recurrent embedding mechanism, the features of the previous video frame need to be aligned to the current frame, and then connected with the current video frame information in the channel dimension as a query element and input into the self-attention mechanism. This process can be expressed by the following formula:
[0077]
[0078]
[0079] Among them, x f-1 ,x f Represent the features of the f-1th and fth video frames respectively, F w ,F c Represents alignment and splicing operations.
[0080] like Figure 3 The figure shows a schematic diagram of a cyclic embedding mechanism of the present invention. The features of the previous frame are aligned with the current frame through the distortion module, and then spliced with the features of the current frame. The information of the two is fused through the convolution layer, and the information is input into the attention module as a query element for self-attention calculation. Since the cyclic embedding features can be transmitted frame by frame, long-distance dependencies in the time dimension can be established, further enhancing robustness.
[0081] like Figure 4 As shown, in order to perform video deblurring at multiple scales and improve the algorithm effect, the optical flow guided self-attention network adopts a multi-scale structure, uses downsampling and upsampling modules, changes the feature resolution, performs cyclic embedding mechanism and optical flow guided self-attention mechanism calculation at different resolutions, and prevents information loss in downsampling operations through skip connections. Among them, the downsampling module uses a convolution layer with a stride of 2, and the upsampling module uses a deconvolution layer with a stride of 2.
[0082] The present invention proposes a loop embedding mechanism to transmit video information frame by frame in a loop embedding mechanism, thereby establishing a long-distance dependency relationship in the time dimension. Combining these two technologies, the video deblurring algorithm proposed in the present invention can surpass the current mainstream video deblurring algorithm with a comparable or even reduced computational cost, and has better deblurring effect and robustness.
[0083] Recurrent Embedding uses reconstructed video frames to enhance the reconstruction of the current frame and establishes long-distance dependencies in the time dimension. At the same time, since the reconstruction of the later frames is based on the reconstruction results of the previous frames, the reconstruction difficulty can be reduced and the ideal reconstruction effect can be achieved at a lower computational cost.
[0084] The loop embedding mechanism stores the information of the previous video frame in the loop embedding feature, passes it to the subsequent video frames, and iterates frame by frame, thereby establishing a long-distance dependency in the time dimension and improving the video deblurring effect.
[0085] Continue as Figure 4 As shown, first, one convolution and five stacked residual blocks are used for feature extraction to convert the video information from color space to feature space; then a multi-scale optical flow guided attention network is used to calculate the optical flow guided attention mechanism and the recurrent embedding mechanism at three scales to obtain a residual video; finally, the residual video is added to the input video to output a clear deblurred video. The optical flow guided self-attention network (FGST) proposed in the present invention is compared with four mainstream video deblurring algorithms, including: spatiotemporal filter adaptive network (STFAN), enhanced deformable convolution network (EDVR), deep video deblurring algorithm (Su), cascaded clarity prior deblurring network (TSP) and other mainstream video deblurring algorithms. As shown in Table 1, it is the comparison result of the above experiment.
[0086] Table 1 Comparison results of the experiments
[0087]
[0088] As can be seen from Table 1, the peak signal-to-noise ratio of the clear video obtained by the method of the present invention reaches 32.90dB and the structural similarity reaches 0.961, which is significantly better than the current mainstream algorithms. Specifically, the peak signal-to-noise ratios of the output videos of STFAN, EDVR, Su, and TSP are 28.59, 26.83, 27.31, and 31.67dB, respectively, and the structural similarity indexes are 0.861, 0.843, 0.826, and 0.928, respectively. In addition, in terms of parameter quantity and computational complexity, the algorithm is also significantly superior to mainstream algorithms such as EDVR and TSP. The algorithm has the advantages of only 41.4% and 59.9% of EDVR and TSP in parameters and 82.7% and 36.8% of EDVR and TSP in computation, but can achieve better deblurring effect; further, in the presence of optical flow-guided sparse window self-attention mechanism (FGSW-MSA), regional features with large motion offset and blur will have more significant response, which shows that the attention mechanism proposed in the present invention can make the model more sensitive to motion information and focus more on blurry areas with high correspondence. This proves the efficiency and robustness of the algorithm.
[0089] The present invention proposes a new video deblurring algorithm, which models the long-distance dependencies in the spatial and temporal dimensions in an efficient way, thereby significantly improving the efficiency and robustness of the video deblurring algorithm.
[0090] The algorithm proposed in the present invention can achieve the currently optimal video deblurring effect on synthetic datasets and real datasets at a relatively low computational cost.
[0091] An embodiment of the present application also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor is used to execute at least the method described above when executing the computer program.
[0092] An embodiment of the present application also provides a storage medium for storing a computer program, which at least performs the method described above when executed.
[0093] An embodiment of the present application further provides a processor, which executes a computer program and at least performs the method described above.
[0094] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAMEnhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0095] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0096] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0097] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0098] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.
[0099] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0100] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0101] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0102] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0103] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A video deblurring method, characterized in that: The steps include: S1: Obtain a video to be processed and extract video features of the video to be processed; S2: Inputting the video to be processed into an optical flow network to estimate an optical flow graph between video frames of the video to be processed; S3: Inputting the video features and the optical flow map into a self-attention network to obtain a residual video; S4: adding the residual video to the video to be processed to obtain a deblurred clear video; Inputting the video to be processed into an optical flow network to estimate an optical flow graph between video frames of the video to be processed specifically includes the following steps: The two video frames of the video to be processed are used as the input of the optical flow network, and the motion information between the two video frames is extracted, specifically: O=F o (f1,f2), Among them, F o represents the optical flow network, f1, f2 represent adjacent video frames, and O represents the optical flow map; A spatial pyramid network or a light field convolutional neural network is used as the optical flow network to extract features of the video frame to be processed, the video information of the video to be processed is mapped from the color space to the feature space, the optical flow of each pyramid level is learned in a coarse-to-fine manner, and finally an accurate optical flow map between the two video frames is calculated.
2. The video deblurring method according to claim 1, wherein: The video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video. The optical flow guided self-attention network uses the optical flow to guide the self-attention mechanism to fuse the information between video frames.
3. The video deblurring method according to claim 2, wherein: The optical flow guided self-attention network uses optical flow to guide the self-attention mechanism to fuse the information between video frames as follows: Among them, q represents the query element, Ω represents the set of address elements, which are selected by guiding the optical flow information; A nqk Represents the similarity between the index element and the address element, calculated by the self-similarity mechanism, W n and W' n is the mapping matrix, is a learnable parameter, and the calculation formula of the self-similar mechanism is: A nqk =softmax(q T U T Vk) Among them, U and V are mapping matrices and are learnable parameters.
4. The video deblurring method according to claim 1, wherein: The video features and the optical flow map are input into an optical flow guided self-attention network to obtain a residual video. The optical flow guided self-attention network uses the optical flow to guide the window attention mechanism to fuse the information between video frames.
5. The video deblurring method according to claim 4, characterized in that: The video features are divided into evenly distributed windows. For each query element in the window, not only the address element corresponding to the query element is referenced, but also the address elements corresponding to other query elements in the window are referenced. This process is expressed by the following formula: FGSW-MSA(Φ,Ψ)={FGS-MSA(q,Ψ)|q∈Φ} Wherein, q represents a query element, Φ represents a query element set in the window, and Ψ represents an address element corresponding to the query element set.
6. The video deblurring method according to claim 3 or 5, characterized in that: The self-attention network also includes: A recurrent embedding mechanism stores the information of the previous video frame in the recurrent embedding feature before calculating the optical flow to guide the self-attention mechanism, and then splices it with the query element of the current frame, and then performs the attention mechanism calculation to pass the information to subsequent video frames, iterating frame by frame, and then establishing long-distance dependency in the time dimension.
7. The video deblurring method according to claim 6, wherein: In the recurrent embedding mechanism, the features of the previous video frame need to be aligned to the current frame, and then connected with the features of the current video frame in the channel dimension as a query element and input into the self-attention mechanism. This process can be expressed by the following formula: Among them, x f-1 ,x f Represent the features of the f-1th and fth video frames respectively, F w ,F c Represents alignment and splicing operations.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
