A two-stage video deblurring method and system based on mixed motion states

Abstract

The present invention provides a two-stage video deblurring method and system based on mixed motion states, and relates to the technical field of image processing. The present invention divides the repair process of mixed motion blur into a global motion blur removal stage and a local target blur removal stage through a two-stage deblurring strategy, and performs targeted modeling and processing on motion blur features of different scales respectively. In the global motion blur removal stage, a fused hybrid expert network is constructed in the spatial domain to aggregate intra-frame features, and a frequency domain attention module is used to modulate and align the spectrum in the Fourier frequency domain, effectively improving the removal quality of global motion blur and breaking through the performance bottleneck of traditional methods that rely only on spatial domain processing. In the local target blur removal stage, an optical flow estimation method is used to predict local motion trends, and a TransMamba module that combines a spatiotemporal attention mechanism with Mamba is introduced to achieve high-precision propagation and restoration of inter-frame detail features.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a two-stage video deblurring method and system based on mixed motion states. Background Art

[0002] When an object moves at a high speed relative to an imaging device (such as a camera or mobile phone camera), the device's photosensitive element continuously receives light from different positions on the object during the exposure time. This light forms a continuous light trail on the photosensitive element, resulting in motion blur. Simply put, the object moves a certain distance during the imaging process, resulting in it appearing as a continuous path on the image rather than a clear, static image. Motion blur blurs the edges of objects in the image, blurring and diffusing previously sharp outlines. This loss of detail severely impacts image clarity and resolution, making it difficult for viewers to accurately discern the object's shape, texture, and other features.

[0003] In order to effectively utilize redundant pixels in space to regress the original pixel spatial positions, DSTNet proposed the use of spatial wavelet transform at the CVPR conference, which performs multi-scale spatial attention through a wavelet perspective. However, while wavelet frequency transform can locally calculate blurred low frequencies, it cannot achieve spatial offsets for changes in high-dimensional temporal dimension arrangements. VRT at the ICCV conference believed that global temporal and spatial arrangements were required for video sequences, and introduced a global spatiotemporal attention algorithm. However, the high cost hindered the deployment and implementation of this method. TurbSR at the CVPR conference proposed using template matching to align the central area between frames. However, this alignment method is not suitable for scenes with camera movement due to edge cropping.

[0004] Current research and engineering challenges often focus on addressing the loss of edge detail caused by high-speed motion of the target, while ignoring complex scenarios such as camera shake, dolly motion, and high-speed relative motion of the target (where both foreground and background shift simultaneously). Distortion in complex real-world scenarios typically involves global blurring due to camera motion and localized distortion due to relative motion between the captured targets.

[0005] Hybrid motion mechanisms cause pixel displacement in space to be driven by two or more vectors, making it difficult to capture pixel motion trajectories due to mutual interference. Object-based restoration algorithms offer some improvement in static scenes with an unchanging background. However, in scenes with high-speed camera movement and a large amount of redundant low-frequency information, they lack global motion capture, hindering effective motion blur removal and significantly reducing restoration effectiveness.

[0006] Therefore, we need to provide a two-stage video deblurring method and system based on mixed motion states to solve the technical problem that the existing technology ignores complex task scenarios, resulting in poor repair effect when dealing with the mixed superposition of local target motion blur and overall lens motion blur. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a two-stage video deblurring method and system based on mixed motion states. The method aims to divide the repair process of mixed motion blur into a global motion blur removal stage and a local target blur removal stage, and to perform targeted modeling and processing on motion blur features of different scales, thereby enhancing the detailed expression of the target edge area at the pixel level and alleviating the problem of insufficient edge sharpness of local target images.

[0008] To achieve the above objectives, this application proposes a two-stage video deblurring system based on mixed motion states, comprising:

[0009] Data acquisition unit: collects motion-distorted videos of various motion states and uses them as datasets for training and testing a hybrid motion two-stage video deblurring model; wherein the hybrid motion two-stage video deblurring model includes a global motion deblurring unit and a local motion deblurring unit;

[0010] Two-stage task setup unit: By studying and analyzing blurred videos under mixed motion, a physical motion model driven by global and local motion is derived. Based on the end-to-end two-stage restoration scheme HMD-TVDF, the blurred video restoration task under mixed motion is decomposed into the global motion deblurring stage and the local motion deblurring stage.

[0011] Global motion deblurring unit: used to perform the global motion deblurring stage, fuse the features processed in the spatial domain and the Fourier frequency domain, and obtain a video frame with global blur removed; it includes a hybrid expert module and a frequency domain energy attention module; wherein the hybrid expert module is used to extract large edge features in the spatial domain and obtain a weighted fusion multi-expert output; the frequency domain energy attention module is used to correct abnormal fluctuations in global motion energy in the Fourier frequency domain;

[0012] Local motion deblurring unit: used to perform the local motion deblurring stage and ultimately output a clear video with local motion blur removed, including an optical flow estimation module and a hybrid module; wherein the optical flow estimation module is used to predict the motion of local areas between different frames and implement a coarse aggregation operation of target inter-frame features based on the pixel offset of the optical flow; the hybrid module is used to establish spatiotemporal context dependencies for local targets and repair the non-uniform blur of the moving target by capturing the pixel offset of the inter-frame motion within the 3D window.

[0013] As a further implementation scheme, the training dataset and the test dataset include a synthetic blurred video dataset and a real blurred video dataset; wherein, the dataset includes a local target motion blurred video, a global motion blurred video, and a mixed motion blurred video in which global motion blur and target motion exist simultaneously.

[0014] As a further embodiment, the physical motion model is set by the following modeling formula:

[0015]

[0016] Where X represents the mixed motion blurred video generated by the mixed motion vector joint drive; δ is the local blur caused by the target motion; It is a blurred frame presentation after the entire frame is disturbed by global motion; Y represents the corresponding clear video of the mixed motion blurred video; ω represents the global motion vector of the overall movement of the lens; n is the number of video frames in the frame sequence; ɛ represents the driving vector of the local target motion; h 、 w Respectively represent the height and width of the video frame; Y(i,j) Represents the coordinates of the local target high-speed motion area; t is the time parameter.

[0017] As a further embodiment, the hybrid expert module is a large-core hybrid expert module, including a weight distribution module and a hybrid expert network;

[0018] The hybrid expert network includes a chief expert module and several deputy expert modules, and the weight distribution module includes a 3D convolution module, a pooling module, a fully connected layer module and a Softmax module;

[0019] The frequency domain energy attention module includes a convolution module, a Mamba module and a channel attention module.

[0020] As a further implementation scheme, the optical flow estimation module is set by a pre-trained RAFT model, and the hybrid module is set by a hybrid network TransMamba constructed by combining a Mamba module and a Transformer model, including a TemporalMamba module and a 3D SwinTransformer module.

[0021] On the other hand, the present invention also provides a two-stage video deblurring method based on mixed motion states, which is applied to a two-stage video deblurring system based on mixed motion states as described in any of the above items. When executing the end-to-end two-stage restoration scheme HMD-TVDF: input the blurred video frame and use the 3D convolution module to embed the channel dimension and map it to a high-dimensional space, perform preliminary feature extraction in the high-dimensional space, and use the channel attention module to stabilize the feature channel after dimensionality increase of the preliminary extracted features to obtain the embedded video frame after dimensionality increase; wherein, the embedded video frame after dimensionality increase will be input into a U-shaped network with three different resolution scales, and the global motion deblurring stage and the local motion deblurring stage will be performed for each resolution scale respectively.

[0022] As a further implementation, when the U-network performs the global motion deblurring stage:

[0023] The input of each resolution scale is first passed to the global deblurring stage, which fuses the frequency domain and spatial domain features to output a globally clear frame with locally blurred frames. F non-g ;

[0024] Global clear frame F non-g Two branches are input respectively, one branch is used as a skip connection at the same scale, and the other branch is a global deblurring stage that downsamples the input to the next resolution scale through convolution operation;

[0025] The global stage output at the minimum resolution scale is passed into two branches. One branch serves as the input of the local motion stage at the same scale, and the other branch performs feature fusion with the skip connection of the global output of the second scale under convolution upsampling and channel attention stabilization. The fused features execute the same process as above.

[0026] As a further implementation, when the U-network performs the local deblurring stage:

[0027] The local deblurring stage performs bidirectional propagation of inter-frame features and pixel-level detail enhancement to obtain a sharpened video frame with a clear moving target area. F non-l ;

[0028] The decoder fuses the outputs of the local deblurring stages at three different resolution scales and compresses the channel dimension to generate a video restoration frame without global motion blur and local motion blur.

[0029] As a further embodiment, the global motion deblurring unit performs the global deblurring stage by:

[0030] For global motion restoration in the spatial domain, 3D convolution modules, pooling modules, fully connected layer modules, and Softmax modules are used to dynamically calculate different sub-expert weights.

[0031] The blurred video frame is evenly divided into N-1 groups of partial features along the channel dimension. Each group of features is input into the sub-expert module of the corresponding channel group. Each sub-expert module is reorganized along the channel by the sub-expert weight assigned by the dot product to obtain the sub-expert group feature F sec The chief expert module inputs the unsegmented fuzzy frame and calculates the chief discriminant output F master ;

[0032] Use a learnable parameter as a coefficient to adjust the balance between the chief expert module and each deputy expert module, and weightedly integrate the deputy expert group features F sec and the chief discriminant output F master ;

[0033] The channel attention module is used to strengthen and stabilize the channel dimension of the fusion features of all experts; the multi-expert feature capture design eliminates the dependence on a single feature and generates a clear feature F in the spatial domain. spa ;

[0034] To repair global motion blur in the frequency domain, a spectrum is generated through Fourier transform, and the amplitude spectrum and phase spectrum of the blurred frame are gradually decomposed along each channel.

[0035] The 1x1 convolution is used to modulate the spectrum signal of the amplitude spectrum and phase spectrum, and transform it into a recognizable signal that changes with continuous fluctuations in the time domain;

[0036] Then, layer normalization is used to compress the spectrum signal, and the amplitude spectrum and phase spectrum are rearranged and expanded in one dimension along the time-frequency direction. The correct frequency distribution between the spectra is calculated based on the mamba module.

[0037] After correcting the frequency distribution and adding residual connection, the phase spectrum and amplitude spectrum are fused channel by channel to reconstruct the Fourier spectrum, and the Fourier spectrum with accurate frequency distribution is inversely Fourier transformed to obtain the spectrum enhanced spatial domain mapping F fre , using channel attention to stabilize feature mapping along the channel dimension;

[0038] Based on feedforward network and normalization fusion spatial domain clear feature F spa and spatial domain mapping F fre , and obtain a clear frame with global motion blur removed.

[0039] As a further embodiment, the local motion deblurring unit is performed by the following steps:

[0040] The clear frames with global motion blur removed are input into the pre-trained RAFT model to calculate the pixel-level displacement of the relative local motion between objects between adjacent video frames, obtaining preliminary optical flow estimation results and enriching the intra-frame features of the previous and next moments.

[0041] After optical flow alignment, the image values ​​are compressed through layer normalization, and bidirectional window rearrangement is performed according to the time sequence. The TemporalMamba module establishes spatiotemporal context dependencies through bidirectional feature propagation and uses a feedforward network to fuse bidirectional features.

[0042] The fused features are passed to the 3D SwinTransformer module using a 3D shift window. The pixel blocks within the 3D shift window are divided into query Q, key K and value V. Multi-head self-attention calculation is performed by embedding the position code and the shift mask of the 3D window.

[0043] The output attention results are constrained by residual connections and weighted by the discriminative features of the feedforward network to obtain video frames with clear local motion areas.

[0044] Compared with related technologies, the two-stage video deblurring method and system based on mixed motion states provided by the present invention has the following advantages:

[0045] 1. This invention decomposes mixed motion blur into a progressive process, from processing global motion blur caused by lens movement to repairing target motion blur. This two-stage framework design clearly overcomes the drawback of conventional algorithms that can only process a single state of target motion and cannot repair global motion, achieving significant repair performance for mixed motion blur in complex scenes.

[0046] 2. The present invention proposes a hybrid expert module based on a sparse large kernel. This module uses multiple sub-experts with the same internal structure to perform sparse calculations on some features after the clipping channel. While maintaining the same computing resources, it enables multiple expert networks to perceive different feature distributions. In a single expert network, we use a large-kernel dilated convolution group for sparse sampling, which enables the expert network to focus on the extraction of large features. By designing multiple experts to perform sparse calculations, the problem of redundant calculations caused by extracting repeated features between dense networks is solved. The design of the chief expert solves the problem of branch representation degradation in traditional gated networks.

[0047] 3. This paper proposes a frequency-domain energy attention (FEA) module based on Fourier transforms, convolution, and a Mamba module (selective spatial state model). Through a cross-domain perspective, this method accurately utilizes the frequency information of high-frequency edges down-converted and diffused to low-frequency space due to global motion, achieving frequency modulation and calibration alignment, mitigating frequency fluctuations and restoring a stable frequency structure. An auxiliary hybrid expert module utilizes cross-frequency information fusion to overcome the performance bottleneck of mainstream algorithms for aggregating single spatial features, optimizing global blur recovery.

[0048] 4. This paper develops a hybrid architecture module (TransMamba) that combines Mamba and Transformer to optimize the bidirectional propagation of information between frames and utilizes the latent state space to construct spatiotemporal contextual dependencies. This technology improves the accuracy of multi-frame image alignment through spatiotemporal attention, refines the rich features distributed across multiple frames, and makes the restoration of local details in the video more natural and clear. This also improves the problem of insufficient features within a single frame to restore clear details. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 This is a schematic diagram of the steps of a two-stage video deblurring system proposed by the present invention;

[0052] Figure 2 Schematic diagram of a multi-scale hybrid motion deblurring network HMD-TVDF proposed in the present invention;

[0053] Figure 3 2. It is a schematic diagram of the multi-domain parallel structure of the global motion removal stage provided by the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of a frequency domain energy attention module proposed in the present invention;

[0055] Figure 5 This is a flow chart of the hybrid expert sparse computing strategy proposed in the present invention;

[0056] Figure 6 This is a structural diagram of an expert weight dynamic distribution module proposed by the present invention;

[0057] Figure 7 This is a schematic diagram of the structure of a sparse large-kernel hybrid expert network proposed in the present invention;

[0058] Figure 8 is a schematic diagram of the structure of the TranMamba module proposed in the present invention; wherein, Figure 8 a is a schematic diagram of the TransMamba structure, Figure 8 b is a diagram of the processing flow of the mamba module. Figure 8 c is a schematic diagram of the TemporalMamba module processing flow;

[0059] Figure 9 This is a schematic diagram of a fuzzy mapping weighted 3D Transformer module provided by the present invention.

[0060] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0062] Example 1

[0063] See also Figure 1 , an embodiment of the present application provides a two-stage video deblurring system based on mixed motion states, comprising:

[0064] Data acquisition unit: collects motion-distorted videos of various motion states and uses them as datasets for training and testing a hybrid motion two-stage video deblurring model; wherein the hybrid motion two-stage video deblurring model includes a global motion deblurring unit and a local motion deblurring unit;

[0065] Two-stage task setup unit: By studying and analyzing blurred videos under mixed motion, a physical motion model driven by global and local motion is derived. Based on the end-to-end two-stage restoration scheme HMD-TVDF, the blurred video restoration task under mixed motion is decomposed into the global motion deblurring stage and the local motion deblurring stage.

[0066] Global Motion Deblurring Unit: This unit is used to perform the global motion deblurring phase, fusing features processed in the spatial domain and the Fourier frequency domain to obtain a video frame with global blur removed. It includes a hybrid expert module and a frequency domain energy attention module. The hybrid expert module extracts large edge features in the spatial domain and obtains a weighted fusion multi-expert output. The frequency domain energy attention module corrects abnormal fluctuations in global motion energy in the Fourier frequency domain.

[0067] Local motion deblurring unit: used to perform the local motion deblurring stage and ultimately output a clear video with local motion blur removed, including an optical flow estimation module and a hybrid module; the optical flow estimation module is used to predict motion in local areas between different frames and implement a coarse aggregation operation of target inter-frame features based on the pixel offset of the optical flow; the hybrid module is used to establish spatiotemporal context dependencies for local targets and repair the non-uniform blur of moving targets by capturing the pixel offset of inter-frame motion within a 3D window.

[0068] It should be noted that the present invention proposes a two-stage deblurring strategy, which divides the repair process of mixed motion blur into a global motion blur removal stage and a local target blur removal stage, and performs targeted modeling and processing on motion blur features of different scales respectively.

[0069] In the global motion blur removal stage, a fused hybrid expert network is constructed in the spatial domain to aggregate intra-frame features, and the frequency domain attention module is used to modulate the aligned spectrum in the Fourier frequency domain, effectively improving the quality of global motion blur removal and breaking through the performance bottleneck of traditional methods that only rely on spatial domain processing.

[0070] During the local object deblurring stage, optical flow estimation is used to predict local motion trends. The TransMamba module, which combines the spatiotemporal attention mechanism with Mamba, is introduced to achieve high-precision propagation and restoration of detailed features between frames. This module enhances the detail representation of object edge areas at the pixel level, alleviating the problem of insufficient edge sharpness in localized object images.

[0071] Experimental results show that the present invention exhibits superior deblurring effects in a variety of complex motion interference scenarios, and has good robustness and practical application value.

[0072] The data set includes a synthetic blurred video data set and a real blurred video data set; the data set includes a local target motion blurred video, a global motion blurred video, and a mixed motion blurred video in which global motion blur and target motion exist simultaneously.

[0073] As a further implementation, the physical motion model is set up by the following modeling formula:

[0074]

[0075] Where X represents the mixed motion blurred video generated by the mixed motion vector joint drive; δ is the local blur caused by the target motion; It is a blurred frame presentation after the entire frame is disturbed by global motion; Y represents the corresponding clear video of the mixed motion blurred video; ω represents the global motion vector of the overall movement of the lens; n is the number of video frames in the frame sequence; ɛ represents the driving vector of the local target motion; h 、 w Respectively represent the height and width of the video frame; Y(i,j) Represents the coordinates of the local target high-speed motion area; t is the time parameter.

[0076] like Figure 5 As shown, in the preliminary global motion blur removal stage at each scale, this embodiment designs a spatial domain-oriented hybrid expert network based on a sparse large kernel, including a weight distribution module and a hybrid expert network. The weight distribution module includes a 3D convolution module, a pooling module, a fully connected layer module, and a softmax module. The hybrid expert network consists of a chief expert module and N-1 sub-expert modules that calculate spatial features of some channel dimensions. Specifically:

[0077]

[0078] Among them, N is the number of all experts, and m is the depth of the U-shaped architecture at the current scale.

[0079] Assume the input blurred frame is of a sick patient. The N expert network modules represent a team of experts consisting of a chief general practitioner and N-1 specialists. Each specialist examines the corresponding parts of the body (eyes, hands, mouth, and nose) based on their expertise and issues a diagnosis. The general practitioner then provides a comprehensive diagnosis based on a full-body examination. Ultimately, the optimal solution is derived by weighted fusion of all the expert judgments.

[0080] like Figure 6 As shown in the figure, the weight distribution module performs maximum pooling and average pooling on the input frame sequence. After pooling, the feature sequence is compressed from a high-dimensional representation through a linear layer to obtain an N-1-dimensional weight tensor. The Softmax module is used to generate N-1 expert weights for different sub-expert modules.

[0081] Before the features are input to the sub-expert group, the full channel features are segmented, extracting N-1 independent channel features and distributing them to N-1 sub-experts. Each N-1 sub-expert independently captures a set of Sec[i] feature map representations with different biases based on its learned parameters. Because each sub-expert module only calculates 1 / N-1 of the original features, the computational resource consumption is only 1 / N-1 of the full feature computation. While the total resource consumption of all expert modules remains unchanged, the network's learning capacity is enhanced by (N / 2) times the original value. This efficient representation also ensures richer, clearer details within the frame.

[0082] Because the features calculated by each sub-expert module are subsets of the channel dimension, a chief expert module is designed in addition to the sub-expert modules to process all features. The chief expert module is designed to act as a constraint, expanding the feature capture capabilities of multiple experts while preventing the entire network's feature extraction from being completely biased towards partially independent channel features, which could suppress the expressions of other experts. The chief expert module ensures that the calculated global features are not compressed, avoiding data overfitting during the model network training and learning phases. While ensuring the network model's generalization ability to cope with different scenarios, it also plays a role in rapid convergence and suppressing divergent expressions.

[0083] Moreover, the hybrid expert strategy of the present invention distributes the data features assigned to the sub-expert module Sec evenly, and does not perform expert priority sorting. In contrast, the Top K strategy that only retains the k experts with the largest weights for calculations can easily lead to the permanent degradation of the sub-expert modules that are not selected because they have not learned the correct features. This degradation further suppresses the feature expression capabilities of the subsequent NK expert modules, which in turn will lead to a vicious cycle of reverse overfitting, thereby reducing the feature capture capabilities of the overall expert team. At the same time, in order to prevent the weight of the chief expert module from being replaced by the sub-expert group, we initialized a learnable weight coefficient to ensure that while maintaining the importance of the chief expert module, it avoids mutual suppression with the sub-expert group Sec and balances the relationship between global features and local features.

[0084] like Figure 7 As shown in the figure, each expert module, upon receiving a sequence of input video frames, feeds it into a group of sparse atrous convolutions with kernels of varying sizes. Sparse and dense kernels of varying sizes capture different perceptual features of motion. The expert module constructs a sparse sampling window with a large perceptual field of view by adjusting the stride and performing parallel 9×9, 5×5, and 3×3 dilated convolutions alongside traditional convolutions. Normalization ensures the stability of network learning with different kernel sampling. The sampled features from the large field of view are then modulated into global motion direction trend information through a 1×1 convolution. After adding the residual to ensure the result is constrained, the result is then dot-multiplied with the expert weights assigned by each expert, ultimately outputting the computational representation of a single expert.

[0085] Specifically, the hybrid expert network is carried out through the following steps:

[0086] In the weight generation and distribution stage, all channel features are passed into the routing network, such as Figure 6 As shown, average pooling and maximum pooling operations are implemented on the video frame embedding to condense the feature information; LeakyRelu Nonlinear activation function enhances the dynamic expression ability of different input frames; linear layer network is used to compress the full features into N-1 Dimension corresponds to the number of experts; then, Softmax calculation generates N-1 Expert weights are assigned and finally the weights are distributed;

[0087] In the calculation phase of the chief expert module, a weight lower limit L, a weight threshold upper limit H and a learnable adaptive parameter of the chief expert module are initialized. ∈[L,H]. The present invention initializes L=0.5 and H=0.8. The weight range of the chief expert module is adjusted by this learnable parameter to remain in the interval [L,H], ensuring that the chief expert module and the deputy expert module adaptively balance the weights while learning the global motion adjustment, so that all expert modules do not degenerate. Input the full channel features into the features learned by the Master network F master ∈ Contains all channels and is related to the weight parameter multiplication;

[0088] In the sub-expert group calculation stage, the original full-channel features are evenly divided into N-1 groups, enter them into their respective Sec [i] Calculating some features in expert networks F [i] ∈ .right Sec [i] Calculated local channel feature weights W [i] , weighted features F [i] • W [i] Then stack them in the original channel segmentation order to obtain multi-expert mixed features F sec , where i=1,2,3...N-1;

[0089]

[0090] In the hybrid expert aggregation stage, multi-expert sparse calculation can provide more global motion features. The global features generated by the chief expert moduleF master and deputy expert module groups F sec , weighted fusion;

[0091]

[0092] In the feature alignment stage, after obtaining all expert fusion features F' Then, use channel attention to strengthen F' feature channels to obtain stable feature expressions F spa .

[0093] like Figure 4 As shown, this embodiment designs a frequency domain energy attention module for global motion restoration in the Fourier frequency domain, including a convolution module, a Mamba module, and a channel attention module; specifically:

[0094] The digital signal of the image is a low-frequency representation for the smooth area of ​​the visual image, and the edge detail features are represented by high-frequency signals. The global motion causes an increase in the low-frequency signal of the global image. We found that the global motion appears in the Fourier frequency domain with visual violent energy fluctuations, while the local motion has weak fluctuations. This is because the Fourier transform calculates the digital frequency distribution of all pixel bits in the entire frame. Based on this, the frequency domain energy attention module can effectively find the motion increment of the global low-frequency signal from the 3D image space to the Fourier frequency domain, and reverse the abnormal increment of the callback frequency fluctuation by aligning the normal frequency of the time dimension, thereby enhancing the frequency feature distribution expression that cannot be captured in the image space. Its workflow is as follows: Figure 4 As shown:

[0095] First, the 3D spatial feature sequence F s Perform Fourier transform calculation and convert it into a 3D Fourier spectrum sequence F f , which is conducive to directly processing frequency domain information;

[0096] For each channel spectrum sequence F f Decompose and separate the Fourier energy amplitude Magnitude and phase. Stack them into frequency-amplitude tensors of the same size. F m and the phase tensor F p ,It helps to observe the frequency fluctuation generated by global distortion from the energy amplitude and phase respectively;

[0097] Parallel F m and F pFrequency modulation is performed through 1×1 3D convolution combined with activation function, and nonlinear modulation is converted into a learnable frequency feature expression. F' m和 F' p ;

[0098] The modulated features are first expanded from the time dimension and normalized before being input into the Mamba module to accelerate convergence, so that the adjacent frame spectra of the same 2D coordinate are aligned along the time dimension for autoregressive correction. Mamba outputs the corrected frequency and amplitude features F'' m and phase characteristics F'' p The frequency and phase of each channel are aggregated into a spectrum of channel characteristics. F' ;

[0099] Perform inverse Fourier transform on the spatial spectrum of all channels to generate refined features after frequency domain repair F' fs , aggregate the features processed by the hybrid expert network and the frequency domain restoration features to generate global clear features without the global motion driving vector F non-g .

[0100] Furthermore, the optical flow estimation module is set by the pre-trained RAFT model, and the hybrid module is set by the hybrid network TransMamba constructed by combining the Mamba module and the Transformer model, including the TemporalMamba module and the 3D SwinTransformer module.

[0101] like Figure 8 As shown in Figure 1, the present invention designs a hybrid network, TransMamba, that combines the selective spatial state model Mamba with an attention mechanism to process inter-frame spatiotemporal features. TransMamba, a hybrid network, comprises two modules for processing temporal features: the TemporalMamba module and the 3D SwinTransformer module. After alignment of video frame features using optical flow estimation, multiple redundant details are present within a single frame.

[0102] Figure 8 As shown in c, the TemporalMamba module uses two parallel mamba branches to bidirectionally propagate features. It normalizes the distribution of the compressed image and rearranges the pixel blocks using a 3D window of the same size in the 3D SwinTransformer module.

[0103] Figure 8 As shown in b, the selective representation process of the hidden state space by the SSM module in the mamba module is as follows:

[0104]

[0105] 3D window expands into one-dimensional input , hidden state , M is the size of the hidden state, 、 、 are all continuous parameters;

[0106] The SSM, which is equivalent to the global CNN effect, establishes the initial spatiotemporal context dependency of diffuse adjacent frames, then reorganizes the windowed tensors output by the mamba into a normal frame format, integrating the forward and backward target motion trends.

[0107] 3D SwinTransformer further sharpens the edge information of objects in the video frame based on the initial windowing dependency. Each scale contains 2 n 3D SwinTransformer blocks, n∈(1,..,3) represents the depth of the U-shaped network where the scale is located. Figure 9 As shown, each 3D SwinTransformer embeds a blur map after calculating the attention score using a multi-head attention mechanism. Compared to the spatial position embedding without blur map, the blur map can use the blur energy field generated by optical flow to simulate the numerical representation of regions with different blur levels. The relative position offset of the blur map guides attention to focus more on the blurred object area.

[0108] In the spatiotemporal self-attention calculation stage, the window features are flattened and normalized to compress the distribution expression to generate the attention query. Q ,key K ,value V . Calculate the attention score A 1 =Q @ K , attention score plus position embedding encoding , fuzzy mapping map And the window encoding of the shift window b , get the updated attention score representation A 2 =A 1 +P b +map+b The attention calculation formula is as follows:

[0109]

[0110] In the spatiotemporal mutual attention stage, the data is divided into two groups of equal size along the time dimension. Q (i) K (i) V(i) {i∈(1, 2)} ,Dual frame exchange attention query Q ,

[0111] A (1) =softmax(Q (2) @ K (1) )V (1)

[0112] A (2) =softmax(Q (1) @ K (2) )V (2)

[0113] The output attention results are constrained by residual connections and discriminated by feedforward networks to obtain video frames with clear local motion areas.

[0114] Loss function:

[0115] We design a loss function for global motion frequency fluctuations in the Fourier frequency domain, performing loss perception from the phase and amplitude perspectives respectively.

[0116]

[0117] Blurmap:

[0118] The forward and backward optical flows generated by the RAFT network are normalized, and the high-speed motion areas are weighted and segmented (the higher the speed, the more likely it is to be blurred). The vector directionality of the optical flow prediction is removed, and the offset value is retained.

[0119] Map =∑ (O (t,t+1) ) 2 +(O (t,t-1) ) 2

[0120] Example 2

[0121] See also Figure 2 , an embodiment of the present application also provides a two-stage video deblurring method based on mixed motion states, which is applied to a two-stage video deblurring system based on mixed motion states as any of the above items, when executing the end-to-end two-stage recovery scheme HMD-TVDF: input the blurred video frame and use the 3D convolution module to embed the channel dimension and map it to the high-dimensional space, perform preliminary feature extraction in the high-dimensional space, use the channel attention module to stabilize the feature channel after dimensionality increase of the preliminary extracted features, and obtain the embedded video frame after dimensionality increase; wherein, the embedded video frame after dimensionality increase will be input into a U-shaped network with three different resolution scales, and the global motion deblurring stage and the local motion deblurring stage will be performed for each resolution scale respectively.

[0122] Through motion vector decomposition, the algorithm is divided into two progressive stages: global motion deblurring and local motion deblurring. The algorithm extracts global features in the spatial domain, frequency motion fluctuations in the Fourier frequency domain, and establishes spatiotemporal contextual relationships, effectively capturing blurred region information at different scales.

[0123] For the input of N consecutive blurred frame sequences, 3D convolution is used to embed the channel dimension into a high-dimensional space, and the features are preliminarily extracted in the high-dimensional space. The channel attention block is used to stabilize the feature channels after dimensionality increase.

[0124] The video frame after embedding is fed into a U-shaped network with three different resolution scales, each of which includes a global motion deblurring stage. G and local motion deblurring stage L ; From top to bottom it is represented as ( G 1， L 1) 、 ( G 2 ，L 2) 、 ( G 3 ，L 3), the number of channels and resolution of each scale are set to (48,256*256), (96,128*128), (192,64*64);

[0125] The input of each scale is first passed to the global deblurring stage G The global motion deblurring stage fuses frequency domain and spatial domain features to output a globally clear and locally blurred video frame. F non-g ;

[0126] Global clear frame F non-g Two branches are input respectively, one is used as a skip connection at the same scale, and the other branch is a global deblurring stage that downsamples the input to a smaller resolution scale through convolution operation;

[0127] Minimum resolution scale G The global stage output of 3 is fed into two branches, one branch serves as the local motion stage at the same scale L 3 input, the other branch is stabilized by convolution upsampling and channel attention, and the global G 2. The skip connection of the output is used for feature fusion, and the fused features execute the same process as above.

[0128] Local deblurring stage LThe features between frames are bidirectionally propagated and pixel-level details are enhanced to obtain sharpened video frames with clear moving target areas. F non-l ;

[0129] Finally, the decoder fuses the local stage outputs at three scales and compresses the channel dimension to generate video restoration frames with global and local motion blur removed.

[0130] Global motion deblurring stage:

[0131] To restore global motion blur in the spatial domain, the large-kernel hybrid expert module of our invention achieves a broader perceptual field of view, ensuring that the model can capture large, global blur features and contextual information, and determine the direction of feature motion across the video as it shifts with camera offset. The hybrid channel features extracted by the hybrid expert network are further weighted using channel attention, stabilizing the channel representations of the global blur region across different experts.

[0132] Multi-expert feature capture eliminates the redundancy of single features and provides more judgments on the global motion offset of the camera.

[0133] Because global motion causes spectral fluctuations in the Fourier frequency domain, the video sequence is converted to the Fourier frequency domain using a discrete fast Fourier transform (DFT) to sense and align the motion energy in the frequency domain. The converted video information is separated into frequency amplitude energy and phase information, and the frequency energy and phase information are spectrally modulated, transforming them into recognizable signals that continuously fluctuate in the time domain. Finally, the time-frequency aligned amplitude energy and phase are aggregated into a new Fourier spectrum sequence. This Fourier spectrum sequence is then converted into a spatial mapping using an inverse Fourier transform.

[0134] like Figure 3 As shown in Figure 3, the outputs of the parallel branches consisting of the hybrid expert and frequency domain attention are fused to obtain a globally clear video frame sequence with global motion blur removed.

[0135] Local motion deblurring stage:

[0136] The prior map of optical flow estimation in locally blurred scenes is very accurate. However, the accuracy of optical flow estimation decreases significantly under the interference of global blur. Therefore, the optical flow alignment part is prioritized in the local motion deblurring stage to remove global blur.

[0137] First, the output of the global stage is passed into the pre-trained RAFT optical flow model to calculate the pixel-level displacement of the relative local motion between objects between adjacent video frames, obtaining a preliminary optical flow estimation result. The inter-frame information is aligned using temporal bidirectional optical flow estimation, and the frame sequences before and after alignment are stacked to reduce feature misalignment caused by parallax and motion, while enriching the features within a single frame at the previous and next moments.

[0138] Subsequently, an interleaved temporal dimension processing strategy is adopted to perform temporal bidirectional scanning on the frame sequence information using multiple Mamba blocks to perceive the preliminary spatiotemporal offset trend of local motion pixels.

[0139] Based on the preliminary modeling of spatiotemporal dependencies, the 3D SwinTransformer module combined with the spatiotemporal attention mechanism further refines the dynamic features of local motion areas by strengthening attention to the contextual relationships between adjacent frames.

[0140] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A two-stage video deblurring system based on mixed motion states, characterized in that: include: Data acquisition unit: collects motion-distorted videos of various motion states and uses them as datasets for training and testing a hybrid motion two-stage video deblurring model; wherein the hybrid motion two-stage video deblurring model includes a global motion deblurring unit and a local motion deblurring unit; Two-stage task setup unit: By studying and analyzing blurred videos under mixed motion, a physical motion model driven by global and local motion is derived. Based on the end-to-end two-stage restoration scheme HMD-TVDF, the blurred video restoration task under mixed motion is decomposed into the global motion deblurring stage and the local motion deblurring stage. Global motion deblurring unit: used to perform the global motion deblurring stage, fuse the features processed in the spatial domain and the Fourier frequency domain, and obtain a video frame with global blur removed; it includes a hybrid expert module and a frequency domain energy attention module; wherein the hybrid expert module is used to extract large edge features in the spatial domain and obtain a weighted fusion multi-expert output; the frequency domain energy attention module is used to correct abnormal fluctuations in global motion energy in the Fourier frequency domain; Local motion deblurring unit: used to perform the local motion deblurring stage and ultimately output a clear video with local motion blur removed, including an optical flow estimation module and a hybrid module; wherein the optical flow estimation module is used to predict the motion of local areas between different frames and implement a coarse aggregation operation of target inter-frame features based on the pixel offset of the optical flow; the hybrid module is used to establish spatiotemporal context dependencies for local targets and repair the non-uniform blur of the moving target by capturing the pixel offset of the inter-frame motion within the 3D window.

2. The two-stage video deblurring system based on mixed motion states according to claim 1, characterized in that: The dataset includes a training dataset and a test dataset; wherein, the dataset includes a synthetic blurred video dataset and a real blurred video dataset; wherein, the dataset includes a local target motion blurred video, a global motion blurred video, and a mixed motion blurred video in which global motion blur and target motion exist simultaneously.

3. The two-stage video deblurring system based on mixed motion states according to claim 2, characterized in that: The physical motion model is set by the following modeling formula: Where X represents the mixed motion blurred video generated by the mixed motion vector joint drive; δ is the local blur caused by the target motion; It is a blurred frame presentation after the entire frame is disturbed by global motion; Y represents the corresponding clear video of the mixed motion blurred video; ω represents the global motion vector of the overall movement of the lens; n is the number of video frames in the frame sequence; ɛ represents the driving vector of the local target motion; h 、 w Respectively represent the height and width of the video frame; Y(i,j) Represents the coordinates of the local target high-speed motion area; t is the time parameter.

4. The two-stage video deblurring system based on mixed motion states according to claim 1, characterized in that: The hybrid expert module is a large-core hybrid expert module, including a weight distribution module and a hybrid expert network; The hybrid expert network includes a chief expert module and several deputy expert modules, and the weight distribution module includes a 3D convolution module, a pooling module, a fully connected layer module and a Softmax module; The frequency domain energy attention module includes a convolution module, a Mamba module and a channel attention module.

5. The two-stage video deblurring system based on mixed motion states according to claim 1, characterized in that: The optical flow estimation module is set by a pre-trained RAFT model, and the hybrid module is set by a hybrid network TransMamba constructed by combining a Mamba module and a Transformer model, including a TemporalMamba module and a 3DSwinTransformer module.

6. A two-stage video deblurring method based on mixed motion states, applied to a two-stage video deblurring system based on mixed motion states as claimed in any one of claims 1 to 5, characterized in that: When executing the end-to-end two-stage restoration scheme HMD-TVDF: input blurred video frames and use the 3D convolution module to embed and map the channel dimensions into a high-dimensional space, perform preliminary feature extraction in the high-dimensional space, and use the channel attention module to stabilize the feature channels after dimensionality increase of the preliminarily extracted features to obtain the embedded video frames after dimensionality increase; wherein, the embedded video frames after dimensionality increase will be input into a U-shaped network with three different resolution scales, and the global motion deblurring stage and the local motion deblurring stage will be performed for each resolution scale respectively.

7. The two-stage video deblurring method based on mixed motion states according to claim 6, characterized in that: When the U-network performs the global motion deblurring stage: The input of each resolution scale is first passed to the global deblurring stage, which fuses the frequency domain and spatial domain features to output a globally clear frame with locally blurred frames. F non-g ; Global clear frame F non-g Two branches are input respectively, one branch is used as a skip connection at the same scale, and the other branch is a global deblurring stage that downsamples the input to the next resolution scale through convolution operation; The global stage output at the minimum resolution scale is passed into two branches. One branch serves as the input of the local motion stage at the same scale, and the other branch performs feature fusion with the skip connection of the global output of the second scale under convolution upsampling and channel attention stabilization. The fused features execute the same process as above.

8. The two-stage video deblurring method based on mixed motion states according to claim 7, characterized in that: When the U-network performs the local deblurring stage: The local deblurring stage performs bidirectional propagation of inter-frame features and pixel-level detail enhancement to obtain a sharpened video frame with a clear moving target area. F non-l ; The decoder fuses the outputs of the local deblurring stages at three different resolution scales and compresses the channel dimension to generate a video restoration frame without global motion blur and local motion blur.

9. The two-stage video deblurring method based on mixed motion states according to claim 6, characterized in that: The global motion deblurring unit performs the global deblurring stage by following the steps below: For global motion restoration in the spatial domain, 3D convolution modules, pooling modules, fully connected layer modules, and Softmax modules are used to dynamically calculate different sub-expert weights. The blurred video frame is evenly divided into N-1 groups of partial features along the channel dimension. Each group of features is input into the sub-expert module of the corresponding channel group. Each sub-expert module is reorganized along the channel by the sub-expert weight assigned by the dot product to obtain the sub-expert group feature F sec The chief expert module inputs the unsegmented fuzzy frame and calculates the chief discriminant output F master ; Use a learnable parameter as a coefficient to adjust the balance between the chief expert module and each deputy expert module, and weightedly integrate the deputy expert group features F sec and the chief discriminant output F master ; The channel attention module is used to strengthen and stabilize the channel dimension of the fusion features of all experts; the multi-expert feature capture design eliminates the dependence on a single feature and generates a clear feature F in the spatial domain. spa ; To repair global motion blur in the frequency domain, a spectrum is generated through Fourier transform, and the amplitude spectrum and phase spectrum of the blurred frame are gradually decomposed along each channel. The 1x1 convolution is used to modulate the spectrum signal of the amplitude spectrum and phase spectrum, and transform it into a recognizable signal that changes with continuous fluctuations in the time domain; Then, layer normalization is used to compress the spectrum signal, and the amplitude spectrum and phase spectrum are rearranged and expanded in one dimension along the time-frequency direction. The correct frequency distribution between the spectra is calculated based on the mamba module. After correcting the frequency distribution and adding residual connection, the phase spectrum and amplitude spectrum are fused channel by channel to reconstruct the Fourier spectrum, and the Fourier spectrum with accurate frequency distribution is inversely Fourier transformed to obtain the spectrum enhanced spatial domain mapping F fre , using channel attention to stabilize feature mapping along the channel dimension; Based on feedforward network and normalization fusion spatial domain clear feature F spa and spatial domain mapping F fre , and obtain a clear frame with global motion blur removed.

10. The two-stage video deblurring method based on mixed motion states according to claim 9, characterized in that: The local motion deblurring unit is performed by the following steps: The clear frames with global motion blur removed are input into the pre-trained RAFT model to calculate the pixel-level displacement of the relative local motion between objects between adjacent video frames, obtaining preliminary optical flow estimation results and enriching the intra-frame features of the previous and next moments. After optical flow alignment, the image values ​​are compressed through layer normalization, and bidirectional window rearrangement is performed according to the time sequence. The TemporalMamba module establishes spatiotemporal context dependencies through bidirectional feature propagation and uses a feedforward network to fuse bidirectional features. The fused features are passed to the 3D SwinTransformer module using a 3D shift window. The pixel blocks within the 3D shift window are divided into query Q, key K and value V. Multi-head self-attention calculation is performed by embedding the position code and the shift mask of the 3D window. The output attention results are constrained by residual connections and weighted by the discriminative features of the feedforward network to obtain video frames with clear local motion areas.

Images