Multi-small-block progressive video rain removal method based on event camera motion perception estimation

Abstract

The invention provides a multi-small-block progressive video rain removal method based on event camera motion perception estimation, and belongs to the field of computer vision and image processing. A multi-small-block multi-level structure is adopted, event data and video frame data are subjected to feature extraction after being partitioned, then the event data and the video frame data are input into an event sensing motion module to be processed, and finally the event data and the video frame data are input into a cross-modal fusion module to be subjected to cross-modal fusion, so that complementary information is mined from the event data and the frame data, rain information is eliminated, and background information is enhanced. According to the method, the event camera is introduced into a traditional video rain removal method, and the advantages of high tense resolution and high dynamic range of the event camera are utilized, so that complex rain time-space characteristics can be simulated, and a rain layer and a background layer can be distinguished more easily.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and relates to a multi-patch progressive video de-raining method based on event camera motion perception estimation. Background Art

[0002] Outdoor cameras often face some harsh weather conditions, such as rainy days. Rain not only reduces the visual quality of captured images and videos, but also affects the performance of downstream multimedia tasks that rely on clear video frames, such as object tracking, person re-identification, semantic segmentation, and object detection. Under the current urgent demand for rain-free videos, it is crucial to study an effective video de-raining algorithm.

[0003] Up to now, many video de-raining methods have been proposed and achieved many results. The video de-raining algorithm was first proposed by Garg and Nayar [1] and is an extension of single-image de-raining. They solved this problem by separating raindrops and non-rain areas based on raindrop dynamics and photometric analysis. Barnum et al. [2] systematically analyzed the spatio-temporal characteristics of rainy-day videos and constructed a statistical model in the frequency space to detect rain areas. Chen [3] proposed a low-rank model to encode the spatio-temporal characteristics of raindrops in videos. Chen [4] regarded rain detection as a problem of pixel-level motion segmentation, taking into account the photometric and color constraints of rain. Jiang et al. [5] considered all known discriminative characteristics of rain and formulated them into four loss terms. However, due to the complex background situations and drastic motions in actual rainy-day videos, this may lead to errors in the sparsity and dynamic assumptions of raindrops and reduce the robustness of these model-based methods.

[0004] In recent years, with the rapid development of deep learning, some deep learning-based methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have been proven to be extremely effective in video deraining. Chen et al. [6] used superpixel segmentation technology to divide frames into meaningful regions, and then transmitted the segmented features into a CNN to obtain enhanced results. And in [7], the rich temporal redundancy in videos was analyzed, and a new rainy-day image model was established to illustrate the occlusion effect brought by the rain layer. Based on the proposed new model, Chen used a deep recurrent neural network to jointly complete the deraining and reconstruction tasks. The designed network (J4R-Net) has integrated the classification of rain degradation, deraining based on spatial texture knowledge, and background restoration based on temporal correlation. To process the dynamically detected video context, a parallel technical route was selected, and a dynamic routing network (D3R-Net) [8] was further developed. Inspired by the deep learning-based dynamic rainfall generator, S2VD (semi-supervised video deraining) [9], a semi-supervised framework consisting of a deraining CNN and a rain generation CNN, was developed to bridge the gap between real rainfall videos and synthetic rainfall. Yang et al.

[10] constructed a two-stage recurrent network that first captures motion information and then maintains the motion consistency between frames to remove rain. Zhang et al.

[11] proposed a novel video deraining framework named ESTINet, which can extract more powerful spatio-temporal information from consecutive frames to achieve high-quality and high-speed video deraining. This framework can not only learn spatial and temporal cues, but also introduces a novel convolutional bidirectional long short-term memory network.

[0005] Although the above neural network-based methods can achieve considerable deraining performance, they always focus on complex architectures and have difficulties in distinguishing raindrops of different shapes and sizes, modeling raindrop characteristics and raindrop dynamic information. The present invention adopts an event camera to help video deraining provide more useful rain streak motion information.

[0006] An event camera is a new type of neuromorphic camera, such as DVS (Dynamic Vision Sensor)

[12] , ATIS (Asynchronous Time-based Image Sensor)

[13] , and DAVIS (Dynamic and Active Pixel Vision Sensor) [14, 15].

[0007] The Chinese invention patent with the application number 202011252900.7 has applied for a raindrop detection method based on an event camera. Without prior information and ground truth images, it can utilize the real data collected outdoors in rainy and snowy weather to complete raindrop recognition well. However, this patent application only relies on the data collected by the event camera for raindrop recognition and does not fully utilize the possibility of multi-modal information fusion of other sensors (such as traditional optical cameras, radars, etc.), resulting in a decline in detection performance in some specific environments.

[0008] The Chinese invention patent with the application number 202011252900.7 has applied for a method and system for eliminating background noise based on an event camera. It collects event stream data containing only moving objects and event stream data containing only background noise for combined processing, overcoming the problem of difficult sample collection in actual situations. Through a self-supervised method, it uses existing data to generate labeled sample data to solve the problem of difficult collection of labeled samples in supervised learning. However, the labeled samples generated by this patent application using self-supervised learning may not be able to fully simulate the complex situations in the real world. The labeled samples self-supervised generated based on a specific dataset may not be able to generalize well to new real scenarios, resulting in poor background noise elimination effect.

[0009] The Chinese invention patent with the application number 202011252900.7 has applied for a video de-raining method, system, device, and storage medium. It introduces an event camera into the video de-raining task, uses the event camera to capture fast-moving rain lines, effectively overcoming the drawback that traditional RGB cameras with a fixed frame rate cannot capture the movement of rain lines. It effectively fuses video frame data and event data using a neural network, enabling explicit modeling of the spatio-temporal dynamic distribution of rain lines, and finally accurately and effectively separating the rain layer and the clean background layer from a rainy video. However, this patent application needs to better solve the problem of efficient fusion of video frame data and event data by the neural network to improve the robustness and accuracy of the algorithm in complex scenarios.

[0010] In summary, the existing video de-raining methods cannot fully distinguish raindrops of different shapes and sizes, nor can they fully model raindrop characteristics, resulting in incomplete raindrop removal and blurred background edges. The present invention proposes a multi-patch progressive video de-raining method based on event camera motion perception estimation, which divides the input data into different numbers of patch structures to achieve a rich receptive field, and performs cross-modal information fusion on the temporal information of frame features and event features to provide richer environmental information, helping to improve the accuracy and robustness of raindrop recognition.

[0011] Related literature:

[0012] [1]. Garg K, Nayar S K. Detection and removal of rain from videos[C] / / Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. IEEE, 2004, 1: I-I.

[0013] [2]. Barnum P, Kanade T, Narasimhan S G. Spatio-temporal frequency analysis for removing rain and snow from videos[C] / / Proceedings of the First International Workshop on Photometric Analysis For Computer Vision - PACV 2007. INRIA, 2007: 8p.

[0014] [3]. Chen Y L, Hsu C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks[C] / / Proceedings of the IEEE international conference on computer vision. 2013: 1968 - 1975.

[0015] [4]. Chen J, Chau L P. A rain pixel recovery algorithm for videos with highly dynamic scenes[J]. IEEE transactions on image processing, 2013, 23(3): 1097 - 1104.

[0016] [5].Jiang T X,Huang T Z,Zhao X L,et al.Fastderain:Anovel video rainstreak removal method using directional gradient priors[J].IEEE Transactionson Image Processing,2018,28(4):2089-2102.

[0017] [6].Chen J,Tan C H,Hou J,et al.Robust video content alignment andcompensation for rain removal in a cnn framework[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2018:6286-6295.

[0018] [7].Liu J,Yang W,Yang S,et al.Erase or fill?deep joint recurrent rainremoval and reconstruction in videos[C] / / Proceedings of the IEEE conferenceon computer vision and pattern recognition.2018:3233-3242.

[0019] [8].Liu J,Yang W,Yang S,et al.D3r-net:Dynamicrouting residuerecurrent network for video rain removal[J].IEEE Transactions on ImageProcessing,2018,28(2):699-712.

[0020] [9].Yue Z,Xie J,Zhao Q,et al.Semi-supervised video deraining withdynamical rain generator[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2021:642-652.

[0021]

[10] .Yang W,Liu J,Feng J.Frame-consistent recurrent video derainingwith dual-level flow[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2019:1661-1670.

[0022]

[11] .Zhang K,Li D,Luo W,et al.Enhanced spatio-temporal interactionlearning for video deraining:faster and better[J].IEEE Transactions onPattern Analysis and Machine Intelligence,2022,45(1):1287-1293.

[0023]

[12] .Lichtsteiner P,Posch C,Delbruck T.A128×128 120dB 15μs latencyasynchronous temporal contrast vision sensor[J].IEEE journal of solid-statecircuits,2008,43(2):566-576.

[0024]

[13] .Posch C,Matolin D,Wohlgenannt R.AQVGA 143dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[J].IEEE Journal of Solid-State Circuits,2010,46(1):259-275.

[0025]

[14] .Brandli C,Berner R,Yang M,et al.A240×180 130db 3μs latency global shutter spatiotemporal vision sensor[J].IEEE Journal of Solid-State Circuits,2014,49(10):2333-2341.

[0026]

[15] .Hui T W,Loy C C.Liteflownet3:Resolving correspondence ambiguity for more accurate optical flow estimation[C] / / Computer Vision–ECCV 2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XX 16.Springer International Publishing,2020:169-184. Summary of the Invention

[0027] In view of the problems existing in the existing video de-raining algorithms, such as the inability to achieve satisfactory de-raining effects in scenarios with rain layers having complex spatio-temporal distributions, the inability to distinguish various different patterns of rain streaks, and the large time consumption and inaccuracy of optical flow alignment, the present invention incorporates an event camera into the video de-raining network, utilizes the advantages of the high dynamic range and low latency of the event camera to provide more accurate raindrop motion information for frame data. At the same time, the present invention adopts a multi-block recursive progressive learning structure, divides the data stream into blocks, thereby expanding the receptive field, and helps the network learn rain streaks of different shapes and sizes. The proposed event perception module and cross-modal fusion module can respectively help the frame data identify the raindrop area and play the roles of enhancing background information and removing raindrops, so as to make full use of the motion information contained in the event camera and achieve better de-raining performance.

[0028] The present invention proposes a multi-small-block structure progressive video de-raining method based on event camera motion perception estimation.

[0029] Event cameras can asynchronously capture intensity changes at the pixel level and microsecond level, and have the advantages of high dynamic range and low latency compared with traditional cameras. They can be used in visual tasks such as stereo vision, pose estimation, video de-blurring, visual tracking, video interpolation, optical flow estimation, image reconstruction, HDR reconstruction, etc. These tasks mainly utilize the high temporal resolution of event data to supplement the information missing from traditional cameras. The present invention uses event data as the input of the video de-raining network, and provides an enhancement and supplement of raindrop motion information for frame data in the event perception motion module. And it plays the roles of raindrop removal and background information enhancement in the subsequent cross-modal fusion module.

[0030] The present invention proposes a multi-small-block progressive video de-raining method based on event camera motion perception estimation, including the following steps:

[0031] Step 1: Obtain a sequence composed of N events captured by the event camera, denoted as and the frame sequence in the same time period collected by the frame camera

[0032] Adopt a multi-small-block multi-level structure. In each level, first divide the frame sequence with a length of T and respectively into blocks to obtain the block-divided event data and video frame data.

[0033] The frame data respectively pass through the feature extraction module to extract the event feature F e and the frame feature F f ; The feature extraction module includes an event feature extraction module and a video feature extraction module;

[0034] Step 2: The extracted event feature F e and the frame feature Ff It is input into the event-aware motion module for the following processing: to extract motion information to better generate the modeling of the rain layer, and to correct the frame features and event features.

[0035] First, the event feature F e is respectively convolved through 3×3, 5×5, and 7×7 to expand the receptive field of the event feature, and all are processed through 1×1 convolution and concatenated in the channel dimension, and then processed through 3×3 convolution and the Sigmoid activation function to obtain the motion perception mask M;

[0036] The motion perception mask M and the frame feature F f are multiplied element by element and then processed by a 3×3 convolutional layer to obtain the corrected frame feature F m ;

[0037] The corrected frame feature F is processed by a three-dimensional convolutional layer m to generate a motion-enhanced frame feature The event feature F is processed by a three-dimensional convolutional layer e to generate a motion-enhanced event feature

[0038] Step 3: The frame feature and the event feature are jointly input into the cross-modal fusion module for cross-modal fusion to mine complementary information from the event and frame data to eliminate rain information and enhance background information.

[0039] Step 4: The multi-small-block multi-level structure is used to progressively and recursively execute Steps 1 to 3 to expand the range of the receptive field and pay attention to raindrop streaks of various sizes and patterns.

[0040] Preferably, Step 3 specifically includes the following steps:

[0041] First, both the frame feature and the event feature are processed by the encoder for better fusion. The event feature is extracted through the rain edge attention module composed of four groups of channel-spatial attention to obtain the rain feature in the event domain. Then, the event feature is used to subtract the rain feature element by element to obtain the background feature

[0042] Subsequently, the background feature and the frame feature are added element by element and then passed through a 1×1 convolution to obtain the fused frame feature To enhance the background details in the frame field and eliminate edge blurring and rain streaks. Input them into the residual block and the convolutional block in sequence.

[0043] Preferably, in step one, each of the events is denoted as e i =(x i , y i , t i , p i );

[0044] where: x i ∈[1, W] and y i ∈[1, H] are the spatial indices of the pixels, W and H respectively represent the width and height of the event data, t i is the time position of the event, and the polarity p i ∈{-1, +1} indicates the direction of the intensity change.

[0045] Preferably, in step four, the multi-small-block multi-level structure is used to progressively and recursively execute steps one to three, which specifically includes the following steps:

[0046] Represent the frame sequence as Its true video sequence is represented as

[0047] Divide the frame sequence into P small blocks, denoted as I P , where the value of P is adjusted through experiments to select the value with the best performance.

[0048] Denote the process of processing using the event-aware motion module in step two as g.

[0049] Denote the cross-modal fusion process and the subsequent residual block and convolutional block processing in step three as h.

[0050] Then the data stream at a single level is represented as:

[0051]

[0052] where f L represents the above entire process. and are respectively the input and intermediate output at the l-th level, and P l is the number of small blocks. Then at the (l + 1)-th level to the last level, the concatenation of and the original rain sequence is used as the input for recursion.

[0053] The recursive process is as follows:

[0054]

[0055] where is the input of the de-rained video frame at the level above the (l + 1)-th level, is the input of the original video frame at the (l + 1)-th level, is the total input at the (l + 1)-th level, and C[·] is the concatenation operation along the channel dimension.

[0056] However, the above f L is only allowed once at each level, which means that the network pays equal attention to patches of various sizes. However, intuitively, larger patches require more revisions and learning iterations. Therefore, in this method, the network is reconstructed into a progressive structure at each level. At the l-th level, f L is iterated multiple times before being merged into the larger patches at the next level, denoted as:

[0057]

[0058] where: r l is the number of loop iterations for the patches at the l-th level. Finally, the overall progressive recursive framework is as follows:

[0059]

[0060] where: represents the de-rained output at the (l + 1)-th level, and represent the total network input at the l-th level and the total network input at the (l + 1)-th level, respectively.

[0061] Preferably, the multi-level structure is specifically a three-level sub-patch structure. At each level, a frame sequence of length T and are divided into P sub-patches;

[0062] At the first level, the number of sub-patches P is set to 4. The frame sequence is divided into 2 sub-patches, is divided into 2 sub-patches, where one is a vertical split and one is a horizontal split;

[0063] At the second level, the number of sub-patches P is set to 2. The frame sequences and are divided into 2 sub-patches, and the frame sequences and each become 1 sub-patch;

[0064] At the third level, the number of sub-patches P is set to 1. The frame sequences and are regarded as a whole as one sub-patch.

[0065] To solve the problem that current video de-raining methods cannot distinguish rain streaks of various shapes and sizes, the present invention divides the input data into small block structures of different numbers. Multiple block structures achieve receptive fields of different sizes through partitioned tiles, and thus can learn rain streaks of various shapes and sizes. At the same time, the number of progressive recursive learning times at different block levels is also different. The more blocks are divided, the fewer times they enter the network for learning, and vice versa. The data stream after being divided into blocks will be input into the event-aware motion module simultaneously. This module can obtain the motion information of rain streaks through further extraction of event data, and then use this motion information to emphasize the rain regions in the frame features, helping to remove rain streaks more effectively. This module will also strengthen the frame features and event features in terms of temporal information. The strengthened frame and event features will be input into the cross-modal fusion module. This module will separate the rain features in the event domain, obtain the background features in the event domain, and fuse the background features with the features in the frame domain to strengthen the background information and remove rain streaks and edge blurring. Finally, the fused features will pass through residual blocks and convolutional blocks to obtain the intermediate de-rained video frames. The above process will be carried out according to block-level recursion and finally obtain the final de-rained video frames. The de-raining method of the present invention is a segment-to-segment de-raining network, and finally obtains a progressive video de-raining network process through recursive learning at different block levels.

[0066] The advantages of the present invention are as follows: The present invention provides a multi-small-block progressive video de-raining method for motion perception estimation using an event camera and combined with a traditional frame camera. By performing small-block processing on the input frame data stream and event data stream, the present invention can be adapted to various situations and rain streaks of various sizes. The addition of event data and its estimation of the motion area also better helps the frame camera to separate the background from the rain streaks, enabling excellent de-raining performance while effectively retaining edge and background details. The present invention can make the de-rained video retain more background detail information, avoid blurring, and be applicable to large motion scenes and some situations with fine rain streaks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flowchart of the small-block levels of two and one in the multi-small-block progressive video de-raining network based on event camera motion perception estimation;

[0068] Figure 2 is the structure of the event-aware motion module;

[0069] Figure 3 is the structure of the cross-modal fusion module;

[0070] Figure 4 In (a) is a visualization diagram of adjacent event data;

[0071] Figure 4In (b) is the visualization diagram of the event data by the present invention;

[0072] Figure 5 In (a) is a video frame with rain streaks in the synthetic dataset NTURain;

[0073] Figure 5 In (b) is the rain removal result of the present invention for this video frame;

[0074] Figure 5 In (c) is the corresponding real video frame without rain streaks;

[0075] Figure 6 In (a) is a video frame with heavy rain in the synthetic dataset RainSynHeavy25;

[0076] Figure 6 In (b) is the rain removal result of the present invention for this video frame;

[0077] Figure 6 In (c) is the corresponding real video frame without rain streaks;

[0078] Figure 7 In (a) is a video frame with light rain in the synthetic dataset RainSynLight25;

[0079] Figure 7 In (b) is the rain removal result of the present invention for this video frame;

[0080] Figure 7 In (c) is the corresponding real video frame without rain streaks;

[0081] Figure 8 In (a) is a video frame in a real video with rain streaks;

[0082] Figure 8 In (b) is the rain removal result of the present invention for the real rain streak video frame; Specific embodiments

[0083] A method for motion perception estimation of rain streaks based on an event camera, combining with a traditional frame camera, and finally performing progressive video rain removal for multiple small structures is as follows:

[0084] Step 1: Processing of the input data of the event camera and the input video frames of the traditional frame camera.

[0085] Compared with traditional cameras, the sensor of the event camera consists of a W×H pixel matrix. When the intensity change exceeds the threshold of the inherent intensity change ΔL, these pixels will respond independently and asynchronously. Assuming a total of N events are captured, they are stored and processed in a stream, expressed as Each event can be represented as e i =(x i ,y i ,t i ,p i ).

[0086] Where: W, H represent the width and height of the event data respectively, ΔL is a set constant, x i ∈[1,W] and y i ∈[1,H] is the spatial index of the pixel, t i is the time position of the event. Polarity p i ∈{-1,+1} represents the direction of intensity change (p i = 0 means no event is captured). In order to solve the difficulty of existing video deraining methods in removing some tiny raindrops, this method adopts a multi-block cross-level structure to better expand the receptive field. This method adopts a multi-level structure. In each level, a frame sequence of length T and sequence of events It is divided into P small blocks before being input into the proposed network.

[0087] Where: N represents the number of events captured. Then the continuous video frames will be spliced ​​into a whole, and the event data and video frame data will be simultaneously passed through a feature extraction module to extract high-level event features F e and frame feature F f .

[0088] F f =FeatureExtractor(I i )

[0089] F e =FeatureExtractor(e i )

[0090] Step 2: Input the extracted event features and frame features into the event-aware motion module to estimate and extract motion information, so as to better generate modeling for the rain layer and correct the frame features and event features.

[0091] Video deraining can utilize the additional temporal information between adjacent frames for deraining. However, directly inputting multi-frame features into the network is not necessarily effective. Instead, it will increase the burden of the deraining task by introducing too much redundant information. Motion information can usually be used as a clue that is beneficial to modeling the dynamic generation process of the rain layer and can effectively identify rainy areas. This method uses the characteristics of event cameras that can accurately detect motion changes in large motion and low-light scenes to selectively extract adjacent motion-aware information.

[0092] First, the event feature F e is respectively convolved by 3×3, 5×5, and 7×7 to expand the receptive field of the event feature, and all pass through the processing of 1×1 convolution and are concatenated in the channel dimension. Subsequently, it undergoes the processing of 3×3 convolution and the Sigmoid activation function, restricting the output to between (0, 1), and finally obtaining the motion perception information mask.

[0093]

[0094] Where: M is the motion mask, σ is the Sigmoid activation function, φ 1×1 and φ 3×3 and φ 5×5 and φ 7×7 respectively represent convolutional layers with convolutional kernel sizes of 1×1, 3×3, 5×5, and 7×7, is the intermediate feature result before Sigmoid processing. Subsequently, the predicted motion perception mask M is used to correct the frame feature.

[0095]

[0096] Where: F m is the frame feature corrected by the motion perception mask, is element-wise multiplication. Subsequently, a three-dimensional convolutional layer is used to aggregate the temporal information contained in the multi-frame context and further enhance the motion perception information within the frame domain, generating motion-enhanced frame features. Similarly, in the event domain, a three-dimensional convolutional layer is also used to aggregate event features, generating motion-enhanced event features.

[0097]

[0098] Where: are respectively the motion-enhanced frame feature and event feature, and φ 3D is the three-dimensional convolutional layer.

[0099] Step 3: Simultaneously input the motion-enhanced frame feature and event feature obtained in the previous step into the cross-modal fusion module to utilize the unique advantages of the event camera and the frame camera respectively, mine complementary information from event and frame data, eliminate rain information, and enhance background information.

[0100] Due to the limited imaging ability of the frame camera, there will be some additional background noise in the background of moving edges such as object contours and texture boundaries. On the contrary, due to the advantages of high temporal resolution and high dynamic range of the event camera, it can better capture background information to help separate rain and the background. First, both the enhanced frame features and event features are processed by the encoder for better fusion. To separate the background features in the event domain, the event features need to pass through the rain edge attention module (which consists of four groups of channel-spatial attention) to extract the rain features in the event domain, and then subtract the rain features from the event features to obtain the background features in the event domain.

[0101]

[0102] Where: is the rain feature in the event domain, is the background feature in the event domain, REA is the rain edge attention module, is element-wise subtraction. Subsequently, the background feature in the event domain is added to the frame feature element-wise and then passed through a 1×1 convolution to complete the complementary fusion of the background feature and frame feature in the event domain, thereby enhancing the background details in the frame domain and eliminating edge blurring and rain streaks.

[0103]

[0104] Where: is the fused frame feature, is element-wise addition.

[0105] Step 4. Execute the above process recursively in a multi-patch structure progressively.

[0106] Since some raindrop streaks closer to the camera are much larger than usual and thus more difficult to remove. On the other hand, some small raindrop streaks that are subtly mixed with the background are also difficult to remove. By adopting the method of multi-patch progressive rain removal, the size of the patch can be gradually increased, thereby expanding the range of the receptive field and focusing on raindrop streaks of various sizes and patterns.

[0107] This method adopts a hierarchical structure with a three-level structure. For simplicity of representation, the input sequence is denoted as Its true video sequence is denoted as The P-patch tensor of the input sequence can be denoted as I P , and the data input to the event motion perception process is simplified as g, the cross-modal fusion process and the subsequent residual blocks and convolutional blocks are simplified as h, then the data stream of a single level can be denoted as

[0108]

[0109] Where fL represents the entire above process and are the input and intermediate output of the l-th level respectively, and P l is the number of patches. Then, from the (l + 1)-th level to the last level, the same network recursively takes the concatenation of and the original rain sequence as the input. The recursive process of the entire framework is as follows:

[0110]

[0111] where is the input de-rained video frame of the level above the (l + 1)-th level, is the original video frame input of the network at the (l + 1)-th level, is the total input of the network at the (l + 1)-th level, and C[·] is the concatenation operation along the channel dimension.

[0112] However, the above f L is only allowed once at each level, which means the network pays equal attention to patches of various sizes. However, intuitively, larger patches require more revisions and learning iterations. Therefore, in this method, the network is reconstructed into a progressive structure at each level. At the l-th level, f L iterates multiple times before being merged into the larger patches of the next level, which can be expressed as:

[0113]

[0114] where: r l is the number of loop iterations of the patches at the l-th level. Finally, the overall progressive recursive framework is as follows:

[0115]

[0116] where: represents the de-rained output at the l-th level, and represent the total input of the network at the l-th level and the total input of the network at the (l + 1)-th level respectively.

[0117] The following further describes the specific implementation manners of this aspect in combination with the accompanying drawings and technical solutions.

[0118] The specific implementation process of the present invention is as shown in Figure 1 (This figure only shows the cases of levels two and one). Three consecutive frames and the events between the frames are used as input data, where the event data is processed and transformed into a fixed-size representation Where B represents the batch size, H and W represent the height and width respectively, and this method adopts a three-level sub-block structure. At each level, the frame sequence of length T is divided into P sub-blocks before being input into the proposed network. Among the three levels, the number of sub-blocks is set to {4, 2, 1} respectively. This means that at the first level, and are divided into 4 sub-blocks (one vertical division and one horizontal division), while at the last level, and are regarded as a whole as one sub-block. Before entering the motion perception estimation, shallow features are extracted from the events between three frames and the frames. These features are used for frames and events respectively, and the feature extractors do not share parameters. Subsequently, these two features are input into the event-aware motion module, which estimates the raindrop motion at multiple scales in the event domain and forms a motion region mask. This mask further modifies the frame domain so that the regions with raindrop motion can receive more attention. At the same time, a three-dimensional convolutional layer is used to jointly combine the temporal information in the frequency domain and the event domain and input it into the cross-modal fusion module. The cross-modal fusion module processes the two motion-enhanced features using the same encoder for subsequent fusion. In the event domain, a rain edge attention module is used to separate the rain features in the event domain, and the rain features are subtracted to obtain the background features in the event domain. Then, the background features are fused with the features in the frame domain to utilize the unique advantages of the event camera and the frame camera respectively, mine complementary information from event and frame data, enhance the background and remove rain streaks. Finally, to solve the difficulty in removing tiny raindrops, a multi-sub-block structure is adopted to obtain more receptive fields. The above process will be recursively performed r l ∈{1, 2, 4} times at three levels of P ∈ {4, 2, 1}, and finally the clean rain-removed video frames are obtained.

[0119] The present invention uses PyTorch as the simulation experiment platform and conducts training on the public dataset NUTRain.

[0120] The present invention adopts an end-to-end deep convolutional neural network to implement the entire video rain-removal algorithm. During the training process, the present invention uses the Adam optimizer to optimize the parameters of the entire network, and the learning rate of the entire network is set to 0.001. Every 60 rounds of training, the learning rate is reduced to half of the original until the overall loss of the network does not decrease within 100 rounds, or the loss fluctuates within 0.001 within the last 50 rounds, then the training of the entire network ends.

[0121] The present invention is a video enhancement algorithm. The loss function consists of two parts, namely the self-consistency loss and the self-alignment loss. The self-consistency loss is used to improve the quality of the de-rained video frames, while the self-alignment loss is used to further improve the temporal consistency of adjacent video frames.

[0122] L = L1 + λL2

[0123] Among them, the value of λ is set to 0.3. L1 represents the spatio-temporal interaction loss, as follows:

[0124]

[0125] Among them, B represents the real rain-free frame, represents the de-rained video frame obtained by the de-raining network, and L2 represents the self-alignment loss, as follows:

[0126]

[0127] Among them, A is the alignment function. In this method, LightFlowNet3

[15] is used as the A function to estimate the optical flow and warp the adjacent frames. T is the time series, respectively represent the corresponding de-rained video frames at time t and time t + 1.

[0128] Table 1 PSNR and SSIM performance of the present invention and some other video de-raining methods on the synthetic dataset

[0129]

Claims

1. A multi-patch progressive video de-raining method based on event camera motion perception estimation, characterized in that Including the following steps: Step 1: Obtain a sequence composed of N events captured by an event camera, denoted as and the frame sequence of the same time period collected by a frame camera Adopt a multi - small - block multi - level structure. In each level, first, for the frame sequence of length T and perform chunking respectively to obtain the chunked event data and video frame data; The frame data passes through the feature extraction module respectively to extract the event feature F e and the frame feature F f ; the feature extraction module includes an event feature extraction module and a video feature extraction module; Step 2: Input the extracted event feature F e and the frame feature F f into the event-aware motion module for the following processing: First, the event feature F e is respectively convolved by 3×3, 5×5, and 7×7 convolutions to expand the receptive field of the event feature, and all are processed by 1×1 convolution and concatenated in the channel dimension, and then processed by 3×3 convolution and Sigmoid activation function to obtain the motion perception mask M; Multiply the motion perception mask M and the frame feature F f element-wise, and then process it using a 3×3 convolutional layer to obtain the corrected frame feature F m ; Process the corrected frame feature F using a three-dimensional convolutional layer m to generate a motion-enhanced frame feature Process the event feature F using a three-dimensional convolutional layer e to generate a motion-enhanced event feature Step 3: Co-input the frame feature and the event feature into the cross-modal fusion module for cross-modal fusion, so as to mine complementary information from the event and frame data to eliminate rain information and enhance background information; Step Four: Use a multi-small-block multi-level structure to progressively and recursively execute Steps One to Three to expand the receptive field range and focus on raindrop streaks of various sizes and patterns.

2. The multi-patch progressive video de-raining method based on event camera motion perception and estimation according to claim 1, wherein Step Three specifically includes the following steps: First, the frame features and the event features are both processed by the encoder for better fusion. The event features are passed through the rain edge attention module composed of four groups of channel-spatial attention to extract the rain features in the event domain. Then, the event features are used to perform element-wise subtraction on the rain features to obtain the background features Subsequently, the background features and the frame features are added element - by - element and then passed through a 1×1 convolution to obtain the fused frame features The are sequentially input into the residual block and the convolutional block.

3. The multi-patch progressive video de-raining method based on event camera motion perception estimation according to claim 1, characterized in that In Step 1, each of the said events is denoted as e i =(x i ,y i ,t i ,p i ); where: x i ∈ [1, W] and y i ∈ [1, H] are the spatial indices of the pixels, where W and H represent the width and height of the event data respectively, t i is the time position of the event, and the polarity p i ∈ {-1, +1} indicates the direction of the intensity change.

4. The multi-patch progressive video de-raining method based on event camera motion perception estimation according to claim 2, wherein In Step Four, the progressive recursive execution of Steps One to Three using a multi-small-block multi-level structure specifically includes the following steps: Represent the frame sequence as Its true video sequence representation is Divide the frame sequence into P small blocks, denoted as I P , Denote the process of processing using the event-aware motion module in Step Two as g. Denote the cross-modal fusion process and the subsequent residual block and convolutional block processing in Step Three as h. Then the data stream at a single level is represented as: where f L represents the entire above process, and are the input and intermediate output of the l-th level respectively, and P l is the number of small blocks; then from the (l + 1)-th level to the last level, the concatenation of and the original rain sequence is used as the input for recursion; The recursive process is as follows: Among them is the input of the de-rained video frame of the level above the (l + 1)-th level, is the input of the original video frame of the (l + 1)-th level, is the total input of the (l + 1)-th level, and C[·] is the concatenation operation along the channel dimension; at the l-th level, f L is iterated multiple times before being merged into larger blocks at the next level, expressed as: where: r l is the number of loop iterations of the l-level small block; the final overall progressive recursive framework is as follows: Wherein: represents the de-rained output at the (l + 1)-th level, and respectively represent the total network input at the l-th level and the total network input at the (l + 1)-th level.

5. The multi-small-block progressive video rain removal method based on event camera motion perception estimation according to claim 4, characterized in that The multi-level structure is specifically a three-level sub-block structure. At each level, a frame sequence of length T and is divided into P sub-blocks; In the first level, the number P of small blocks is set to 4, and the frame sequence is divided into 2 small blocks, is divided into 2 small blocks, where one is a vertical division and one is a horizontal division; In the second level, the number P of small blocks is set to 2, and the frame sequences and are divided into two small blocks, and the frame sequences and each become one small block; In the third level, the number P of small blocks is set to 1, and the frame sequence and are regarded as a whole as a small block.

Images