Event-guided online video deblurring method, system, and readable storage medium

Abstract

The application provides an event-guided online video deblurring method, a system and a readable storage medium, and relates to the technical field of image processing. The event-guided online video deblurring method comprises the following steps: according to obtained previous frame images and current frame images, estimating forward optical flow information from the previous frame images to the current frame images and reverse optical flow information from the current frame images to the previous frame images; obtaining image features according to the current frame images, the forward optical flow information and the reverse optical flow information; weighting the image features according to obtained event stream information corresponding to the previous frame images and the current frame images to obtain weighted features; and performing deblurring processing on the current frame images according to the weighted features; wherein the previous frame images and the current frame images are two adjacent video frame images in a forward propagation process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an event-guided online video deblurring method, system and readable storage medium. BACKGROUND

[0002] With the rapid development of portable imaging devices based on visible light and on-board imaging devices, video has gained more and more attention and application in the field of computer vision. However, in the actual shooting process, due to the relative motion between the portable imaging device and the dynamic target in the scene, the imaging result of the portable device inevitably has the phenomenon of motion blur. In order to alleviate or even eliminate the adverse effects of image blur on downstream vision tasks such as SLAM, 3D reconstruction, target tracking, etc., more and more researchers have gradually invested in the image or video deblurring task.

[0003] In addition, these portable imaging devices often have high latency, making it difficult to be directly applied in vision tasks such as target detection and unmanned driving, which have high requirements on image frame rate. On the other hand, due to the low dynamic range of such imaging devices, during actual shooting of high-contrast scenes, overexposure often occurs in areas with high scene brightness, and it is often difficult to collect clear texture information of scene targets in areas with very low scene brightness, making it difficult to effectively implement and conduct downstream vision tasks. Therefore, video deblurring based on multi-modal information has gradually attracted the attention of researchers.

[0004] For online video, since it can only obtain video images at the current time and images between the current time, and the video image information after the current time is in a missing state, it is difficult to implement online video deblurring processing using traditional deblurring methods. SUMMARY

[0005] The purposes of the present application include providing an event-guided online video deblurring method which can realize the deblurring processing of the current frame image in the online video.

[0006] The purposes of the present application also include providing an event-guided online video deblurring network system which can realize the deblurring processing of the current frame image in the online video.

[0007] The purposes of the present application also include providing a readable storage medium which can realize the deblurring processing of the current frame image in the online video.

[0008] The embodiments of the present application can be implemented in the following ways:

[0009] An event-guided online video deblurring method comprises:

[0010] According to the obtained previous frame image and the current frame image, forward optical flow information from the previous frame image to the current frame image and reverse optical flow information from the current frame image to the previous frame image are estimated;

[0011] Image features are extracted according to the current frame image, the forward optical flow information and the reverse optical flow information;

[0012] The image features are weighted according to event stream information corresponding to the obtained previous frame image and the current frame image to obtain weighted features;

[0013] The current frame image is deblurred according to the weighted features;

[0014] The previous frame image and the current frame image are adjacent two frame video frame images in a forward propagation process.

[0015] Optionally, the step of estimating the forward optical flow information from the previous frame image to the current frame image and the reverse optical flow information from the current frame image to the previous frame image comprises:

[0016] Feature extraction and down-sampling operations are performed on the previous frame image and the current frame image;

[0017] The extracted features of the previous frame image are warped by an up-sampled optical flow map to align with the features of the current frame image, so as to obtain the forward optical flow information;

[0018] The extracted features of the current frame image are warped by an up-sampled optical flow map to align with the features of the previous frame image, so as to obtain the reverse optical flow information.

[0019] Optionally, the step of extracting image features according to the current frame image, the forward optical flow information and the reverse optical flow information comprises:

[0020] A first latent clear image between the previous frame image and the current frame image is obtained according to the current frame image and the reverse optical flow information;

[0021] A second latent clear image between the current frame image and a next frame is obtained according to the current frame image and the forward optical flow information;

[0022] Features of an image sequence composed of the current frame image, the first latent clear image and the second latent clear image are extracted and encoded to obtain the image features.

[0023] Optionally, the step of weighting the image feature according to the obtained event stream information corresponding to the previous frame image and the current frame image to obtain a weighted feature comprises:

[0024] obtaining a feature in the event stream information and encoding the feature to obtain an event feature;

[0025] obtaining a feature weight parameter according to the event feature and the image feature;

[0026] weighting the image feature according to the feature weight parameter to obtain the weighted feature;

[0027] the step of performing deblurring processing on the current frame image according to the weighted feature comprises:

[0028] decoding the weighted feature to obtain a clear image of the current frame image, thereby realizing deblurring processing on the current frame image.

[0029] Optionally, the feature weight parameter is obtained by the following formula:

[0030]

[0031] wherein, weight refers to the feature weight parameter, softmax(·) refers to a normalized exponential function processing, e refers to the event feature, z refers to the image feature, W1 refers to a learnable parameter of a convolutional neural network for processing the event feature, W2 refers to a learnable parameter of a convolutional neural network for processing the image feature, T refers to a matrix transposition operation, and e and z have the same dimension.

[0032] Optionally, the process of performing deblurring processing on the current frame image is expressed by the following formula:

[0033] I′ i ← F(cat(I i-1 (x + f i-1→i ), I i ), E i , E i-1 )

[0034] wherein, I′ i refers to a clear image of the current frame image, F(·) represents a network system operation, cat(·) represents image connection along the channel dimension, I i-1 refers to the previous frame image, x refers to a pixel point on the previous frame image, f i-1→i refers to the forward optical flow information, I i refers to the current frame image, E i refers to the event stream information corresponding to the current frame image, and E i-1corresponding to the previous frame image.

[0035] An event-guided online video deblurring network system, comprising:

[0036] an optical flow estimation sub-network configured to estimate forward optical flow information from the previous frame image to the current frame image and reverse optical flow information from the current frame image to the previous frame image according to the obtained previous frame image and the current frame image;

[0037] an extraction module configured to extract image features according to the current frame image, the forward optical flow information and the reverse optical flow information;

[0038] an event data fusion module configured to weight the image features according to event stream information corresponding to the obtained previous frame image and the current frame image to obtain weighted features; and

[0039] an image reconstruction convolutional sub-network configured to deblur the current frame image according to the weighted features.

[0040] wherein the previous frame image and the current frame image are two adjacent video frame images in a forward propagation process.

[0041] Optionally, the optical flow estimation sub-network comprises:

[0042] a feature pyramid module configured to extract features of the previous frame image and the current frame image and perform down-sampling operation;

[0043] a warping operation module configured to warp the extracted features of the previous frame image through up-sampled optical flow map to align with the features of the current frame image, and configured to warp the extracted features of the current frame image through up-sampled optical flow map to align with the features of the previous frame image; and

[0044] a cost volume module configured to calculate the matching degree between the warped features of the previous frame image to the features of the current frame image and the original features of the current frame image, and to calculate the matching degree between the warped features of the current frame image to the features of the previous frame image and the original features of the previous frame image.

[0045] Optionally, the event data fusion module comprises a feature extractor, a first convolutional neural network, a second convolutional neural network and a weighting module, and the extraction module comprises a reconstruction network encoder.

[0046] The feature extractor is configured to extract and encode features in the event stream information to obtain event features;

[0047] The reconstruction network encoder is configured to encode features of an image sequence obtained according to the current frame image and the forward optical flow information and the reverse optical flow information obtained by estimation, to extract image features;

[0048] The first convolutional neural network is configured to process the event features, and the second convolutional neural network is configured to process the image features, and a feature weight parameter is obtained by a normalized exponential function;

[0049] The weighting module is configured to weight the image features according to the feature weight parameter to obtain the weighted features.

[0050] A readable storage medium stores a computer program, and the computer program is executed by a processor to implement the event-guided online video deblurring method.

[0051] The event-guided online video deblurring method, system and readable storage medium provided by the embodiments of the present application have the following beneficial effects:

[0052] The embodiments of the present application provide an event-guided online video deblurring method, which comprises: estimating forward optical flow information from a previous frame image to a current frame image and reverse optical flow information from the current frame image to the previous frame image according to the obtained previous frame image and current frame image; extracting image features according to the current frame image, the forward optical flow information and the reverse optical flow information; weighting the image features according to event stream information corresponding to the obtained previous frame image and current frame image to obtain weighted features; and performing deblurring processing on the current frame image according to the weighted features; wherein the previous frame image and the current frame image are two adjacent video frame images in a forward propagation process. The event stream information contains rich time sequence information, and the image features are weighted according to the event stream information, which can better describe the pixel changes between the previous frame image and the current frame image, so as to help realize the deblurring processing on the current frame image in the case of missing image information after the current frame image.

[0053] The embodiments of the present application also provide an event-guided online video deblurring network system, which comprises an optical flow estimation sub-network, an acquisition module and an extraction module, and through processing of the forward optical flow information, the reverse optical flow information and the event stream information, the current frame of the online video can be deblurred.

[0054] The embodiment of the present application also provides a readable storage medium, a computer program stored in the readable storage medium is executed by a processor to realize the event-guided online video deblurring method. BRIEF DESCRIPTION OF DRAWINGS

[0055] The above features and advantages of the present application can be better understood by reading the following detailed description of embodiments of the present application in conjunction with the drawings, in which:

[0056] Figure 1 A step diagram of the event-guided online video deblurring method according to an aspect of the present application is shown;

[0057] Figure 2 A flowchart of the event-guided online video deblurring method according to an aspect of the present application is shown;

[0058] Figure 3 A process diagram of the event data fusion module obtaining a feature weight parameter according to an aspect of the present application is shown;

[0059] Figure 4 A comparison diagram of qualitative experimental results of the event-guided online video deblurring method according to an aspect of the present application on a GoPro dataset is shown;

[0060] Figure 5 Another comparison diagram of qualitative experimental results of the event-guided online video deblurring method according to an aspect of the present application on a GoPro dataset is shown. DETAILED DESCRIPTION

[0061] The present application is described in detail below in conjunction with the drawings and specific embodiments. Note that the aspects described below in conjunction with the drawings and specific embodiments are merely exemplary and should not be understood as limiting the scope of protection of the present application in any way.

[0062] Recently, a type of visual sensor capable of recording changes in scene brightness—the dynamic visual sensor—has emerged in researchers' field of vision. Event cameras have been developed and evolved by deploying dynamic visual sensors on imaging devices. Relying on dynamic visual sensors, event cameras can record changes in pixel brightness in scene images at microsecond rates. Therefore, compared to traditional visual imaging devices, event cameras can more reliably and effectively acquire scene target information and have lower latency. Furthermore, by measuring brightness changes between image pixels, event cameras can capture richer target motion information. The inventors believe that by combining event data acquired by dynamic visual sensors with image data acquired by traditional visual sensors, the problem of insufficient motion information between video frames caused by the high latency of traditional imaging devices can be compensated for to some extent. At the same time, the high dynamic range of event data will also facilitate the full utilization of video frame information, playing an important auxiliary role in the deblurring operation of video frames, thereby contributing to the deblurring of online videos. Based on the above analysis theory, this invention provides an event-guided online video deblurring method. The method uses event stream data to supplement the motion information lost between image frames, and at the same time repairs the problem of image frames losing detail information such as image texture due to overexposure or underexposure, which leads to downstream visual tasks arbitrarily extracting effective features, thereby achieving deblurring of images in online videos.

[0063] Figure 1 This is a flowchart illustrating the steps of the event-guided online video deblurring method provided in this embodiment. Figure 2 This is a flowchart of the event-guided online video deblurring method provided in this embodiment. Please refer to the attached document. Figure 1 and Figure 2 This embodiment provides an event-guided online video deblurring method and an event-guided online video deblurring network system. This system is essentially a learnable network framework structure, and the aforementioned event-guided online video deblurring method can be implemented based on this network framework structure.

[0064] Framework design theory and approach:

[0065] In the problem of deblurring offline videos, traditional variational model-based methods often assume that there are a large number of sharp latent frames between two adjacent video image frames. Therefore, for the i-th image frame in a video sequence, it is often regarded as the result of the accumulation of all latent frames from the previous time step to the next time step. This idea can be modeled as follows:

[0066]

[0067] in, I represents the i-th blurred image frame in the video sequence. i This represents the sharp image frame in the image sequence, where t represents the relative exposure time of the camera. The clear image I at relative time k i with I i+1 Potential frames between, The clear image I at relative time k i with I i-1 Potential frames between.

[0068] If, based on the above idea, the method of acquiring potentially sharp frames is geometrically correlated with the optical flow information between video frames, then the above equation can be written as:

[0069]

[0070] Among them, f i→i+1 The optical flow information from the i-th video frame to the (i+1)-th video frame, f i→i-1 This refers to the optical flow information from the i-th video frame to the (i-1)-th video frame, where x refers to a pixel in the image. Specifically, I i (x+kf i→i+1 In the image, x refers to the pixel in the current frame.

[0071] However, unlike offline video deblurring tasks, in online video image deblurring tasks, the model can only obtain the video image frame at the current moment (i.e., the current frame image) and all image frames before the current frame, while the video image information after the current frame is missing.

[0072] To address this problem, the inventors treat the video frame at the next moment (i.e., the next frame image) as the result of the current frame video being warped by forward optical flow. The current frame image can then be seen as the result of accumulating all potential frames along the time dimension between the previous frame image and the warped image frame. Based on this idea, the production method of the i-th blurred frame video in a video image sequence, i.e., the content of the above formula, can be written in the following form:

[0073]

[0074] Therefore, in order to obtain a clear latent frame image between the current frame and the previous frame, as well as a virtual clear latent frame image between the current frame and the next frame, forward and reverse optical flow information is required.

[0075] Based on the above ideas, the event-guided online video deblurring method provided in this embodiment includes:

[0076] S1: According to the obtained previous frame image and the current frame image, the forward optical flow information from the previous frame image to the current frame image and the reverse optical flow information from the current frame image to the previous frame image are estimated.

[0077] The event-guided online video deblurring network system includes an optical flow estimation sub-network (i.e., PWC-Net), which takes two adjacent video images (i.e., a previous frame image and a current frame image) as input, respectively estimates the forward optical flow information from the previous frame image to the current frame image, and the reverse optical flow information from the current frame image to the previous frame image.

[0078] It should be noted that in the description of the embodiment, the order of "previous", "current" and "next" is defined according to the order of forward propagation of the video, and accordingly, "forward" is the direction consistent with the forward propagation direction of the video, and "reverse" is the direction opposite to the forward propagation direction of the video.

[0079] Specifically, the basic assumption of the optical flow estimation method is that in the two frames of images, the pixel position of the scene target on the image may shift a certain distance, but the brightness of the image on the corresponding pixel often remains unchanged, that is, the brightness consistency assumption. Therefore, by adopting the form of a sliding window, a corresponding pixel window can be obtained in one image frame, and the most closely matched pixel window with the pixel window of the one image frame can be found in another image frame. Through such a pixel window matching method, the optical flow information between the image frames is ultimately estimated.

[0080] The optical flow estimation sub-network includes a feature pyramid module, a warping operation module and a cost volume calculation module, and the execution of step S1 is realized through the optical flow estimation sub-network.

[0081] Specifically, step S1 includes:

[0082] S11: Feature extraction and down-sampling operation is performed on the previous frame image and the current frame image.

[0083] The optical flow estimation sub-network uses the feature pyramid module to perform feature extraction and down-sampling operation on the input two adjacent frames of images, so that the resolution of the obtained features is lower, and by performing pixel window matching operation on such low-resolution features, the calculation complexity of pixel matching can be effectively reduced, and the pixel displacement vector obtained by calculation can provide effective guidance for the matching operation of the pixel window in the next layer of the pyramid.

[0084] S12: The features extracted from the previous frame image are warped by the up-sampled optical flow to align with the features of the current frame image, so as to obtain the forward optical flow information.

[0085] The optical flow estimation sub-network warps the features extracted from the previous frame by the up-sampled optical flow through a warping operation module, and aligns the warped features with the features of the current frame video.

[0086] In addition, the optical flow estimation sub-network also calculates the matching degree between the features warped and transformed from the features of the previous frame to the features of the current frame through a cost volume calculation module set, to ensure that the matching degree between the features warped and transformed from the features of the previous frame to the features of the current frame is high when the warping operation module performs the warping operation. The higher the matching degree is, the more accurate the estimated forward optical flow information is.

[0087] S13: Warp the features extracted from the current frame by the up-sampled optical flow map to align them with the features of the previous frame image, thereby obtaining the reverse optical flow information.

[0088] The optical flow estimation sub-network warps the features extracted from the current frame by the up-sampled optical flow through a warping operation module, and aligns the warped features with the features of the previous frame video.

[0089] In addition, the optical flow estimation sub-network also calculates the matching degree between the features warped and transformed from the features of the previous frame to the features of the current frame through a cost volume calculation module set, to ensure that the matching degree between the features warped and transformed from the features of the previous frame to the features of the current frame is high when the warping operation module performs the warping operation. The higher the matching degree is, the more accurate the estimated forward optical flow information is.

[0090] In this way, the optical flow estimation sub-network outputs the forward optical flow information f i-1→i and the reverse optical flow information f i→i-1 In the present embodiment, the current frame image is denoted as I i , the previous frame image is denoted as I i-1 , and correspondingly, i-1→i denotes from the previous frame image to the current frame image; i→i-1 denotes from the current frame image to the previous frame image.

[0091] S2: Extract image features according to the current frame image, the forward optical flow information and the reverse optical flow information.

[0092] On the basis of the current frame image, the forward optical flow information and the reverse optical flow information, the potential images before and after the current frame image can be obtained. Further, the image features required for reconstructing the clear image of the current frame image are extracted.

[0093] The event-guided online video deblurring network system comprises an extraction module to realize the execution of step S2 through the extraction module.

[0094] Specifically, for a pixel (x, y) contained in the current frame image, the process of reconstructing a certain latent sharp image I i (x+kf i→i-1 ) is as follows: an offset (u, v) of the pixel point coordinate (x, y) is obtained according to the scene optical flow data (i.e., a combination of the forward optical flow information and the reverse optical flow information), so that the pixel position of the pixel point on the reconstructed image, i.e., the pixel point (x+u, y+v), can be obtained, and then the pixel value of the pixel point (x+u, y+v) is set. It should be noted that, since the output result of the optical flow estimation sub-network is not necessarily an integer, and the value range of k in the expression of the latent sharp image does not impose an integer limitation, the pixel point (x+u, y+v) of the reconstructed latent sharp image is not necessarily an integer, and therefore the reconstructed pixel point coordinate obtained above needs to be subjected to a difference operation to achieve the purpose of integerizing the pixel point coordinate. Here, k refers to a certain time in t.

[0095] Through the above processing, the latent sharp image I i (x+kf i→i-1 ) between the previous frame image and the current frame image can be obtained, which is a first latent sharp image, and the virtual latent sharp image I i (x+kf i-1→i ) between the current frame image and the next frame can be obtained, which is a second latent sharp image. Correspondingly, the extraction module contains. According to different values of k, latent sharp images at different times can be obtained, in other words, according to the above expression of the latent sharp image, a series of images can be obtained, and the virtual next frame image I i+1 can be expressed by the following formula:

[0096] I i+1 =I i (x+f i-1→i )

[0097] The features of the image sequence composed of the current frame image, the first latent sharp image, and the second latent sharp image are encoded to obtain image features.

[0098] The extraction module further includes a reconstruction network encoder, which encodes the features of the image sequence composed of the current frame image, the first latent sharp image, and the second latent sharp image to obtain image features z.

[0099] S3: The image features are weighted according to the event stream information 10 corresponding to the previous frame image and the current frame image to obtain weighted features.

[0100] For an event camera, when a piece of image data is given, the event stream information corresponding to the image data can also be obtained at the same time. The event stream information is generally expressed in the form of (t, x, y, p), where t is the timestamp of the event camera collecting data, (x, y) is the pixel coordinate of the event data, and the polarity p is generally +1 or -1, indicating that the brightness at the pixel point increases or decreases.

[0101] The event stream information contains rich spatial information and event information, which can indirectly describe the motion blur in the image spatial domain. On the other hand, the high frame rate characteristics of the event stream information can make up for the lack of events in the sequence caused by the high time delay of image data to a certain extent.

[0102] Specifically, the event-guided online video deblurring network system includes an event data fusion module, and the execution of step S3 is realized through the event data fusion module.

[0103] Step S3 includes:

[0104] S31: Obtain and encode the features in the event stream information 10 to obtain event features.

[0105] The event data fusion module includes a feature extractor, which extracts and encodes the features in the event stream information 10 to obtain event features e. Optionally, the feature extractor is ResNet-152.

[0106] S32: Obtain feature weight parameters according to the event features e and the image features z.

[0107] Figure 3 The process diagram of the event data fusion module obtaining the feature weight parameters in this embodiment is shown. Please refer to Figures 1-3 The event data fusion module further includes two layers of learnable convolutional neural networks, which are a first convolutional neural network conv1 and a second convolutional neural network conv2. The event features e are processed through the first convolutional neural network conv1, and the image features z are processed through the second convolutional neural network conv2. After processing, the feature weight parameters are obtained through a normalization exponential function softMax.

[0108] Specifically, the feature weight parameters weight are obtained by the following formula:

[0109]

[0110] wherein, W1 refers to the learnable parameters of the first convolutional neural network conv1, W2 refers to the learnable parameters of the second convolutional neural network conv2, T refers to the matrix transposition operation, and e and z have the same dimension. It should be noted that the values of W1 and W2 are obtained through training.

[0111] S33: The image features are weighted according to the feature weight parameter to obtain weighted features.

[0112] The event data fusion module further comprises a weighting module which weights the image features according to the feature weight parameter weight obtained through calculation, thereby obtaining weighted features. By weighting the image features, the features which are beneficial to subsequent deblurring processing can be retained or enhanced, while the influence of the features which are not beneficial can be weakened, thereby ultimately improving the deblurring capability and performance of the event-guided online video deblurring network system.

[0113] S4: The current frame image is deblurred according to the weighted features.

[0114] According to the above analysis, for the current frame image in the video sequence, it is often regarded as the result accumulated by all potential frame images between the image at the previous moment (i.e. the previous frame image) and the image at the next moment (i.e. the next frame image). However, in the actual application process, k cannot take all values from 0 to t, i.e. in the actual application process, the number of all potential frame images from the image at the previous moment to the image at the next moment is limited. Therefore, for the actual online video image deblurring task, the clear potential frame image can be obtained through the following formula:

[0115]

[0116] If t = 1, the above formula can be expressed as:

[0117]

[0118] wherein, I' i is the clear image obtained by reconstructing the current frame image I i .

[0119] However, the inventors have found that, since I i+1 (x+f i→i-1 ) and I i-1 (x+f i-1→i ) are not aligned with I i at the pixel points, the above formula is not the best way to obtain the image after deblurring, which may cause the clear image obtained through recovery to have artifacts, and therefore the following formula is used to obtain a clear frame image with higher quality:

[0120]

[0121] where, since I i+1 = I i (x+f i-1→i ), the above can be expressed in the following form within the error tolerance range:

[0122]

[0123] However, the inventors continue to study and find that such an averaging method is not the best deblurring scheme, and therefore the event-guided online video deblurring network system in the present application generates a clear image of the blurred current frame image through a learnable manner, which can be expressed by the following formula:

[0124] I i ← F(cat(I i-1 (x+f i-1→i ), E i , E i ) i-1 )

[0125] where cat(·) represents the connection of images along the channel dimension, F(·) represents the operation of the network system (i.e., the event-guided online video deblurring network system), E i denotes the event stream information corresponding to the current frame image, E i-1 denotes the event stream information corresponding to the previous frame image, I i-1 (x+f i-1→i ) denotes the pixel point on the previous frame image.

[0126] The event-guided online video deblurring network system further comprises an image reconstruction convolutional subnetwork, which processes the weighted features to reconstruct a clear current frame image, thereby realizing deblurring processing of the current frame image. Specifically, the image reconstruction convolutional subnetwork essentially forms a decoder structure, which constitutes the image reconstruction subnetwork of the event-guided online video deblurring network system, and reconstructs the clear image 20 of the current frame image by decoding the weighted features.

[0127] The loss function of the event-guided online video deblurring network system is designed to consist of three parts, namely, the clear image reconstruction loss L rec , the contrast regularization loss L CR , and the image structural similarity loss L SSIM , and the calculation formula of the overall loss function L total is as follows:

[0128] L total= L rec + 0.2L CR + 0.5L SSIM

[0129] Clear image reconstruction loss L rec : The clear image reconstruction loss trains the overall network by calculating the L1 norm of the reconstructed clear image and the true value clear image to constrain the pixel value of the reconstructed image in the spatial domain. Specifically, the loss function calculates the difference between the reconstructed clear image and the true value clear image in the pixel value, and calculates the L1 norm of the difference to avoid negative values. The specific calculation formula of the loss function is as follows:

[0130]

[0131] Where F is the overall network system, M is the image batch size used in the training process, and a batch size contains M groups of data samples

[0132] Contrast regularization loss L CR : Unlike the clear image reconstruction loss L rec , the contrast regularization loss L CR considers processing the reconstructed image in the image feature domain. The main idea is to use a pre-trained encoder, VGG19 in this experiment, to extract corresponding feature information from the reconstructed image, the blurred image and the true value clear image. The true value clear image is regarded as a positive sample, and the extracted feature information is positive feature information. The blurred image is regarded as a negative sample, and the extracted feature information is negative feature information. The main purpose of the contrast regularization loss L CR is to make the reconstructed image features more similar to the positive feature information and more distant from the negative feature information. The algorithm in this application uses the features extracted by the first layer, the third layer, the fifth layer and the ninth layer of the VGG19 network to calculate in the training stage. The contrast regularization loss L CR calculated by each feature layer is weighted and summed with different weights. Specifically, the contrast regularization loss function calculation formula is as follows:

[0133]

[0134] Where G represents the VGG19 encoder, G j (·) represents the feature map extracted by the jth layer of the VGG19 encoder, ω i is the weighting coefficient. The weights for calculating the features extracted by the first layer, the third layer, the fifth layer, the ninth layer and the thirteenth layer of the VGG19 network are

[0135] Image structural similarity loss L SSIM The image structural similarity function SSIM is an index for measuring the similarity of two images in brightness, contrast and structure. The actual calculation result of the SSIM index ranges from 0 to 1, where 0 indicates that the similarity of the two images is low, and 1 indicates that the two images are extremely similar in brightness, contrast and structure. The specific calculation method of the function is as follows:

[0136]

[0137] Wherein, μ x and μ y are the average values of x and y, respectively, and are the variances of x and y, respectively, σ xy is the covariance of x and y, and c1 and c2 are constants, which can be selected as c1 = 0.0001 and c2 = 0.0009.

[0138] The design of the image structural similarity loss LSSIM is mainly based on the SSIM index for measuring image quality. Since the network system of the present embodiment constrains and optimizes the model parameters by minimizing the overall loss function during the training process, the image structural similarity loss function L SSIM is used as follows:

[0139]

[0140] The event-guided online video deblurring network system provided in the present embodiment adopts Pytorch for framework building, uses Adam optimizer for training and optimization, and the algorithm model is trained on an NVIDIA Tesla V100 GPU.

[0141] Since the deblurring process of the present embodiment requires video images and corresponding event stream information as input, the present embodiment uses synthetic event stream information and real-time stream information for training and testing, respectively. For synthetic event stream information, the currently public GoPro video deblurring dataset is used, supplemented by an ESIM event simulator to generate corresponding event stream information for each frame of video image. Thus, based on the GoPro video deblurring dataset, blurred image data, ground truth sharp image data and event stream information can be obtained respectively. By using these data, the event-guided online video deblurring network system built in the present embodiment can be trained.

[0142] In the training stage, the model is trained and optimized by using an Adam optimizer, wherein the parameters of the optimizer are respectively β1=0.9 and β2=0.999. The algorithm model provided in the embodiment is trained on an NVIDIA Tesla V100 GPU. In the experiment, the batch size of the image is set to 8, and the input size of each image is 256*256. The learning rate used in the experiment is 1*10 -4 , and the decay mode of the learning rate is to decrease by 50% for every 100 epochs. The system iterates for a total of 200 epochs.

[0143] The embodiment of the application also provides a readable storage medium, which stores a computer program readable by a computer. The computer program is executed by a processor, so as to realize the event-guided online video deblurring method.

[0144] Experimental example

[0145] Figure 4 FIG. 1 is a comparison diagram of qualitative experimental results of the event-guided online video deblurring method provided in the embodiment on a GoPro dataset, Figure 5 FIG. 2 is another comparison diagram of qualitative experimental results of the event-guided online video deblurring method provided in the embodiment on the GoPro dataset. Please refer to Figure 4 and Figure 5 The quantitative and qualitative tests on the test set in the GoPro video deblurring dataset are performed by using the event-guided online video deblurring network system provided in the embodiment, so as to verify the deblurring effect of the event-guided online video deblurring method provided in the embodiment.

[0146] Specifically, in the index of the quantitative results, two evaluation indexes commonly used in the field of video deblurring are adopted, which are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Table 1 is the quantitative experimental results of the method of the application on the GoPro dataset:

[0147] Table 1

[0148] GoPro PSNR 32.13 SSIM 0.938

[0149] In Table 1, the PSNR and SSIM of all the processed images are calculated after all the images in the GoPro dataset are processed by using the method of the application, and then the average values are obtained, Figure 4 and Figure 5 FIGS. 3 and 4 respectively show the comparison diagram of one image in the GoPro dataset before and after processing.

Claims

1. An event-guided online video deblurring method, characterized in that, include: Based on the obtained previous frame image and current frame image, estimate the forward optical flow information from the previous frame image to the current frame image and the reverse optical flow information from the current frame image to the previous frame image; Image features are extracted based on the current frame image, the forward optical flow information, and the reverse optical flow information; The image features are weighted based on the event stream information corresponding to the previous frame image and the current frame image to obtain weighted features; Based on the weighted features, the current frame image is deblurred; Wherein, the previous frame image and the current frame image are two adjacent video frame images during one forward propagation process; The step of weighting the image features based on the event stream information corresponding to the previous frame image and the current frame image to obtain weighted features includes: The features in the event stream information are obtained and encoded to obtain event features; Based on the event features and the image features, feature weight parameters are obtained; The image features are weighted according to the feature weight parameters to obtain the weighted features; The step of deblurring the current frame image based on the weighted features includes: The weighted features are decoded to obtain a clear image of the current frame image, thereby achieving region blurring processing of the current frame image; The feature weight parameters are obtained through the following formula: Wherein, weight refers to the feature weight parameter, softmax(·) refers to the normalization exponential function processing, e refers to the event feature, z refers to the image feature, W1 refers to the learnable parameters of the convolutional neural network processing the event feature, W2 refers to the learnable parameters of the convolutional neural network processing the image feature, T refers to the matrix transpose operation, and e and z have the same dimension.

2. The event-guided online video deblurring method according to claim 1, characterized in that, The step of estimating the forward optical flow information from the previous frame image to the current frame image and the reverse optical flow information from the current frame image to the previous frame image includes: Feature extraction and downsampling operations are performed on the previous frame image and the current frame image; The features extracted from the previous frame image are distorted by upsampling the optical flow map to align them with the features of the current frame image, thereby obtaining the forward optical flow information; The features extracted from the current frame image are distorted by upsampling the optical flow map to align them with the features of the previous frame image, thereby obtaining the reverse optical flow information.

3. The event-guided online video deblurring method according to claim 1, characterized in that, The step of extracting image features based on the current frame image, the forward optical flow information, and the reverse optical flow information includes: A first potentially clear image between the previous frame image and the current frame image is obtained based on the current frame image and the reverse optical flow information; A virtual second potential clear image between the current frame image and the next frame is obtained based on the current frame image and the forward optical flow information; The features of the image sequence consisting of the current frame image, the first potentially sharp image, and the second potentially sharp image are extracted and encoded to obtain the image features.

4. The event-guided online video deblurring method according to claim 1, characterized in that, The process of deblurring the current frame image is expressed by the following formula: I i ’ ←F(cat(I i-1 (x+f i-1→i ),I i ),E i ,E i-1 ) Among them, I i ’ The current frame image is the clear image, F(·) represents the network system operation, cat(·) represents the image concatenation along the channel dimension, and I i-1 The previous frame refers to the image in question, x refers to the pixel in the previous frame, and f refers to the previous frame. i-1→i The forward optical flow information, I i Refers to the current frame image, E i Refers to the event stream information corresponding to the current frame image, E i-1 This refers to the event stream information corresponding to the previous frame image.

5. An event-guided online video deblurring network system, characterized in that, include: An optical flow estimation subnetwork is used to estimate the forward optical flow information from the previous frame image to the current frame image and the reverse optical flow information from the current frame image to the previous frame image based on the obtained previous frame image and current frame image. The extraction module is used to extract image features based on the current frame image, the forward optical flow information, and the reverse optical flow information; An event data fusion module is used to weight the image features based on the event stream information corresponding to the previous frame image and the current frame image to obtain weighted features; and An image reconstruction convolutional subnetwork is used to deblur the current frame image based on the weighted features. Wherein, the previous frame image and the current frame image are two adjacent video frame images during one forward propagation process; The optical flow estimation subnetwork includes: The feature pyramid module is used to extract features from the previous frame image and the current frame image, and perform downsampling operations. A distortion operation module is configured to distort the features extracted from the previous frame image using an upsampled optical flow map, aligning them with the features of the current frame image; and the distortion operation module is also configured to distort the features extracted from the current frame image using an upsampled optical flow map, aligning them with the features of the previous frame image; and The cost capacity module is used to calculate the matching degree between the features of the previous frame image transformed into the features of the current frame image and the original features of the current frame image, and to calculate the matching degree between the features of the current frame image transformed into the features of the previous frame image and the original features of the previous frame image.

6. The event-guided online video deblurring network system according to claim 5, characterized in that the event data fusion module includes a feature extractor, a first convolutional neural network, a second convolutional neural network, and a weighting module, and the extraction module includes a reconstruction network encoder; The feature extractor is used to extract and encode features from the event stream information to obtain event features; The reconstruction network encoder is used to encode the features of the image sequence obtained based on the current frame image and the estimated forward optical flow information and the backward optical flow information to obtain image features; The first convolutional neural network is used to process the event features, and the second convolutional neural network is used to process the image features, and the feature weight parameters are obtained through a normalized exponential function; The weighting module is used to weight the image features according to the feature weight parameters to obtain the weighted features.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program that is executed by a processor to implement the event-guided online video deblurring method according to any one of claims 1-4.

Images