An event camera based self-supervised rolling shutter image rectification method and device
By employing a self-supervised learning method based on event cameras in the field of image correction, the problem of correcting rolling shutter images was solved, achieving high frame rate image reconstruction results in real-world scenes.
Patent Information
- Application Number
- CN202310325247.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-30
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2043-03-30
AI Technical Summary
Existing technologies struggle to effectively correct rolling shutter images in complex motion scenarios, and event camera-based methods suffer from performance degradation in real-world scenes. In particular, the reconstructed global shutter images under nonlinear motion conditions are not ideal, and discrepancies exist between real and simulated data, affecting the correction effect.
A self-supervised rolling shutter image correction method based on event cameras is adopted. By combining the self-supervised learning method of event cameras and training with convolutional neural networks, a unified event-based intra-frame and out-of-frame compensator is constructed using a self-supervised framework and unified temporal and spatial transformations to realize the conversion between rolling shutter and global shutter images.
It achieves effective correction of rolling shutter images in real-world scenarios, solves the problem of intra-frame/out-of-frame information loss, improves image quality and clarity, and enhances the image correction effect.
Smart Images
Figure CN116579931B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of image processing, and relates to rectification and high-frame-rate image reconstruction of rolling shutter images using an event camera. BACKGROUND
[0002] Traditional cameras are mainly divided into global shutter (GS) cameras and rolling shutter (RS) cameras. The image frame captured by a global shutter camera is exposed simultaneously by all pixels, which can directly reflect rich scene information. A rolling shutter camera starts exposure in a row-by-row manner from the first row to the last row (or from the last row to the first row) at a fixed time interval, and the exposure time of each row is equal. Therefore, the exposure interval of each row is different, and when an object moves rapidly, the object will easily deform, causing a decrease in visual effect, which is also known as the rolling shutter effect, which can seriously affect subsequent algorithm processing, so it is necessary to rectify the rolling shutter image to a global shutter image to ensure subsequent target detection, three-dimensional reconstruction and other algorithm processing and provide a good visual experience.
[0003] Current rectification algorithms based on frame images usually assume that the motion between two frames is linear, and a global shutter image is reconstructed from two or more consecutive rolling shutter images. However, this motion assumption is difficult to guarantee in actual scenes, especially when complex non-linear motion is encountered, and the reconstructed global shutter image is often unsatisfactory.
[0004] A visual sensor based on biological neural mechanisms, an event camera, has become a breakthrough to solve this problem. Unlike traditional optical cameras, an event camera can record changes in brightness, generate asynchronous events with a delay of microseconds, record the coordinates of the change, the time and the change polarity, and has high temporal resolution, low power consumption and low bandwidth requirements. The event camera can record the edge texture of the target and the inter-frame motion information in a high-speed motion scene.
[0005] Current rolling shutter image rectification methods based on event cameras can only reconstruct a global shutter image within the exposure time range. In actual situations, there is a time interval between the exposure intervals of consecutive rolling shutter images, which will cause the reconstructed result to be discontinuous, with missing frames in time, resulting in poor visual experience, which has prompted the need for continuous global shutter image reconstruction within / without frames.
[0006] In addition, due to the difficulty in obtaining real data sets, current rolling shutter image rectification methods based on event cameras are based on simulation data sets. In fact, due to various factors such as camera response function, event trigger threshold, output delay caused by transmission bandwidth, etc., there is a difference in data distribution between real data and simulation data. In addition, due to the modal difference between events and images, the connection between the two kinds of data established by the existing method is easily affected by the data distribution, and its performance often decreases in real scene test, prompting the demand for a self-supervised framework. SUMMARY
[0007] Based on the above analysis, the present application provides a self-supervised rolling shutter image rectification method based on an event camera. The method takes advantage of the high temporal resolution of the event camera to reconstruct a global shutter image sequence within / outside the continuous frame from a rolling shutter image. A unified transformation of time and space is proposed to flexibly realize the conversion between rolling shutter images and global shutter images. A self-supervised learning framework is established, so that the algorithm can be directly trained without ground truth.
[0008] The self-supervised rolling shutter image rectification method based on the event camera provided by the present application comprises the following specific steps:
[0009] Step 1: Simulate the generation of rolling shutter images and event streams by shooting global shutter videos with a high-speed camera, or directly shoot rolling shutter images and event streams.
[0010] Step 2: Event stream preprocessing: for each set of input data, the event stream is divided according to the target time to be reconstructed and preprocessed to convert it into a multi-channel image format as the input of the network.
[0011] Step 3: Based on the event-based time domain conversion and spatial domain conversion, an event-based unified intra-frame and inter-frame compensator is constructed, and a convolutional neural network is established based on the compensator.
[0012] Step 4: Network training using a self-supervised framework: first, reconstruct a high-frame-rate global shutter video from a rolling shutter image, and then reconstruct the original rolling shutter image using the reconstructed global shutter video. The mutual relationship between the rolling shutter image, the reconstructed global shutter image and the reconstructed rolling shutter image is used for self-supervised training.
[0013] Step 5: Capture event streams using an event camera and rolling shutter images using a rolling shutter camera. After processing the event stream in step 2, an event frame that can be input to the network is obtained. The event frame and the rolling shutter image are reconstructed into a high-frame-rate global shutter video through the optimized network model.
[0014] The specific details of each step are as follows:
[0015] For real scenes, the rolling shutter images in step 1 can be taken by a rolling shutter camera and an event camera and event stream .
[0016] In step 1, a rolling shutter image and an event stream can also be simulated by taking a global shutter video with a high-speed camera. The specific steps are as follows: according to the exposure mode of the rolling shutter, select consecutive frames of global shutter images from the global shutter video, select the th row of the th global shutter image according to the time sequence, and then splice the single-row images row by row to form a rolling shutter image. Select different frames in the same way to generate consecutive rolling shutter videos, and finally generate an event stream from the taken global shutter video.
[0017] The specific simulation data generation steps are as follows:
[0018] A global shutter image , the formation process is
[0019]
[0020] , where is a continuous global shutter video, is time, ( x, y ) is the coordinate position of the pixel.
[0021] The rolling shutter image in step 1 is exposed row by row, and the formation process is
[0022]
[0023] , where is an event movement operator related to the pixel position, and the superscript indicates that the movement process is performed row by row.
[0024]
[0025] represents the height of the image, represents the exposure time interval of the image, is the vertical coordinate of the pixel.
[0026] The event stream generated in step 1 is
[0027] According to the event stream Relationship between rolling shutter video and global shutter video When the log domain luminance variation exceeds an event trigger threshold η > 0:
[0028]
[0029] From time to time , video The amount of luminance log value variation at coordinates exceeds a threshold An event point , The event polarity, 1 when the luminance log value is increasing, -1 when decreasing; the event stream represents the set of events triggered in the entire time period and
[0030] Through step 1, continuous rolling shutter images and event streams are obtained.
[0031] Select two consecutive rolling shutter images and the corresponding event stream from the data as a group of data, set the target time as the global shutter image time to be reconstructed, which is expressed as follows:
[0032]
[0033] Where represents the previous rolling shutter image, represents the next rolling shutter image, represents the event stream between the exposure times of the two rolling shutter images, is the global shutter image time to be reconstructed.
[0034] The event stream segmentation method in step 2 is to divide into and according to the target time .
[0035] The preprocessing method in step 2 is two kinds, the first kind converts the absolute timestamp of the event stream into the time displacement relative to the starting time, denoted as (·). The processing method of a single event in the event stream is expressed as
[0036]
[0037] The second kind converts the absolute timestamp of the event stream into the time displacement relative to the ending time and performs inversion, then performs polarity inversion of the events, denoted as (·) One event stream
[0038]
[0039] After two preprocessing methods, four event streams can be obtained, which can also be written as . The superscripts and respectively represent the event stream after (·) and (·) preprocessing. Finally, the is processed by (·) to obtain .
[0040] The specific process of converting the event stream into a multi-channel image format is as follows: each event stream is divided into event streams according to the vertical coordinates of the event points, and each event stream is divided into time intervals according to the time span of each event stream, and the event points are accumulated in each time interval to form an event frame with a size of (2 represents positive and negative polarity, and 1 and represent the height and width of the event frame), and finally event frames are formed.
[0041] Time domain conversion in step 3: the gray value change of a pixel point from the RS graph to the GS graph in the time domain can be represented by events as
[0042]
[0043] where represents the brightness change between and
[0044]
[0045] where uses the Dirac function to represent continuous event streams .
[0046] Spatial domain conversion in step three: the motion information in the scene is implied in the events, so the displacement of the pixel points is predicted through the event stream
[0047]
[0048] wherein represents the flow of events estimated optical flow.
[0049] The network model in step 3 is divided into three parts: a compensator in the spatial domain , a compensator in the temporal domain and a fusion module . The compensator in the spatial domain is composed of a Shuffle layer, a convolution layer, a residual dense module Reshuffle layer and a RELU activation function. Each part of the compensator in the temporal domain and the fusion module is composed of a convolution layer, a residual module and a RELU activation function. Only using spatial domain changes or only using temporal domain changes will be affected by the instability of the event threshold and event noise, so the fusion module is added to improve the network performance.
[0050] After constructing the network, the event stream and the rolling shutter image are first input into the network, and the spatial change result from the RS image to the GS image is calculated by the following formula and the temporal change result
[0051] ,
[0052] ,
[0053] Then through the fusion module, the advantages of the compensator in the spatial domain and the compensator in the temporal domain are combined to estimate the final result
[0054] .
[0055] The overall network model is
[0056] .
[0057] Step 4: training the network using a self-supervised framework: first, reconstruct a high-frame-rate global shutter video from the rolling shutter image, and then reconstruct the original rolling shutter image using the reconstructed global shutter video. The mutual relationship between the rolling shutter image, the reconstructed global shutter image and the reconstructed rolling shutter image is used for self-supervised training.
[0058] The self-supervised framework described in step 4 is
[0059]
[0060] wherein is a potential consistency constraint; is a cyclic consistency constraint; is a timing consistency constraint; is a weight scalar.
[0061] Potential consistency constraint is defined as
[0062] .
[0063] Cyclic consistency constraint is defined as
[0064] .
[0065] Timing consistency constraint is defined as
[0066] .
[0067] denotes norm.
[0068] The optimization method in step 4 is an Adam optimization algorithm, which designs independent adaptive learning rates for different parameters by calculating the first moment estimation and second moment estimation of the gradient, and iteratively updates the neural network convolutional layer parameters based on the training data.
[0069] The optimized multi-layer convolutional neural network is constructed according to the optimized parameters of each convolutional layer.
[0070] Based on the same inventive concept, the present application also designs an electronic device, comprising:
[0071] one or more processors;
[0072] a storage device for storing one or more programs;
[0073] When the one or more programs are executed by the one or more processors, the one or more processors implement the event camera-based self-supervised rolling shutter image rectification method.
[0074] Based on the same inventive concept, the present application also designs a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the event camera-based self-supervised rolling shutter image rectification method.
[0075] Compared with the prior art, the present application has the following advantages and beneficial effects:
[0076] The application proposes an event-based unified intra-frame and inter-frame compensator for rolling shutter image correction based on an event camera, which can realize unified transformation of rolling shutter images and global shutter images in the time domain and the spatial domain.
[0077] The application can reconstruct a continuous global shutter sequence of intra-frame / inter-frame from a rolling shutter image and a continuous event stream.
[0078] Finally, the application proposes a self-supervised framework for training the network.
[0079] The application solves the problems of rolling shutter effect and intra-frame / inter-frame information loss by taking advantage of the low latency of the event camera, and designs a self-supervised learning framework so that the network can be directly trained on a real shooting dataset, and has good correction and video reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS
[0080] Figure 1 , 2, 3 are flowcharts of three constraints in the self-supervised framework proposed by the application: latent consistency constraint, cycle consistency constraint and temporal consistency constraint.
[0081] Figure 4 It is a schematic diagram of the event-based unified intra-frame and inter-frame compensator proposed by the application.
[0082] Figure 5 It is a schematic diagram of the network model SelfUnroll proposed by the application.
[0083] Figure 6 It is a comparison result diagram of the current rolling shutter image correction algorithm on the simulation dataset Fastec-RS.
[0084] Figure 7 It is a comparison result diagram of the current rolling shutter image correction algorithm on the simulation dataset Gev-RS.
[0085] Figure 8 It is a comparison result diagram of the current rolling shutter image correction algorithm on the real dataset. DETAILED DESCRIPTION
[0086] In order to more clearly understand the application, the technical content of the application will be specifically introduced below.
[0087] Step 1: Shoot a global shutter video by a high-speed camera to simulate a rolling shutter image and an event stream , or directly shoot a rolling shutter image and an event stream .
[0088] For real scenes, the rolling shutter images in step 1 can be taken by rolling shutter cameras and event cameras and event streams .
[0089] The specific steps for simulating the generation of rolling shutter images and event streams in step 1 by taking global shutter videos with high-speed cameras are as follows: according to the exposure mode of rolling shutter, select consecutive frames of global shutter images from the global shutter video, select the th row of the th global shutter image according to the time sequence, and then stitch the single-row images row by row to form a rolling shutter image. Select different frames in the same way to generate consecutive rolling shutter videos, and finally generate event streams from the taken global shutter videos.
[0090] The specific simulation data generation steps are as follows:
[0091] A global shutter image , the formation process of which is
[0092]
[0093] where is a continuous global shutter video, is time, ( x, y ) is the coordinate position of the pixel.
[0094] The rolling shutter image in step 1 is exposed row by row, and the formation process is
[0095]
[0096] where is an event movement operator related to the pixel position, and the superscript indicates that the movement process is performed row by row.
[0097]
[0098] represents the height of the image, represents the exposure time interval of the image, is the vertical coordinate of the pixel.
[0099] The event stream generated in step 1 is
[0100] According to the event stream and the GS video the relationship between the log of the luminance change and the event trigger threshold η > 0:
[0101]
[0102] from time to time , the video at coordinates exceeds the threshold , an event point , is the event polarity, 1 if the log of the luminance is increasing, -1 if it is decreasing; the event stream represents the set of events triggered over the entire time period and
[0103] By step 1, we can obtain continuous rolling shutter images and event streams .
[0104] Select two consecutive rolling shutter images and the corresponding event stream from the data as a group of data, set the target time as the global shutter image time to be reconstructed, which is expressed as follows:
[0105]
[0106] where represents the previous rolling shutter image, represents the next rolling shutter image, represents the event stream between the exposure times of the two rolling shutter images, is the global shutter image time to be reconstructed.
[0107] Step 2 event stream preprocessing: for each group of input data, the event stream is divided according to the target time to be reconstructed and preprocessed to convert it into a multi-channel image format as the input of the network.
[0108] The event stream division method in step 2 is to divide into and according to the target time .
[0109] The preprocessing method in step 2 is two kinds, the first kind converts the absolute timestamp of the event stream into the time displacement relative to the starting time, denoted as (·). The processing method of a single event in the event stream is expressed as
[0110]
[0111] The second way is to convert the absolute timestamp of the event stream into the time displacement relative to the end time and flip it, and then perform the polarity inversion of the event, denoted as (·). The processing mode of a single event in a section of event stream is represented as .
[0112]
[0113] and After the two preprocessing methods, four sections of event streams , , and are obtained. The superscripts and respectively represent that the event stream has undergone (·) and (·) preprocessing. Finally, the is processed by (·) to obtain .
[0114] The specific process of converting the event stream into a multi-channel image format is as follows: each section of event stream is divided into sections according to the longitudinal coordinates of the event points, and is divided into time intervals according to the time span of each section of event stream, and the event points are accumulated in each time interval to form an event frame with a size of (2 represents the positive and negative polarity, and 1 and represent the height and width of the event frame), and finally event frames are formed.
[0115] Step 3: Based on the time domain conversion and spatial domain conversion based on events, a unified intra-frame and inter-frame compensator based on events is constructed, and a convolutional neural network is established based on the compensator.
[0116] Time domain conversion in step 3: the change of the gray value of a pixel point in the time domain from the RS graph to the GS graph can be represented by events as
[0117]
[0118] where represents the brightness change between and
[0119]
[0120] where The Dirac function is used to represent the continuous event stream . .
[0121] The spatial domain conversion in step three: the motion information in the scene is implied in the events, so the displacement of the pixels is predicted through the event stream
[0122]
[0123] where represents the optical flow estimated through the event stream .
[0124] The network model in step 3 is divided into three parts: the compensator in the spatial domain , the compensator in the temporal domain and the fusion module . The compensator in the spatial domain is composed of a Shuffle layer, a convolution layer, a residual dense module, a Reshuffle layer and a RELU activation function. Each part of the compensator in the temporal domain and the fusion module is composed of a convolution layer, a residual module and a RELU activation function. Using only the spatial domain variation or only the temporal domain variation will be affected by the instability of the event threshold and the event noise, so we add the fusion module to improve the network performance.
[0125] After constructing the network, the event stream and the rolling shutter image are first input into the network, and the spatial variation result from the RS image to the GS image is calculated by the following formula and the temporal variation result
[0126]
[0127]
[0128] Then through the fusion module, the advantages of the compensator in the spatial domain and the compensator in the temporal domain are combined to estimate the final result
[0129]
[0130] The overall network model is
[0131]
[0132] The compensator in the spatial domain in step 3 It consists of one Reshuffle layer, one Shuffle layer, six convolutional layers, ten residual dense modules, and a ReLU activation function. It concatenates event frames and rolling shutter images, passes them through the Reshuffle layer, and outputs them to convolutional layer 1 to obtain intermediate results. .Will The input is fed into a concatenation of 10 residual dense modules, with the result from convolutional layer 2 as the input. The outputs of the 10 residual dense modules are then concatenated to obtain... .Will The input is concatenated into convolutional layers 3 and 4, and the output is added... get .Will The input is fed into a convolutional layer 5, a reshuffle layer, and a convolutional layer 6, which scales the output resolution to the size of the input rolling shutter image to obtain the residual. The residual The final output is obtained by adding the input rolling shutter image to the input image. The convolution kernel of convolutional layer 3 is... In addition, the kernels of all convolutional layers are .
[0133] Time-domain compensator in step 3 and fusion module It consists of 7 convolutional layers, 1 residual module, and a ReLU activation function. The input passes through concatenated convolutional layers 1, 2, 3, and 4, yielding intermediate results as follows: ,Will After passing through the residual module, the result is... .Will and The concatenation is obtained through convolutional layer 5. , and The concatenation is obtained through convolutional layer 6. , and The output is obtained by concatenating the data through convolutional layer 7. All convolutional layers have convolutional kernels of type 1. .
[0134] Step 4: Train the network using a self-supervised framework: First, reconstruct a high frame rate global shutter video from the rolling shutter image, then reconstruct the original rolling shutter image using the reconstructed global shutter video. Use the rolling shutter image and the relationship between the reconstructed global shutter image and the reconstructed rolling shutter image for self-supervised training.
[0135] The self-supervised framework described in step 4 is
[0136]
[0137] in Potential consistency constraint Cyclic consistency constraint Temporal consistency constraint Weighted scalar, which can be taken as {1, 1, 1} in the embodiment.
[0138] Potential consistency constraint is defined as
[0139]
[0140] Cyclic consistency constraint is defined as
[0141]
[0142] Temporal consistency constraint is defined as
[0143]
[0144] represents norm.
[0145] The optimization method in step 4 is an Adam optimization algorithm, which designs independent adaptive learning rates for different parameters by calculating the first moment estimate and the second moment estimate of the gradient, and iteratively updates the neural network convolutional layer parameters based on the training data.
[0146] The optimized multi-layer convolutional neural network is constructed according to the optimized parameters of each convolutional layer.
[0147] Step 5: Capture event stream using event camera, capture rolling shutter image using rolling shutter camera, process event stream through step 2 to obtain event frame that can be input to the network, and input event frame and rolling shutter image to the optimized network model to reconstruct high-frame-rate global shutter video.
[0148] By comparing several rolling shutter correction methods:
[0149] 1) DSUN: image correction method based on deep learning and supervision.
[0150] 2) SUNET: image correction method based on deep learning and supervision.
[0151] 3) RSSR: image correction and video reconstruction method based on deep learning and supervision.
[0152] 4) CVR: image correction and video reconstruction method based on deep learning and supervision.
[0153] 5) EvUnroll: Event stream and image rectification and video reconstruction method based on deep learning and supervision.
[0154] 6) SelfUnroll (Ours): Event stream and image rectification and video reconstruction method based on deep learning and self-supervision.
[0155] Gev-RS and Fastec-RS datasets are generated by simulating the RS images and event streams from the videos captured by high-speed cameras following the simulation process in Step 1.
[0156] We measure the numerical indicators under the same dataset:
[0157] Table 1. Single reconstruction and sequence reconstruction test results of the model on the Gev-RS dataset
[0158]
[0159] Table 2. Single reconstruction test results of the model on the Fastec-RS dataset
[0160]
[0161] In the experiment, the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and image perceptual similarity (LPIPS) are used as measurement indicators, and their specific definitions are as follows:
[0162]
[0163]
[0164] where , , . represent the mean of , and and represent the variance and covariance between the two, respectively.
[0165]
[0166]
[0167] where represents the distance between in the feature. The feature stack is extracted from the L layer and unit-normalized in the channel dimension. The vector is used to scale the number of activation channels and calculate the L2 distance. Finally, it is averaged in space and summed in channels.
[0168]
[0169] The higher the PSNR and SSIM values and the lower the LPIPS value, the better the reconstruction effect. As shown in the numerical results in Tables 1 and 2, the present application exhibits high performance in both single and multi-image reconstruction tasks for rolling shutter image rectification, verifying the effectiveness of the present application.
[0170] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art to which the present application belongs can make various modifications or supplements to the described specific embodiments or replace them with similar ways, without departing from the spirit of the present application or exceeding the scope defined by the appended claims.
Claims
1. An event camera based self-supervised rolling shutter image rectification method, characterized in that, The method comprises the following steps: Step 1: simulate the generation of rolling shutter image and event stream by shooting global shutter video by high-speed camera, or directly shoot rolling shutter image and event stream; Step 2: event stream preprocessing: for each group of input data, the event stream is divided according to the target time to be reconstructed and preprocessed to convert into a multi-channel image format as the input of the network; Step 3, according to the event-based time domain conversion and the space domain conversion, an event-based unified intra-frame and inter-frame compensator is constructed, and a convolutional neural network is established based on the compensator; the convolutional neural network is divided into three parts, including a space domain compensator , a time domain compensator and a fusion module ; Step 4: training of the network using a self-supervised framework: first, reconstruct the high-frame-rate global shutter video from the rolling shutter image, and then reconstruct the original rolling shutter image using the reconstructed global shutter video; the mutual relationship among the rolling shutter image, the reconstructed global shutter image and the reconstructed rolling shutter image is used for self-supervised training; The self-supervised framework of step 4 is wherein is a potential consistency constraint; is a cyclic consistency constraint; is a timing consistency constraint; is a weight scalar; Potential consistency constraints are defined as Cyclic consistency constraint timing consistency constraint is defined as denotes norm The optimization method used in step 4 is: Adam optimization algorithm, which designs independent adaptive learning rates for different parameters by calculating the first moment estimate and the second moment estimate of the gradient, and iteratively updates the neural network convolutional layer parameters based on the training data; The optimized multi-layer convolutional neural network is a multi-layer convolutional neural network constructed according to the optimized parameters of each layer of the convolutional neural network; wherein: according to the target time will be cut into and , represents the flow of events between the exposure times of the two rolling shutter images, the rolling shutter images ; Step 5: capture the event stream by the event camera, capture the rolling shutter image by the rolling shutter camera, and obtain the event frame that can be input to the network after the event stream is processed in step 2; the event frame and the rolling shutter image are input into the optimized network model to reconstruct the high-frame-rate global shutter video.
2. The event camera-based self-supervised rolling shutter image correction method according to claim 1, wherein: The specific steps in step 1 for simulating the generation of rolling shutter images and event streams by capturing global shutter video with a high-speed camera are as follows: Following the exposure mode of a rolling shutter, select consecutive frames from the global shutter video... The frame of the global shutter image is selected according to the time sequence. The first of the global shutter images Afterwards, A rolling shutter image is created by stitching together individual single-line images. Different frames are selected in the same way to generate a continuous rolling shutter video. Finally, an event stream is generated from the captured global shutter video.
3. The event camera-based self-supervised rolling shutter image correction method according to claim 2, wherein: The specific process of simulating the generation of rolling shutter image and event stream in step 1 is: One GS image in step 1 The forming process is wherein is a continuous global shutter video, is time, ( x, y ) is the coordinate position of the pixel; Rolling shutter image in step 1 In the case where the exposure process is performed line by line, the process is wherein is an event shift operator related to the pixel position, the superscript indicates that the shift is performed line by line for the shift process; denotes the height of the image, T denotes the exposure time interval of the image, is the ordinate of the pixel; According to the relationship between the event stream and global shutter video when the luminance change in the logarithmic domain exceeds the event trigger threshold : From time to time , video brightness at coordinates changes by more than a threshold , an event point , is generated, with event polarity 1 if the brightness is increasing, -1 if it is decreasing; the event stream represents the set of events triggered over the entire time period Select two consecutive rolling shutter images and the corresponding event stream from the simulated or real captured data as a set of data, set the target time The global shutter image time to be reconstructed is represented as follows: wherein denotes the previous rolling shutter image, denotes the next rolling shutter image, denotes the event stream between the exposure times of the two rolling shutter images, is the global shutter image time to be reconstructed.
4. The event camera-based self-supervised rolling shutter image correction method according to claim 1, wherein: The event stream in Step 2 is split according to the target time Will be split into and , represents the event stream between the exposure times of the two rolling shutter images; The specific procedure of converting the event stream into a multi-channel image format is as follows: each segment of the event stream is divided into segments according to the longitudinal coordinates of the event points, each segment of the event stream is divided into time intervals according to the time span of each segment of the event stream, and the event points are accumulated in the time intervals to form an event frame with a size of , and finally event frames are formed; wherein 2 represents positive and negative polarity, and 1 and represent the height and width of the event frame.
5. The event camera-based self-supervised rolling shutter image correction method according to claim 1, wherein: There are two preprocessing methods. The first method converts the absolute timestamps of the event stream into time offsets relative to the start time, denoted as... Event Stream Single event The processing method is expressed as The second way is to convert the absolute timestamp of the event stream into a time displacement relative to the end time, and then flip it, and then perform the polarity inversion of the event, denoted as , a segment of event stream , a single event , the processing mode wherein, is time, x is the coordinate position of a pixel, is the event polarity, and After two preprocessing methods can be obtained four event stream and , also known as and , the superscript and respectively indicate that the event stream through and preprocessing, finally through processing to get .
6. The event camera-based self-supervised rolling shutter image correction method according to claim 1, wherein: The time domain conversion and spatial domain conversion in step 3 are as follows: Time domain conversion: the change of the gray value of a pixel point in the time domain from the rolling shutter RS image to the global shutter GS image can be represented by events as wherein represents to a change in luminance between wherein when using Dirac functions to represent continuous event streams ; Spatial domain conversion: the displacement of a pixel point is predicted by the event stream wherein represents a flow of events estimated optical flow, target time , is a continuous global shutter video, x is a coordinate position of a pixel.
7. The event camera-based self-supervised rolling shutter image correction method according to claim 1, wherein: The network model in step 3 is divided into three parts: a compensator in the spatial domain , a compensator in the time domain , and a fusion module ; the compensator in the spatial domain is composed of a Shuffle layer, a convolution layer, a residual dense module Reshuffle layer, and a RELU activation function; each part of the compensator in the time domain and the fusion module is composed of a convolution layer, a residual module, and a RELU activation function; using only the spatial domain variation or only the time domain variation will be affected by the instability of the event threshold and event noise, so we add the fusion module to improve the network performance; and After two pre-processing ways can get four segments of event stream and , After the network is constructed, the event stream and the rolling shutter image are input into the network, and the spatial variation result from the rolling shutter RS image to the global shutter GS image is calculated by the following formula and the temporal variation result ， ， Then through the fusion module, the advantages of the compensator in the spatial domain and the compensator in the time domain are combined to estimate the final result The overall network model is 。 8. An electronic device, comprising: It comprises: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.
9. A computer readable medium having stored thereon a computer program, characterized in that: The program is executed by the processor to implement the method of any one of claims 1-7.