A method for reducing global motion and rolling shutter effects in a dual camera system

Abstract

The present invention provides a method for reducing motion blur and rolling shutter effects, comprising: receiving a main image frame, a main timing and a main exposure from a main camera; receiving a sub-image frame, a sub-timing and a sub-exposure from a sub-camera; correcting the sub-image frame to the main image frame; determining an incremental timing based on the main timing and the sub-timing; determining an incremental exposure based on the main exposure and the sub-exposure; determining discrete motion of an offset sequence based on the sub-image frame; determining a row-wise motion blur kernel based on the discrete motion of the main image frame and the offset sequence; and determining spatial variation kernel deconvolution based on the main image frame and the row-wise motion blur kernel.

Description

Technical Field

[0001] The present invention relates to the technical field of blur reduction, and more particularly, to the reduction of global motion and rolling shutter effects in a dual camera system. Background Art

[0002] Currently, capturing scenes in low light lengthens the exposure time of frames, making them susceptible to jitter, which results in global motion blur in the output image.

[0003] Common commercial cameras usually capture images using a rolling shutter, which may produce unwanted artifacts when there is motion. Summary of the invention

[0004] An exemplary dual-camera system includes: a primary camera using a rolling shutter; and a secondary camera using a global shutter.

[0005] An exemplary method for simultaneously reducing motion blur and rolling shutter effects includes: receiving a main image frame, a main timing, and a main exposure from a main camera; receiving a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera; correcting the secondary image frame to the main image frame; determining an incremental timing based on the main timing and the secondary timing; determining an incremental exposure based on the main exposure and the secondary exposure; determining a discrete motion of an offset sequence based on the secondary image frame; determining a row-wise motion blur kernel based on the discrete motion of the main image frame and the offset sequence; and determining a spatially varying kernel deconvolution based on the main image frame and the row-wise motion blur kernel.

[0006] Another exemplary method for simultaneously reducing motion blur and rolling shutter effects includes: receiving a main image frame, a main timing, and a main exposure from a main camera; receiving a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera; determining an incremental timing based on the main timing and the secondary timing; determining an incremental exposure based on the main exposure and the secondary exposure; determining a discrete motion of an offset sequence based on the secondary image frame; determining a row-wise motion blur kernel based on the discrete motion of the main image frame and the offset sequence; remapping the row-wise motion blur kernel to the main image frame; and determining a spatially varying kernel deconvolution based on the main image frame and the row-wise motion blur kernel. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram showing a first example system according to an embodiment of the present disclosure is shown;

[0008] Figure 2 A schematic diagram showing a second example system according to an embodiment of the present disclosure is shown;

[0009] Figure 3An optical schematic diagram of camera motion blur according to an embodiment of the present disclosure is shown;

[0010] Figure 4 A schematic diagram of a rolling shutter and a global shutter according to an embodiment of the present disclosure is shown;

[0011] Figure 5 A schematic diagram of a dual camera system according to an embodiment of the present disclosure is shown;

[0012] Figure 6 A schematic diagram showing the timing of a main camera and a secondary camera capturing images at different frame rates under controlled synchronization according to one embodiment of the present disclosure;

[0013] Figure 7 A schematic diagram of a full motion blur kernel according to one embodiment of the present disclosure is shown;

[0014] Figure 8 A schematic diagram showing a row-wise blur kernel in a rolling shutter image according to an embodiment of the present disclosure is shown;

[0015] Fig. 9 A schematic diagram of a process of a rolling shutter main camera remapping a kernel of a secondary camera frame to a primary image geometry according to an embodiment of the present disclosure is shown;

[0016] Fig.10 A schematic diagram of a process of a rolling shutter main camera correcting a secondary camera frame to a main image geometry according to an embodiment of the present disclosure is shown;

[0017] Fig.11 A schematic diagram of a process of generating a row-wise blur kernel according to an embodiment of the present disclosure is shown;

[0018] Fig.12 A first example method of simultaneously reducing motion blur and rolling shutter effects according to one embodiment of the present disclosure is shown;

[0019] Fig.13 An example method of generating a temporal density segmentation according to one embodiment of the present disclosure is shown; and

[0020] Fig.14 A second example method of quantitative evaluation according to one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The embodiments listed below are only used to illustrate the application of the device and method, and are not intended to limit the scope. Equivalent modifications to the device and method should fall within the scope of protection of the claims.

[0022] Certain terms are used in the following description and claims to refer to specific system components. As those skilled in the art will appreciate, different companies may refer to components and / or methods by different names. This document should not distinguish between components and / or methods that have different names but the same function.

[0023] In the following discussion and in the claims, the terms "including" and "comprising" are used in an open-ended fashion, and thus may be interpreted to mean "including, but not limited to..." Additionally, the term "connected" is intended to mean either an indirect or direct connection. Thus, if a first device connects to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0024] Figure 1 An exemplary hybrid computing system 100 is depicted that can be used to implement a neural network associated with the operation of one or more portions or steps of a process. In this example, processors associated with the hybrid system include a field programmable gate array (FPGA) 122, a graphics processing unit (GPU) 120, and a central processing unit (CPU) 118.

[0025] The CPU 118, GPU 120, and FPGA 122 have the capability to provide a neural network. The CPU is a general purpose processor that can perform many different functions, and its versatility results in the ability to perform a variety of different tasks, but its processing of multiple data streams is limited, and its capabilities related to neural networks are also limited. The GPU is a graphics processor that has many small processing cores that can handle parallel tasks sequentially. The FPGA is a field programmable device that can be reconfigured and perform any function that may be programmed into the CPU or GPU in a hardwired circuit manner. Because the FPGA is programmed in circuit form, its speed is many times faster than the CPU, and significantly faster than the GPU.

[0026] Systems may contain other types of processors, such as accelerated processing units (APUs), which include a CPU with GPU elements on the chip, and digital signal processors (DSPs) designed to perform high-speed digital data processing. Application-specific integrated circuits (ASICs) can also perform the hardwired functions of an FPGA; however, the lead time to design and produce an ASIC is on the order of several quarters of a year, as opposed to the quick turnaround implementation available in programming an FPGA.

[0027] Graphics processor unit 120, central processing unit 118, and field programmable gate array 122 are connected and connected to memory interface and controller 112. FPGA is connected to memory interface through programmable logic circuit and memory interconnect 130. Since FPGA operates at a very large bandwidth, in order to minimize the circuit used by FPGA, this additional device is used to perform memory tasks. Memory interface and controller 112 is additionally connected to disk 110, system memory 114, and read-only memory (ROM) 116.

[0028] Figure 1 The system can be used to program and train the FPGA. The GPU works well with unstructured data and can be used for training. Once the data is trained, a deterministic inference model can be found and the CPU can program the FPGA using the model data determined by the GPU.

[0029] The memory interface controller is connected to a central interconnect 124, which is further connected to GPU 120, CPU 118, and FPGA 122. Central interconnect 124 is further connected to input and output interfaces 128 and network interface 126.

[0030] Figure 2 A second example hybrid computing system 200 is depicted that may be used to implement a neural network associated with the operation of one or more portions or steps of process 1000. In this example, the processor associated with the hybrid system includes a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220.

[0031] The FPGA is electrically connected to an FPGA controller 212, which interfaces with a direct memory access (DMA) 218. The DMA is connected to an input buffer 214 and an output buffer 216, which are coupled to the FPGA to buffer data stored in and data retrieved from the FPGA, respectively. The DMA 218 includes two first-in-first-out (FIFO) buffers, one for the host CPU and the other for the FPGA, and the DMA allows data to be written to and read from the appropriate buffers.

[0032] The CPU side of the DMA has a master switch 228 that transfers data and commands to the DMA. The DMA is also connected to the SDRAM controller 224, which allows data to go to and from the FPGA and the CPU 220, and the SDRAM controller is also connected to the external SDRAM 226 and the CPU 220. The master switch 228 is connected to the peripheral interface 230. The flash controller 222 controls the permanent storage and is connected to the CPU 220.

[0033] The image sensor may be a rolling shutter sensor. A higher resolution primary camera having a capture resolution of, for example, 12 mega-pixels (12MP) may capture images at 30 frames per second (fps) or a slower frame rate. A secondary camera having a lower resolution of, for example, 2MP may have a higher frame rate, for example, 60fps. The secondary camera may be mounted adjacent to the primary camera, and the optical axes of the primary and secondary cameras may be similar.

[0034] Due to the difference in frame rates, for one main image frame captured by the main camera, there may be multiple secondary image frames captured by the secondary camera. The global motion can be determined by matching the information between the secondary image frames and the main image frame during the main image frame integration time.

[0035] Different pixel lines in the main image frame can share a time-delayed motion kernel of the continuous motion kernel. Line-by-line kernel deconvolution can be applied to the main image frame to achieve synchronous de-motion blur and de-rolling shutter effect.

[0036] The secondary image frames from the secondary camera can provide information of time-sampled motion information, from which a motion blur kernel can be constructed. In addition, the secondary image frames can provide continuous time motion information, which helps to remove motion blur and remove rolling shutter effects simultaneously. The low-resolution, high-frame-rate secondary camera can be a global shutter camera, which can provide an estimate of the motion kernel.

[0037] In a CMOS image sensor, light is captured from the environment through the camera lens onto the sensor surface. The sensor converts these photons into an electrical current and therefore generates the brightness of a two-dimensional image (single-channel grayscale or multi-channel color levels) by integrating the light captured during the exposure time. If the camera moves during the integration time, the integration may be contaminated by this motion. This type of movement is common during the integration process and can be caused, for example, by an unstable camera mount, hand shaking, etc. This type of contamination is called motion blur.

[0038] In higher resolution sensors, the buffer for reading out the data may be large, and the data may not be stored by the sensor. Therefore, rolling shutter technology is currently used to integrate and read out the data of high resolution sensors line by line.

[0039] The camera imaging system can be described by an integration process. Figure 3The effect of global motion blur driven by the motion of the camera during integration is depicted. For physical point P 310, when the camera is stable at position A 314, the optical center of the lens labeled OA 312 projects the image of that point onto the sensor pixel array plane as PA 322. When the camera moves to position B 318, the optical center moves to position OB 316, and its projected image may move to PB 324. When this motion occurs during exposure, i.e., within a single frame integration time, the camera optical center will move from position A 314 to position B 318 along the camera motion trajectory 320. This motion may be projected onto the sensor pixel array plane, i.e., from position PA 322 to PB 324, and thus the resulting image of physical point P 310 may show camera motion blur.

[0040] An example of rolling shutter is that some rows or columns of pixels may start integrating at a delayed time than the previous row or column. If there is motion, either camera or object motion, this can cause row and column dependent distortion. A global shutter can be free of this rolling shutter artifact limited by the memory / buffer of the sensor.

[0041] Figure 4 A rolling shutter is depicted, including, for image row 410, an Nth frame 414 and an (N+1)th frame 416 taken over time 412. In a rolling shutter, the integration start time for a row is delayed from the previous row. For image row 418, a global shutter occurs at time 420. In a global shutter, an Nth frame 422 and an (N+1)th frame 424 are taken over time 420. In a global shutter, the integration start times for the rows begin at the same point in time. The simultaneous reduction of rolling shutter artifacts and global motion blur effects is essentially a reduction in coupled deformations.

[0042] Figure 5 An exemplary dual camera system is depicted, where a primary camera and a secondary camera are adjacent. In this example, the primary camera has a higher resolution than the secondary camera, and the primary camera frame rate is slower than the secondary camera frame rate. In this example, the primary camera is a digital image camera with a resolution of 12MP or higher and a frame rate of 30fps or lower. The secondary camera has a lower resolution than the primary camera and a higher frame rate, such as 4 times or more than the primary camera.

[0043] like Figure 5As shown, a scene 510 can be captured by a main camera 512 with high resolution, low frame rate and rolling shutter to obtain a main image 514, and captured by a secondary camera 516 with low resolution, high frame rate and global shutter to obtain a secondary image 518. Due to the high frame rate of the secondary camera, a monochrome filter can be used to ensure acceptable brightness. The two cameras can be geometrically calibrated, and the output images are rectified to output similar views and formats. The calibration and correction used can be performed by direct linear transformation.

[0044] Figure 6 Depicts the timing sequence of the primary and secondary cameras when capturing images at different frame rates under controlled synchronization.

[0045] The exposure time of the rolling shutter main camera, i.e. the integral, is denoted as T expMain The total time from the start of integration to the end of the last line of integration is expressed as T FrameMax .

[0046] The secondary camera uses a full-frame exposure global shutter camera, so the exposure time is T expSec Almost the same as the exposure period, where T expSecs Smaller than the main camera's T expMain The frame rate of the secondary camera is greater than the frame rate of the primary camera.

[0047] The secondary camera provides a sequence of image frames taken at fixed intervals during the exposure of the primary camera. The image from the secondary camera can be represented as I sec (i), i=0,1,2,….I sec The exposure duration of (i) can be from TStart(i) to TEnd(i). Since the secondary camera operates in high frame rate mode, the exposure time is relatively very short, so the secondary image is captured at T(i) = (TStart(i) + TEnd(i)) / 2.

[0048] To reduce global motion blur, deconvolution can be used. Therefore, the global alignment of the image with its previous frame is as follows:

[0049]

[0050] Where (x, y) are the horizontal and vertical coordinates of the pixel in the frame. After rotating by θ and translating by (Δx, Δy), the coordinates are transformed to (u, v) and aligned with the previous frame. Since the deconvolution process is implemented in the imaging plane, the rotation by θ can be further omitted, that is, the transformation follows the optical flow constraint, so the objective function is

[0051]

[0052] The first frame is obtained by translating one frame from the reference frame, which is represented by (Δx i ,Δy i ), i = 0, 1, 2, ... In one example, the motion should have second-order smoothness and continuity, for example, through spline interpolation, a continuous motion path C(t) = {x(t), y(t)} can be estimated.

[0053] C(t)={x(t),y(t)}=interpolate{{x i ,y i},T i} (3)

[0054] In this example trajectory, the tangent vector might represent the velocity vector of the camera's motion when the sequence was captured.

[0055]

[0056] The faster the motion, the weaker the kernel strength at that location. Therefore, the initial unnormalized discretization kernel can be set to:

[0057]

[0058] Using Discrete Kernel Data

[0059] in,

[0060] The global continuous kernel, the point spread function, can be estimated as:

[0061]

[0062]

[0063] Use range [0,T Framemax ].

[0064] exist Figure 6 In the example, the first row of the main image frame is at time [0,T expMain ], and the second line integrates over time [Δt row ,Δt row +T expMain ], and so on, where Δt row is the speed of the rolling shutter, successive lines start at Δt row Typically, the kth row of the image is integrated at time [k×Δt row ,k×Δt row +T expMain ], k = 0, 1, 2, …H-1.

[0065] The solution of the example uses [k×Δt row ,k×Δt row +T expMain ] to estimate the kernel for a single row (the kth row) of the data in .

[0066]

[0067]

[0068] PSF row(k) ,k=0,1,2,…similar to PSF in equation (6) frame This allows the PSF row(k) Since in equation (6), the non-zero values ​​are located at (x,y)∈C(t), the PSF Frame can be rewritten as:

[0069]

[0070] therefore,

[0071] Figure 7 A total motion kernel 710 is depicted.

[0072] Figure 8 A row-wise motion blur kernel 810 is depicted in a rolling shutter image, with kernel 812 on row 0, kernel 814 on row 1, and kernel 816 on row N.

[0073] So far, the phrase PSF does not imply normalization. Next, the PSF applied to the image will be normalized with conservative energy.

[0074] Therefore, the blur model involving rolling shutter and global motion blur is:

[0075] I observed (x,y)=I clear (x,y)*PSF row(y) (x,y)+Noise(x,y) (10)

[0076] Among them, Noise(x,y) is the system additional noise.

[0077] Based on the convolution model in matrix-vector product, the blur model can be expressed as:

[0078] B=KL+N (11)

[0079] Where L, B and N represent I clear (latent image), I observed (blurred image) and the noise in column vector form.

[0080] K is the image filter matrix to which the convolution is applied. The rows of K are motion blur kernels placed at pixel locations and decomposed into row vectors. K is also called the blur matrix. In spatially invariant blur, the rows have the same value and move in position. The spatially varying kernel can be a row-wise motion blur kernel, so for pixels in the same row, the rows of K corresponding to these pixels are the same and move in position.

[0081] Given a blurred image B, a blur matrix K and noise levels Noise(x,y)~N(0,σ 2 ), the solution of the latent image L can be determined.

[0082] A dual camera system includes a primary camera with a rolling shutter and a secondary camera with a global shutter. In the dual camera system, there may be controlled synchronization between the rolling shutter of the primary camera and the global shutter of the secondary camera.

[0083] Fig. 9 An example process is depicted for mapping an estimated row-wise motion blur kernel to match the geometry of a primary image using a rolling shutter primary camera and a global shutter secondary camera. The process receives 910 secondary image frames from the secondary camera and performs motion analysis 916 on the secondary image frames, outputting a sequence of discrete motion offsets. The timing and exposure 912 between the primary camera and the secondary camera is received by the motion blur kernel estimation 918, which also receives the sequence of discrete motion offsets from the motion analysis 916. The motion blur kernel estimation 918 outputs a row-wise motion blur kernel and remaps 924 the row-wise motion blur kernel to the primary image frame. In addition to the remapped row-wise motion blur kernel, the spatially varying kernel deconvolution 920 also receives 914 a primary image frame 914 from the primary camera. The output 922 of the spatially varying kernel deconvolution 920 is a blur-reduced image.

[0084] Fig.10An example process is depicted for correcting secondary image frames to primary image geometry prior to motion analysis using a rolling shutter primary camera and a global shutter secondary camera. The process receives 1010 secondary image frames from the secondary camera, corrects 1024 the secondary image frames to primary image frames, and performs motion analysis 1016 on these secondary image frames, outputting a sequence of discrete motion offsets. Motion kernel estimation 1018 receives the timing and exposure 1012 between the primary camera and the secondary camera, and the motion kernel estimation 1018 also receives the sequence of discrete motion offsets from the motion analysis 1016 and outputs a row-wise motion blur kernel. In addition to the row-wise motion blur kernel, spatially varying kernel deconvolution 1020 also receives 1014 primary image frames from the primary camera. The output of the spatially varying kernel deconvolution 1020 is a blur-reduced image 1022.

[0085] Fig.11 An example process for generating a row-wise motion blur kernel is described. Discrete motion offset sequence 1110 and discrete sequence timing 1112 are received by interpolation module 1114. Interpolation module outputs to tangent vector estimation module 1116 and continuous global point spread function estimation and time density module 1118. Continuous global point spread function estimation and time density module 1118 receives discrete sequence timing 1112 output by interpolation module 1114 and tangent vector estimation module 1116, and sends its output to time density segmentation module 1120, which then outputs to row-wise motion blur kernel module 1122.

[0086] Fig.12 A first example method for simultaneously reducing motion blur and rolling shutter effects is depicted, including receiving 1210 a primary image frame, a primary timing, and a primary exposure from a primary camera, and receiving 1212 a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera. The method includes correcting 1214 the secondary image frame to the primary image frame, determining 1216 an incremental timing based on the primary timing and the secondary timing, and determining 1218 an incremental exposure based on the primary exposure and the secondary exposure. The method also includes determining 1220 a discrete motion of an offset sequence based on the secondary image frame, determining 1222 a row-wise motion blur kernel based on the primary image frame and the discrete motion of the offset sequence, and determining 1224 a spatially varying kernel deconvolution based on the primary image frame and the row-wise motion blur kernel. The method may also perform geometric correction of the primary image frame based on the secondary image frame.

[0087] Fig.13An example method of generating a temporal density segmentation is described, including determining 1310 a timing of discrete motion of an offset sequence, and interpolating 1312 a trajectory based on the discrete motion of the offset sequence and the timing of the discrete motion of the offset sequence. The method also includes estimating 1314 a tangent vector based on the interpolation, estimating 1316 a global point spread function and a temporal density based on the timing, trajectory, and tangent vector of the discrete motion of the offset sequence, and determining 1318 a temporal density segmentation.

[0088] Fig.14 A second example method of reducing motion blur and rolling shutter effects is depicted, including receiving 1410 a primary image frame, a primary timing, and a primary exposure from a primary camera, and receiving 1412 a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera. The method includes determining 1414 an incremental timing based on the primary timing and the secondary timing, determining 1416 an incremental exposure based on the primary exposure and the secondary exposure, and determining 1418 a discrete motion of an offset sequence based on the secondary image frame. The method also includes determining 1420 a row-wise motion blur kernel based on the primary image frame and the discrete motion of the offset sequence, remapping 1422 the row-wise motion blur kernel to the primary image frame, and determining 1424 a spatially varying kernel deconvolution based on the primary image frame and the row-wise motion blur kernel.

[0089] Those skilled in the art will appreciate that the various illustrative blocks, modules, elements, assemblies, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, assemblies, methods and algorithms have been described above generally according to their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the system. Skilled technicians can implement the described functions in different ways for each specific application. Without departing from the scope of the subject technology, various components and blocks can be arranged differently, for example, arranged in different orders, or divided in different ways.

[0090] It should be understood that the specific order or hierarchy of steps in the above disclosed process is an illustration of an exemplary method. Based on design preferences, it is understood that the specific order or hierarchy of steps in the process can be rearranged. Some steps can be performed simultaneously. The attached method claims present elements of the various steps in an exemplary order and are not meant to be limited to the specific order or hierarchy presented.

[0091] The foregoing description is used to enable any person skilled in the art to implement the various aspects described herein. The foregoing description provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the present application is not intended to be limited to the aspects shown herein, but to comply with the full scope consistent with the language claims, wherein unless specifically stated so, the elements mentioned in the singular form are not intended to represent "one and only one", but "one or more". Unless otherwise clearly stated, the term "some" refers to one or more. Positive pronouns (e.g., he) include female and neutral genders (e.g., she and it), and vice versa. Titles and subtitles, if any, are used only for convenience and do not limit the present invention. The words "configured to", "operated to" and "programmed to" do not mean any specific tangible or intangible modification to the subject matter, but are intended to be used interchangeably. For example, a processor or component configured to monitor and control operations can also mean a processor programmed to monitor and control operations or a processor operated to monitor and control operations. Likewise, a processor configured to execute code may be construed as a processor programmed to execute code or operated to execute code.

[0092] Words such as "aspects" do not mean that the aspect is essential to the subject technology or that the aspect applies to the configuration of the subject technology. Disclosures related to an aspect may apply to the configuration, or one or more configurations. An aspect may provide one or more examples. Phrases such as aspects may refer to one or more aspects, and vice versa. Words such as "embodiments" do not mean that such embodiments are essential to the subject technology or that such embodiments apply to the configuration of the subject technology. Disclosures related to embodiments may apply to embodiments or one or more embodiments. An embodiment may provide one or more examples. Words such as "embodiments" may refer to one or more embodiments, and vice versa. Words such as "configurations" do not mean that such configurations are essential to the subject technology or that such configurations apply to the configuration of the subject technology. Disclosures related to configurations may apply to configurations or one or more configurations. A configuration may provide one or more examples. Words such as "configurations" may refer to one or more configurations, and vice versa.

[0093] The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0094] The structural and functional equivalents of the elements of the various aspects described in this disclosure are known or will be known to those of ordinary skill in the art, are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, no matter whether such disclosure is explicitly listed in the claims, anything disclosed herein is not intended to be dedicated to the public. In addition, if "including", "having" or similar words are used in the specification or claims, the words are intended to be included in a manner similar to the word "comprising" when used as a transitional word of the claims.

[0095] References to "an embodiment," "one embodiment," "some embodiments," "various embodiments," etc. indicate that a particular element or feature is included in at least one embodiment of the present invention. Although these words may appear in different places, they do not necessarily refer to the same embodiment. In conjunction with this disclosure, those skilled in the art will be able to design and combine any of a variety of mechanisms suitable for achieving the above functions.

[0096] It should be understood that the present disclosure teaches only one example of illustrative embodiments and that those skilled in the art may readily devise many variations of the present invention after reading the present disclosure and that the scope of the present invention will be defined by the claims.

Claims

1. A method for reducing motion blur and rolling shutter effects, characterized in that: include: receiving a master image frame, a master timing, and a master exposure from a master camera; receiving a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera; Correcting the auxiliary image frame to the main image frame; determining an incremental timing based on the primary timing and the secondary timing; determining an incremental exposure based on the main exposure and the secondary exposure; determining discrete motion of an offset sequence based on the secondary image frame; Determining a row-wise motion blur kernel based on discrete motions of the primary image frame and the offset sequence; as well as A spatially varying kernel deconvolution is determined based on the primary image frame and the row-wise motion blur kernel.

2. The method for reducing motion blur and rolling shutter effect according to claim 1, characterized in that: Also includes: determining the timing of discrete movements of the offset sequence; a timed interpolated trajectory based on discrete motions of the offset sequence and discrete motions of the offset sequence; Estimate the tangent vector based on interpolation; estimating a global point spread function and temporal density based on the timing, trajectory and tangent vector of discrete motions of the offset sequence; and Determine the temporal density segmentation.

3. The method for reducing motion blur and rolling shutter effect according to claim 1, characterized in that: The resolution of the main camera is higher than that of the secondary camera.

4. The method for reducing motion blur and rolling shutter effect according to claim 1, characterized in that: The frame rate of the main camera is slower than the frame rate of the secondary camera.

5. The method for reducing motion blur and rolling shutter effect according to claim 1, characterized in that: The secondary camera has a monochrome filter.

6. The method of reducing motion blur and rolling shutter effect according to claim 1, characterized in that: Also includes: Calibrate the secondary camera to the primary camera; and The auxiliary image frame is corrected to the main image frame.

7. The method of reducing motion blur and rolling shutter effect according to claim 6, characterized in that: Calibration and correction are direct linear transformations.

8. A method for reducing motion blur and rolling shutter effects, characterized in that: include: receiving a master image frame, a master timing, and a master exposure from a master camera; receiving a secondary image frame, a secondary timing, and a secondary exposure from a secondary camera; determining an incremental timing based on the primary timing and the secondary timing; determining an incremental exposure based on the main exposure and the secondary exposure; determining discrete motion of an offset sequence based on the secondary image frame; Determining a row-wise motion blur kernel based on discrete motions of the primary image frame and the offset sequence; remapping the row-wise motion blur kernel to the primary image frame; as well as A spatially varying kernel deconvolution is determined based on the primary image frame and the row-wise motion blur kernel.

9. The method of reducing motion blur and rolling shutter effect according to claim 8, characterized in that: Also includes: determining the timing of discrete movements of the offset sequence; a timed interpolated trajectory based on discrete motions of the offset sequence and discrete motions of the offset sequence; Estimate the tangent vector based on interpolation; estimating a global point spread function and temporal density based on the timing, trajectory and tangent vector of discrete motions of the offset sequence; and Determine the temporal density segmentation.

10. The method of reducing motion blur and rolling shutter effect according to claim 8, characterized in that: Also includes: Calibrate the secondary camera to the primary camera; and The auxiliary image frame is corrected to the main image frame.

Images