How to register each pixel of an event camera to a frame camera.

Abstract

To provide a method, a stereo camera device and a program for registering pixels provided in a pixel event stream by an event-based camera into pixels in an image provided by a frame-based camera.SOLUTION: A stereo camera device includes generating each image frame acquired from a frame-based camera FMC by associating a first point matrix with an acquisition period of the image frame from one or more of image frames, acquiring a pixel event stream generated during the acquisition period, generating a second point matrix from a pixel event of a pixel event stream occurring during acquisition of the first point matrix, calculating a correlation scoring function to be applied to at least some of points of the first point matrix and the second point matrix, and maximizing the correlation scoring function to estimate the position of each of the points of the second point matrix in the first point matrix by the depth of the points of the first point matrix related to the second point matrix.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure relates to a method and apparatus for pixel-by-pixel registration of an event-based camera to a frame-based camera in the presence of a scene showing objects at different distances from the camera, using stereo mapping. [Background technology]

[0002] An event-based camera does not capture an image or frame using a shutter like the image sensor of a conventional frame-based camera, but includes a visual sensor where each pixel operates independently and asynchronously to respond to local changes in brightness, reporting when relative changes in brightness occur and remaining silent otherwise.

[0003] Image registration is the process of establishing a pixel-by-pixel mapping or link between objects captured by two cameras viewing the same scene. Such registration is required when sensors are incorporated into two separate cameras. Thus, the two cameras inevitably have different viewpoints. The scene depth (single or multiple) of objects viewed by the cameras from different viewpoints results in parallax, which in turn results in disparity in the image objects projected onto the focal plane of the camera sensors.

[0004] Such registrations are performed for event enhancement, such as improving images and videos using events or event noise reduction. They are also performed to generate depth or disparity maps for use in any application requiring 3D information of the scene. Solutions exist for 3D estimation involving two or more cameras, either frame-based or event-based, of the same type.

[0005] Traditional stereo vision uses two cameras spaced apart and aligned horizontally to obtain two different field-of-view images of a scene, similar to human binocular vision. By correlating the objects captured in these two images, relative depth information can be obtained in the form of a disparity map that encodes the difference in the horizontal coordinates of corresponding image points. The value of the disparity map is inversely proportional to the scene depth at the corresponding pixel position.

[0006] For a human to compare two images, the images must be superimposed using a stereoscopic device, with the image from the right camera shown to the observer's right eye and the image from the left camera shown to their left eye. In computer vision systems, several pre-processing steps may be performed: 1. Distortions are removed from each image so that barrel distortion and tangential distortion are eliminated. This ensures that the observed image matches the projection of an ideal pinhole camera. 2. Each image is projected onto a common plane to allow for comparison of image pairs; this is known as image parallelization. 3. The information measure comparing the two images is minimized. This gives the best estimate of the location of features in the two images and creates a disparity map. 4. Optionally, a disparity map is projected onto the 3D point cloud to form a depth map. By using the camera's projection parameters, the point cloud can be calculated to provide measurements at a known scale.

[0007] Many solutions already exist for registering between two frame-based cameras. These solutions have been developed to extract depth information from dual-camera systems using the principle of binocular vision ([1][2][3][4][5]). These solutions often require the following steps: Camera calibration: This process is performed offline and requires a thorough understanding of the camera system. Camera parallelization: This process modifies the camera's viewpoint so that the search for corresponding pixels is performed along the pixel rows of the sensor. Without camera parallelization, the search can still be performed by calculating the epipolar curve per pixel, which does not align with the pixel rows of the sensor. Therefore, the parallelization algorithm becomes less efficient. Matching: Matching often involves an optimization algorithm that attempts to minimize the photometric distance between pixels, and further includes a normalization term that smoothly varies the shift (also called parallax) between pixels.

[0008] [6, 14] shows that it is possible to use temporal matching instead of photometric matching to register between two event-based cameras.

[0009] Due to the differing nature of the signals coming from the cameras, these methods, commonly known as stereo methods, cannot be easily generalized to a camera pair that combines one frame-based camera with one event-based camera. A frame-based camera's 2D array of pixels generates a 2D image by measuring the intensity (or color intensity) of incident light at regularly sampled time intervals, while an event-based camera does not generate a 2D image but generates a stream of discrete events, which are generated asynchronously as soon as relative brightness changes of incident light are detected.

[0010] In addition, event-based cameras may have a different (typically lower) spatial resolution than frame-based cameras. Therefore, since one pixel in an event-based camera may correspond to many pixels in a frame-based camera, for example, the mapping between the two cameras cannot be bijective.

[0011] Some frame-based cameras do not capture a snapshot of the entire scene at a single moment as when using a global shutter, but instead have a rolling shutter sensor that captures each frame by rapidly scanning vertically or horizontally across the scene. In general, one pixel row of the rolling shutter sensor or one group of several pixel rows is scanned at a single moment, and multiple pixel rows of the sensor or multiple groups of pixel rows are scanned at different moments. If not explicitly addressed, the rolling shutter may bias the depth estimated in the stereo system, as shown in [5].

[0012] Accordingly, it may be desirable to establish pixel-by-pixel registration between pixel events from an event-based sensor and images from a frame-based sensor. It may also be desirable to address cases where the frame-based camera has a rolling shutter sensor. SUMMARY OF THE INVENTION

[0013] A method for registering pixels provided in a pixel event stream by an event-based camera to pixels in an image provided by a frame-based camera is described. The method may include obtaining image frames from the frame-based camera, wherein each image frame is generated using an exposure period, generating a first point matrix from one or more of the image frames, wherein the first point matrix is associated with an acquisition period of the image frames, obtaining the pixel event stream generated during the acquisition period, generating a second point matrix from pixel events of the pixel event stream that occur during the acquisition period of the first point matrix, calculating a correlation scoring function applied to at least some of the points of the first and second point matrices, and estimating the respective position of each point of the second point matrix in the first point matrix according to the depth of the points of the first point matrix relative to the second point matrix by maximizing the correlation scoring function.

[0014] According to one embodiment, the method further includes generating a parallelized first point matrix obtained by projecting a first point matrix onto a common image plane at a first image resolution, and generating a parallelized second point matrix obtained by projecting a second point matrix onto a common image plane at a first image resolution, wherein the estimation of the position of each point in the second point matrix is ​​performed from the parallelized first and second point matrices.

[0015] According to one embodiment, the positions of points in a second point matrix in a first point matrix are represented by a disparity map or depth map of at least one patch of adjacent points, or used to generate a modified image based on one of the image frames.

[0016] According to the embodiment, the method further includes generating and deparallaxing a parallax or depth map, or deparallaxing the modified image to the image plane of a frame-based or event-based camera.

[0017] According to one embodiment, the correlation scoring function evaluates the smoothness of at least one patch of adjacent points in the combined point matrix resulting from the combination of the first and second point matrices.

[0018] According to the embodiment, the correlation scoring function is one of the following functions: the norm of the 2D gradient of the combined point matrix, the variance of the local patches of the combined point matrix, and a cross-correlation function applied to the first and second point matrices.

[0019] According to the embodiment, the method further includes generating a third point matrix by integrating pixel events from a pixel event stream occurring during the exposure period of a first point matrix; parallelizing the third point matrix to project the points of the third point matrix onto a common image plane at the resolution of the image frame; generating a transformed point matrix by transforming the points of the parallelized third point matrix by applying a point transformation defined by a disparity map; deparallelizing the transformed point matrix to project the transformed image onto the image plane of a frame-based camera; and combining one of the image frames used to generate the first point matrix with the deparallelized transformed point matrix to obtain a blur-free image.

[0020] According to one embodiment, the first point matrix is ​​one of the image frames acquired from a frame-based camera, and the second point matrix is ​​generated by integrating the pixel events from the pixel event stream that occur during the exposure period of the first point matrix.

[0021] According to the embodiment, the second or third point matrix is ​​a correction point matrix that is multiplied by one of the image frames to obtain a deblurred image, the deblurred image being generated at the frame rate of the image frame or at the interval between two consecutive image frames of the image frame to provide a slow-motion video stream at a frame rate higher than the frame rate of the image frame.

[0022] According to one embodiment, a first point matrix is ​​generated by calculating the ratio of two consecutive image frames acquired from a frame-based camera, and a second point matrix is ​​generated by integrating pixel events from a pixel event stream occurring during a period delimited by the moments corresponding to the two consecutive image frames.

[0023] According to the embodiment, the correlation scoring function is one of the following functions: a cross-correlation function applied to at least one patch of adjacent points in the first and second point matrices; a sum of point-by-point absolute differences applied to at least one patch of adjacent points in the first and second point matrices; and a sum of point-by-point square differences applied to at least one patch of adjacent points in the first and second point matrices.

[0024] According to the embodiment, the points of the first point matrix have binary values ​​determined based on a comparison made between corresponding pixels in two consecutive image frames acquired from a frame-based camera, and the points of the second point matrix have binary values ​​set to 0 or 1 depending on the presence of at least one event at the corresponding pixel of an event stream during a period separated by moments corresponding to the two consecutive image frames.

[0025] According to one embodiment, the scoring function evaluates the similarity between the first and second point matrices by using logical operations applied to adjacent point patches of the first and second point matrices.

[0026] Embodiments may also relate to stereo camera devices including frame-based cameras, event-based cameras, and a processor for receiving and processing signals provided by the frame-based cameras and event-based cameras, wherein the processor is configured to implement steps of the method as defined above.

[0027] The embodiment may also relate to a computer program product that includes a portion of code that can be directly loaded into the computer's internal memory and, when executed by the computer, configures the computer to implement steps of the method defined above.

[0028] The methods and / or apparatus described herein may be better understood with reference to the following drawings and descriptions. A non-limiting and non-exclusive description is provided with the following drawings. In the drawings, similar reference numerals may refer to the same parts through different drawings unless otherwise specified. [Brief explanation of the drawing]

[0029] [Figure 1] This is a block diagram of a stereo camera system according to an embodiment. [Figure 2] This is a block diagram showing the steps of a method for registering each pixel of an event-based camera to a frame-based camera according to an embodiment. [Figure 3A] The time-varying curves for the ideal and real pixel signals, the correction signal, and the combined real and correction signals are shown, respectively. [Figure 3B] The time-varying curves for the ideal and real pixel signals, the correction signal, and the combined real and correction signals are shown, respectively. [Figure 3C] The time-varying curves for the ideal and real pixel signals, the correction signal, and the combined real and correction signals are shown, respectively. [Figure 4] This block diagram shows the integration or exposure period in rolling shutter cameras and global shutter cameras. [Figure 5] This is a block diagram showing the steps of a method for registering each pixel of an event-based camera to a frame-based camera according to another embodiment. [Figure 6] This is a block diagram showing the steps of a method for registering each pixel of an event-based camera to a frame-based camera according to another embodiment. [Modes for carrying out the invention]

[0030] Figure 1 shows a stereo camera system SVS including a frame-based camera FMC, an event-based camera EVC, and a processor PRC that receives and processes signals provided by the cameras FMC and EVC. The frame-based camera FMC and the event-based camera EVC observe the same scene, including objects at different distances from the cameras FMC and EVC. The frame-based camera FMC captures and generates image frames of the scene. The image frames generated by the camera FMC may be grayscale or color images such as RGB images. The event-based camera EVC generates a stream of asynchronous events related to moving objects detected in the scene and captured during image frame acquisition. Each of the frame-based camera FMC and the event-based camera EVC is incorporated into the camera system SVS so as to have their respective direction z1, z2 and respective image planes (O1, x1, y1) and (O2, x2, y2). Points O1 and O2 may be located at the center (or upper left corner) of the sensors of the cameras FMC and EVC, respectively. Points O1 and O2 are separated by a distance dx from each other. The indicated directions z1 and z2 are not necessarily the same. The same applies to the axes x1, x2 and axes y1, y2 of the image planes (O1, x1, y1) and (O2, x2, y2). In addition, event-based vision sensors typically have lower resolution than frame-based sensors, and the lenses that focus the scene onto the respective image planes of the camera FMC and EVC may have their own distortions.

[0031] The following discloses different methods for pixel-by-pixel registration between pixels of a frame provided by a frame-based camera (FMC) and pixel events provided by an event-based camera (EVC). Such registration is necessary, in particular, because each of the cameras (FMC, EVC) has its own viewpoint, and the depth of objects in the observed scene introduces some parallax, which results in disparity or shift between the image objects formed on the sensors of the cameras (FMC, EVC) respectively.

[0032] The first crucial element of registration is the precise synchronization between events from the camera EVC and frames from the camera FMC. While the camera FMC may generate one image frame with a frame duration of 33ms, the camera EVC can generate 1.2·10 per second. 9 The event, namely, during the frame period 4.10 7 It is known that they can offer events.

[0033] According to the embodiment, the clocks of the camera FMC and EVC are synchronized using a synchronization signal transmitted from one of the camera FMC and EVC to the other camera. Such a synchronization signal may be a flash signal, typically provided by the camera FMC, that triggers a flash and records the start of the exposure time. If the camera FMC has a global shutter sensor, the exposure time of each image frame from the camera FMC is also transmitted to the processor PRC. If the camera FMC has a rolling shutter sensor, the exposure time of the first row and the start time of exposure of the last row of the image sensor relative to the start time of exposure of the first row are transmitted to the processor PRC.

[0034] A second crucial element of registration is determining the stereo matching distance between the pixels of the image frame and the events. According to the embodiment, the matching is determined between an image frame I and a set of events E that occur during the exposure period of image frame I. It is assumed that there are no significant depth changes in the observed scene during the exposure period of frame I, and as a result, the mapping between image frame I and the set of events E is constant.

[0035] Figure 2 shows steps S01-S08 of an embodiment of a method for pixel-by-pixel registration that can be implemented by the processor PRC. Registration takes place between pixels of a frame provided by a frame-based camera FMC and pixel events provided by an event-based camera EVC. The camera FMC generates an image frame I at instant f, where I(x1,y1) is the pixel coordinate of a pixel in the image plane (O1,x1,y1) of the camera FMC with respect to exposure time T (see Figure 1). The camera EVC generates a pixel event denoted as [x2,y2,t0,σ], where (x2,y2) is the pixel coordinate of a pixel in the image plane (O2,x2,y2) of the camera EVC, t0 is the time of the event, and σ is the polarity, equal to +1 if the luminance change is positive with respect to a logarithmic luminance change threshold c, and equal to -1 if it is negative. Note that the luminance threshold c may have different values ​​for positive and negative polarity, but this possibility is omitted for clarity.

[0036] In step S01, image frame I is parallelized to produce a parallelized image RI with a given size of W × H pixels, where the pixels of frame I are projected onto a common image plane (O,x,y). The parallelization calculation may include linear transformations, rotations around axes x1, y1, and z1, scaling transformations to obtain the image size W × H, and other transformations to correct image distortion. Some of the parameters defining these transformations can be obtained by calibration.

[0037] In parallel with step S01, in step S02, the point matrix J is constructed on the image plane (O2, x2, y2) of the camera FMC. Step S02 is performed using the instantaneous f and the duration T of the exposure period.

[0038] The relationship between a potentially blurred image I and the corresponding event is given by the following equation:

number

number

[0039] The luminance threshold c may have different values ​​for positive and negative polarity, but this possibility is omitted for clarity.

[0040] The mapping function m can theoretically be any arbitrary mapping from the coordinate plane (O1, x1, y1) of the frame-based camera FMC to the coordinate plane (O2, x2, y2) of the event-based camera EVC. However, it is constrained under the widely used pinhole camera model (see Chapter 6 of

[10] ) and the thin lens assumption (see Chapter 3 of

[11] ). In practice, the mapping function m accounts for and removes lens distortion (using a model such as that in

[12] ). According to the algorithm disclosed in

[13] , image rectification can ensure that a point from one camera can be found at the same vertical coordinate y in the other camera. Rectification applies homography R F and R E to each camera image from camera FMC and camera EVC, respectively. The transformation m is defined by the following equation: m(x,y,p)=dist E (R E -1 (R F dist F -1 ([x,y] T )+([p,0] T ))) for a pixel at coordinates (x,y), where dist E and dist F -1 are respectively the forward distortion function of camera EVC and the inverse distortion function of camera FMC, which model camera lens distortion that deviates from the perfect pinhole projection model, [x,y] T denotes the transpose of vector [x,y], and p is the parameter required to align the coordinates to the events. In the literature, p=d(x,y) is referred to as a "disparity map" and corresponds to the horizontal shift in a rectified camera system. Such a function can typically be obtained from a camera calibration procedure.

[0041] Equation (1) can be written as follows: [Mathematical Formula] Since h is an arbitrary time, equations (3) and (4) combine image I and the point matrix J to obtain image I(x1,y1,[fT / 2,f+T / 2]) at an exposure period T taken at time f to image I at an instantaneous exposure period at any time h s It can be used to generate (x1, y1, h).

[0042] Equations (3) and (4) make it possible to provide a single deblurred image Is for each blurred frame I, thereby providing a deblurred video stream having the same frame rate as the initial video stream from the frame-based camera. According to embodiments, equations (3) and (4) are applied multiple times by varying the value of h to obtain multiple images between each pair of consecutive image frames I, thereby providing a slow-motion video stream at a higher frame rate than that of image frames I.

[0043] Image J is calculated in step S02 using the above equation (4) in the image plane (O1, x1, y1).

[0044] In step S03, the point matrix J is parallelized to provide a parallelized point matrix RJ having a given size W × H points and where the points at coordinates (x,y) are located within the common image plane (O1,x,y). The parallelization of the images in steps S01 and S03 is performed so that the parallelized image RI and the parallelized point matrix RJ appear to have the same focal length.

[0045] Image I may be blurred due to moving objects in the scene observed during the exposure time operated by the frame-based camera FMC. The blur can be corrected by integrating the luminance changes recorded by the stream of events [x2, y2, t0, σ] during the frame-based exposure period T, using equation (3).

[0046] The points RJ(x,y,h,p) in the point matrix RJ represent the deblurring correction signal applied to the parallelized image RI. If the parameter p is correct, the position of points RJ(x,y,h,p) in the image RI(x,y) is properly evaluated to generate a deblurred image Is, which will have sharp edges along with a smooth intensity gradient for non-edge pixels. Smoothness is a measure of the similarity of color or gray levels in the immediate vicinity.

[0047] However, if the exposure duration T of the frame-based camera FMC is sufficiently short, the image frame I may not be blurred and may have sharp edges similar to those of the point matrix 1 / RJ(x,y,h,p). If the spatial alignment is correct between image I(x,y) and the point matrix RJ(x,y,h,p), the sharp structures will match, as opposed to having double edges or ghost edges due to misalignment, and the overall smoothness of image Is will be preserved. Measuring the smoothness of Is can be used to verify the spatial alignment between the pixels of image I and the events from the event-based camera EVC.

[0048] Figures 3A, 3B, and 3C show curves C1 to C5 of line values ​​for pixels in images I and Is, illustrating the purpose of spatial alignment according to the embodiment. In Figure 3A, curves C1 and C2 show the corresponding lines for pixels in the obtained image Is and blurred image I, respectively. The pixel lines appear sharper in image Is(C1) than in image I(C2). Figure 3B shows curve C3 representing the corresponding lines for points in the point matrix 1 / J defined to improve the sharpness of image I and obtain image Is. In the example of Figure 3B, curve C3 corresponds to the result of dividing each pixel of the sharp signal C1 in image Is by the blurred signal C2 in image I. Figure 3C shows curves C4 and C5 of the estimated corresponding lines for pixels in image Is, respectively, when the spatial alignment between image I and image Is is correct and when it is incorrect. Curve C5 shows that spatial misalignment or misalignment results in double / ghost edges, and therefore reduces the smoothness of the reconstructed signal. Therefore, the objective is to estimate the parameter p by maximizing the smoothness of RI(x,y) / RJ(x,y,h,p) for each pixel in at least one patch of adjacent pixels.

[0049] In step S04, smoothness is evaluated using a selected scoring function S(x,y,p) applied to the following function F: F(x,y,p)=RI(x,y) / RJ(x,y,h,p)

[0050] In step S05, the parameter p is estimated by maximizing or minimizing the scoring function S within a given disparity range for at least one patch of adjacent pixels. The estimated parameter p obtained may vary for each pixel (x,y) or may be calculated for at least one patch of adjacent pixels having the same transformation value.

[0051] The following scoring functions can be used to maximize smoothness: - The L1 or L2 norm of the 2D gradient of F(x,y,p) that is minimized:

number

[0052] The function F, variance, or cross-correlation function can be pre-transformed by a logarithmic function. Furthermore, the scoring function S may have a normalization term that smoothly modifies the transformation field, for example, by weighting the L1 norm of the gradient of the transformation map.

[0053] According to one embodiment, the smoothness of the transformation field can be enhanced by using a cost-aggregated method of the semi-global matching algorithm [1] and replacing the mutual information criterion with the L1 or L2 norm of the 2D gradient of the function F(x,y,p), or the variance of local patches, or a measure of the cross-correlation between patches of the image RI(x,y) and the point matrix 1 / RJ(x,y,h,p).

[0054] In another embodiment, for example, another function such as a logarithmic function is applied to the image RI(x,y) and the point matrix RJ(x,y,p), and as a result their relationship is log(I)-log(J)=log(I s The use of logarithmic functions can lighten the computation required to maximize smoothness when finding the best transformation m(x,y,p) by eliminating one division.

[0055] The method disclosed above for maximizing smoothness can be applied iteratively, where an approximation of the parameter p is first used and then iteratively refined using the results of the cost function applied in the previous iteration.

[0056] In another embodiment, the parameter p is determined using a function learned from many examples of ground truth or from a sharpness queue.

[0057] Steps S01 and S03 use a stereo frame parallelization mechanism to guide both image I and point matrix J to an intermediate resolution W × H, which is preferably the resolution of an event-based visual EVS that has fewer calculations (smaller) in the calculation of the parameter p or transformation map m(x,y,p). Even smaller resolutions may be selected for faster processing.

[0058] Note that parallelization steps S01 and S03 are optional. When image I and point matrix J are parallelized, the transformation map m(x,y,p) is given by m(x,y,d(x,y))=dist E (R E -1 (R F dist F -1 ([x,y] T )+([d(x,y),0] T It can be written as ))), where the parameter p=d(x,y) represents the parallax or shift along the x-axis.

[0059] If image I and the point matrix J are not parallelized, the parallax is no longer horizontal (along the x-coordinate), and the transformation m(x,y,p) is m(x,y,d x (x,y),d y (x,y) = dist E (dist F -1 ([x,y] T )+([d x (x,y),d y (x,y)] T It can be written as ))), and in the formula, d x (x,y) and d y (x,y) represents the parallax components of the pixel at coordinates (x,y) in the x and y directions. The set of parallax values ​​at each pixel position forms a parallax map.

[0060] According to the embodiment, the depth of an object imaged by a pixel is inversely proportional to its parallax on the sensor; therefore, the estimated parallax map or parameter p is used to determine the depth map for some or all pixel events provided by the camera EVC. The depth of a pixel at coordinates (x,y) is given by the following equation: d(x,y)=dx·fl / z (5) It can be calculated using the formula, where dx is the distance [O1,O2] between the origins O1 and O2 of the parallelized image plane of the camera FMC and EVC, fl is the focal length of the parallelized camera, and z is the depth of the object imaged by the pixel at coordinate (x,y). The parallax or depth map thus obtained is defined on the coordinate plane (O1,x,y). The parallax or depth map can be deparallelized and defined on the coordinate plane (O1,x1,y1) or (O2,x2,y2). The depth map gives the depth of moving objects in the frame provided by the camera FMC.

[0061] According to the embodiment, steps S06 to S08 are performed after step S05. In step S06, the point matrix RJ is corrected by applying the following equation: RJ'(x,y,h)=RJ(m(x,y,p),h) to each point RJ(x,y,h,p) of the point matrix RJ using the parameter p estimated in step S05. It should be noted here that the transformation m(x,y,p) does not necessarily provide an integer. Therefore, step S06 may result in interpolation calculations between pixels transformed by m(x,y,p) to obtain pixel values ​​with integer coordinates.

[0062] In step S07, the point matrix RJ' obtained in step S06 is oversampled back to the resolution of the frame-based camera FMC to obtain a deparallelized point matrix J' on the image plane (O1, x1, y1). Thus, the point matrix J' is defined on the image plane of the camera FMC. In step S08, the deblurred image Is is calculated by applying equation (3) to all pixels of image I and the points of the point matrix J', where J' represents the deblurred signal. Thus, each pixel of the deblurred image Is is calculated as follows: Is(x1,y1,h,p)=I(x1,y1) / J'(x1,y1,h,p)

[0063] For a rolling shutter frame-based camera, the point matrix J is given by the following equation in step S02:

number

[0064] Figure 4 shows the exposure timing for a vertical rolling shutter sensor. Each row of pixels in the rolling shutter sensor, R0, R1, ..., Rn, is exposed for the entire duration of the same exposure period Tr, indicated by the rectangle RE in Figure 4. The start of the exposure period RE for each row R0 to Rn is shifted by the rolling shutter skew Rs compared to the previous row. In contrast, the exposure periods for each pixel row of a global shutter sensor are all identical, as indicated by the single central rectangle GE in Figure 4. The duration T of the exposure period for the global shutter sensor may be the same as or different from the duration Tr of the row exposure period.

[0065] If the sensor has a vertical rolling shutter that starts from top to bottom, and the y-coordinate changes from 0 (top of the sensor) to height -1 (bottom of the sensor or last row), then the function Es(x,y) can be calculated as Es(x,y) = ts + y·Rs. The function Es(x,y) has the same value for all pixels in the same row. The function Ee(x,y) can be calculated as Ee(x,y) = Es(x,y) + Tr, where Tr is the duration of the exposure period for one row of image I. Alternatively, the function Ee can also be calculated as Ee(x,y) = te + y·Rs, where te is the end of the exposure period for the first row of frame I.

[0066] It can be noted that the point matrix J is defined only for points corresponding to pixel events that occur during the exposure period T. Other points are not defined and can be set to any value, such as 0 or the value of the nearest defined point.

[0067] The method in Figure 2 appears efficient when the frame exposure time T is long enough to record a sufficient number of events (e.g., three or more events) within the exposure time. In practice, if only a few events are recorded, there may not be enough events to match the image I recorded by the frame-based camera FMC. In this case, or more generally, it is desirable to register events occurring outside the frame exposure period T, and the integral of the brightness changes recorded by these events is given by the following equation: I(x1,y1,f+F)=I(x1,y1,f)e c·INT(x1,y1,f,f+F,p) (7) or

number

[0068] Figure 5 shows steps S11-S16 of another embodiment of a method for pixel-by-pixel registration between pixels of a frame provided by a frame-based camera FMC and pixel events provided by an event-based camera EVC, which can be implemented by a processor PRC. In step S11, two consecutive image frames I(f) and I(f+F) provided by the frame-based camera FMC are processed to create a point matrix R for each time f as follows:

number

[0069] Steps S13 and S14 may be performed in parallel with steps S11 and S12. In step S13, the point matrix K is constructed on the image plane (O2, x2, y2) of the camera EVC from the event stream [x2, y2, t0, σ] generated by the camera EVC. Step S13 is performed using the instant f and the frame period F provided by the camera FMC, and the point matrix K is given by the following equation: K(x1,y1,p)=e c·INT(x1,y1,f,f+F,p) (10) Calculated using INT(x1,y1,f,f+F,p), where INT(x1,y1,f,f+F,p) is defined by equation (2). The points in the point matrix K represent changes in luminance ratio recorded by pixel events from the event stream provided by the camera EVC, occurring during the frame duration F. Then, in step S14, the point matrix K is parallelized to give a parallelized point matrix RK having a given size of W×H points, where the points are located within a common image plane (O1,x,y).

[0070] In step S15, a scoring function S(RR(x,y),RK(x,y,p)) is evaluated to compare the image RR and the point matrix RK. The scoring function S can evaluate the correlation between adjacent point patches of the point matrices RR and RK by summing the point-by-point products of their values, or by summing the point-by-point absolute or squared differences.

[0071] In another example, the scoring function is the following function F': F'(x,y,p)=RR(x,y) / RK(x,y,p) This can be applied to the equation, where the value of the parameter p is estimated such that for each point in at least one patch of adjacent points, F'(x,y,p) is as close to 1 as possible.

[0072] In step S16, the parameter p is estimated for each point in one or more patches of adjacent points in the parallelized point matrices RR and RK. This operation can be performed by maximizing a scoring function S when evaluating the correlation between points in the point matrices RR and RK. The correlation between points can be evaluated by calculating the sum of the absolute or squared differences between adjacent patches of points in both point matrices. In this case, a small sum of the calculated differences indicates a high correlation.

[0073] As shown in Figure 2, the parameter p may be used to calculate a parallax or depth map for one or more pixel patches of moving objects present in a frame provided by, for example, the camera FMC, or for all image pixels. The parallax or depth map may then be deparallelized to have dimensions of the image plane (O2, x2, y2) of the event-based camera EVC. The deparallelization operation may also be performed to have dimensions of the image plane (O1, x1, y1) of the frame-based camera FMC.

[0074] If the frame camera FMC uses a rolling shutter sensor, equation (10) is as follows: K(x1,y1,p)=e c·INT(x1,y1,Ee(x1,y1),EsF(x1,y1),p) (11) In this equation, Ee(x1,y1) and EsF(x1,y1) are functions that give the end of the exposure period for pixel (x1,y1) in the first frame and the start of the exposure period for pixel (x1,y1) in the second frame, respectively.

[0075] Figure 6 shows steps S21-S26 of another embodiment of a method for pixel-by-pixel registration between pixels of a frame provided by a frame-based camera FMC and pixel events provided by an event-based camera EVC, which can be implemented by a processor PRC. The method in Figure 6 has the advantage of having a much lower computational complexity than the methods in Figures 2 and 5. In step S21, two consecutive image frames I(x1,y1,f) and I(x1,y1,f+F) provided by the frame-based camera FMC are processed to form a binary point matrix BF defined as follows:

number

[0076] Therefore, when comparing corresponding pixels at the same coordinates (x1,y1) in images I(f) and I(f+f), if a change in brightness occurs in the logarithmic domain, the value of the point at coordinates (x1,y1) in the point matrix BF is set to 1; otherwise, it is set to 0. The binary point matrix BF is calculated on the image plane (O1,x1,y1) of the camera FMC. Then, in step S22, the point matrix BF is parallelized to provide a parallelized point matrix RBF having a given size of W×H points and where the points are located within the common image plane (O,x,y). The origin O may be chosen to coincide with point O1.

[0077] Steps S23 and S24 may be performed in parallel with steps S21 and S22. In step S23, the binary point matrix BE is constructed on the image plane (O2, x2, y2) of the camera EVC from the event stream [x2, y2, t0, σ] generated by the camera EVC. Step S23 is performed using the instant f and the frame period F provided by the camera FMC, and the point matrix BE is given by the following equation:

number

[0078] Next, in step S24, the point matrix BE is parallelized to provide a parallelized point matrix RBE having a given size W × H of points and where the points are located within a common image plane (O,x,y). According to the embodiment, the parallelization is performed in such a manner that the points in the resulting parallelized point matrix RBE remain either 0 or 1.

[0079] In step S25, the correlation between the point matrix RBF and the point matrix RBE is evaluated using the selected scoring function S(RBF(x,y),RBE(x,y,p)).

[0080] For example, the scoring function S is the following function F1: F1(x,y,p)=RBF(x,y)-RBE(x,y,p) This applies to the matrix where the value RBF(x,y) or RBE(x,y,p) for each point is equal to either 0 or 1. In this case, the scoring function S can be the sum of the absolute or square values ​​of F1 across one or more patches of adjacent points. Thus, the scoring function S(F1) should estimate the noncorrelation between the point matrix RBF and the point matrix RBE and be minimized over at least one patch of adjacent points.

[0081] In another example, the scoring function S is the following function F2: F2(x,y,p)=RBF(x,y)(AND)RBE(x,y,p) This applies to the following: The scoring function S may be equal to the number of F2 values ​​equal to 1 across one or more patches of points. In this case, the scoring function S(F2) should estimate the correlation between the point matrix RBF and the point matrix RBE and be maximized for at least one patch of adjacent points.

[0082] In step S26, the parameter p is estimated over one or more patches of adjacent points in the parallelized point matrices RBF and RBE. This operation can be performed by minimizing the scoring function S(F1) or maximizing the scoring function S(F2).

[0083] As shown in Figure 2, the parameter p can be used to calculate a disparity map d(x,y) or depth map for one or more pixel patches of moving objects present in a frame provided by, for example, the camera FMC, or for all image pixels. The disparity or depth map can then be deparallelized to a disparity or depth map in the image plane (O2,x2,y2) of the event-based camera EVC or in the image plane (O1,x1,y1) with respect to the dimensions of the image frame of the camera FMC.

[0084] Steps S02, S03, and S05-S08 are performed after step S16 or S26 to remove blur from image frame I.

[0085] It should be noted that the methods in Figures 5 and 6 do not consider events occurring during the exposure period of the corresponding frames, but they do consider all events occurring between the two frames.

[0086] The descriptions provided herein are intended to provide a general understanding of the structures of various embodiments. These descriptions are not intended to serve as a complete description of all elements and features of devices, processors, and systems that utilize the structures or methods described herein. Many other embodiments or combinations thereof may become apparent to those skilled in the art when considering this disclosure in conjunction with the disclosed embodiments. Other embodiments can be utilized and derived from this disclosure, and as a result, structural and logical substitutions and modifications can be made without departing from the scope of this disclosure.

[0087] In this regard, the methods disclosed in Figures 2, 5, and 6 can be performed without calculating the positions (x,y) of all points for the complete point matrices RJ, RK, or RBE. In addition, it is not necessary to save all pixel positions (x,y) and disparity maps in order to calculate the points of the point matrix RJ' in step S06 (Figure 2) or to calculate the depth map.

[0088] Furthermore, this disclosure and description should be considered illustrative rather than restrictive, and the attached claims are intended to include all such modifications, enhancements, and other embodiments that fall within the true intent and scope of the description. Accordingly, the following claims should be determined by the broadest permissible interpretation of the claims and their equivalents, and should not be limited or restricted by the foregoing description.

[0089] References [1] H. Hirschmuller, "Stereo processing by semiglobal matching and mutual information". IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328-341 (2007) [2] G. Van Meerbergen, M. Vergauwen, M. Pollefeys et al. "A Hierarchical Symmetric Stereo Algorithm Using Dynamic Programming", International Journal of Computer Vision, 47, 275-285 (2002). [3] Jian Sun, Nan-Ning Zheng and Heung-Yeung Shum, "Stereo matching using belief propagation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003 [4] Y. Boykov, O. Veksler and R. Zabih, "Fast approximate energy minimization via graph cuts," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001 [5] O. Saurer, K. Koser, J. Bouguet and M. Pollefeys, "Rolling Shutter Stereo," IEEE International Conference on Computer Vision, Sydney, NSW, pp. 465-472 (2013) [6] P. Rogister, R. Benosman, S. Ieng, P. Lichtsteiner and T. Delbruck, "Asynchronous Event-Based Binocular Stereo Matching," IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no. 2, pp. 347-353, Feb. 2012 [7] L. Pan, R. Hartley, C. Scheerlinck, M. Liu, X. Yu and Y. Dai, "High Frame Rate Video Reconstruction based on an Event Camera," IEEE Transactions on Pattern Analysis and Machine Intelligence (2020) [8] G. Gallego, M. Gehrig and D. Scaramuzza, "Focus Is All You Need: Loss Functions for Event-Based Vision," IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 12272-12281 (2019) [9] D. Perrone, P. Favaro, “A Logarithmic Image Prior for Blind Deconvolution”, Int J Comput Vis 117, 159-172 (2016)

[10] R. Hartley and A. Zisserman, "Multiple View Geometry in Computer Vision" (2nd. ed.), 2003, Cambridge University Press, USA.

[11] Y. Ma, S. Soatto, J. Koseck, and S. Shankar Sastry, "An Invitation to 3-D Vision: From Images to Geometric Models", 2010, Springer Publishing Co., Inc.

[12] J. Weng, P. Cohen, and M. Herniou, "Camera calibration with distortion models and accuracy evaluation", IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(10):965-980, Oct. 1992

[13] M. Pollefeys, R. Koch, L. Van Gool, "A simple and efficient rectification method for general motion", Proc. International Conference on Computer Vision, 496-501, (1999), Retrieved 2011-01-19

[14] R. Benosman, S. Sio-Hoi Ieng, P. Rogister and C. Posch, "Asynchronous Event-Based Hebbian Epipolar Geometry", in IEEE Transactions on Neural Networks, vol. 22, no. 11, pp. 1723-1734, Nov. 2011

Claims

1. A method for registering pixels provided to a pixel event stream by an event-based camera to pixels in an image provided by a frame-based camera, wherein the method is: This involves acquiring image frames from a frame-based camera, where each image frame is generated using an exposure period. The method involves generating a first point matrix from one or more of the aforementioned image frames, wherein the first point matrix is ​​associated with the acquisition period of the aforementioned image frames. The acquisition of the pixel event stream generated during the aforementioned acquisition period, A second point matrix is ​​generated from the pixel events of the pixel event stream that occur during the acquisition period of the first point matrix, The calculation of a correlation scoring function applied to at least some of the points in the first and second point matrices, By maximizing the correlation scoring function, the position of each point in the second point matrix is ​​estimated based on the depth of the points in the first point matrix related to the second point matrix. Methods that include...

2. To generate a parallelized first point matrix obtained by projecting the first point matrix onto a common image plane with a first image resolution, The method involves generating a parallelized second point matrix obtained by projecting the second point matrix onto the common image plane at the first image resolution, wherein the estimation of the position of each point in the second point matrix is ​​performed from the parallelized first and second point matrices. The method according to claim 1, further comprising:

3. The method according to claim 1 or 2, wherein the position of a point in the second point matrix in the first point matrix is ​​represented by a disparity map or depth map of at least one patch of adjacent points, or is used to generate a modified image based on one of the image frames.

4. The method according to claim 3, further comprising generating and deparallelizing the disparity map or the depth map, or deparallelizing the modified image to the image plane of the frame-based camera or the event-based camera.

5. The method according to claim 2, wherein the correlation scoring function evaluates the smoothness of at least one patch of adjacent points in the combined point matrix resulting from the combination of the first and second point matrices.

6. The correlation scoring function is as follows: The norm of the 2D gradient of the aforementioned combined point matrix, The dispersion of local patches of the combined point matrix, and Cross-correlation function applied to the first and second point matrices The method according to claim 5, which is one of the methods.

7. A third point matrix is ​​generated by integrating the pixel events from the pixel event stream that occur during the exposure period of the first point matrix, The third point matrix is ​​parallelized, and the points of the third point matrix are projected onto the common image plane with the resolution of the image frame, The transformed point matrix is ​​generated by applying a point transformation defined by the disparity map to transform the points of the parallelized third point matrix, The transformed point matrix is ​​deparallelized, and the transformed image is projected onto the image plane of the frame-based camera. The method involves combining one of the image frames used to generate the first point matrix with the deparallelized transformed point matrix to obtain a blur-free image. The method according to claim 2, further comprising:

8. The method according to claim 1, wherein the first point matrix is ​​one of the image frames acquired from the frame-based camera, and the second point matrix is ​​generated by integrating pixel events from the pixel event stream that occur during the exposure period of the first point matrix.

9. The method according to claim 8, wherein the second or third point matrix is ​​a correction point matrix multiplied by one of the image frames to obtain a deblurred image, the deblurred image is generated at the frame rate of the image frame or at the interval between two consecutive image frames of the image frame to provide a slow-motion video stream at a frame rate higher than the frame rate of the image frame.

10. The method according to claim 1, wherein the first point matrix is ​​generated by calculating the ratio of two consecutive image frames acquired from the frame-based camera, and the second point matrix is ​​generated by integrating pixel events from the pixel event stream occurring during a period delimited by the moments corresponding to the two consecutive image frames.

11. The correlation scoring function is as follows: A cross-correlation function applied to at least one patch of adjacent points in the first and second point matrices, The sum of the absolute differences between points applied to at least one patch of adjacent points in the first and second point matrices, and The sum of the point-by-point squared differences applied to at least one patch of adjacent points in the first and second point matrices. The method according to claim 10, which is one of the methods.

12. The method according to claim 1, wherein the points in the first point matrix have binary values ​​determined based on a comparison made between corresponding pixels in two consecutive image frames acquired from the frame-based camera, and the points in the second point matrix have binary values ​​set to 0 or 1 depending on the presence of at least one event at a corresponding pixel in an event stream during a period separated by moments corresponding to the two consecutive image frames.

13. The method according to claim 12, wherein the correlation scoring function evaluates the similarity between the first and second point matrices using logical operations applied to adjacent point patches of the first and second point matrices.

14. A stereo camera apparatus comprising a frame-based camera, an event-based camera, and a processor for receiving and processing signals provided by the frame-based camera and the event-based camera, wherein the processor is configured to implement steps of the method according to any one of claims 1, 2, 4, 6, 8, 9, 10, 11, 12, or 13.

15. A computer program product comprising a code portion that can be directly loaded into the internal memory of the computer and, when executed by the computer, constitutes the computer to implement the steps of the method according to any one of claims 1, 2, 4, 6, 8, 9, 10, 11, 12, or 13.

Images