A method for detecting motion artifacts in a TOF camera using an event camera

Abstract

This invention relates to a method for detecting motion artifacts in a Time-of-Flight (TOF) camera using an event camera, comprising: acquiring a mapping relationship between pixel coordinates between the event camera and the TOF camera; acquiring events output by the event camera and taking pictures by the TOF camera simultaneously until the TOF camera obtains a TOF camera depth image; based on the mapping relationship, mapping each event output by the event camera in the first time period to the coordinate system of the TOF camera depth image to obtain the coordinates of each event mapped to the coordinate system of the TOF camera depth image; based on the coordinates of each event mapped to the coordinate system of the TOF camera depth image, determining the probability of a moving object at a specified pixel coordinate point corresponding to each event; and acquiring a TOF camera depth image with motion artifacts removed based on the TOF camera depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting motion artifacts in a TOF camera using an event camera. Background Technology

[0002] A TOF camera continuously captures multiple images and then uses the phase relationship between these images to calculate an intensity map and a depth map. However, when an object or the camera moves, pixels at the same location in these images may correspond to objects at different distances, leading to errors in the phase relationship and consequently causing ranging errors.

[0003] An event camera is a type of camera that differs from a traditional camera. Traditional cameras, such as Time-of-Flight (TOF) cameras, integrate the charge on pixels over a fixed period of time to output an image. Event cameras, however, record changes in light intensity on pixels. When a change in light exceeds a threshold, they output a signal containing a timestamp, pixel location, and whether the brightness has increased or decreased. A common method for removing motion artifacts is to compare the relationships between a set of data used in the TOF camera's imaging process to see if they meet expectations. For example, a four-phase camera might calculate the sum of phase 1 and phase 2, and the sum of phase 3 and phase 4. If the difference between the two sums is less than a threshold, it is retained; otherwise, it is discarded as outlier data due to motion artifacts.

[0004] Because of the signal quality issues of the phase map itself used in the existing technology that uses the theoretical relationship between the phases of the TOF camera itself and the threshold to remove motion artifacts, the phase itself will fluctuate. When the reflectivity of the object is low, the fluctuation will be greater. In this case, in order to avoid normal signals being rejected as motion artifacts, the threshold for judging motion artifacts must be increased. However, this will cause some data that should have been rejected as motion artifacts to be retained, affecting the subsequent use of the data. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for detecting motion artifacts of a TOF camera using an event camera, which solves the technical problem that the noise of the TOF camera phase map itself leads to inaccurate motion artifact detection in the prior art.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] This invention provides a method for detecting motion artifacts in a Time-of-Flight (TOF) camera using an event camera, the method comprising:

[0010] S1. Obtain the mapping relationship of pixel coordinates between the event camera and the TOF camera;

[0011] S2. Starting from the same time, acquire the events output by the event camera and the TOF camera to take pictures until the TOF camera obtains the TOF camera depth image;

[0012] S3. Based on the mapping relationship between the pixel coordinates of the event camera and the TOF camera, map each event output by the event camera in the first time period to the coordinate system of the TOF camera depth image to obtain the coordinates of each event mapped to the coordinate system of the TOF camera depth image.

[0013] S4. Based on the coordinates of each event mapped to the coordinate system of the TOF camera depth image, determine the probability of a moving object at a specified pixel coordinate point corresponding to each event.

[0014] S5. Based on the TOF camera depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold, obtain a TOF camera depth image with motion artifacts removed.

[0015] Preferably,

[0016] The first time period is the period from the start of shooting by the TOF camera until the end of the TOF camera depth image obtained by the TOF camera.

[0017] Preferably, S4 specifically includes:

[0018] S41. For each event mapped to the coordinates (x, y) of the TOF camera depth image, round down the horizontal coordinate x and the vertical coordinate y to obtain new horizontal coordinate x0 and vertical coordinate y0.

[0019] S42. For each event mapped to the new horizontal coordinate x0 and vertical coordinate y0 in the coordinate system of the TOF camera depth image, a pre-set calculation strategy is used to calculate the probability that there is a moving object at the pixel coordinate points (x0, y0), (x0, y1), (x1, y0), and (x1, y1), and the probability is accumulated on the preset initial image Mask to obtain the determined image Mask.

[0020] Let x0+1 be x1 and y0+1 be y1.

[0021] Preferably,

[0022] The initial image Mask has the same size as the TOF camera depth image, and the initial value of the initial image Mask is 0.

[0023] Preferably,

[0024] In S42, the probability that there is a moving object at pixel coordinate point (x0, y0) is 1-dx+1-dy;

[0025] Where dx = x - x0; dy = y - y0;

[0026] The probability that there is a moving object at pixel coordinates (x1, y1) is dx+dy;

[0027] The probability that there is a moving object at pixel coordinates (x0, y1) is 1-dx+dy;

[0028] The probability that there is a moving object at pixel coordinates (x1, y0) is dx+1-dy.

[0029] Preferably, S5 specifically includes:

[0030] S51. Determine whether the probability of each pixel in the TOF camera depth image and the corresponding pixel coordinate in the determined image Mask is less than the pre-specified threshold. If it is less than the pre-specified threshold, retain it; if it is greater than or equal to the pre-specified threshold, delete the pixel in the TOF camera depth image.

[0031] S52. Repeat step S51 until the probability of all pixels in the TOF camera depth image corresponding to the pixel coordinates in the determined image Mask is less than the pre-specified threshold, and a TOF camera depth image with motion artifacts removed is obtained.

[0032] Preferably,

[0033] The mapping relationship between the pixel coordinates of the event camera and the TOF camera is: HP = Q;

[0034] P is the coordinate system of the event camera;

[0035] Q is the coordinate system of the TOF camera;

[0036] H is the affine transformation matrix for the pixel coordinates from the event camera to the TOF camera.

[0037] Preferably, S1 specifically includes:

[0038] A checkerboard pattern was used to calibrate the mapping relationship of pixel coordinates between the event camera and the TOF camera.

[0039] Preferably,

[0040] The pre-specified threshold is M;

[0041] Where M = kW + d;

[0042] W is the average probability of all pixel coordinates in the determined image mask;

[0043] k is a pre-set coefficient;

[0044] d is a pre-defined constant.

[0045] (III) Beneficial Effects

[0046] The beneficial effects of this invention are as follows: This invention provides a method for detecting motion artifacts in a Time-of-Flight (TOF) camera using an event camera. By acquiring events output by the event camera and capturing images by the TOF camera simultaneously until the TOF camera obtains a TOF camera depth image, and based on the mapping relationship between the pixel coordinates of the event camera and the TOF camera, mapping each event output by the event camera in a first time period to the coordinate system of the TOF camera depth image, obtaining the coordinates of each event mapped to the TOF camera depth image coordinate system, and further determining the coordinates of each event mapped to the TOF camera depth image coordinate system based on the coordinates of each event mapped to the TOF camera depth image coordinate system. The probability of a moving object at a specified pixel coordinate point corresponding to an event is calculated. Finally, based on the TOF camera depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold, a TOF camera depth image with motion artifacts removed is obtained. Compared with the prior art, the method of detecting motion artifacts of a TOF camera using an event camera can take advantage of the fact that the event camera has a microsecond-level delay and only generates signals of pixel positions with changing brightness. The edges of objects will continuously generate signals when moving. Collecting these low-latency signals can be used as a basis for judging which pixels are unstable during the TOF camera imaging process. The effect is stable and very simple to implement. Attached Figure Description

[0047] Figure 1 This is a flowchart of a method for detecting motion artifacts of a TOF camera using an event camera, according to the present invention. Detailed Implementation

[0048] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0050] See Figure 1 This embodiment provides a method for detecting motion artifacts of a TOF camera using an event camera, the method comprising:

[0051] S1. Obtain the mapping relationship of pixel coordinates between the event camera and the TOF camera.

[0052] Specifically, the mapping relationship between the pixel coordinates of the event camera and the TOF camera is: HP = Q.

[0053] P is the coordinate system of the event camera.

[0054] Q represents the coordinate system of the TOF camera.

[0055] H is the affine transformation matrix for the pixel coordinates from the event camera to the TOF camera.

[0056] S2. Starting from the same time, acquire the events output by the event camera and the TOF camera to take pictures until the TOF camera obtains the TOF camera depth image.

[0057] S3. Based on the mapping relationship between the pixel coordinates of the event camera and the TOF camera, map each event output by the event camera in the first time period to the coordinate system of the TOF camera depth image to obtain the coordinates of each event mapped to the coordinate system of the TOF camera depth image.

[0058] The first time period is the period from the start of shooting by the TOF camera until the end of the TOF camera depth image obtained by the TOF camera.

[0059] S4. Based on the coordinates of each event mapped to the coordinate system of the TOF camera depth image, determine the probability of a moving object at a specified pixel coordinate point corresponding to each event.

[0060] S4 specifically includes:

[0061] S41. For each event mapped to the coordinates (x, y) of the TOF camera depth image, round down the horizontal coordinate x and the vertical coordinate y to obtain new horizontal coordinate x0 and vertical coordinate y0.

[0062] S42. For each event mapped to the new horizontal coordinate x0 and vertical coordinate y0 in the coordinate system of the TOF camera depth image, a pre-set calculation strategy is used to calculate the probability that there is a moving object at the pixel coordinate points (x0, y0), (x0, y1), (x1, y0), and (x1, y1), and the probability is accumulated on the preset initial image Mask to obtain the determined image Mask.

[0063] The initial image Mask has the same size as the TOF camera depth image, and the initial value of the initial image Mask is 0.

[0064] Let x0+1 be x1 and y0+1 be y1.

[0065] In S42, the probability that there is a moving object at pixel coordinate point (x0, y0) is 1-dx+1-dy;

[0066] Where dx = x - x0; dy = y - y0.

[0067] The probability of a moving object at pixel coordinates (x1, y1) is dx+dy.

[0068] The probability of a moving object at pixel coordinates (x0, y1) is 1-dx+dy.

[0069] The probability that there is a moving object at pixel coordinates (x1, y0) is dx+1-dy.

[0070] S5. Based on the TOF camera depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold, obtain a TOF camera depth image with motion artifacts removed.

[0071] In the practical application of this embodiment, S5 specifically includes:

[0072] S51. Determine whether the probability of each pixel in the TOF camera depth image corresponding to the pixel coordinates in the determined image Mask is less than the pre-specified threshold. If it is less than the pre-specified threshold, retain it; if it is greater than or equal to the pre-specified threshold, delete the pixel in the TOF camera depth image.

[0073] The pre-specified threshold is M.

[0074] Where M = kW + d.

[0075] W is the average probability of all pixel coordinates in the determined image mask.

[0076] k is a pre-set coefficient.

[0077] d is a pre-defined constant.

[0078] S52. Repeat step S51 until the probability of all pixels in the TOF camera depth image corresponding to the pixel coordinates in the determined image Mask is less than the pre-specified threshold, and a TOF camera depth image with motion artifacts removed is obtained.

[0079] Specifically, S1 includes:

[0080] A checkerboard pattern was used to calibrate the mapping relationship of pixel coordinates between the event camera and the TOF camera.

[0081] After fixing the event camera and TOF camera lenses in the same direction and minimizing the lens spacing as much as possible, the mapping relationship of pixel coordinates between the event camera and TOF camera is calibrated.

[0082] Switch the event camera to integration mode; in this mode, the event camera can be treated as a traditional camera.

[0083] Place a checkerboard grid so that it exists in the field of view of both cameras simultaneously. Record the coordinates of the checkerboard grid corner points as Pi in the event camera coordinate system and Qi in the TOF camera coordinate system, where i is the checkerboard grid corner point number.

[0084] Repeatedly move the chessboard grid to record more coordinates of the chessboard grid corner points in the event camera coordinate system and the coordinates of the chessboard grid corner points in the TOF camera coordinate system. After calibration, switch the event camera to event mode.

[0085] Using the coordinates of the checkerboard corner points in the event camera coordinate system and the coordinates of the checkerboard corner points in the TOF camera coordinate system, solve for the affine transformation matrix H of the pixel coordinates from the event camera to the TOF camera. The mapping relationship is HP = Q.

[0086] Meanwhile, in addition to using checkerboard corner points to calibrate the mapping relationship between the TOF camera and the event camera, other calibration objects can also be used. This embodiment does not impose any specific restrictions on this.

[0087] This embodiment presents a method for detecting motion artifacts in a Time-of-Flight (TOF) camera using an event camera. This method acquires events output by the event camera and captures images from the TOF camera simultaneously until a TOF depth image is obtained. Based on the pixel coordinate mapping between the event camera and the TOF camera, each event output by the event camera in the first time period is mapped to the coordinate system of the TOF depth image, obtaining the coordinates of each event mapped to the TOF depth image. Further, based on the coordinates of each event mapped to the TOF depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event is determined. Finally, based on the TOF depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold, a TOF depth image with motion artifacts removed is obtained. Compared to existing technologies, this method utilizes the fact that the event camera has a microsecond-level delay and only generates signals from pixel positions with varying brightness. As objects move, they continuously generate signals; collecting these low-latency signals can serve as a basis for determining which pixels are unstable during TOF camera imaging. The method is stable and very simple to implement.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0090] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0091] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0092] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for detecting motion artifacts in a TOF camera using an event camera, characterized in that, The method includes: S1. Obtain the mapping relationship of pixel coordinates between the event camera and the TOF camera; S2. Starting from the same time, acquire the events output by the event camera and the TOF camera to take pictures until the TOF camera obtains the TOF camera depth image; S3. Based on the mapping relationship between the pixel coordinates of the event camera and the TOF camera, map each event output by the event camera in the first time period to the coordinate system of the TOF camera depth image to obtain the coordinates of each event mapped to the coordinate system of the TOF camera depth image. S4. Based on the coordinates of each event mapped to the coordinate system of the TOF camera depth image, determine the probability of a moving object at a specified pixel coordinate point corresponding to each event. S5. Based on the TOF camera depth image, the probability of a moving object at a specified pixel coordinate point corresponding to each event, and a pre-specified threshold, obtain a TOF camera depth image with motion artifacts removed. S4 specifically includes: S41. For each event mapped to the coordinates (x, y) of the TOF camera depth image, round down the horizontal coordinate x and the vertical coordinate y to obtain new horizontal coordinate x0 and vertical coordinate y0. S42. For each event mapped to the new horizontal coordinate x0 and vertical coordinate y0 in the coordinate system of the TOF camera depth image, a pre-set calculation strategy is used to calculate the probability that there is a moving object at pixel coordinate points (x0, y0), (x0, y1), (x1, y0), and (x1, y1), and the probability is accumulated onto a preset initial image mask to obtain a determined image mask; wherein, x0+1 is x1 and y0+1 is y1; wherein, the initial image mask has the same size as the TOF camera depth image, and the initial value of the initial image mask is 0; In S42, the probability that there is a moving object at pixel coordinate point (x0, y0) is 1-dx+1-dy; Where dx = x - x0; dy = y - y0; the probability of a moving object at pixel coordinate point (x1, y1) is dx + dy; the probability of a moving object at pixel coordinate point (x0, y1) is 1 - dx + dy; and the probability of a moving object at pixel coordinate point (x1, y0) is dx + 1 - dy.

2. The method according to claim 1, characterized in that, The first time period is the period from the start of shooting by the TOF camera until the end of the TOF camera depth image obtained by the TOF camera.

3. The method according to claim 2, characterized in that, S5 specifically includes: S51. Determine whether the probability of each pixel in the TOF camera depth image and the corresponding pixel coordinate in the determined image Mask is less than the pre-specified threshold. If it is less than the pre-specified threshold, retain it; if it is greater than or equal to the pre-specified threshold, delete the pixel in the TOF camera depth image. S52. Repeat step S51 until the probability of all pixels in the TOF camera depth image corresponding to the pixel coordinates in the determined image Mask is less than the pre-specified threshold, and a TOF camera depth image with motion artifacts removed is obtained.

4. The method according to claim 3, characterized in that, The mapping relationship between the pixel coordinates of the event camera and the TOF camera is: HP = Q; P is the coordinate system of the event camera; Q is the coordinate system of the TOF camera; H is the affine transformation matrix for the pixel coordinates from the event camera to the TOF camera.

5. The method according to claim 4, characterized in that, S1 specifically includes: A checkerboard pattern was used to calibrate the mapping relationship of pixel coordinates between the event camera and the TOF camera.

6. The method according to claim 4, characterized in that, The pre-specified threshold is M; Where M = kW + d; W is the average probability of all pixel coordinates in the determined image mask; k is a pre-set coefficient; d is a pre-defined constant.

Images