Method and apparatus for determining relative motion between a time-of-flight camera and an object

Abstract

The present disclosure relates to a method and apparatus for determining relative motion between a time-of-flight camera and an object. A method for determining relative motion between a time-of-flight camera and an object sensed by the camera in a scene is provided. The object emits a modulated light signal. The method comprises: receiving a set of at least two original images of a scene from a ToF camera. The original images are based on a correlation of a modulated reference signal with a measurement signal of the ToF camera. The sets of at least two original images each include at least one original image. The measurement signal is based on a modulated light signal emitted by the object. The method comprises: each set of original images determines a value based on a corresponding set of original images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. The method comprises: determining information of relative motion between the ToF camera and the object based on the value indicating the phase difference. The method comprises: outputting information of relative motion between the ToF camera and the object.

Description

Technical Field

[0001] Examples relate to time-of-flight (ToF) sensing. Specifically, examples relate to a method and apparatus for determining relative motion between a ToF camera and an object sensed in a scene by the ToF camera. Background Art

[0002] In various applications, ToF sensors measure the distance to electronic devices. The range and image (frame) rate of ToF sensors are limited, which limits the ability of ToF sensors to sense the motion of objects relative to the ToF sensor. Summary of the invention

[0003] Therefore, there may be a need for an improved technique for determining relative motion between a ToF camera and an object.

[0004] This requirement can be met through the following topics.

[0005] One example relates to a method for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene. The object emits a modulated light signal. The method includes: receiving a set of at least two raw images of the scene from the ToF camera. The raw images are based on a correlation of a modulated reference signal with a measurement signal of the ToF camera. The sets of at least two raw images each include at least one raw image. The measurement signal is based on the modulated light signal emitted by the object. Further, the method includes: determining a value for each set of raw images based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. Additionally, the method includes: determining information about the relative motion between the ToF camera and the object based on the value indicating the phase difference. Further, the method includes: outputting information about the relative motion between the ToF camera and the object.

[0006] Another example relates to a device for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene. The object emits a modulated light signal. The device includes an input circuit configured to receive a set of multiple raw images of the scene from the ToF camera. The sets of at least two raw images each include at least one raw image. The raw image is based on a correlation between a modulated reference signal and a measurement signal of the ToF camera. The measurement signal is based on the modulated light signal emitted by the object. The device also includes a processing circuit configured to determine a value for each set of raw images based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. Additionally, the processing circuit is configured to determine information about the relative motion between the ToF camera and the object based on the value indicating the phase difference. Further, the device includes an output circuit configured to output information about the relative motion between the ToF camera and the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Some examples of apparatus and / or methods are described below by way of example only and with reference to the accompanying drawings, in which

[0008] Figure 1 A flow chart illustrating an example of a method for determining relative motion between a ToF camera and an object sensed in a scene by the ToF camera;

[0009] Figure 2 An example of a ToF camera sensing an object is shown;

[0010] Figure 3 Two exemplary phasors are shown; and

[0011] Figure 4 An example of an apparatus for determining relative motion between a ToF camera and an object sensed in a scene by the ToF camera is shown. DETAILED DESCRIPTION

[0012] Various examples will now be described more fully with reference to the accompanying drawings in which some examples are shown. In the accompanying drawings, the thickness of lines, layers and / or regions may be exaggerated for clarity.

[0013] Thus, although other examples can have various modifications and alternative forms, some specific examples thereof are shown in the drawings and will be described in detail later. However, this specific embodiment will not limit other examples to the specific forms described. Other examples may cover all modifications, equivalents and alternatives falling within the scope of the present disclosure. Throughout the description of the drawings, the same or similar reference numerals refer to similar or similar elements, which can be implemented in an equivalent or modified form when compared to each other while providing the same or similar functions.

[0014] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, these elements may be directly connected or coupled via one or more intermediate elements. If two elements A and B are combined using "or", it should be understood that all possible combinations are disclosed, i.e., only A, only B, and A and B (if not otherwise explicitly or implicitly defined). Alternative wording for the same combination is "at least one of A and B" or "A and / or B". The same applies to combinations of more than two elements, mutatis mutandis.

[0015] The terms used to describe specific examples herein are not intended to limit other examples. Whenever a singular form such as "one", "an" and "the" is used and only a single element is used, that is, it is not explicitly or implicitly defined as mandatory, other examples may also use multiple elements to implement the same function. Similarly, when a function is subsequently described as being implemented using multiple elements, other examples may use a single element or processing entity to implement the same function. It should also be understood that the terms "comprises", "comprising", "includes" and / or "including" specify the presence of the features, integers, steps, operations, processes, actions, elements and / or parts when used, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, actions, elements, parts and / or any groups thereof.

[0016] Unless otherwise defined, all terms (including technical and scientific terms) are used herein in accordance with the common meaning in the art to which the examples belong.

[0017] Figure 1 A flow chart of a method 100 for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene is shown. Figure 2 ,For pedagogical reasons, the following paragraphs will introduce some basics of traditional ToF depth measurement.

[0018] The ToF camera 200 includes an illumination element 210 for illuminating a scene with modulated light 211 (e.g., infrared light). The illumination element 210 generates the modulated light 211 based on an (electrically) modulated reference signal such as a continuous wave modulated signal (e.g., by controlling one or more light emitting diodes (LEDs) or one or more laser diodes based on the modulated reference signal). An object 230 in the scene illuminated by the modulated light 211 reflects at least a portion of the modulated light 211 back to a light capturing element 220 (e.g., including optics, an image sensor, and driver electronics) of the ToF camera 200. In other words, in conventional ToF sensing, the light capturing element 220 receives light reflected from the object 230.

[0019] The image sensor of the light capturing element 220 is pixelated and each pixel measures a portion of the reflected light. Thus, an (electrical) measurement signal is generated based on the light reflected from the scene. For example, each pixel may include a photonic mixer device (PMD) for measuring the reflected light.

[0020] According to the distance d between the ToF camera 200 and the object 230 obj , i.e., depending on the depth, the reflected light exhibits a delay relative to the emission of the modulated light 211. Thus, the measurement signal experiences a distance-dependent (depth-dependent) phase shift relative to the modulated reference signal.

[0021] In order to obtain the correlation value L of each pixel, according to the (auto)correlation function The modulated reference signal is correlated with the measured signal of the corresponding pixel. Correlation function The phase distance function is simulated, which describes the distance d between the modulated reference signal and the measured signal. obj The relationship between the phase.

[0022] Related functions The output of the PMD is a correlation value L for each pixel. Then, the determined correlation values ​​L are combined into an original image (sometimes also referred to as a "phase image"). That is, the original image includes a plurality of pixels, each of which represents a corresponding correlation value L. Each of the correlation values ​​indicates a corresponding phase difference between the reflected light (represented by the corresponding measurement signal) and the modulated reference signal. For example, if the PMD is used to measure the reflected light, the steps of generating the measurement signal and correlating the measurement signal with the modulated reference signal can be combined into a single step.

[0023] To sample the correlation function A plurality of raw images are generated. The phase shift δ between the modulated reference signal and the measurement signal used for correlation varies between the raw images. In other words, to obtain each raw image, a different phase shift δ is used to correlate the modulated signal with the measurement signal.

[0024] Sampling the same object at the same distance and exhibiting the same reflectivity enables the correlation function For example, phase shifts of 0°, 90°, 180°, and 270° can be used to generate four original images, each of which includes a corresponding correlation value L 0° , L 90° , L 180° and L 270° multiple pixels.

[0025] For the relevant value L 0° , related functions The distance-dependent phase shift between the measured signal and the modulated reference signal of the pixel is shifted relative to the zero value of the argument of the function

[0026] Use the pair correlation function The four relevant values ​​L for sampling 0° , L 90° , L 180° and L 270° , phase shift (phase angle) It can be determined as follows:

[0027]

[0028] Considering the speed of light c and the modulation frequency f of the emitted light 211 p (ie, the modulation frequency of the modulated reference signal), the distance d from the object 230 obj (ie, depth) can be calculated as follows:

[0029]

[0030] In contrast to conventional ToF depth sensing as described above, in method 100, object 230 actively emits modulated light 231, i.e., ToF camera 200 does not illuminate the scene for determining the relative motion between ToF camera 200 and object 230. The light received by the capture element 220 of the ToF camera is modulated light 231 actively emitted by object 230. Modulated light 231 may be, for example, an infrared signal emitted by one or more LEDs or laser diodes of object 230.

[0031] The method 100 comprises receiving a set of at least two raw images of a scene including an object 230 from a ToF camera 200. The sets of at least two raw images each comprise at least one raw image. In other words, each received set of raw images comprises one or more raw images (e.g., 2, 4, 8, or 16 raw images). Similar to the above, the raw images are based on a correlation of a modulated reference signal with a measurement signal of a capture element 220 of the ToF camera 200. Thus, the measurement signal is based on a modulated light signal 231 emitted by the object 230. Thus, the (correlation) values ​​represented by the pixels of one or more images of each set of raw images indicate a corresponding phase difference of the modulated light signal with respect to the modulated reference signal. For example, two phase images with a phase shift δ of 0° and 90° may be taken, or four phase images with a phase shift δ equidistant may be taken. The time difference (s) between the capture times of the respective raw images within the corresponding set of raw images may be smaller than the time difference between two immediately consecutive raw images belonging to different sets of raw images (e.g., the last raw image of the first set of raw images and the first raw image of the second set of raw images that follows it). Thus, the set of raw images characterizes the position of the object 230 relative to the ToF camera at different times.

[0032] Further, the method 100 comprises: determining 104 a value for each set of raw images based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. That is, for a first set of raw images in the set of at least two raw images, a first value indicating a first phase difference between the modulated light signal and the modulated reference signal is determined based only on one or more raw images in the first set of raw images, and for a second set of raw images in the set of at least two raw images, a second value indicating a second phase difference between the modulated light signal and the modulated reference signal is determined based only on one or more raw images in the second set of raw images.

[0033] The step of determining 104 a respective value indicating a phase difference between a modulated light signal and a modulated reference signal for a set of raw images of the set of at least two raw images may for example comprise: applying a calibration correction and / or an error correction to the value indicated by the pixel of one or more raw images of the respective set of raw images. For example, one or more raw images of the respective set of raw images may be corrected for fixed patterns, temperature variations, wiggles, etc.

[0034] If one of the sets of original images includes more than one original image, the step of determining 104 a corresponding value of a phase difference between a modulated light signal and a modulated reference signal of the set of original images may alternatively or additionally include: combining phase information contained in the original images in the set of original images. For example, the value indicating the phase difference between the modulated light signal and the modulated reference signal may be determined according to the above expression (1), wherein the correlation value in expression (1) may be an unmodified correlation value of the original image or a calibrated and / or error-corrected correlation value.

[0035] Additionally, the method 100 comprises determining 106 information about a relative motion between the ToF camera 200 and the object 230 based on the value indicative of the phase difference.

[0036] In other words, the ToF camera continuously captures the phase difference by capturing a collection of individual single or multiple raw images. Considering the speed of light c and the modulation frequency f of the modulated reference signal p , the change in phase difference can be converted into relative motion:

[0037]

[0038] where Δp represents the difference between the values ​​indicating the phase difference of two consecutive (eg, directly following) sets of original images, and Δd obj The representation indicates the change in the relative distance between the ToF camera 200 and the object 230 between the capture times of two consecutive sets of raw images.

[0039] Similarly, the relative motion speed Δv between the ToF camera 200 and the object 230 can be determined obj :

[0040]

[0041] where Δt represents the time difference between the capture times of two consecutive sets of raw images.

[0042] Therefore, the information about the relative motion between the ToF camera 200 and the object 230 may indicate that the relative distance between the ToF camera 200 and the object 230 changes by Δd obj Or relative speed Δv obj At least one of .

[0043] Further, the method 100 includes outputting 108 information about the relative motion between the ToF camera 200 and the object 230. For example, an image including a plurality of pixels indicating the motion between the ToF camera 200 and the object 230 may be output. Alternatively or additionally, a stream or an array of values ​​indicating the motion between the ToF camera 200 and the object 230 may be output.

[0044] The method 100 may enable the measurement of relative motion (e.g., relative distance change) between the ToF camera 200 and the modulated light source 230. In other words, the method 100 uses the ToF camera 200 and the object 230 in a collaborative manner by equipping the object 230 with an active illumination unit. Compared to conventional ToF depth sensing, the ToF camera 200 is able to sample the modulated light signal 231 of the cooperative object 230 over a wider range of distances with a shorter exposure time. Since the ToF camera 200 does not need to emit light, the sampling rate can be improved (increased) without considering eye safety. Even though the method 100 in its most basic implementation may not allow the determination of the absolute distance between the ToF camera 200 and the object 230, it may also allow the measurement of relative motion between the ToF camera 200 and the object 230.

[0045] In order to be able to correctly determine the corresponding value indicative of the phase difference, the modulation frequency of the modulated reference signal used by the light capturing element 220 of the ToF camera 200 is synchronized with the modulation frequency of the modulated light signal 231 emitted by the object 230. Therefore, the method 100 additionally includes: controlling the ToF camera 200 to synchronize the modulation frequency of the modulated reference signal with the modulation frequency of the modulated light signal 231 emitted by the object 230. In other words, the ToF camera approximates its pixel modulation clock to the frequency of the modulated light signal 231 emitted by the object 230 using a synchronization mechanism.

[0046] In practice, perfect synchronization is not possible, and thus the modulation frequency of the modulated reference signal used by the ToF camera 200 may differ from the modulation frequency of the modulated light signal 231 emitted by the object 230 (e.g., by a few Hertz). In order to be able to measure relative distance changes, synchronization errors need to be compensated. This is Figure 3 As exemplified in Figure 3 The phasor p of two successive sets of the original image is depicted 1 and p 2 .

[0047] A phasor (also called a phase vector) is a complex number that represents a ToF measurement. The phasor consists of an angle that describes the (depth / distance-dependent) phase shift of the measurement signal measured by the light-capturing element of the ToF camera relative to the modulated reference signal. Further, the phasor consists of a length that describes the intensity of the measurement signal (i.e., the measured intensity of the measured light). Since the phase angle is depth / distance-dependent, sensing objects at different distances relative to the ToF camera will cause the phasor to rotate.

[0048] Figure 3 The left part shows the phasor p of the first set of original images 1 . Phase p 1 Display Intensity I 1 and angle Figure 3 The right side shows the phasor p of the second set of successive sets for the original image. 2 . Phase p 2 Display Intensity I 2 and angle

[0049] The time span Δt occurs between the capture of the first set of raw images and the capture of the second set of raw images. As described above, the movement of the object 230 causes the phasor to rotate. Further, the synchronization error of the modulation frequency of the modulated reference signal used by the ToF camera 200 relative to the modulation frequency of the modulated light signal 231 emitted by the object 230 causes the phasor to rotate. In mathematical terms, this can be expressed as follows:

[0050]

[0051] in represents the phase shift caused by the relative motion between the ToF camera 200 and the object 230, and represents a phase shift caused by a synchronization error of the modulation frequency of the modulated reference signal used by the ToF camera 200 .

[0052] To obtain the correct phase for the second set of successive original images To measure the phase Subtract the phase shift caused by synchronization error from (therefore can be understood as phase error):

[0053]

[0054] Phase Error (e.g., in light spots) by the synchronization error f error (e.g., in Hz) and can be determined as follows:

[0055]

[0056] Therefore, the method 100 further includes: determining the synchronization error f based on the modulation frequency of the modulated reference signal relative to the modulation frequency of the modulated optical signal 231 emitted by the object 230. error That is, a compensated (synchronization error corrected) value indicating the phase difference is obtained so that the value caused by the synchronization error f can be compensated (corrected) after each measurement (for obtaining a set of original images). error Caused by phase drift.

[0057] Therefore, information on the relative motion between the ToF camera 200 and the object 230 is determined based on the compensation value.

[0058] The modulation frequency of the modulated reference signal can be synchronized with the modulation frequency of the modulated light signal 231 emitted by the object 230 in many different ways. In the following, some exemplary options for synchronization will be described. However, the proposed technology is not limited to the following examples.

[0059] In some examples, the modulation frequency of the modulated reference signal may be synchronized with the modulation frequency of the modulated light signal 231 emitted by the object 230 according to the method described in document DE 10 2017 105142 A1, which is incorporated herein by reference.

[0060] For example, the method 100 may include controlling the ToF camera 200 to synchronize a modulation frequency of a modulated reference signal with a radio frequency (RF) signal that can be received by the ToF camera 200 and the object 230. In other words, the two systems may synchronize their reference frequencies with a phase of a common RF signal such as a mobile communication signal, such as a mobile communication signal (e.g., a signal according to the 3rd Generation Partnership Project (3GPP) standard) or a global navigation satellite system (GNSS) signal, such as a global positioning system (GPS) signal.

[0061] Alternatively, the ToF camera 200 may provide an RF synchronization signal to the object 230, and vice versa. That is, in some examples, the method 100 may include: controlling the ToF camera 200 to synchronize the modulation frequency of the modulated reference signal with the RF reference signal; and controlling the ToF camera 200 to transmit the RF reference signal to the object 230. In other examples, the method 100 may include: controlling the ToF camera 200 to receive the RF reference signal from the object 230; and controlling the ToF camera 200 to synchronize the modulation frequency of the modulated reference signal with the received RF reference signal.

[0062] The following is a description of the synchronization error f for measuring the modulation frequency of the modulated reference signal. error However, it should be noted that method 100 is not limited to this technique.

[0063] The method 100 may further include determining a synchronization error f of a modulation frequency of the modulated reference signal based on a first auxiliary set of original images of the scene. error . The raw images in the first auxiliary raw image set are captured using the same modulated reference signal (i.e., a constant reference phase for the pixels of the light capturing element). The corresponding first signal intensities of the modulated light signal 231 emitted by the object 230 in each raw image of the first auxiliary set of raw images are determined. Further, the first waveform is determined using the corresponding first signal intensities for the modulated light signal 231 as a sample of the first waveform. In other words, the average signal value of the light source 230 is extracted from each image of the first auxiliary set of raw images and used as the estimated synchronization error f error That is, the average signal of the object 230 on the pixel array of the light capturing element 220 is placed in the waveform. Further, the synchronization error f is determined error The frequency of the first waveform is determined to be a synchronization error f of the modulation frequency of the modulated reference signal 231. error In other words, the frequency of the waveform determined by the average signal value of the object 230 is the synchronization error f error .

[0064] To increase the sampling rate, the ToF camera 200 may sample only sub-images. In other words, each raw image in the first auxiliary set of raw images may include fewer pixels than raw images in the first set and / or the second set of raw images for determining relative motion.

[0065] In order to determine the synchronization error f error Whether the sign is positive or negative, the frequency of the modulated reference signal is changed by a predetermined positive frequency step, and another series of original images can be captured.

[0066] In other words, determine the synchronization error f error The method further comprises: determining a corresponding second signal strength of the modulated light signal 231 in each raw image in the second auxiliary set of raw images of the scene. The raw images in the first auxiliary set of raw images and the second auxiliary set of raw images are based on the correlation of the measurement signal of the ToF camera with two different modulated reference signals, and the two different modulated reference signals show different modulation frequencies. Further, the synchronization error f is determined errorThe method comprises: determining a second waveform as a sample of the second waveform using a corresponding second signal strength for the modulated optical signal, and determining a frequency of the second waveform. Determining a synchronization error f of a modulation frequency of a modulated reference signal based on a comparison of the frequencies of the first waveform and the second waveform error For example, if the frequency of the second waveform is higher than the frequency of the first waveform, the synchronization error f errror The sign of can be determined to be positive and vice versa.

[0067] Once the synchronization error f is determined error The modulated reference signal can be reset to its original frequency by using the symbol of

[0068] The method 100 may also be used to infer a known absolute distance between the ToF camera 200 and the object 230. For example, the method 100 may further include receiving information about the absolute distance between the ToF camera 200 and the object 230. The method 100 may include updating the information about the absolute distance between the ToF camera 200 and the object 230 based on the determined information about the relative motion between the ToF camera 200 and the object 230.

[0069] For example, the object may transmit information about the absolute distance between the ToF camera 200 and the object 230 to the ToF camera 200. Alternatively, an element including / holding the ToF camera 200 may transmit information about the absolute distance between the ToF camera 200 and the object 230 to the ToF camera 200. The information may be transmitted to the ToF camera, for example, using RF communication or optical communication. In some examples, the information about the absolute distance between the ToF camera 200 and the object 230 may be encoded as a modulated light signal emitted by the object 230. In other words, relative motion determination may be combined with optical communication. For example, the modulated light signal 231 emitted by the object 230 may be modulated by information about the absolute distance between the ToF camera 200 and the object 230 by means of phase shift keying (PSK). The ToF camera 200 is capable of observing a phase difference for determining relative motion in the PSK modulated light signal.

[0070] Alternatively, the information about the absolute distance between the ToF camera 200 and the object 230 may be determined by the ToF camera 200 itself using conventional ToF depth sensing. Thus, in some examples, the method 100 may include: controlling the ToF camera 200 to determine the information about the absolute distance between the ToF camera 200 and the object 230 by illuminating the scene (using the modulated reference signal). Thus, the information about the absolute distance between the ToF camera 200 and the object 230 is updated again based on the information about the absolute distance between the ToF camera 200 and the object 230. The ToF camera 200 is controlled to illuminate the scene while the object 230 pauses emitting the modulated light signal 231, so that the conventional ToF depth measurement is not affected / interfered by the modulated light signal 231. In other words, the cooperating object 230 pauses the modulated light signal 231 for example for certain intervals to allow the ToF camera 200 to measure the absolute distance. After determining the absolute distance, the relative measurement may be used to infer the absolute distance. Thus, the amount of light emitted by the ToF camera 200 may be reduced.

[0071] The proposed technique can be used, for example, as a redundant high-frequency distance estimator between vehicles in traffic. Further, the proposed technique can be used in a control loop to regulate the distance between vehicles. If the proposed system is synchronized only by the ToF sensor synchronization method as described above, both speed and distance are relative to the start of the proposed phase tracking process. In vehicle-to-vehicle communication, active illumination of one of the vehicles can therefore transmit its current speed or distance via the emitted light (pulses). If the other car is within the range of the ToF camera of the other vehicle, the ToF camera of the other vehicle can make a reference depth measurement to derive the absolute speed or distance of the vehicle and infer it according to the proposed technique.

[0072] Further, the proposed technique may be used, for example, in motion tracking systems or indoor positioning systems.The proposed technique may allow absolute measurements to be inferred from relative measurements, thereby enabling high positioning update rates.

[0073] Since the proposed method also tracks the two-dimensional position of the object 230, a vector pointing in the direction in which the current object is moving can be determined and output. If the object 230 is located in three dimensions, the motion of the object 230 can be predicted.

[0074] Further, Figure 4An example of an apparatus 400 for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene according to the proposed concepts is shown in FIG. The apparatus 400 includes a processing circuit 420. The processing circuit 420 may be a single dedicated processor, a single shared processor, or multiple individual processors (some or all of which may be shared), a digital signal processor (DSP) hardware, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processing circuit 420 may optionally be coupled to a read-only memory (ROM), a random access memory (RAM), and / or a non-volatile memory, for example, for storing software. The apparatus 400 may also include other hardware (legacy and / or custom hardware).

[0075] The apparatus 400 comprises an input circuit 410 configured to receive input data 401 representing a set of at least two raw images of a scene from a ToF camera. A processing circuit 420 processes the input data 401 according to the above-mentioned concept for determining the relative motion between the ToF camera and the object. Thus, the output circuit 430 of the apparatus outputs output data 402 representing information about the relative motion between the ToF camera and the object.

[0076] For example, the functionality of apparatus 400 may be implemented in an application processor coupled to a ToF camera module that provides a set of at least two raw images, or in the ToF camera module itself.

[0077] The examples described in this article can be summarized as follows:

[0078] Some examples relate to a method for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene. The object emits a modulated light signal. The method includes: receiving a set of at least two raw images of the scene from the ToF camera. The raw images are based on a correlation of a modulated reference signal with a measurement signal of the ToF camera. The sets of at least two raw images each include at least one raw image. The measurement signal is based on the modulated light signal emitted by the object. Further, the method includes: determining a value for each set of raw images based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. Additionally, the method includes: determining information about the relative motion between the ToF camera and the object based on the value indicating the phase difference. Further, the method includes: outputting information about the relative motion between the ToF camera and the object.

[0079] In some examples, the method further includes: controlling the ToF camera to synchronize a modulation frequency of the modulated reference signal with a modulation frequency of the modulated light signal emitted by the object. Additionally, the method includes: modifying a value indicating a phase difference based on a synchronization error of the modulation frequency of the modulated reference signal to obtain a compensation value. Information about the relative motion between the ToF camera and the object is determined based on the compensation value.

[0080] According to some examples, controlling the ToF camera to synchronize a modulation frequency of a modulated reference signal with a modulation frequency of a modulated light signal emitted by the object includes controlling the ToF camera to synchronize a modulation frequency of the modulated reference signal with an RF signal receivable by the ToF camera and the object.

[0081] In some examples, controlling the ToF camera to synchronize a modulation frequency of a modulated reference signal with a modulation frequency of a modulated light signal emitted by the object includes: controlling the ToF camera to synchronize a modulation frequency of the modulated reference signal with an RF reference signal; and controlling the ToF camera to transmit the RF reference signal to the object.

[0082] According to some examples, controlling a ToF camera to synchronize a modulation frequency of a modulated reference signal with a modulation frequency of a modulated light signal emitted by an object includes: controlling the ToF camera to receive a radio frequency reference signal from the object; and controlling the ToF camera to synchronize the modulation frequency of the modulated reference signal with the radio frequency reference signal.

[0083] In some examples, the method further includes determining a synchronization error of a modulation frequency of the modulated reference signal based on the first auxiliary set of original images of the scene.

[0084] According to some examples, determining a synchronization error of a modulation frequency of a modulated reference signal includes: determining a corresponding first signal strength for a modulated light signal in each original image of a first auxiliary set of original images; determining a first waveform as a sample of the first waveform using the corresponding first signal strength of the modulated light signal; and determining the frequency of the first waveform as the synchronization error of the modulation frequency of the modulated reference signal.

[0085] In some examples, determining a synchronization error of a modulation frequency of a modulated reference signal also includes: determining a corresponding second signal strength for a modulated light signal in each original image of a second auxiliary set of original images of a scene, wherein the original images of the first auxiliary set of original images and the second auxiliary set of original images are based on correlations of a measurement signal of a ToF camera with two different modulated reference signals; determining a second waveform as a sample of the second waveform using the corresponding second signal strength for the modulated light signal; determining a frequency of the second waveform; and determining a sign of the synchronization error of the modulation frequency of the modulated reference signal based on a comparison of the frequencies of the first waveform and the second waveform.

[0086] According to some examples, two different modulated reference signals exhibit different modulation frequencies.

[0087] In some examples, the method further includes: receiving information about an absolute distance between the ToF camera and the object; and updating the information about the absolute distance between the ToF camera and the object based on the information about the relative motion between the ToF camera and the object.

[0088] According to some examples, information about the absolute distance between a ToF camera and an object is encoded as a modulated light signal emitted by the object.

[0089] In some examples, the method further includes: controlling the ToF camera to determine information about the absolute distance between the ToF camera and the object by illuminating the scene; and updating the information about the absolute distance between the ToF camera and the object based on the information about the relative motion between the ToF camera and the object.

[0090] According to some examples, the ToF camera is controlled to illuminate the scene while the object pauses emitting the modulated light signal.

[0091] In some examples, the information about the relative motion between the ToF camera and the object indicates at least one of a relative change in distance between the ToF camera and the object or a relative speed between the ToF camera and the object.

[0092] Another example relates to a device for determining relative motion between a time-of-flight camera and an object sensed by a ToF camera in a scene. The object emits a modulated light signal. The device includes an input circuit configured to receive a set of multiple raw images of a scene from a ToF camera. At least two sets of raw images each include at least one raw image. The raw image is based on a correlation between a modulated reference signal and a measurement signal of the ToF camera. The measurement signal is based on a modulated light signal emitted by the object. Additionally, the device includes a processing circuit configured to determine a value for each set of raw images based on a corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. Additionally, the processing circuit is configured to determine information about the relative motion between the time-of-flight camera and the object based on the value indicating the phase difference. Further, the device includes an output circuit configured to output information about the relative motion between the time-of-flight camera and the object.

[0093] Other examples relate to another apparatus for determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene. The object emits a modulated light signal. The apparatus includes a device for receiving a set of at least two raw images of the scene from the ToF camera. The sets of at least two raw images each include at least one raw image. The raw images are based on a correlation of a modulated reference signal with a measurement signal of the ToF camera. The measurement signal is based on a modulated light signal emitted by the object. Further, the apparatus includes a device for determining a value for each set of raw images based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated light signal and the modulated reference signal. Additionally, the apparatus includes a device for determining information about the relative motion between the ToF camera and the object based on the value indicating the phase difference. Further, the apparatus includes a device for outputting information about the relative motion between the ToF camera and the object.

[0094] An example relates to a non-transitory machine-readable medium having stored thereon a program having program code for performing, when the program is executed on a processor or programmable hardware, a method of determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene as described herein.

[0095] Other examples relate to a program having a program code for performing, when the program is executed on a processor or programmable hardware, a method of determining relative motion between a ToF camera and an object sensed by the ToF camera in a scene as described herein.

[0096] Examples according to the proposed concept may allow the use of a ToF sensor to measure the relative distance of the ToF sensor and a modulated light source. Using a modulated light source as an object may allow the capture of changes in relative distance at a longer distance range with a higher sampling rate of the ToF sensor.

[0097] The description and drawings illustrate only the principles of the present disclosure. Furthermore, all examples listed herein are in principle explicitly intended to be used for illustrative purposes only, to help the reader understand the principles of the present disclosure and the concepts contributed by the (multiple) inventors to further develop the art. All statements citing the principles, aspects and examples of the present disclosure and their specific examples herein are intended to cover their equivalents.

[0098] Block diagrams, for example, may illustrate high-level circuit diagrams that implement the principles of the present disclosure. Likewise, flow charts, flow diagrams, state transition diagrams, pseudocode, and the like may represent various processes, operations, or steps, which, for example, may be substantially represented in a computer-readable medium and thus executed by a computer or processor (whether or not such a computer or processor is explicitly shown). The methods disclosed herein may be implemented by an apparatus having means for performing each of the corresponding actions of the methods.

[0099] It should be understood that, unless otherwise explicitly or implicitly indicated for example for technical reasons, the disclosure of multiple actions, processes, operations, steps or functions disclosed herein may not be interpreted as being in a specific order. Therefore, unless these actions or functions are not interchangeable for technical reasons, the disclosure of multiple actions or functions will not limit them to a specific order. Further, in some examples, a single action, function, process, operation or step may include or may be decomposed into multiple sub-actions, sub-functions, sub-processes, sub-operations or sub-steps, respectively. Unless explicitly excluded, such sub-actions may be included and are part of the disclosure of the single action.

[0100] Furthermore, the following are hereby incorporated into the detailed description, wherein each item can be independently regarded as a separate example. Although each item can be independently regarded as a separate example.

Claims

1. A method (100) for determining relative motion between a time-of-flight camera and an object sensed by the time-of-flight camera in a scene, wherein the object emits a modulated light signal, the method include: receiving (102) a set of at least two raw images of the scene from the time-of-flight camera, wherein the sets of at least two raw images each include at least one raw image, wherein the raw images are based on a correlation of a modulated reference signal with a measurement signal of the time-of-flight camera, and wherein the measurement signal is based on the modulated light signal emitted by the object; determining (104) for each set of raw images a value based on the corresponding set of raw images, the value indicating a corresponding phase difference between the modulated optical signal and the modulated reference signal; determining (106) information about relative motion between the time-of-flight camera and the object based on the value indicative of the phase difference; outputting (108) said information regarding the relative motion between said time-of-flight camera and said object; controlling the time-of-flight camera to synchronize a modulation frequency of the modulated reference signal with a modulation frequency of the modulated light signal emitted by the object; as well as The value indicative of the phase difference is modified based on a synchronization error of the modulation frequency of the modulated reference signal to obtain a compensation value, wherein the information about the relative motion between the time-of-flight camera and the object is determined based on the compensation value.

2. The method of claim 1 , wherein the time-of-flight camera is controlled to synchronize the modulation frequency of the modulated reference signal with the modulation frequency of the modulated light signal emitted by the object. include: The time-of-flight camera is controlled to synchronize the modulation frequency of the modulated reference signal with a radio frequency signal receivable by the time-of-flight camera and the object.

3. The method of claim 1 , wherein the time-of-flight camera is controlled to synchronize the modulation frequency of the modulated reference signal with the modulation frequency of the modulated light signal emitted by the object. include: controlling the time-of-flight camera to synchronize the modulation frequency of the modulated reference signal with a radio frequency reference signal; as well as The time-of-flight camera is controlled to transmit the radio frequency reference signal to the object.

4. The method of claim 1 , wherein the time-of-flight camera is controlled to synchronize the modulation frequency of the modulated reference signal with the modulation frequency of the modulated light signal emitted by the object. include: controlling the time-of-flight camera to receive a radio frequency reference signal from the object; as well as The time-of-flight camera is controlled to synchronize the modulation frequency of the modulated reference signal with the radio frequency reference signal.

5. The method according to claim 1, further comprising: include: The synchronization error of the modulation frequency of the modulated reference signal is determined based on a first auxiliary set of original images of the scene.

6. The method according to claim 5, wherein the synchronization error of the modulation frequency of the modulated reference signal is determined include: determining a corresponding first signal strength for the modulated light signal in each original image of the first auxiliary set of original images; determining a first waveform as a sample of the first waveform using the corresponding first signal strength for the modulated optical signal; as well as The frequency of the first waveform is determined to be the synchronization error of the modulation frequency of the modulated reference signal.

7. The method of claim 6 , wherein determining the synchronization error of the modulation frequency of the modulated reference signal further comprises: include: determining a respective second signal strength for the modulated light signal in each raw image of a second auxiliary set of raw images of the scene, wherein the raw images of the first auxiliary set of raw images and the second auxiliary set of raw images are based on correlations of a measurement signal of the time-of-flight camera with two different modulated reference signals; determining a second waveform using the corresponding second signal strength for the modulated optical signal as a sample of the second waveform; determining a frequency of the second waveform; as well as The sign of the synchronization error of the modulation frequency of the modulated reference signal is determined based on a comparison of the frequencies of the first waveform and the second waveform. The method of claim 7 , wherein the two different modulated reference signals exhibit different modulation frequencies.

9. The method according to claim 1, further comprising: include: receiving information regarding an absolute distance between the time-of-flight camera and the object; as well as Information about an absolute distance between the time-of-flight camera and the object is updated based on the information about the relative motion between the time-of-flight camera and the object.

10. The method of claim 9, wherein the information about the absolute distance between the time-of-flight camera and the object is encoded into the modulated light signal emitted by the object.

11. The method according to claim 1, further comprising: include: controlling the time-of-flight camera to determine information about an absolute distance between the time-of-flight camera and the object by illuminating the scene; as well as The information about the absolute distance between the time-of-flight camera and the object is updated based on the information about the relative motion between the time-of-flight camera and the object. 12 . The method of claim 11 , wherein the time-of-flight camera is controlled to illuminate the scene while the object is pausing to emit the modulated light signal.

13. The method of claim 1, wherein the information about the relative motion between the time-of-flight camera and the object indicates at least one of a change in relative distance between the time-of-flight camera and the object or a relative speed between the time-of-flight camera and the object.

Images