Information processing devices, methods and systems

Abstract

An information processing device, including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal; and circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

Description

[0001] INFORMATION PROCESSING DEVICES, METHODSAND SYSTEMS

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to information processing devices, methods and systems.

[0004] TECHNICAL BACKGROUND

[0005] Generally, time-of-flight (“TOF”) devices are known which, for example, may either be configured as a direct TOF (“dTOF”) devices or indirect TOF (“iTOF”) devices.

[0006] These TOF devices may be embedded in a mobile device, or generally an information processing device, to support a wide range of different applications which require distance information, e.g., augmented reality (“AR”), face recognition etc.

[0007] Both types of TOF devices emit a modulated light signal to a scene and detect the reflected part of the emitted modulated light signal to determine a distance to one or more points in the scene.

[0008] In dTOF devices the modulated light signal is typically a (train of) single light pulse(s) and the distance is determined based on a round-trip time of the emitted light pulse.

[0009] Whereas in iTOF devices the modulated light signal is a periodic (e.g., a sinusoidal, triangular, rectangular etc.) light signal and the distance is determined based on a phase shift of the emitted light signal.

[0010] However, each TOF device requires one or more (light) emitters which increases complexity and chip area and thus costs of the TOF devices.

[0011] Although there exist techniques for performing distance measurements, it is generally desirable to improve the existing techniques.

[0012] SUMMARY

[0013] According to a first aspect, the disclosure provides an information processing device, comprising: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal; and circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the determined distance to the respective point in the scene.

[0014] According to a second aspect, the disclosure provides a method, comprising: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; obtaining, by a circuitry, a distance to each illuminated point in the scene; and controlling, by the circuitry, each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

[0015] According to a third aspect, the disclosure provides an information processing device, comprising: a plurality of receivers, wherein each receiver is configured to detect a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; and circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0016] According to a fourth aspect, the disclosure provides a method, comprising: detecting, by each of a plurality of receivers, a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; determining for each receiver, by a circuitry, based on the respective detected modulated light signal, a detection parameter; and determining for each receiver, by the circuitry, based on the determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0017] According to a fifth aspect, the disclosure provides a system for assisted optical distance measurement, comprising: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0018] According to a sixth aspect, the disclosure provides an information processing device, comprising: a time-of-flight device including: a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to: detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, detect a second modulated light signal of an external emitter thrown back by the point in the scene; and circuitry configured to: control the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene, determine, based on the respective detected second modulated light signal, a detection parameter, estimate a global offset, determine, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0019] According to a seventh aspect, the disclosure provides a method, comprising: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; detecting, by each of a plurality of receivers of the time-of-flight device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter; detecting, by each of the plurality of receivers, a second modulated light signal of an external emitter thrown back by the point in the scene; controlling, by a circuitry, the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene; determining, by the circuitry, based on the respective detected second modulated light signal, a detection parameter, estimating, by the circuitry, a global offset, determining, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0020] According to an eight aspect, the disclosure provides an information processing device, comprising: an emitter configured to illuminate a point in a scene by emitting a modulated light signal; and circuitry configured to control the emitter to encode information in the modulated light signal.

[0021] According to a nineth aspect, the disclosure provides a method, comprising: illuminating, by an emitter, a point in a scene by emitting a modulated light signal; and controlling, by a circuitry, the emitter to encode information in the modulated light signal.

[0022] According to a tenth aspect, the disclosure provides a system for parallel assisted optical distance measurement and optical information exchange, comprising: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene, control the plurality of emitters to encode information in the respective modulated signal for transmitting a message; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset, detect the encoded information, decode the encoded information for obtaining the message.

[0023] According to an eleventh aspect, the disclosure provides a system for assisted optical measurement for estimating position information, comprising: a plurality of stationary emitters, wherein each emitter is configured to illuminate a different point in a scene with a train of light pulses with a predetermined periodicity; an information processing device including: a plurality of receivers, wherein each receiver is configured to detect the train of light pulses thrown back by the different point in the scene, circuitry configured to: determine for each receiver, based on the respective detected train of light pulses during a detection period, a time-of-arrival of the train of light pulses to extract position information.

[0024] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0027] Fig. 1 schematically illustrates different configurations of a system for assisted optical distance measurement and optical information exchange;

[0028] Fig. 2 schematically illustrates an embodiment of a system for assisted optical distance measurement;

[0029] Fig. 3 schematically illustrates the embodiment of the system of Fig. 2 in more detail; Fig. 4 schematically illustrates in a flow diagram a method that is performed by the system of Fig. 3;

[0030] Fig. 5 schematically illustrates in Fig. 5 A a shift of an emission start timing, in Fig. 5B a result of the shift of the emission start timing and in Fig. 5C a detection timing;

[0031] Fig. 6 schematically illustrates in Fig. 6A an on / off k-th repetition encoding and in Fig. 6B a pulse-position modulation encoding;

[0032] Fig. 7 schematically illustrates a system for assisted optical distance measurement in which three-dimensional information of a virtual object is conveyed;

[0033] Fig. 8 schematically illustrates in a flow diagram a method that is performed by the system of

[0034] Fig. 7;

[0035] Fig. 9 schematically illustrates in a flow diagram an embodiment of a method;

[0036] Fig. 10 schematically illustrates in a flow diagram an embodiment of a method;

[0037] Fig. 11 schematically illustrates a system for assisted optical distance measurement;

[0038] Fig. 12 schematically illustrates a method that is performed by the system of Fig. 11;

[0039] Fig. 13 schematically illustrates in a flow diagram an embodiment of a method;

[0040] Fig. 14 schematically illustrates in a block diagram an embodiment of an optical communications system;

[0041] Fig. 15 schematically illustrates in a flow diagram an embodiment of a method;

[0042] Fig. 16 schematically illustrates in Fig. 16A a normal operation of a direct time-of-flight device and in Fig. 16B a parallel distance measurement and pulse-position modulation encoding;

[0043] Fig. 17 schematically illustrates an embodiment of a function of a system for assisted optical distance measurement and optical information exchange;

[0044] Fig. 18 schematically illustrates an embodiment of a function of a system for assisted optical distance measurement and optical information exchange; and

[0045] Fig. 19 schematically illustrates in a block diagram an embodiment of a multi-purpose computer.

[0046] DETAILED DESCRIPTION OF EMBODIMENTS

[0047] The present disclosure is structured in four sections, wherein section A and section B covers assisted optical distance measurements, section C covers optical information exchange, section D covers a combination of the technologies described in sections A and C, and section E covers assisted optical measurements for estimating position information.

[0048] Section A

[0049] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0050] As mentioned in the outset, generally, time-of-flight (“TOF”) devices are known which, for example, may either be configured as a direct TOF (“dTOF”) devices or indirect TOF (“iTOF”) devices.

[0051] These TOF devices may be embedded in a mobile device, or generally an information processing device, to support a wide range of different applications which require distance information, e.g., augmented reality (“AR”), face recognition etc.

[0052] Both types of TOF devices emit a modulated light signal to a scene and detect the reflected part of the emitted modulated light signal to determine a distance to one or more points in the scene.

[0053] Each receiver of a TOF device is associated with a different emitter in the sense that the modulated light signal of an emitter reflected in the scene is imaged onto the respective receiver if the point in the scene is within the filed-of-view (“FOV”) of the respective receiver.

[0054] In dTOF devices the modulated light signal is typically a (train of) single light pulse(s) and the distance is determined based on a round-trip time of the emitted light pulse.

[0055] Whereas in iTOF devices the modulated light signal is a periodic (e.g., a sinusoidal, triangular, rectangular etc.) light signal and the distance is determined based on a phase shift of the emitted light signal.

[0056] The emitter and receiver are associated with each other in terms of synchronization.

[0057] For example, in the case of a dTOF device, at the emission start timing of the emitter a time counter may start and, e.g., a single-photon avalanche diode ("SPAD”) may be activated such that a light detection signal generated by the SPAD can be used to determine a detection timing indicative of the round-trip time of the modulated light signal such that a histogram can be built to determine the distance to the respective point in the scene.

[0058] For example, in the case of an iTOF device, a phase between a modulation signal applied to the emitter and a demodulation signal applied, e.g., to a current-assisted photonic demodulator ("CAPD”) is kept constant during a correlation measurement, which may also be referred to as emission phase. Typically, the iTOF device performs four correlation measurements with phase shifts of 0°, 90°, 180° and 270° to accurately determine the phase of the received modulated light signal with respect to the emitted modulated light signal, which may also be referred to as detection phase and which is indicative for the distance to the respective point in the scene.

[0059] However, each TOF device requires one or more (light) emitters which increases complexity and chip area and thus costs of the TOF devices.

[0060] Hence, some embodiments pertain to a first information processing device, including: a TOF device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal; and first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

[0061] The first information processing device may be or may include a room scanning apparatus or a mobile device such as a smartphone, smart glasses, a laptop or the like.

[0062] In some embodiments, the TOF device is a dTOF device and the emission parameter is an emission start timing, or the TOF device is an iTOF device and the emission parameter is an emission phase.

[0063] Each emitter may be or may include a light emitting diode (“LED”), a laser diode, a verticalcavity surface-emitting laser (“VCSEL”) or the like.

[0064] In some embodiments, the TOF device includes a plurality of first receivers, wherein each first receiver is associated with a different emitter, and wherein each first receiver is configured to detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter.

[0065] Each first receiver may be or may include, in the case of a dTOF device, a SPAD or the like.

[0066] Each first receiver may be or may include, in the case of an iTOF device, a CAPD or the like. Each CAPD may have one tap, two taps, four taps etc., as generally known.

[0067] The modulated light signal may be or may include, in the case of a dTOF device, a (train of) single light pulse(s).

[0068] The modulated light signal may be or may include, in the case of an iTOF device, a periodic (e.g., a sinusoidal, triangular, rectangular etc.) light signal. The first circuitry may include one or more processors, memory, buses, input / output interfaces, communication or network interfaces (e.g., for transmitting and receiving data via a wired or wireless connection) etc. configured to achieve the functionality as described herein. The first circuitry may include or may be implemented by typical electronic components configured to achieve the functionality as described herein. The functionality of the first circuitry may be provided by hardware or by software or in parts by hardware and in parts by software.

[0069] In some embodiments, the first information processing device obtains a distance to each (illuminated) point in the scene from a third party, e.g., from another information processing device which has previously sensed the scene.

[0070] In some embodiments, the circuitry controls the TOF device to perform a TOF measurement to obtain the distance to each (illuminated) point in the scene.

[0071] Once the first information processing device has obtained the distance to each point in the scene, for example, by instructing the TOF device to perform a TOF measurement, the first information processing device determines the time or phase shift for each emitter based on the determined distance to the respective point in the scene. The TOF device drives each emitter accordingly.

[0072] In the case of a dTOF device, the circuitry shifts each modulated light signal such that all emitted modulated light signals arrive at the same time at their respective point in the scene.

[0073] The whole process of TOF measurement, distance determination and pulse train adaption may be repeated regularly to account for scene updates or mobility of the first information processing device.

[0074] Thus, a second information processing device can use the circumstance that an emission parameter of each modulated light signal is shifted in accordance with the distance to the respective point in the scene for determining a relative distance to each point in the scene within its FOV up to a global offset. Therefore, the second information processing device may not be required to have its own light emitters or may not be required to have a full array of own light emitters, as will also be discussed under reference of Figs. 3, 4, 5 and 6 further below.

[0075] Hence, some embodiments pertain to a second information processing device, including: a plurality of second receivers, wherein each second receiver is configured to detect a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; and second circuitry configured to: determine for each second receiver, based on the respective detected modulated light signal, a detection parameter, determine for each second receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0076] Each external emitter corresponds to an emitter of the first information processing device.

[0077] Each second receiver may be or may include, in the case of a dTOF device, a SPAD or the like.

[0078] Each second receiver may be or may include, in the case of an iTOF device, a CAPD or the like. Each CAPD may have one tap, two taps, four taps etc., as generally known.

[0079] The second circuitry may include one or more processors, memory, buses, input / output interfaces, communication or network interfaces (e.g., for transmitting and receiving data via a wired or wireless connection) etc. configured to achieve the functionality as described herein. The second circuitry may include or may be implemented by typical electronic components configured to achieve the functionality as described herein. The functionality of the second circuitry may be implemented by hardware or by software or in parts by hardware and in parts by software.

[0080] The second circuitry may be configured to implement or may include a signal processing block as generally in dTOF such as histogram processing.

[0081] In some embodiments, the TOF device is a dTOF device and the detection parameter is a time- of-arrival of a train of light pulses of the detected modulated light signal.

[0082] In some embodiments, the TOF device is an iTOF device and the detection parameter is a phase of the detected modulated light signal.

[0083] Some embodiments pertain to a first method, including: illuminating, by each of a plurality of emitters of a TOF device, a different point in a scene by emitting a modulated light signal; obtaining, by a first circuitry, a distance to each illuminated point in the scene; and controlling, by the first circuitry, each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

[0084] The first method may be performed by the first information processing device.

[0085] Some embodiments pertain to a second method, including: detecting, by each of a plurality of second receivers, a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; determining for each second receiver, by a second circuitry, based on the respective detected modulated light signal, a detection parameter; and determining for each second receiver, by the second circuitry, based on the determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0086] The second method may be performed by the second information processing device.

[0087] Some embodiments pertain to a system for assisted optical distance measurement, including: a first information processing device including: a TOF device including: a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene; and a second information processing device including: a plurality of second receivers, wherein each second receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each second receiver, based on the respective detected modulated light signal, a detection parameter, determine for each second receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0088] It has been further recognized that three-dimensional (“3D”) information of a virtual object may be conveyed from the first information processing device to the second information processing device by shifting the emission parameter further in accordance with the 3D information of the virtual object, as will also be discussed under reference of Figs. 7 and 8 further below.

[0089] Hence, in some embodiments, the first circuitry is further configured to obtain three-dimensional information of a virtual object and to control each of the plurality of emitters individually to shift the emission parameter further in accordance with the obtained three-dimensional information of the virtual object. In such embodiments, the second circuitry is further configured to obtain, based on the determined relative distances, three-dimensional information of a virtual object.

[0090] As mentioned above, the second information processing device determines a relative distance to each point in the scene up to a global offset.

[0091] In some embodiments, the second circuitry is further configured to estimate the global offset.

[0092] In some embodiments, the second information processing device further includes an emitter, the emitter being associated with one of the plurality of receivers, wherein the emitter is configured to illuminate a point in the scene by emitting a second modulated light signal.

[0093] In such embodiments, the second circuitry is further configured to control the emitter and the respective receiver to perform a time-of-flight measurement to determine a distance to the point in the scene and to estimate the global offset based on the distance to the point in the scene.

[0094] In some embodiments, the second information processing device further includes an image sensor configured to capture an image of the scene.

[0095] In such embodiments, the second circuitry is further configured to input the captured image and the determined relative distances into a computer vision algorithm or signal processing algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the captured image and the determined relative distances.

[0096] In some embodiments, the computer vision algorithm is a neural network trained to estimate the global offset based on the captured image and the determined relative distances.

[0097] In some embodiments, the second circuitry is configured to input the determined relative distances into a computer vision algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the determined relative distances.

[0098] In some embodiments, the computer vision algorithm is a neural network trained to estimate the global offset based on the determined relative distances.

[0099] In some embodiments, the first information processing device transmits information to the second information processing device which allow the second information processing device to estimate the global offset.

[0100] Hence, in some embodiments, the first circuitry is further configured to control an emitter of the plurality of emitters to emit a second modulated light signal in which additional information is encoded which allow a second information processing device to decode its own distances correctly. In some embodiments, the additional information correspond to the distances of at least four points in the scene that lie on the same plane. In some embodiments the additional information correspond to one point per detected object.

[0101] In such embodiments, each of the plurality of second receivers is further configured to detect a second modulated light signal in which additional information is encoded, and the second circuitry is further configured to estimate the global offset based on the additional information.

[0102] In some embodiments, the additional information correspond to the distances of at least four points in the scene that lie on the same plane. In some embodiments the additional information correspond to one point per detected object.

[0103] Generally, the first information processing device may additionally have the functions and configurations of the second information processing device and the second information processing device may additionally have the functions and configurations of the first information processing device.

[0104] Returning to Fig. 1, there are schematically illustrated different configurations of a system for assisted optical distance measurement, which are discussed in the following.

[0105] As depicted in the upper part, the system may support stationary device to mobile device assistance or vice versa.

[0106] As depicted in the middle part, the system may support duplex and simplex assistance.

[0107] As depicted in the lower part, the system may support assistance for multiple devices.

[0108] In the following, for the sake of illustration only, a case of simplex dToF one-to-one assistance for the optical distance measurement is assumed without limiting the present disclosure to this case.

[0109] Fig. 2 schematically illustrates an embodiment of a system 1 for assisted optical distance measurement, which is discussed in the following.

[0110] The system 1 includes a first information processing device 2, referred to as Device A in the following, and a second information processing device 3, referred to as Device B in the following.

[0111] The Device A performs a distance measurement (TOF measurement) to determine distances to points in a scene 4. After determining the distances, the Device A offsets the emission start timings TX of its emitters individually to allow the Device B to perform its own distance measurement without being required to have own light emitters.

[0112] Fig. 3 schematically illustrates the embodiment of the system 1 in more detail, which is discussed in the following under reference of Figs. 3, 4, 5 and 6.

[0113] The scene 4 of Fig. 2 is here a room with a ceiling 4a, a side wall 4b and furniture 4c. For the way of illustration, a first point Pl and a second point P2 are indicated on the ceiling 4a.

[0114] Referring now to Fig. 4, which schematically illustrates in a flow diagram a method 10 that is performed by the system 1, at 11, the Device A performs a TOF measurement and determines the distance distAi to the point Pl and the distance distA2 to the point P2 (and all other points of the scene within its FOV).

[0115] At 12, the Device A offsets (shifts) the emission start timing TX of each emitter (e.g., VCSEL) according to the respective point distance, as schematically illustrated in Fig. 5 A.

[0116] At 13, as a result, the pulses of each emitter arrive at the illuminated surface all at the same time, as schematically illustrated in Fig. 5B.

[0117] At 14, the Device B activates its receivers (e.g., SPADs) and receives the pulses that are reflected from the surfaces in the scene, wherein the pulses need a time to arrive at the Device B corresponding to the distance to the Device B, as illustrated schematically in Fig. 5C.

[0118] However, the activation of the receivers by the Device B may not be synchronized with the time at which all pulses from Device A arrive at the surfaces of the scene.

[0119] Hence, the Device B calculates, at 15, relative distances to the illuminated points in the scene.

[0120] In particular, the Device B calculates for point Pl the relative distance (distm + global offset) and for point P2 the relative distance (distB2 + global offset).

[0121] But it has been recognized that the Device B can estimate the global offset with various techniques, which are discussed in the following.

[0122] Option 1:

[0123] In some embodiments, the Device B has, e.g., one or more emitters itself which forms with one of the plurality of receivers a TOF device such that the Device B measures the distance to one or a few points, respectively, on its own and uses this information for estimating the global offset. In particular, the Device B estimates the time difference between the relative measurement as received from the Device A and its own measurement. This difference corresponds to the global offset, which can be used to correct all other points.

[0124] In some embodiments, the Device B inputs the distances of the one or few points into a computer vision algorithm, e.g., for estimating a plane in the scene such that Device B can estimate the global offset based on the estimated plane. The distances of the one or few points may be obtained by a TOF measurement of Device B, as discussed above, or may be obtained externally, e.g., from Device A as discussed under Option 3 below.

[0125] Option 2:

[0126] In some embodiments, the Device B uses an additional sensor (e.g., an RGB (“red-green-blue”) sensor) and a computer vision algorithm to estimate the global offset.

[0127] In another embodiment, Device B uses photon count measurements from an additional sensor or using its own SPAD sensor, and a computer vision algorithm to estimate the global offset.

[0128] Option 3:

[0129] In some embodiments, the Device A transmits additional information that assist the Device B to estimate the global time offset. Such information may be the distances to at least four points that he on the same plane such that the Device B can use the difference between them to calculate the global offset. In some embodiments, the additional information correspond to one point per detected object.

[0130] The transmission of the information may be performed optically by the Device A which encodes the information in a second modulation light signal emitted by one of the emitters of Device A. The Device B detects the second modulated light signal with one of its receivers and decodes it to obtain the information.

[0131] The encoding may be performed as schematically illustrated in Fig. 6A and in Fig. 6B, which are discussed in the following.

[0132] As depicted in Fig. 6A, the second modulation signal may be turned on or off every k-th (“OOK”) repetition in accordance with the bit sequence to transmit, wherein the distances are encoded in the bit sequence. In other words, the (whole) train of pulses is turned on an off every k-th repetition in accordance with the bit sequence to transmit.

[0133] As an example of OOK, the Device A wants to send 4 bits. Each bit may correspond to 100 repetitions (or cycles). The message may be: 1010. Thus, in the first 100 cycles, the Device A sends (consecutively) 100 pulses. In the next 100 cycles, the device will not send anything (will be kept silent, as the second bit is 0). In the next 3rd round of 100 cycles, the Device A again sends 100 cycles (as the third bit is 1), and in the last 100 cycles, the Device A is silent (as the fourth bit is 0). Generally, if total number of cycles is N, and number of bits is K, the Device A switches from transmission of pulses to silence every N / K (k-th) cycles in accordance with the bit sequence to transmit.

[0134] As depicted in Fig. 6B, the shift of the emission start timing of the second modulation signal may be determined according to the bit pattern of a data block (“Pulse-Position Modulation”;

[0135] “PPM”), wherein the distances are encoded in one or more data blocks. In some embodiments, the whole measurement window of the receivers of the Device A may be shifted according to the bit pattern of the data block.

[0136] As depicted in Fig. 6B, each pulse may have a different emission start timing according to the data block to be transmitted. In such embodiments, the train of light pulses - in which each pulse is shifted according to the assigned data block - may be repeated several times for increasing a probability of correct transmission of the message.

[0137] In other embodiments, the train of light pulses is divided in groups and each group of pulses of the train of light pulses may have the same emission start timing according to the data block to be transmitted for increasing a probability of correct transmission of the message.

[0138] The above modulation in OOK may happen also in PPM. As an example, the Device A has 20 bits to send and a total of 1000 repetitions. For example, PPM of 4 bits is applied a time such that the 20 bits are divided into 5 groups of 4 bits. Each group may be transmitted over (1000 / 5=200) consecutive pulses that will be shifted in time according to the 4 bits.

[0139] Generally, the second information processing device (Device B) may use all of these techniques (Options 1, 2, 3) in combination to improve the estimate of the global offset.

[0140] Hence, returning to Fig. 4, at 16, the Device B estimates the global offset of all points.

[0141] At 17, the Device B corrects the relative time offsets of all points with the global time estimation.

[0142] It has been recognized that the above technique can be used to convey three-dimensional (“3D”) information of a virtual object from the Device A to the Device B, as will be discussed under reference of Fig. 7 and 8 in the following. The device A can convey 3D information of a virtual object to the Device B by further shifting the emission start timing of the modulated light signal in accordance with the 3D information of the virtual object.

[0143] For example, as illustrated in Fig. 7, can use the side wall 4b to project the virtual object by encoding the 3D information in the emission start timings of the emitters.

[0144] Referring now to Fig. 8, at first, at 21, the Device A performs a TOF measurement and determines the distance distAi to the point Pl and the distance distA2 to the point P2.

[0145] At 22, the Device A obtains three-dimensional information of a virtual object, wherein the Device A determines the distance from the surface Fi for every point of the virtual object to be seen by the Device B. In this way, the Device A is able to generate a virtual object for the Device B.

[0146] In some embodiments, the Device A knows the location and orientation (pose) of the Device B such that the Device A is able to adjust the Fi according to the pose of the Device B, since the proportions of the virtual object as seen by the Device B may depend on the angle and distance to the surface on which the virtual object is projected.

[0147] In some embodiments, the Device A knows the pose of the Device B by previously performing the method of Fig. 12 described further below.

[0148] In other embodiments, the Device A does not know the pose of the Device B and assumes that the Device B is at a certain position relative to the surface (e.g., perpendicular at specific distance) to the surface.

[0149] In such embodiments, the system 1 performs, e.g., at first the method 10 of Fig. 4 such that the Device B can calculate its pose based on the distances to the points in the scene.

[0150] Then, the Device B is able to estimate the surface normal and correct the measurements accordingly.

[0151] At 23, the Device A offsets the emission start timing of each emitter by (-distAi - Fi).

[0152] At 24, the Device B activates, e.g., its SPAD receivers and receives the pulses that are reflected from surfaces in the scene.

[0153] At 25, the Device calculates the relative distances of all returned points: (distBi - Fi + global offset).

[0154] At 26, the Device B estimates the global offset, as discussed under reference of Fig. 4. At 27, the Device B generates the 3D point cloud of the virtual object out of the measurements that represent the virtual object.

[0155] Fig. 9 schematically illustrates in a flow diagram an embodiment of a method 100, which is discussed in the following.

[0156] The method 100 may be performed by the first information processing device as described herein.

[0157] At 101, by each of a plurality of emitters of a TOF device, a different point in a scene is illuminated by emitting a modulated light signal, as discussed herein.

[0158] At 102, by each of a plurality of receivers of the TOF device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter is detected, as discussed herein.

[0159] At 103, by a circuitry, the TOF device is controlled to perform a TOF measurement to determine a distance to each illuminated point in the scene, as discussed herein.

[0160] At 104, by the circuitry, each of the plurality of emitters is individually controlled to shift an emission parameter in accordance with the determined distance to the respective point in the scene, as discussed herein.

[0161] Fig. 10 schematically illustrates in a flow diagram an embodiment of a method 200, which is discussed in the following.

[0162] The method 200 may be performed by the second information processing device as described herein.

[0163] At 201, by each of a plurality of receivers, a modulated light signal thrown back by a different point in the scene is detected, wherein each modulated light signal is emitted by a different external emitter, as discussed herein.

[0164] At 202, by a circuitry, based on the respective detected modulated light signal, a detection parameter is determined for each receiver, as discussed herein.

[0165] At 203, by the circuitry, based on the determined detection parameter, a relative distance to the respective point in the scene up to a global offset is determined for each receiver, as discussed herein.

[0166] Section B Before a detailed description of the embodiments under reference of Fig. 11 is given, general explanations are made.

[0167] Generally, the information processing device described in section B may additionally have the functions and configurations of each of the first and second information processing device described in section A.

[0168] The general explanations in section A regarding, e.g., the information processing device, the emitter(s), the receiver(s), the modulated light signal, the circuitry, the TOF device, the TOF measurement etc. also apply in section B.

[0169] Some embodiments pertain to an information processing device, including: a TOF device including: a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to: detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, detect a second modulated light signal of an external emitter thrown back by the point in the scene; and circuitry configured to: control the TOF device to perform a TOF measurement to determine a distance to each illuminated point in the scene, determine, based on the respective detected second modulated light signal, a detection parameter, estimate a global offset, determine, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0170] Some embodiments pertain to a (corresponding) method, including: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; detecting, by each of a plurality of receivers of the time-of-flight device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter; detecting, by each of the plurality of receivers, a second modulated light signal of an external emitter thrown back by the point in the scene; controlling, by a circuitry, the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene; determining, by the circuitry, based on the respective detected second modulated light signal, a detection parameter, estimating, by the circuitry, a global offset, determining, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0171] Returning to Fig. 11, which schematically illustrates a system 1 for assisted optical distance measurement, which is discussed in the following under reference of Figs. 11 and 12.

[0172] The system 1 includes a first information processing device 2, referred to as Device A in the following, and a second information processing device 3, referred to as Device B in the following.

[0173] A scene is here a room with a ceiling 4a, a side wall 4b and furniture 4c. For the way of illustration, a first point Pl and a second point P2 are indicated on the ceiling 4a.

[0174] Referring now to Fig. 12, which schematically illustrates in a flow diagram a method 30 that is performed by the system 1, at 31, the Device A performs a TOF measurement and determines the distance distAi to the point Pl and the distance distA2 to the point P2 (and all other points of the scene within its FOV).

[0175] At 32, the Device B activates its VCSELs and illuminates points in its FOV.

[0176] At 33, the Device A activates its SPAD receiver and receives the pulses that are reflected from the objects and surfaces in the scene (distAi + distBi + global offset).

[0177] At 34, the Device A estimates (distBi + global offset) for every point in the scene.

[0178] At 35, the Device A estimates the global offset in the same way as discussed under reference of Fig. 4 in section A.

[0179] At 36, the Device A calculates (distBi) for every point in the scene.

[0180] At 37, the Device A estimates the pose (location and orientation) of the Device B.

[0181] Fig. 13 schematically illustrates in a flow diagram an embodiment of a method 300, which is discussed in the following. The method 300 may be performed by the information processing device as described herein.

[0182] At 301, by each of a plurality of emitters of a TOF device, a different point in a scene is illuminated by emitting a modulated light signal, as discussed herein.

[0183] At 302, by each of a plurality of receivers of the TOF device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter is detected, as discussed herein.

[0184] At 303, by each of the plurality of receivers, a second modulated light signal of an external emitter thrown back by the point in the scene is detected, as discussed herein.

[0185] At 304, by a circuitry, the TOF device is controlled to perform a TOF measurement to determine a distance to each illuminated point in the scene, as discussed herein.

[0186] At 305, by the circuitry, based on the respective detected second modulated light signal, a detection parameter is determined, as discussed herein.

[0187] At 306, by the circuitry, a global offset is estimated, as discussed herein.

[0188] At 307, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter is determined, as discussed herein.

[0189] Section C

[0190] Before a detailed description of the embodiments under reference of Fig. 14 is given, general explanations are made.

[0191] Generally, the information processing device described in section C may additionally have the functions and configurations of each of the first and second information processing device described in section A and may additionally have the functions and configurations of the information processing device described in section B.

[0192] The general explanations in section A regarding, e.g., the information processing device, the emitter(s), the receiver(s), the modulated light signal, the circuitry, the TOF device, the TOF measurement etc. also apply in section C.

[0193] Some embodiments pertain to an information processing device, including: an emitter configured to illuminate a point in a scene by emitting a modulated light signal; and circuitry configured to control the emitter to encode information in the modulated light signal.

[0194] In some embodiments, the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information.

[0195] In some embodiments, the information processing device further includes a TOF device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene, wherein the circuitry is further configured to control the TOF device to perform a time- of-flight measurement to obtain a distance to the illuminated point in the scene.

[0196] In some embodiments, the circuitry is further configured to stop histogram processing when the train of pulses is not emitted.

[0197] In some embodiments, the information processing device further includes a TOF device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene, wherein the circuitry is further configured to control the TOF device to perform a time- of-flight measurement to obtain a distance to the illuminated point in the scene.

[0198] In some embodiments, the circuitry is further configured to shift histogram processing according to a shift of the emission start timing.

[0199] In some embodiments, the information is encoded by controlling an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

[0200] Some embodiments pertain to a (corresponding) method, including: illuminating, by an emitter, a point in a scene by emitting a modulated light signal; and controlling, by a circuitry, the emitter to encode information in the modulated light signal.

[0201] In some embodiments, the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information or by controlling an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block. In such embodiments, the information processing device may include a dTOF device including the emitter. In some embodiments, the information is encoded by modifying a phase of the modulated light signal in accordance with a bit pattern representing the information. In such embodiments, the information processing device may include an iTOF device including the emitter.

[0202] Returning to Fig. 14, which schematically illustrates in a block diagram an embodiment of an optical communications system 40, which is discussed in the following.

[0203] A Device A (transmitter), an information processing device, has data to be transmitted to a Device B (receiver), another information processing device.

[0204] Optionally, the Device A performs at first cyclic redundancy coding (“CRC”).

[0205] Then, optionally, forward error correction (“FEC”) at rate k / n is performed and afterwards input to a modulation module.

[0206] The modulation module performs PPM or OOK modulation, as also discussed under reference of Fig. 6.

[0207] In OOK, the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information.

[0208] In PPM, the information is encoded by controlling an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

[0209] Then, serial to parallel conversion is applied and the result is mapped to the VCSELs.

[0210] Then, the pulse repetitions are generated, and the modulated light signal is emitted to a scene, wherein the information is encoded in the modulated light signal.

[0211] The Device B monitors the scene and checks its SPAD readings and combines the repetitions in accordance with a performed synchronization and tracking.

[0212] Then, the combined repetitions are converted from parallel to serial and the result is demodulated.

[0213] Optionally, if performed by Device A, afterwards FEC decoding, and CRC check is applied by Device B to obtain the information.

[0214] Fig. 15 schematically illustrates in a flow diagram an embodiment of a method 400, which is discussed in the following.

[0215] At 401, by an emitter, a point in a scene is illuminated by emitting a modulated light signal, as discussed herein. At 402, by a circuitry, the emitter is controlled to encode information in the modulated light signal, as discussed herein.

[0216] Generally, while device A encodes data to be transmitted, it can perform its own distance measurements in parallel, both in case of OOK and PPM. In case of OOK, it can stop the histogram processing when it does not transmit the pulse train, and in case of PPM, it can cyclically shift the histogram processing according to the message encoded.

[0217] This is illustrated in more detail for the case of PPM in Fig. 16 which schematically illustrates in Fig. 16A a normal operation of a dTOF device and in Fig. 16B a parallel distance measurement and PPM encoding.

[0218] The message to be transmitted may include for the sake of illustration only three data blocks DB1 ("1001”), DB2 ("0000”) and DB3 ("1111”).

[0219] As illustrated in Fig. 16A, the emitter TX of Device A emits a light pulse at T=0 and synchronously starts the time counter of the associated receiver RX which detects the reflected light pulses at a time according to two times the distance of the Device A to the illuminated point in the scene.

[0220] As illustrated in Fig. 16B, the emitter TX shifts its emission start timing according to the bit pattern of each data block DB1, DB2 and DB3.

[0221] Thus, the Device A can shift its histogram processing for distance measurement in a way that the message transmission does not affect its ability to perform a distance measurement on top of a message transmission to Device B.

[0222] Section D

[0223] The techniques of section A and C are combined in the following.

[0224] Hence, some embodiments pertain to a system for parallel assisted optical distance measurement and optical information exchange, including: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene, control the plurality of emitters to encode information in the respective modulated signal for transmitting a message; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset, detect the encoded information, decode the encoded information for obtaining the message.

[0225] In some embodiments, the first circuitry controls the plurality of emitters to encode the distances to each illuminated point in the scene as the information. Such an embodiment is discussed under reference of Fig. 18 below.

[0226] The system is thus configured to support assisted optical distance measurement and optical information exchange, wherein the assisted optical distance measurement and the optical information exchange may be performed in a time serial manner or in a time parallel manner.

[0227] In the following illustrative embodiments, the system performs the assisted optical distance measurement and the optical information exchange in a time parallel manner.

[0228] Fig. 17 schematically illustrates an embodiment of a function of a system 50 for parallel assisted optical distance measurement and optical information exchange, which is discussed in the following.

[0229] The system 50 may be based on the system 1 of Fig. 3 and the system 40 of Fig. 14.

[0230] The Device A knows its distances to the scene and wants, on the one hand, to transmit a message to the Device B and, on the other hand, to allow the Device B to measure at least one distance to the scene.

[0231] The Device A thus shifts an emission start timing of each emitter (the depicted example is for a single emitter) by the respective distance (e.g., distAi for a first emitter, distA2 for a second emitter etc.) and applies PPM (see Fig. 14). For the sake of illustration, the message is represented by three data blocks (DB1, DB2 and DB3). Hence, the time of emission of the first pulse is shifted by (-distA + TbitsDBi), wherein TbitsDBi is the offset according to the bit sequence of the first data block DB1, which may, e.g., four bits.

[0232] The time of emission of the second pulse is shifted by (-distA + TbitsDB2), wherein TbitsDB2 is the offset according to the bit sequence of the second data block DB2.

[0233] The time of emission of the second pulse is shifted by (-distA + TbitsDBs), wherein Ti,itsi )i>,3 is the offset according to the bit sequence of the third data block DB3.

[0234] The distA above may be different for each emitter depending on the distance to the respective illuminated point in the scene.

[0235] Then, Device B receives the modulated light signal and performs the demodulation to obtain the symbols.

[0236] The Device B performs blind decoding to decode the message, wherein the Device B tries different time offsets (global offsets) until the CRC check is passed.

[0237] Then, the Device B has obtained the distB for each receiver (which may be different for each receiver depending on the respective relative distance to the illuminated point in the scene) and the message without activating an emitter on its own. In more detail, the Device B has estimated distB up to a global time offset for each receiver, as discussed in section A above. The Device B may use any of the options described in section A above to estimate the global offset to obtain the absolute distance to each illuminated point in the scene (that is observed by Device B).

[0238] Fig. 18 schematically illustrates an embodiment of a function of a system 60 for parallel assisted optical distance measurement and optical information exchange, which is discussed in the following.

[0239] In this embodiment, the optical information exchange is not intended to carry an arbitrary message, but to transmit the results of the distance measurement to the objects (points) in the scene of Device A to Device B.

[0240] In Frame 1, the Device A measures its distances to the scene and transmits a synchronization signal (“00000”) from which the Device B can estimate the time offset.

[0241] In Frame 2, the Device A transmits the distances (Al) and measures the distances again (A2), wherein the Device B receives the distances Al and extracts Al as the encoded information and extracts the distances Bl in addition, as discussed under reference of Fig. 17. In Frame 3, the Device A transmits the distances (A2) and measures the distances again (A3), wherein the Device B receives the distances A2 and extracts A2 as the encoded information and the distances B2 in addition, as discussed under reference of Fig. 17.

[0242] This may be repeated several times more.

[0243] Section E

[0244] Generally, the information processing device described in section E may additionally have the functions and configurations of each of information processing device described in section A, B, C or D.

[0245] The general explanations in section A regarding, e.g., the information processing device, the emitter(s), the receiver(s), the modulated light signal, the circuitry, the TOF device, the TOF measurement etc. also apply in section E.

[0246] Some embodiments pertain to a system for assisted optical measurement for estimating position and pose information, including: a plurality of stationary emitters, wherein each emitter is configured to illuminate a different point in a scene with a train of light with a predetermined periodicity; an information processing device including: a plurality of receivers, wherein each receiver is configured to detect the train of light pulses thrown back by the different point in the scene, circuitry configured to: determine for each receiver, based on the respective detected train of light pulses during a detection period, a time-of-arrival of the train of light pulses to extract position information.

[0247] In some embodiments, the circuitry is further configured to: determine for each receiver, based on the respective detected train of light pulses during a second detection period, a second time-of-arrival of the train of light pulses, estimate for each receiver, based on the time-of-arrival and the second time-of-arrival, a change in distance to the respective point in the scene, estimate, based on the estimated changes in distance, at least one of a change in position and pose of the information processing device.

[0248] In some embodiments, the information processing device includes at least one sensor configured to acquire sensor information, and the circuitry is further configured to extract, based on the time-of-arrival and the sensor information, the position information. In some embodiments, the at least one sensor includes at least one of an accelerometer, a gyrometer, an inertial measurement unit and an image sensor.

[0249] The position information may be an absolute device position and / or device pose or changes in device position and / or device pose.

[0250] The Device A activates the emitters with predetermined periodicity to emit a train of light pulses.

[0251] The Device B (the information processing device) has receivers and may know the predetermined periodicity.

[0252] If the Device B does not know the predetermined periodicity it may apply computer vision or signal processing algorithms to estimate the periodicity (all illuminated points in the scene shine at the same periodicity).

[0253] Then, the Device B activates, e.g., its SPADs to measure the relative time-of-arrival of the photons (the light pulses) to the SPAD receiver during a detection period.

[0254] This time-of-arrival (of a specific pixel) equals D = distA + dis tp, - global offset, wherein distA is the distance between the emitter and a point in the scene, and distB is the distance between device B and the same corresponding point in the scene, and wherein the global offset is the same for all SPAD receivers, or at least the delta between the global offset of two SPAD receivers is known to the Device B such that it can compensate it.

[0255] The Device B may repeat these measurements (e.g., second detection period, third detection period and so on) at tO, tl,... to get DO, DI,... accordingly for every SPAD receiver.

[0256] Note that Device B is then able to determine D(i) - D(i-l) = distB(i) - distB(i-l) (for corresponding pixels), since the Device A is stationary, and the global offset is kept the same across multiple measurements.

[0257] By utilizing this information, and combining this with sensor information from additional sensors (e.g., accelerometer, gyrometer, or image sensor) of the Device B, and by applying computer vision, signal processing and / or neural network algorithms, Device B is able to extract information about the position and / or pose of Device B. Such information could be used to estimate the change in position and / or pose over time and to perform a kind of “visual inertial odometry”.

[0258] The Device A may be an ordinary light source (e.g., LED) or projector when it emits light with a predetermined modulation frequency which may depend on the electric system, voltage regulators, drivers and projector operations (e.g., raster scan of pixels). In such cases, the emitted light may have a frequency from 100’s of Hz to MHz’s.

[0259] The following embodiment of a multi-purpose computer may apply embodiments of each section A, B, C, D and E.

[0260] Fig. 19 schematically illustrates in a block diagram an embodiment of a multi-purpose computer 130 which can be used for implementing an information processing device.

[0261] The computer 130 can be implemented such that it can basically function as any type of information processing device as described herein. The computer has components 131 to 141, which can form a circuitry, such as any one of the circuitries of any information processing device as described herein.

[0262] Embodiments which use software, firmware, programs or the like for performing the methods as described herein can be installed on computer 130, which is then configured to be suitable for the concrete embodiment.

[0263] The computer 130 has a CPU 131 (Central Processing Unit), which can execute various types of procedures and methods as described herein, for example, in accordance with programs stored in a read-only memory (ROM) 132, stored in a storage 137 and loaded into a random-access memory (RAM) 133, stored on a medium 140 which can be inserted in a respective drive 139, etc.

[0264] The CPU 131, the ROM 132 and the RAM 133 are connected with a bus 141, which in turn is connected to an input / output interface 134. The number of CPUs, memories and storages is only exemplary, and the skilled person will appreciate that the computer 130 can be adapted and configured accordingly for meeting specific requirements which arise, when it functions as an information processing device.

[0265] At the input / output interface 134, several components are connected: an input 135, an output 136, the storage 137, a communication interface 138 and the drive 139, into which a medium 140 (compact disc, digital video disc, compact flash memory, or the like) can be inserted.

[0266] The input 135 can be a pointer device (mouse, graphic table, or the like), a keyboard, a microphone, a receiver, a camera, a touchscreen, a time-of-fight device, etc.

[0267] The output 136 can have a display (liquid crystal display, cathode ray tube display, light emittance diode display, etc.), loudspeakers, an emitter, etc.

[0268] The storage 137 can have a hard disk, a solid-state drive and the like. The communication interface 138 can be adapted to communicate, for example, via a local area network (LAN), wireless local area network (WLAN), mobile telecommunications system (GSM, UMTS, LTE, NR etc.), Bluetooth, infrared, etc.

[0269] It should be noted that the description above only pertains to an example configuration of computer 130. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces or the like. For example, the communication interface 138 may support other radio access technologies than the mentioned UMTS, LTE and NR.

[0270] When the computer 130 functions as an information processing device, the communication interface 138 can further have a respective air interface (providing, e.g., E-UTRA protocols OFDMA (downlink) and SC-FDMA (uplink)) and network interfaces (implementing for example protocols such as Sl-AP, GTP-U, SI -MME, X2-AP, or the like). Moreover, the computer 130 may have one or more antennas and / or an antenna array. The present disclosure is not limited to any particularities of such protocols.

[0271] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0272] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0273] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0274] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0275] Note that the present technology can also be configured as described below. (1) An information processing device, including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal; and circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

[0276] (2) The information processing device of (1), wherein the circuitry is further configured to: obtain three-dimensional information of a virtual object, control each of the plurality of emitters individually to shift the emission parameter further in accordance with the obtained three-dimensional information of the virtual object.

[0277] (3) The information processing device of (1) or (2), wherein the circuitry is further configured to control an emitter of the plurality of emitters to emit a second modulated light signal in which additional information is encoded which allow a second information processing device to decode its own distances correctly.

[0278] (4) The information processing device of anyone of (1) to (3), wherein the time-of-flight device is a direct time-of-flight device and the emission parameter is an emission start timing, or wherein the time-of-flight device is an indirect time-of-flight device and the emission parameter is an emission phase.

[0279] (5) The information processing device of anyone of (1) to (4), wherein the time-of-flight device includes a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain the distance to each illuminated point in the scene.

[0280] (6) A method, including: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; obtaining, by a circuitry, a distance to each illuminated point in the scene; and controlling, by the circuitry, each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

[0281] (7) An information processing device, including: a plurality of receivers, wherein each receiver is configured to detect a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; and circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0282] (8) The information processing device of (7), wherein the circuitry is further configured to obtain, based on the determined relative distances, three-dimensional information of a virtual object.

[0283] (9) The information processing device of (7) or (8), wherein the circuitry is further configured to estimate the global offset.

[0284] (10) The information processing device of (9), further including: an emitter, the emitter being associated with one of the plurality of receivers, wherein the emitter is configured to illuminate a point in the scene by emitting a second modulated light signal; and wherein the circuitry is further configured to: control the emitter and the respective receiver to perform a time-of-flight measurement to determine a distance to the point in the scene, estimate the global offset based on the distance to the point in the scene.

[0285] (11) The information processing device of (9) or (10), further including: an image sensor configured to capture an image of the scene; and wherein the circuitry is further configured to: input the captured image and the determined relative distances into a computer vision algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the captured image and the determined relative distances.

[0286] (12) The information processing device of anyone of (9) to (11), wherein each of the plurality of receivers is further configured to detect a second modulated light signal in which additional information is encoded, and wherein the circuitry is further configured to estimate the global offset based on the additional information. (13) The information processing device of anyone of (9) to (12), wherein the circuitry is further configured to input the determined relative distances into a computer vision algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the determined relative distances.

[0287] (14) A method, including: detecting, by each of a plurality of receivers, a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; determining for each receiver, by a circuitry, based on the respective detected modulated light signal, a detection parameter; and determining for each receiver, by the circuitry, based on the determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0288] (15) A system for assisted optical distance measurement, including: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

[0289] (16) The system of (15), wherein the system supports duplex and simplex assistance for the optical distance measurement between the first and second information processing device.

[0290] (17) The system of (15) or (16), wherein the system supports at least one of one-to-many assistance for the optical distance measurement from the first information processing device to a plurality of the second information processing devices and many-to-many assistance for the optical distance measurement from each of a plurality of the first information processing devices to a plurality of the second information processing devices.

[0291] (18) An information processing device, including: a time-of-flight device including: a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to: detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, detect a second modulated light signal of an external emitter thrown back by the point in the scene; and circuitry configured to: control the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene, determine, based on the respective detected second modulated light signal, a detection parameter, estimate a global offset, determine, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0292] (19) A method, including: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; detecting, by each of a plurality of receivers of the time-of-flight device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter; detecting, by each of the plurality of receivers, a second modulated light signal of an external emitter thrown back by the point in the scene; controlling, by a circuitry, the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene; determining, by the circuitry, based on the respective detected second modulated light signal, a detection parameter, estimating, by the circuitry, a global offset, determining, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

[0293] (20) An information processing device, including: an emitter configured to illuminate a point in a scene by emitting a modulated light signal; and circuitry configured to control the emitter to encode information in the modulated light signal.

[0294] (21) The information processing device of (20), wherein the information is encoded by on / off- modulation of a train of light pulses in accordance with the bit pattern representing the information.

[0295] (22) The information processing device of (21), further including a time-of-flight device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene, wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain a distance to the illuminated point in the scene.

[0296] (23) The information processing device of (22), wherein the circuitry is further configured to stop histogram processing when the train of pulses is not emitted.

[0297] (24) The information processing device of (20), wherein the information is encoded by shifting an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

[0298] (25) The information processing device of (24), further including a time-of-flight device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene, wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain a distance to the illuminated point in the scene.

[0299] (26) The information processing device of (25), wherein the circuitry is further configured to shift histogram processing according to a shift of the emission start timing. (27) A method, including: illuminating, by an emitter, a point in a scene by emitting a modulated light signal; and controlling, by a circuitry, the emitter to encode information in the modulated light signal.

[0300] (28) The method of (27), wherein the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information or by shifting an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

[0301] (29) A system for parallel assisted optical distance measurement and optical information exchange, including: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene, control the plurality of emitters to encode information in the respective modulated signal for transmitting a message; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset, detect the encoded information, decode the encoded information for obtaining the message.

[0302] (30) The system of (29), wherein the first circuitry controls the plurality of emitters to encode the distances to each illuminated point in the scene as the information.

[0303] (31) A system for assisted optical measurement for estimating position information, including: a plurality of stationary emitters, wherein each emitter is configured to illuminate a different point in a scene with a train of light pulses with a predetermined periodicity; an information processing device including: a plurality of receivers, wherein each receiver is configured to detect the train of light pulses thrown back by the different point in the scene, circuitry configured to: determine for each receiver, based on the respective detected train of light pulses during a detection period, a time-of-arrival of the train of light pulses to extract position information.

[0304] (32) The system of (31), wherein the circuitry is further configured to: determine for each receiver, based on the respective detected train of light pulses during a second detection period, a second time-of-arrival of the train of light pulses, estimate for each receiver, based on the time-of-arrival and the second time-of-arrival, a change in distance to the respective point in the scene, estimate, based on the estimated changes in distance, at least one of a change in position and pose of the information processing device.

[0305] (33) The system of (31) or (32), wherein the information processing device includes at least one sensor configured to acquire sensor information, and wherein the circuitry is further configured to extract, based on the time-of-arrival and the sensor information, the position information.

[0306] (34) The system of (33), wherein the at least one sensor includes at least one of an accelerometer, a gyrometer, an inertial measurement unit and an image sensor.

Claims

CLAIMS1. An information processing device, comprising: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal; and circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

2. The information processing device of claim 1, wherein the circuitry is further configured to: obtain three-dimensional information of a virtual object, control each of the plurality of emitters individually to shift the emission parameter further in accordance with the obtained three-dimensional information of the virtual object.

3. The information processing device of claim 1, wherein the circuitry is further configured to control an emitter of the plurality of emitters to emit a second modulated light signal in which additional information is encoded which allow a second information processing device to decode its own distances correctly.

4. The information processing device of claim 1, wherein the time-of-flight device is a direct time-of-flight device and the emission parameter is an emission start timing, or wherein the time-of-flight device is an indirect time-of-flight device and the emission parameter is an emission phase.

5. The information processing device of claim 1, wherein the time-of-flight device includes a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain the distance to each illuminated point in the scene.

6. A method, comprising: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; obtaining, by a circuitry, a distance to each illuminated point in the scene; andcontrolling, by the circuitry, each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene.

7. An information processing device, comprising: a plurality of receivers, wherein each receiver is configured to detect a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; and circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

8. The information processing device of claim 7, wherein the circuitry is further configured to obtain, based on the determined relative distances, three-dimensional information of a virtual object.

9. The information processing device of claim 7, wherein the circuitry is further configured to estimate the global offset.

10. The information processing device of claim 9, further comprising: an emitter, the emitter being associated with one of the plurality of receivers, wherein the emitter is configured to illuminate a point in the scene by emitting a second modulated light signal; and wherein the circuitry is further configured to: control the emitter and the respective receiver to perform a time-of-flight measurement to determine a distance to the point in the scene, estimate the global offset based on the distance to the point in the scene.

11. The information processing device of claim 9, further comprising: an image sensor configured to capture an image of the scene; and wherein the circuitry is further configured to: input the captured image and the determined relative distances into a computer vision algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the captured image and the determined relative distances.

12. The information processing device of claim 9, wherein each of the plurality of receivers is further configured to detect a second modulated light signal in which additional information isencoded, and wherein the circuitry is further configured to estimate the global offset based on the additional information.

13. The information processing device of claim 9, wherein the circuitry is further configured to input the determined relative distances into a computer vision algorithm, wherein the computer vision algorithm is configured to estimate the global offset based on the determined relative distances.

14. A method, comprising: detecting, by each of a plurality of receivers, a modulated light signal thrown back by a different point in the scene, wherein each modulated light signal is emitted by a different external emitter; determining for each receiver, by a circuitry, based on the respective detected modulated light signal, a detection parameter; and determining for each receiver, by the circuitry, based on the determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

15. A system for assisted optical distance measurement, comprising: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset.

16. The system of claim 15, wherein the system supports duplex and simplex assistance for the optical distance measurement between the first and second information processing device.

17. The system of claim 15, wherein the system supports at least one of one-to-many assistance for the optical distance measurement from the first information processing device to a plurality of the second information processing devices and many-to-many assistance for the optical distance measurement from each of a plurality of the first information processing devices to a plurality of the second information processing devices.

18. An information processing device, comprising: a time-of-flight device including: a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, a plurality of receivers, wherein each receiver is associated with a different emitter, and wherein each receiver is configured to: detect the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter, detect a second modulated light signal of an external emitter thrown back by the point in the scene; and circuitry configured to: control the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene, determine, based on the respective detected second modulated light signal, a detection parameter, estimate a global offset, determine, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

19. A method, comprising: illuminating, by each of a plurality of emitters of a time-of-flight device, a different point in a scene by emitting a modulated light signal; detecting, by each of a plurality of receivers of the time-of-flight device associated with a different emitter, the modulated light signal thrown back by the point in the scene that is illuminated by the associated emitter; detecting, by each of the plurality of receivers, a second modulated light signal of an external emitter thrown back by the point in the scene;controlling, by a circuitry, the time-of-flight device to perform a time-of-flight measurement to determine a distance to each illuminated point in the scene; determining, by the circuitry, based on the respective detected second modulated light signal, a detection parameter, estimating, by the circuitry, a global offset, determining, based on the distance to the respective point in the scene and the respective determined detection parameter and the estimated global offset, a distance from the respective point in the scene to the respective external emitter.

20. An information processing device, comprising: an emitter configured to illuminate a point in a scene by emitting a modulated light signal; and circuitry configured to control the emitter to encode information in the modulated light signal.

21. The information processing device of claim 20, wherein the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information.

22. The information processing device of claim 21, further comprising a time-of-flight device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene, wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain a distance to the illuminated point in the scene.

23. The information processing device of claim 22, wherein the circuitry is further configured to stop histogram processing when the train of pulses is not emitted.

24. The information processing device of claim 20, wherein the information is encoded by shifting an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

25. The information processing device of claim 24, further comprising a time-of-flight device, wherein the time-of-flight device includes the emitter and a receiver associated with the emitter, wherein the receiver is configured to detect the modulated light signal thrown back by the different point in the scene,wherein the circuitry is further configured to control the time-of-flight device to perform a time-of-flight measurement to obtain a distance to the illuminated point in the scene.

26. The information processing device of claim 25, wherein the circuitry is further configured to shift histogram processing according to a shift of the emission start timing.

27. A method, comprising: illuminating, by an emitter, a point in a scene by emitting a modulated light signal; and controlling, by a circuitry, the emitter to encode information in the modulated light signal.

28. The method of claim 27, wherein the information is encoded by on / off-modulation of a train of light pulses in accordance with the bit pattern representing the information or by shifting an emission start timing of a light pulse in a train of light pulses in accordance with a bit pattern of a data block.

29. A system for parallel assisted optical distance measurement and optical information exchange, comprising: a first information processing device including: a time-of-flight device including a plurality of emitters, wherein each emitter is configured to illuminate a different point in a scene by emitting a modulated light signal, first circuitry configured to: obtain a distance to each illuminated point in the scene, control each of the plurality of emitters individually to shift an emission parameter in accordance with the obtained distance to the respective point in the scene, control the plurality of emitters to encode information in the respective modulated signal for transmitting a message; and a second information processing device including: a plurality of receivers, wherein each receiver is configured to detect the modulated light signal thrown back by the different point in the scene, second circuitry configured to: determine for each receiver, based on the respective detected modulated light signal, a detection parameter, determine for each receiver, based on the respective determined detection parameter, a relative distance to the respective point in the scene up to a global offset, detect the encoded information, decode the encoded information for obtaining the message.

30. The system of claim 29, wherein the first circuitry controls the plurality of emitters to encode the distances to each illuminated point in the scene as the information.

31. A system for assisted optical measurement for estimating position information, comprising: a plurality of stationary emitters, wherein each emitter is configured to illuminate a different point in a scene with a train of light pulses with a predetermined periodicity; an information processing device including: a plurality of receivers, wherein each receiver is configured to detect the train of light pulses thrown back by the different point in the scene, circuitry configured to: determine for each receiver, based on the respective detected train of light pulses during a detection period, a time-of-arrival of the train of light pulses to extract position information.

32. The system of claim 31, wherein the circuitry is further configured to: determine for each receiver, based on the respective detected train of light pulses during a second detection period, a second time-of-arrival of the train of light pulses, estimate for each receiver, based on the time-of-arrival and the second time-of-arrival, a change in distance to the respective point in the scene, estimate, based on the estimated changes in distance, at least one of a change in position and pose of the information processing device.

33. The system of claim 31, wherein the information processing device includes at least one sensor configured to acquire sensor information, and wherein the circuitry is further configured to extract, based on the time-of-arrival and the sensor information, the position information.

34. The system of claim 33, wherein the at least one sensor includes at least one of an accelerometer, a gyrometer, an inertial measurement unit and an image sensor.

Images