Time-of-flight sensor system

Abstract

A system and method for processing time-of-flight signals from a plurality of time-of-flight sensors. A set of time-of-flight signals is received from each of a plurality of time-of-flight sensors, each set of time-of-flight signals including time-of-flight signals for a plurality of regions of an environment. The sets of time-of-flight signals are processed to identify adjacent time-of-flight sensors and to determine a relative position and orientation between the adjacent time-of-flight sensors.

Description

[0001] Time-of-flight sensor system

[0002] FIELD OF THE INVENTION

[0003] The invention relates to the field of time-of-flight sensors, and in particular, to systems having multiple time-of-flight sensors.

[0004] BACKGROUND OF THE INVENTION

[0005] Time-of-flight, ToF, sensors are used for a wide range of applications, including smart lighting systems, smart appliances, and monitoring applications (e.g. for fall detection). Multi-zone ToF sensors, which are capable of capturing depth information for different zones of an environment simultaneously, are particularly advantageous in applications in which more detailed spatial information is required, such as gesture recognition, automotive applications, robotics and augmented reality.

[0006] The area of an environment covered by a ToF sensor depends on the detection angle of the sensor and the height at which it is mounted (which is limited by the ceiling of the environment). For instance, a ToF sensor with a square field of view (45° x 45°) mounted at a height of 3 meters will cover an area of the floor / ground of 2.6 x 2.6 m. Therefore, in order to cover larger indoor environments, including domestic environments, retail environments, office environments and industrial environments, multiple ToF sensors are required to cover the entire area.

[0007] US2018054876A1 relates to determining respective locations of lighting devices in a service area. It includes a sensing device that receives light signals emitted by a number of lighting devices that are configured in a common plane. The sensing device is located outside the common plane, e.g. below the plane of light outputs of fixture mounted in or below a ceiling. Respective distances between each lighting device and the sensing device are calculated based on the received light signals. The locations of the plurality of lighting devices relative to the sensing device are calculated based on the calculated distances using trilateration, triangulation or parallax.

[0008] US2021056728A1 is about performing an initialization process for a set of FOV sensors using a set of initialization modules that are attachable to or integrated with the set of FOV sensors. A vertical distance associated with the set of FOV sensors is detected. An inter-module distance between a first initialization module and a second initialization module is detected. It is determined whether a placement of the second initialization module with respect to the first initialization module is acceptable based on the vertical distance and the inter-module distance. A feedback signal indicating whether the placement is acceptable is generated. The feedback signal is outputted at one or both of the first initialization module and the second initialization module.

[0009] US2022187462A1 relates to a method implemented by a first time of flight (ToF) sensor includes generating, by the first ToF sensor, a first depth map in accordance with measurements of reflections of an optical signal emitted by the first ToF sensor; communicating, by the first sensor with a second ToF sensor, the first depth map and a second depth map, the second depth map generated by the second ToF sensor; and determining, by the first ToF sensor, a relative location of the first ToF sensor relative to the second ToF sensor in accordance with the first depth map and the second depth map.

[0010] US9109886B1 is about performing calibration of devices to determine a relative location of the devices. The devices may be used to determine distances of surfaces within an environment. The calibration may be performed at different times and / or in response to triggering events to determine a relative location of each of the devices in an environment. After the relative location of the devices is known, the devices may be used to determine a distance of a surface within the environment. In various embodiments, a light sensor may identify a light emitter based on characteristics of the light emitted by the light emitter.

[0011] There is a desire to improve systems having multiple ToF sensors.

[0012] SUMMARY OF THE INVENTION

[0013] The invention is defined by the claims.

[0014] According to examples in accordance with an aspect of the invention, there is provided a computer-implemented method for processing time-of-flight, ToF, signals from a plurality of ToF sensors, the computer-implemented method comprising: receiving, from each of the plurality of ToF sensors, a set of ToF signals acquired by the respective ToF sensor, wherein each set of ToF signals includes ToF signals for a plurality of regions of an environment; processing each set of ToF signals to identify one or more pairs of adjacent ToF sensors, wherein two ToF sensors are considered adjacent if a part of the field of illumination of one ToF sensor is within the field of view of the other ToF sensor; and for each identified pair of adjacent ToF sensors, processing the set of ToF signals acquired by each ToF sensor in the pair to determine a relative position and orientation between the ToF sensors in the pair.

[0015] The inventors have recognized that, when multiple ToF sensors are used to monitor an environment, it is desirable to know the positional relationship(s) between the ToF sensors, and to identify where the fields of view of different ToF sensors overlap, in order to avoid double counting objects (including people) in the environment.

[0016] In some examples, a first ToF sensor and a second ToF sensor are identified as a pair of adjacent ToF sensors in response to identifying a reflection of light emitted by the second sensor in one or more of the ToF signals in the set of ToF signals acquired by the first ToF sensor.

[0017] The inventors have recognized that overlapping fields of view (or overlaps between the field of view of one ToF sensor and the field of illumination of another ToF sensor) may be identified in ToF signals for regions in the overlapping area, as these signals will contain reflected light emitted by more than one sensor.

[0018] In some examples, each set of ToF signals comprises a plurality of ToF signals for each region within a field of view of the ToF sensor that acquired the set of ToF signals; the plurality of signals for each region comprise: a first ToF signal acquired during a time at which light emitted by the plurality of ToF sensors was emitted only by the ToF sensor acquiring the ToF signal; and for each of the plurality of ToF sensors other than the ToF sensor that acquired the set of ToF signals, a further ToF signal acquired during a time at which light emitted by the plurality of ToF sensors was emitted only by the respective ToF sensor and the ToF sensor acquiring the ToF signal; and the step of processing each set of ToF signals to identify one or more pairs of adjacent ToF sensors comprises, for each set of ToF signals: determining a difference in signal strength between the first ToF signal for each region and each further ToF signal for the region; and in response to any difference in signal strength exceeding a predetermined threshold, identifying, as adjacent to the ToF sensor that acquired the set of ToF signals, the ToF sensor corresponding to the further ToF signal for which the difference in signal strength exceeds the predetermined threshold.

[0019] In other words, a ToF signal for a region acquired by a first ToF sensor at a time when a second ToF sensor is inactive is compared with a ToF signal for the same region acquired by the first ToF signal at a time when the second ToF sensor is emitting light. If the difference between these signals exceeds a predetermined threshold, it is determined that the region is within the field of illumination of the second ToF sensor, and that the first and second ToF sensors are therefore adjacent to one another. In some examples, each of the plurality of ToF sensors is configured to emit differently-modulated light; and the step of processing each set of ToF signals to identify one or more pairs of adjacent ToF sensors comprises, for each set of ToF signals, identifying, as adjacent to the ToF sensor that acquired the set of ToF signals, any ToF sensor other than the ToF sensor that acquired the set of ToF signals for which the corresponding differently-modulated light is detected in the set of ToF signals.

[0020] In some examples, the relative position and orientation between the ToF sensors in each pair of adjacent ToF sensors is determined by, for each pair of adjacent ToF sensors: for each ToF sensor in the pair, processing the set of ToF signals acquired by the ToF sensor to define an overlap area in the field of view of the ToF sensor that overlaps the field of view of the other ToF sensor in the pair; and aligning the overlap areas to determine the relative position and orientation between the ToF sensors in the pair.

[0021] In some examples, defining a combined field of view for the plurality of ToF sensors based on the determined relative position and orientation for each pair of adjacent ToF sensors.

[0022] In some examples, processing each set of ToF signals and the defined combined field of view to generate a layout of the environment.

[0023] A layout of the environment defines the size and position of static objects in the environment. In some examples, a type of object may be identified for each object.

[0024] In some examples, each ToF signal is acquired by a single pixel of the ToF sensor that acquired the ToF signal.

[0025] There is also proposed a computer program product comprising computer code means which, when executed on a computing device having a processing system, cause the processing system to perform all of the steps of the method described above.

[0026] According to examples in accordance with an aspect of the invention, there is provided a processing system for processing time-of-flight, ToF, signals from a plurality of ToF sensors, the processing system being configured to: receive, from each of the plurality of ToF sensors, a set of ToF signals acquired by the respective ToF sensor, wherein each set of ToF signals includes ToF signals for a plurality of regions of an environment; process each set of ToF signals to identify one or more pairs of adjacent ToF sensors, wherein two ToF sensors are considered adjacent if a part of the field of illumination of one ToF sensor is within the field of view of the other ToF sensor; and for each identified pair of adjacent ToF sensors, process the set of ToF signals acquired by each ToF sensor in the pair to determine a relative position and orientation between the ToF sensors in the pair. In some examples, the processing system is further configured to define a combined field of view for the plurality of ToF sensors based on the determined relative position and orientation for each pair of adjacent ToF sensors.

[0027] In some examples, the processing system is further configured to process each set of ToF signals and the defined combined field of view to generate a layout of the environment.

[0028] There is also provided a time-of-flight, ToF, sensor system comprising: a plurality of ToF sensors; and the processing system described above.

[0029] In some examples, each ToF sensor comprises a first light source and a second light source, wherein the second light source has a greater power than the first light source; and for each ToF signal in each set of ToF signals, each ToF sensor that emitted light during acquisition of the respective ToF signal emitted light using at least the second light source.

[0030] This enhances the detectability of light emitted by one of the plurality of ToF sensors by adjacent ToF sensors. The second light sources may be disabled during normal operation of the ToF sensors (i.e. once the relative positions and orientations of the ToF sensors have been determined) in order to reduce energy consumption.

[0031] In some examples, each ToF sensor is configured to emit and detect infrared radiation.

[0032] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment s) described hereinafter.

[0033] BRIEF DESCRIPTION OF THE DRAWINGS

[0034] For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0035] Fig. 1 illustrates a time-of-flight, ToF, sensor system, according to an embodiment of the invention;

[0036] Fig. 2 illustrates an example array of signal strengths for ToF signals acquired by a first ToF sensor;

[0037] Fig. 3 illustrates an example array of signal strengths for ToF signals acquired by the same first ToF sensor;

[0038] Fig. 4 illustrates an example array of the difference between the signal strength in the array of Fig. 2 and the signal strength in the array of Fig. 3 for each region; Fig. 5 illustrates the shape of a field of view of each of a first ToF sensor and a second ToF sensor in a ToF sensor system;

[0039] Fig. 6 illustrates a combined field of view for the first and second ToF sensors of Fig. 5;

[0040] Fig. 7 illustrates the field of view for each of a plurality of ToF sensors of a ToF sensor system;

[0041] Fig. 8 illustrates a layout of a room, generated by combining the fields of view of Fig. 7; and

[0042] Fig. 9 illustrates a computer-implemented method for processing time-of-flight, ToF, signals from a plurality of ToF sensors, according to an embodiment of the invention.

[0043] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The invention will be described with reference to the Figures.

[0045] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

[0046] The invention provides a system and method for processing time-of-flight signals from a plurality of time-of-flight sensors. A set of time-of-flight signals is received from each of a plurality of time-of-flight sensors, each set of time-of-flight signals including time-of-flight signals for a plurality of regions of an environment. The sets of time-of-flight signals are processed to identify adjacent time-of-flight sensors and to determine a relative position and orientation between the adjacent time-of-flight sensors.

[0047] Embodiments are at least partly based on the realization that the identification of adjacent time-of-flight sensors may be used to determine the relative position and orientation of time-of-flight sensors in a system comprising a plurality of time-of-flight sensors, and that time-of-flight signals acquired by the sensors may be used to identify adjacent time-of-flight sensors. Illustrative embodiments may, for example, be employed in smart lighting systems, smart appliance systems, security monitoring systems, healthcare monitoring system, and / or fall detection systems.

[0048] Figure 1 illustrates a time-of-flight, ToF, sensor system 100, according to an embodiment of the invention. The ToF sensor system comprises a plurality of ToF sensors 110a, 110b and a processing system 120. The processing system is, itself, an embodiment of the invention.

[0049] Each of the plurality of ToF sensors 110a, 110b may be any sensor configured to emit and detect light (e.g. infrared radiation), and to output, for each of a plurality of regions, a signal responsive to a time of flight of light emitted by the ToF sensor and reflected back to the ToF sensor. For instance, each ToF sensor may emit a short pulse of light, and measure the time taken to detect a reflection of the pulse of light. Alternatively, each ToF sensor may emit a continuous wave of modulated light; the phase of the detected light would then depend on the time of flight of the light.

[0050] Each ToF sensor 110a, 110b may, for example, be mounted on a ceiling of an environment to be monitored. In some examples, one or more of the ToF sensors may be provided within a respective luminaire. The plurality of ToF sensors may, for example, form part of a smart lighting system, a smart appliance system, a security monitoring system, a healthcare monitoring system, or a fall detection system. Further applications for a ToF sensor system will be readily apparent to the skilled person.

[0051] In some examples, each ToF sensor 110a, 110b may comprise a first light source and a second light source, wherein the second light source has a greater power than the first light source. For instance, the first light source may have a power of at least 10 mW and less than 1 W, while the second light source may have a power of at least 1 W (e.g. a power in the range of 1-15 W).

[0052] This allows an increased area of illumination to be achieved (by using the second light source or both light sources) in order to determine a relative position and orientation of each ToF sensor, while the first light source alone may be used during normal operation of each ToF sensor (i.e. after the relative positions and orientations have been determined), in order to reduce energy consumption. In some examples, the power of the second light source may be high enough that light from the sun is not a significant source of interference, enabling the relative position and orientation to be performed at any time of day (i.e. including during the hours of sunlight). In Figure 1, the plurality of ToF sensors comprises a first ToF sensor 110a and a second ToF sensor 110b, mounted in an environment 130. A field of view 11 la of the first ToF sensor 110a covers a first part of the environment, and a field of view 11 lb of the second ToF sensor covers a second part of the environment. The fields of view of the first and second ToF sensors overlap one another in an area 112 of the environment.

[0053] Although Figure 1 illustrates a ToF sensor system having two ToF sensors, the skilled person will readily appreciate that the ToF sensor system may comprise more than two ToF sensors, with the number of ToF sensors depending on a size of the environment to be monitored by the plurality of ToF sensors and a size of the field of view of each ToF sensor.

[0054] Each ToF sensor 110a, 110b is configured to acquire a set of ToF signals 115a, 115b. Each set of ToF signals includes ToF signals for a plurality of regions of the environment 130. Each of the plurality of regions may correspond to a respective pixel of the ToF sensor that acquired the set of ToF signals. In other words, each of the plurality of regions may be defined by a field of view of a pixel of the ToF sensor that acquired the ToF signal for the region. The plurality of regions for a set of ToF signals may together comprise an entire field of view of the ToF sensor that acquired the set of ToF signals.

[0055] In some examples, the sets of ToF signals 115a, 115b may be acquired at night (i.e. at a time between dusk and dawn) so that the plurality of ToF sensors 110a, 110b do not detect light from the sun (e.g. infrared radiation from the sun in the case of ToF sensors configured to emit and detect infrared radiation).

[0056] In examples in which each ToF sensor 110a, 110b comprises a first and second light source (with the second light source having a greater power, as described above), each ToF sensor that emitted light during acquisition of a given ToF signal may have emitted light using at least the second light source (i.e. using the second light source alone or using both light sources). In other words, for each ToF signal in each set of ToF signals, each ToF sensor that is active (i.e. emitting light) during acquisition of the respective ToF signal is emitting light using at least the second light source during acquisition of the respective ToF signal. The second light source may then be disabled once the sets of ToF signals have been acquired, in order to reduce power consumption of the ToF sensors.

[0057] In some examples, each set of ToF signals 115a, 115b may comprise a plurality of ToF signals for each of a plurality of regions within a field of view of the ToF sensor 110a, 110b that acquired the set of ToF signals. The plurality of ToF signals for each region may comprise a first ToF signal and one or more further ToF signals. In some examples, the processing system 120 may control the emission of light by each of the plurality of ToF sensors in order to acquire the first ToF signal and one or more further ToF signals for each region in each set of ToF signals.

[0058] The first ToF signal in the plurality of ToF signals for a region is a ToF signal acquired during a time at which light emitted by the plurality of ToF sensors was emitted only by the ToF sensor acquiring the ToF signal. The one or more further ToF signals comprise, for each of the plurality of ToF sensors other than the ToF sensor that acquired the set of ToF signals, a further ToF signal acquired during a time at which light emitted by the plurality of ToF sensors was emitted only by the respective ToF sensor and the ToF sensor acquiring the ToF signal.

[0059] For instance, in the case of the ToF sensor system 100 illustrated in Figure 1, the set of ToF signals 115a acquired by the first ToF sensor 110a may comprise, for each region within the field of view 11 la of the first ToF sensor, a first ToF signal for the region acquired during a time at which the first ToF sensor was emitting light and the second ToF sensor 110b was not emitting light, and a further ToF signal for the region acquired during a time at which both the first ToF sensor and the second ToF sensor were emitting light.

[0060] Similarly, the set of ToF signals 115b acquired by the second ToF sensor may comprise, for each region within the field of view 11 lb of the second ToF sensor, a first ToF signal for the region acquired during a time at which the second ToF sensor was emitting light and the first ToF sensor was not emitting light, and a further ToF signal for the region acquired during a time at which both the first ToF sensor and the second ToF sensor were emitting light.

[0061] As the skilled person will readily appreciate, this may be extended to any number of ToF sensors. For example, in the case of a ToF sensor system having three sensors, each set of ToF signals may comprise three ToF signals for each region, i.e. the set of ToF signals acquired by a first ToF sensor may comprise, for each region, a first ToF signal acquired during a time at which the first ToF sensor was emitting light and the remaining two ToF sensors were not emitting light, a further ToF signal acquired during a time at which the first ToF sensor and a second ToF sensor were emitting light but a third sensor was not emitting light, and another further ToF signal acquired during a time at which the first ToF sensor and the third ToF sensor were emitting light but the second sensor was not emitting light.

[0062] In other examples, each of the plurality of ToF sensors 110a, 110b is configured to emit differently-modulated light. In this way, light emitted by one of the plurality of ToF sensors may be distinguished from light emitted by any other of the plurality of ToF sensors. Suitable ways to modulate the light emitted by a ToF sensor will be apparent to the skilled person.

[0063] The processing system 120 is configured to receive, from each of the plurality of ToF sensors 110a, 110b, the set of ToF signals 115a, 115b acquired by the respective ToF sensor, and to process each set of ToF signals to identify one or more pairs of adjacent ToF sensors. In Figure 1, each set of ToF signals is received directly from the ToF second that acquired the set of ToF signals; however, as the skilled person will readily appreciate, in some examples, the sets of ToF signals may be received via one or more further devices (e.g. a memory unit storing the sets of ToF signals).

[0064] A first ToF sensor may be considered adjacent to a second ToF sensor if a part of the field of illumination of the second ToF sensor is within the field of view of the first ToF sensor. For instance, in the ToF sensor system 100 illustrated in Figure 1, the first ToF sensor 110a is adjacent to the second ToF sensor 110b. In some cases, adjacent sensors may have overlapping fields of view. In some examples, each ToF sensor may have a field of illumination that is larger than the field of view of the ToF sensor; a pair of sensors may be identified as adjacent even if their fields of view do not overlap, if it is determined that the field of illumination of one sensor is within the field of view of the other.

[0065] As the skilled person will readily appreciate, the technique used to identify adjacent sensors may depend on the ToF signals received from the ToF sensors. In some examples, the set of ToF signals acquired by each ToF sensor may be processed to identify one or more objects in the field of view of the ToF sensor, and to determine a size and position (and optionally, surface reflectance) of each identified object. If an object in the field of view of one ToF sensor is sufficiently similar to an object in the field of view of another ToF sensor (e.g. an object having a same height and surface reflectance), this object may be considered to be the same object and the ToF sensors may be identified as adjacent to one another. Similarly, processing the sets of ToF signals to detect movement across the environment may enable adjacent sensors to be identified. For instance, is movement of a person is detected by a first ToF sensor only, then by the first ToF sensor and a second ToF sensor, and finally by the second ToF sensor only, the first and second ToF sensor may be identified as adjacent to one another.

[0066] In some examples, a first ToF sensor and a second ToF sensor may be identified as a pair of adjacent ToF sensors in response to identifying a reflection of light emitted by the second sensor in one or more of the ToF signals in the set of ToF signals acquired by the first ToF sensor. This may provide a more accurate and reliable determination of the relationships between ToF sensors than by identifying corresponding objects and tracking movement across the fields of view of the ToF sensors.

[0067] In some examples, each set of ToF signals 115a, 115b may comprise a plurality of ToF signals for each region within a field of view of the ToF sensor that acquired the set of ToF signals, where the plurality of ToF signals for each region comprises a first ToF signal and one or more further ToF signals, each corresponding to a different ToF sensor, as described above. The one or more pairs of adjacent ToF sensors may then be identified by processing each set of ToF signals to determine, for each set of ToF signals, a difference in signal strength between the first ToF signal for each region and each further ToF signal for the region. In response to any difference in signal strength exceeding a predetermined threshold, the ToF sensor corresponding to the further ToF signal for which the difference exceeds the predetermined threshold may be identified as adjacent to the ToF sensor that acquired the set of ToF signals. The value of the predetermined threshold for a given ToF sensor may depend on the sensitivity of a receiving element of the ToF sensor. For instance, a suitable predetermined threshold for existing ToF sensors may be between 0.1 and 1 Kcps / SPAD (kilo-count per second per SPAD), depending on sensitivity.

[0068] In some examples, a check may be performed before identifying a ToF sensor corresponding to at least one further ToF signal for which the difference exceeds the predetermined threshold as adjacent to the ToF sensor that acquired the set of ToF signals. For instance, two ToF sensors for which an apparently overlapping area has been identified may be identified as adjacent only if the apparently overlapping area between the two TOF sensors extends to the boundary of the field of view of each of the two ToF sensors and the overlapping area contains a predefined minimum number of regions of the environment (e.g. at least two regions).

[0069] This concept is illustrated by Figures 2 to 4. Figure 2 illustrates an example array 200 of signal strengths for ToF signals acquired by a first ToF sensor (such as the first ToF sensor 110a of the ToF sensor system 100 illustrated in Figure 1), measured in Kcps / SPAD. Each signal strength in the array 200 is the strength of a ToF signal for a respective region within the field of view of the first ToF sensor, acquired during a time at which only the first ToF sensor was emitting light (i.e. no other ToF sensors emitted light during acquisition of each of the ToF signals represented in the array 200). The ToF signals were acquired at night to avoid detecting sunlight. Each region within the field of view corresponds to a single pixel of the first ToF sensor (i.e. each signal strength is the strength of a signal incident on a respective pixel of the ToF sensor). Figure 3 illustrates an example array 300 of signal strengths for ToF signals acquired by the same first ToF sensor. As with the array 200, each signal strength in the array 300 is the strength of a ToF signal for a respective region within the field of view of the first ToF sensor, except the ToF signals represented in the array 300 were acquired during a time at which light was emitted by both the first ToF sensor and a second ToF sensor (but by no other ToF sensor).

[0070] Figure 4 illustrates an example array 400 of the difference between the signal strength in the array 200 and the signal strength in the array 300 for each region (i.e. the array 200 has been subtracted from the array 300). Each region in one half of the field of view (represented by shaded squares in Figure 4) has a difference in signal strength greater than a threshold of 0.5 Kcps / SPAD. In other words, the signal strength of ToF signals for regions in this half of the field of view of the first sensor is significantly higher when the second ToF sensor is also emitting light. This half of the field of view of the first ToF sensor is therefore identified as overlapping with a field of illumination of the second ToF sensor, and the first and second ToF sensors are identified as adjacent to one another.

[0071] Returning to Figure 1, in other examples, each of the plurality of ToF sensors 110a, 110b may be configured to emit differently-modulated light, as described above. The one or more pairs of adjacent ToF sensors may then be identified by processing each set of ToF signals 115a, 115b to identify the source(s) of light detected be each ToF sensor. In particular, for each set of ToF signals, any sensor other than the ToF sensor that acquired the set of ToF signals for which the corresponding differently-modulated light is detected in the set of ToF signals may be identified as adjacent to the ToF sensor that acquired the set of ToF signals.

[0072] For example, in the case of the ToF sensor system 100 illustrated in Figure 1, if each ToF sensor was configured to emit differently-modulated light, pixels of the first ToF sensor 110a that cover the overlap area 112 (i.e. that detect light emitted from the overlap area) would detect the modulated light emitted by the second sensor 110b as well as the modulated light emitted by the first sensor. Similarly, pixels of the second ToF sensor that cover the overlap area would detected the modulated light emitted by the first sensor as well as the modulated light emitted by the second sensor. The first and second sensors would therefore be identified as adjacent to one another.

[0073] Having identified one or more pairs of adjacent sensors, the processing system 120 is configured to, for each identified pair of adjacent sensors, process the set of ToF signals 115a, 115b acquired by each ToF sensor 110a, 110b in the pair to determine a relative position and orientation between the ToF sensors in the pair.

[0074] The relative position and orientation between the ToF sensors in each pair of adjacent ToF sensors may, for example, be determined by processing the set of ToF signals acquired by each ToF sensor in the pair to define an overlap area in the field of view of each ToF sensor in the pair. An overlap area in the field of view of each ToF sensor in the pair is an area that overlaps the field of view of the other ToF sensor in the pair. The overlap areas for the pair of ToF sensors may then be aligned to determine the relative position and orientation between the ToF sensors in the pair.

[0075] This concept is illustrated by Figures 5 and 6. Figure 5 illustrates the shape of a field of view 510 of a first ToF sensor in a ToF sensor system, and the shape of a field of view 520 of a second ToF sensor in the same ToF sensor system. The size and shape of each field of view may be determined by processing the set of ToF signals acquired by the respective ToF sensor to determine a detection plane (e.g. a floor plane) defining the field of view. The shape of the field of view of a ToF sensor indicates the angle at which the ToF sensor is mounted, while the size of the field of view depends on the height at which the ToF sensor is mounted.

[0076] The first ToF sensor is mounted on a ceiling of an environment, facing the floor, and the normal direction of the first ToF sensor is perpendicular to the floor, so the field of view 510 has a square shape. The second ToF sensor is mounted on the ceiling of the environment at a greater height than and at a different angle to the first ToF sensor, resulting in a larger field of view that has trapezoidal shape.

[0077] A region 515 (shaded in Figure 5) of the first field of view 510 of the first ToF sensor has been identified as overlapping with the field of view 520 of the second ToF sensor, while a region 525 of the field of view of the second ToF sensor has been identified as overlapping with the field of view of the first sensor.

[0078] Figure 6 illustrates a combined field of view 600 for the first and second ToF sensors, formed by aligning the overlapping regions 515 and 525. From this combined field of view, the relative position and orientation of the first and second sensors may be determined (each ToF sensor is centered with respect to the field of view of the ToF sensor).

[0079] In some situations, the field of view of one ToF sensor may not overlap with the field of view of an adjacent ToF sensor, but only with the field of illumination of the adjacent ToF sensor. In these cases, the relative position and orientation of the pair of adjacent ToF sensors may be determined by aligning the field of view of a first ToF sensor in the pair with the field of illumination of a second ToF sensor in the pair, based on a first area of overlap between the field of view of the first ToF sensor and the field of illumination of the second ToF sensor, and / or aligning the field of illumination of the first ToF sensor with the field of illumination with the field of view of the second ToF sensor, based on a second area of overlap between the field of illumination of the first ToF sensor and the field of view of the second ToF sensor. Since the shape of the field of view and field of illumination of each ToF sensor, and the position of the ToF sensor with respect to its field of view and field of illumination (e.g. a ToF sensor is typically centered with respect to its own field of view and field of illumination), is known to the respective ToF sensor, the alignment of the field of view of one ToF sensor with the field of illumination of the other ToF sensor in the pair enables the relative position and orientation of the ToF sensors to be determined.

[0080] Returning to Figure 1, in some examples, the processing system 120 may be further configured to define a combined field of view for the plurality of ToF sensors 110a, 110b based on the determined relative position and orientation for each pair of adjacent ToF sensors. In some examples, a combined field of view may be defined for each pair of adjacent sensors in order to determine the relative position and orientation, as described above; in ToF sensor systems with more than two ToF sensors, the combined fields of views for pairs of adjacent sensors may be combined to define an overall combined field of view.

[0081] A combined field of view for the plurality of ToF sensors 110a, 110b may enable more precise tracking of movement in the environment 130 (e.g. when movement of a person is detected in the overlapping area 112, the use of a combined field of view would mean that this would be identified as one person, not two) and more accurate object recognition (e.g. when a piece of furniture is partly within the field of view of one ToF sensor and partly within the field of view of another ToF sensor, the combined field of view would provide the size and shape of the whole of the piece of furniture).

[0082] In some examples, the processing system 120 may be further configured to process each set of ToF signals 115a, 115b and the defined combined field of view to generate a layout of the environment 130. A layout of the environment indicates the size and position of each of a plurality of objects in the environment. The layout of the environment may be used for a variety of applications: for instance, in smart lighting and smart appliance systems, the layout may be used to facilitate personalized and effective rule configuration of the system. In healthcare monitoring systems, the layout may enable the provision of more accurate activity and / or health reports, and may be used to analyze risk factors in the environment 130. Figure 7 illustrates the field of view 710, 720, 730, 740, 750, 760, 770, 780 for each of a plurality of ToF sensors of a ToF sensor system in a single room. In each field of view, one or more objects have been identified in the field of view (represented by shaded blocks in Figure 7). The striped regions in the field of view illustrated regions that are outside the walls of the room). In the example of Figure 7, each ToF sensor is mounted such that a normal direction of the ToF sensor is perpendicular to the floor; each field of view therefore has a square shape.

[0083] Pairs of adjacent ToF sensors may be identified using any of the methods described above (e.g. by identifying that an object in the field of view of one sensor is the same object as an object in the field of view of another sensor based on object height, size and surface reflectance, or by identifying that one sensor is able to detect light emitted by another sensor). For instance, the field of view 710 may be identified as overlapping with the field of view 720, indicating that the ToF sensors having these fields of view are adjacent to one another.

[0084] The relative position and orientation of each ToF sensor in each adjacent pair may then be determined, for example, by aligning overlapping areas in the fields of view. This allows a combined field of view of the plurality of ToF sensors to be defined, and a layout of the environment to be generated.

[0085] Figure 8 illustrates a layout 800 of the room, generated by combining the fields of view 710, 720, 730, 740, 750, 760, 770, 780. The dot at the center of each field of view indicates the position of the ToF sensor having that field of view. The layout of the room indicates the size and position of each object identified by the ToF sensors.

[0086] In some examples, further information may be used to refine the generated layout of the environment, for example to determine a type of object for each object in the generated layout, and / or to include in the layout one or more additional objects that are too small to be identified by the ToF sensors (this may occur particularly where the ToF sensor has a low resolution). For instance, returning to Figure 1, the processing system 120 may be configured to receive a user input indicating a type of object for each object included in the generated layout.

[0087] In another example, the processing system 120 may be configured to receive one or more photographs of the environment 130, and to process the one or more photographs using an object recognition algorithm to identify each object in the one or more photographs. Each object included in the generated layout may then be matched with an object identified in the one or more photographs, based, for example, on the size, position and IR reflectance of the object, as determined based on the ToF signals. Information about the object determined from the one or more photographs may then be added to the generated layout. Further, the one or more photographs may be used to identify one or more additional objects in the environment, and to determine the size of each additional object and the position of each additional object relative to one or more of the objects already included in the generated layout. In this way, the one or more additional objects may be added to the layout.

[0088] Although the fields of view illustrated in the Figures have quadrilateral shapes, the skilled person will readily appreciate that the ToF sensors may have any shape of field of view (e.g. the field of view may be circular or elliptical). Pairs of adjacent ToF sensors and the relative position and orientation of each ToF sensor may be determined as described above for any shape of field of view.

[0089] Figure 9 illustrates a computer-implemented method 900 for processing time-of-flight, ToF, signals from a plurality of ToF sensors, according to an embodiment of the invention.

[0090] The computer-implemented method 900 begins at step 910, at which a set of ToF signals is received from each of the plurality of ToF sensors. Each set of ToF signals is acquired by the respective ToF sensor and includes ToF signals for a plurality of regions of an environment.

[0091] At step 920, each set of ToF signals is processed to identify one or more pairs of adjacent ToF sensors. Two ToF sensors are considered adjacent if a part of the field of illumination of one ToF sensor is within the field of view of the other ToF sensor. In some examples, a first ToF sensor and a second ToF sensor may be identified as a pair of adjacent ToF sensors in response to identifying a reflection of light emitted by the second sensor in one or more of the ToF signals in the set of ToF signals acquired by the first ToF sensor. The one or more pairs of adjacent ToF sensors may be identified using any of the techniques described above.

[0092] At step 930, for each identified pair of adjacent ToF sensors, the set of ToF signals acquired by each ToF sensor in the pair is processed to determine a relative position and orientation between the ToF sensors in the pair. In some examples, the relative position and orientation between the ToF sensors in a pair may comprise defining, for each sensor in the pair, an overlap area in the field of view of one ToF sensor that overlaps the field of view of the other ToF sensor in the pair, and aligning the overlap areas. In some examples, the computer-implemented method 900 may further comprise a step 940 of defining a combined field of view for the plurality of ToF sensors based on the determined relative position and orientation for each pair of adjacent ToF sensors.

[0093] In some examples, the computer-implemented method 900 may further comprise a step 950 of processing each set of ToF signals and the defined combined field of view to generate a layout of the environment.

[0094] It will be understood that the disclosed methods are computer-implemented methods. As such, there is also proposed a concept of a computer program comprising code means for implementing any described method when said program is run on a processing system.

[0095] As discussed above, embodiments make use of a controller. The controller can be implemented in numerous ways, with software and / or hardware, to perform the various functions required. A processor is one example of a controller which employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. A controller may however be implemented with or without employing a processor, and also may be implemented as a combination of dedicated hardware to perform some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

[0096] Examples of controller components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0097] In various implementations, a processor or controller may be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the required functions. Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller.

[0098] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. Functions implemented by a processor may be implemented by a single processor or by multiple separate processing units which may together be considered to constitute a “processor”. Such processing units may in some cases be remote from each other and communicate with each other in a wired or wireless manner.

[0099] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0100] A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0101] If the term “adapted to” is used in the claims or description, it is noted the term “adapted to” is intended to be equivalent to the term “configured to”. If the term “arrangement” is used in the claims or description, it is noted the term “arrangement” is intended to be equivalent to the term “system”, and vice versa.

[0102] Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. A computer-implemented method (900) for processing time-of-flight, ToF, signals from a plurality of fixed mounted multi -zone ToF sensors (110a, 110b), the computer-implemented method comprising:receiving, from each of the plurality of ToF sensors, a set of ToF signals (115a, 115b) acquired by the respective ToF sensor, wherein each set of ToF signals includes ToF signals for a plurality of regions of an environment (130);processing each set of ToF signals to identify one or more pairs of adjacent ToF sensors, wherein a ToF signal for a region acquired by a first ToF sensor of the plurality of ToF sensors at a time when a second ToF sensor of the plurality of ToF sensors is inactive is compared with a ToF signal for the same region acquired by the first ToF at a time when the second ToF sensor is emitting light, and if the difference between these signals exceeds a predetermined threshold, it is determined that the region is within the field of illumination of the second ToF sensor and the region is a part of an overlap area of the first ToF sensor and second ToF sensor, and that the first and second ToF sensors are regarded as adjacent to one another; andfor each identified pair of adjacent ToF sensors, processing the set of ToF signals acquired by each ToF sensor in the pair to determine an overlap area (112, 515, 525) in the field of view (Illa, 510) of the ToF sensor that overlaps the field of view (11 lb, 520) of the other ToF sensor in the pair; andaligning the overlap areas to determine a relative position and orientation between the ToF sensors in the pair.

2. The computer-implemented method (900) of claim 1, wherein a first ToF sensor (110a) and a second ToF sensor (110b) are identified as a pair of adjacent ToF sensors in response to identifying a reflection of light emitted by the second sensor in one or more of the ToF signals in the set of ToF signals (115a) acquired by the first ToF sensor.

3. The computer-implemented method (900) of claim 1 or 2, wherein:each of the plurality of ToF sensors (110a, 110b) is configured to emit differently-modulated light; andthe step of processing each set of ToF signals (115a, 115b) to identify one or more pairs of adjacent ToF sensors comprises, for each set of ToF signals, identifying, as adjacent to the ToF sensor that acquired the set of ToF signals, any ToF sensor other than the ToF sensor that acquired the set of ToF signals for which the corresponding differently-modulated light is detected in the set of ToF signals.

4. The computer-implemented method (900) of any of claims 1 to 3, further comprising defining a combined field of view (600) for the plurality of ToF sensors (110a, 110b) based on the determined relative position and orientation for each pair of adjacent ToF sensors and size and shape of field of view of each ToF sensor.

5. The computer-implemented method (900) of claim 4, wherein the size and shape of field of view of each ToF sensor are determined by processing the set of ToF signals acquired by the respective ToF sensor to determine a detection plane defining the field of view.

6. The computer-implemented method (900) of claim 4 or 5, further comprising processing each set of ToF signals (115a, 115b) and the defined combined field of view (600) to generate a layout (800) of the environment.

7. The computer-implemented method (900) of any of claims 1 to 6, wherein each ToF signal is acquired by a single pixel of the ToF sensor (110a, 110b) that acquired the ToF signal.

8. A computer program product comprising computer code means which, when executed on a computing device having a processing system, cause the processing system to perform all of the steps of the method (900) according to any of claims 1 to 7.

9. A processing system (120) for processing time-of-flight, ToF, signals from a plurality of fixed mounted multi -zone ToF sensors (110a, 110b), the processing system being configured to:receive, from each of the plurality of ToF sensors, a set of ToF signals (115a, 115b) acquired by the respective ToF sensor, wherein each set of ToF signals includes ToF signals for a plurality of regions of an environment (130);process each set of ToF signals to identify one or more pairs of adjacent ToF sensors, wherein a ToF signal for a region acquired by a first ToF sensor of the plurality of ToF sensors at a time when a second ToF sensor of the plurality of ToF sensors is inactive is compared with a ToF signal for the same region acquired by the first ToF at a time when the second ToF sensor is emitting light, and if the difference between these signals exceeds a predetermined threshold, it is determined that the region is within the field of illumination of the second ToF sensor and the region is a part of an overlap area of the first ToF sensor and second ToF sensor, and that the first and second ToF sensors are regarded as adjacent to one another; andfor each identified pair of adjacent ToF sensors, process the set of ToF signals acquired by each ToF sensor in the pair to determine an overlap area (112, 515, 525) in the field of view (Illa, 510) of the ToF sensor that overlaps the field of view (11 lb, 520) of the other ToF sensor in the pair; andaligning the overlap areas to determine a relative position and orientation between the ToF sensors in the pair.

10. The processing system (120) of claim 9, wherein the processing system is further configured to define a combined field of view (600) for the plurality of ToF sensors (110a, 110b) based on the determined relative position and orientation for each pair of adjacent ToF sensors and size and shape of field of view of each ToF sensor.

11. The processing system (120) of claim 10, wherein the processing system is further configured to determine the size and shape of field of view of each ToF sensor by processing the set of ToF signals acquired by the respective ToF sensor to determine a detection plane defining the field of view.

12. The processing system (120) of claim 10 or 11, wherein the processing system is further configured to process each set of ToF signals (115a, 115b) and the defined combined field of view (600) to generate a layout (800) of the environment.

13. A time-of-flight, ToF, sensor system (100) comprising:a plurality of ToF sensors (110a, 110b); andthe processing system (120) of any of claims 9 to 12.

14. The system (100) of claim 13, wherein:each ToF sensor (110a, 110b) comprises a first light source and a second light source, wherein the second light source has a greater power than the first light source; and for each ToF signal in each set of ToF signals (115a, 115b), each ToF sensor that emitted light during acquisition of the respective ToF signal emitted light using at least the second light source.

15. The system (100) of claim 13 or 14, wherein each ToF sensor (110a, 110b) is configured to emit and detect infrared radiation.

Images