SYSTEM AND METHOD FOR OBJECT LOCATION IN AN INDOOR ENVIRONMENT

Description

[0001] The present invention pertains to a system and a method for object localization, particularly in a passenger cabin of a vehicle, especially of an aircraft.

[0002] Although it can be used in many applications, the present invention and the problems underlying it are explained in greater detail in relation to passenger aircraft. However, the methods and devices described can likewise be used in different vehicles and in all sectors of the transport industry, e.g. for road vehicles, for rail vehicles, for general aircraft or for watercraft. Moreover, the systems and methods disclosed herein may be used for other environments, such as automated or semi-automated assembly lines for tracking the flow of components.

[0003] The vast majority of current indoor localization methods, available on the market, requires the installation of additional equipment in the area where the localization is performed. This may be considered a disadvantage in the case of object localization within a vehicle cabin since additional equipment normally implies additional weight as well as extra certification. Hence, there is a need to find simpler solutions for indoor localization that are compatible with the specific requirements of vehicle cabins.

[0004] Document WO 2018 / 094502 A1 describes a device-free localization method within smart indoor environments within an indoor area, e.g. an apartment or other living area, covered by wireless networks utilizing active off-the-shelf-devices. The method exploits existing wireless communication signals and machine learning techniques in order to automatically detect entrance into the area and track the location of a moving subject within the sensing area.

[0005] Document Ma et al., "WiFi Sensing with Channel State Information: A Survey," ACM Computer Survey, 2019, provides an overview of signal processing techniques, algorithms, applications and performance results of WiFi sensing with Channel State Information (CSI). CSI represents how wireless signals propagate from a transmitter to a receiver at certain carrier frequencies along multiple paths. For example, CSI may be a 3D matrix of complex values representing the amplitude attenuation and phase shift of multi-path WiFi channels. A time series of CSI measurements captures how wireless signals travel through surrounding objects and humans in time, frequency and spatial domains, and hence it can be used for different wireless sensing applications. WO 2016 / 011433 A2 discloses wireless positioning systems. XP033586503 discloses CSI-Based Device-Free Indoor Localization Using Convolutional Neural Networks. XP036923017 discloses a robust minimal hardware low-cost device-free WLAN localization system.

[0006] Against this background, it is an object of the present invention to find simple, cost and weight efficient solutions for passively locating objects, for example in an indoor environment such as a vehicle cabin.

[0007] This object is achieved by a system having the features of claim 1 and a method having the features of claim 6. Further embodiments are set out in the dependent claims.

[0008] According to an aspect of the invention, a system for object localization in an indoor environment, particularly in a passenger cabin of a vehicle, in particular of an aircraft, comprises a wireless communication infrastructure adapted to facilitate wireless communication within the indoor environment via a wireless communication network and comprising at least one wireless access point adapted to provide user devices access to the wireless communication network within the indoor environment, wherein the wireless access point is further adapted to acquire signal propagation data of wireless signals transmitted over the wireless communication network along multiple propagation paths; and a computing element configured to analyze the signal propagation data and extract localization data from the signal propagation data, the localization data specifying the position of objects located within the indoor environment.

[0009] According to a further aspect of the invention, an aircraft has a passenger cabin equipped with a system according to the invention.

[0010] According to yet a further aspect of the invention, a method for object localization in a passenger cabin of a vehicle, in particular of an aircraft, using a system according to the invention comprises acquiring signal propagation data of wireless signals transmitted over the wireless communication network along multiple propagation paths; and analyzing the signal propagation data and extracting localization data from the signal propagation data, the localization data specifying the position of objects located within the passenger cabin.

[0011] One idea of the present invention is to use a wireless communication architecture already present inside aircraft and other typical vehicle cabins to passively locate mobile and immobile objects in the indoor environment of the respective cabin. To this end, the present invention utilizes the hardware of the communication devices of the wireless access points provided in the vehicle for wireless internet access of user devices, e.g. personal electronic devices like smartphones, tablet computers and so on. This is based on the inventive insight that these communication devices are principally capable of acquiring all information present in the wireless signals transmitted over the wireless communication network of the cabin and relevant for tracking and / or locating objects within the cabin. In many applications, it is sufficient to merely adjust and / or update the software of the respective communication devices of the wireless access points in order to enable the system for object localization. Radio-frequency signals (e.g. amplitude and phase of RF symbols) used to exchange information between the different access points can then be used to sense and fingerprint the environment and detect changes. Changes can also be detected by analysing the changes over time of the radio signals due to the movement of objects (e.g. Doppler shift), including vibration and rotation. Those changes may then be used for predicting the position of an object.

[0012] The present invention thus is able to locate objects in a vehicle cabin without additional electronics (passive localization) and without adding special sensing equipment. This makes it possible to locate objects, such as forgotten luggage from passengers, for example, or safety equipment like life vests or the like, without the need of additional equipment on the respective object. During aircraft operation, the present invention will reduce the workload of the cabin crew and speed up turnaround times.

[0013] The invention may generally be used for the localization of people, objects and small displacements (e.g. vibrations) inside a vehicle cabin, such as an aircraft cabin. Such localization capabilities can be used to ease deployment of aircraft equipment (e.g. sensors) in the final assembly line, during aircraft operation to support crew activities (e.g. check aircraft status after landing), or for predictive maintenance.

[0014] The computing element of the present invention may be installed anywhere within the indoor environment, such as a vehicle cabin, and may be in wireless or wired data communication with the wireless access points. The system may thus offer a distributed data analysis approach. However, the computing element may also be provided at the respective access point. Moreover, the computing element may even be provided outside of the indoor environment.

[0015] Advantageous embodiments and improvements of the present invention are found in the subordinate claims.

[0016] According to an embodiment of the invention, the wireless communication infrastructure may comprise a plurality of wireless access points. The signal propagation data may comprise correlation data of the wireless access points.

[0017] The present invention basically works with one or more access points. In the presence of multiple access points however, the proposed method is able to exploit the spatial feature of fingerprints from multiple adjacent access points placed inside the indoor environment, such as a vehicle cabin. Such a spatial relationship among adjacent fingerprints makes it robust against signal variations. Further, challenging scenarios such as non-line of sight environments can be tackled. Based on the collected data from the different wireless access points, a preprocessing and normalization process may be applied in order to extract relevant features.

[0018] In order to increase the accuracy of the system, the correlation of the data generated by the various access points available in the indoor environment may be performed. Collected data may be correlated according to their time of measurement in order to have a common measurement of the environment at a given time. This use of multiple access points enables to cover a larger part of the indoor environment from different measurement points, meaning that the area where indoor positioning is done will have better coverage. This is especially helpful in an indoor environment with other objects, since those other objects may block partially the propagation of the RF signals. Multiple access points can also be used to increase the accuracy of the system in line of sight and non-line of sight situations.

[0019] According to an embodiment of the invention, the signal propagation data may comprise channel state information (CSI).

[0020] In wireless communications, CSI characterizes how wireless signals propagate from a transmitter to a receiver at certain carrier frequencies. These information represent the combined effect of, for example, scattering, fading, and power decay with distance. CSI amplitude and phase are impacted by multi-path effects including amplitude attenuation and phase shift. Specifically, the CSI is impacted by the displacements and movements of the transmitter, receiver, and surrounding objects. In other words, CSI captures the wireless characteristics of the environment. These characteristics, assisted by mathematical modelling or machine learning algorithms, can be used for different sensing applications.

[0021] According to an embodiment of the invention, the signal propagation data may comprise spatial, frequency and / or temporal variations of the wireless signals within the passenger cabin.

[0022] For example, CSI amplitude variations in the time domain have different patterns for different movements, which can be used for motion detection, including vibration and rotation. CSI phase shifts in the spatial and frequency domains, i.e., transmit / receive antennas and carrier frequencies, are related to signal transmission delay and direction, which can be used for object localization and tracking.

[0023] According to an embodiment of the invention, each wireless access point may be a standard commercial wireless device operating according to a IEEE 802.11 standard. However, the invention may also rely on other wireless standards in other embodiments.

[0024] For example, the communication architecture may be be based on standard technologies such as IEEE 802.11n or IEEE 802.11ac or the like.

[0025] According to an embodiment of the invention, the computing element may be configured to execute a machine-learning algorithm on the signal propagation data. The machine-learning algorithm may be configured to extract the position data based on training data being generated using an object position measurement system within the passenger cabin. Alternatively or additionally, the machine-learning algorithm may be configured to extract the position data based on training data being generated using a predetermined test object arrangement within the passenger cabin. Correspondingly, the method may comprise executing a machine-learning algorithm on the signal propagation data extracting the position data based on training data being generated using at least one of an object position measurement system and a predetermined test object arrangement within the passenger cabin.

[0026] The data acquired by the wireless access points may be passed to a machine learning algorithm which makes a prediction on the position of objects in the indoor environment. In order to train the algorithm, a training phase may first be performed where the position of objects is measured. In order to find this position, an external positioning system may be used, for example. This external positioning system can be based on a standard computer vision approach. Using one or multiple cameras and markers, specific objects can be marked and their positions can be easily extracted. Another possible external positioning system can be based on the usage of accelerometers or gyroscopes, which are used to compute an object's displacement by transforming the information of an external sensor. Given original coordinates of an object (e.g. aircraft entry door), the final position can be estimated according to the directional displacement.

[0027] The computed positions, based on external positioning systems such as computer vision or accelerometers, may then be used in order to train the machine learning algorithm. Multiple measurements may then be performed in a given target environment, where objects are moved throughout the environment while the raw information from the wireless network architecture is recorded. Standard optimization methods may then be used for training the machine learning algorithm. Once the training is finalized, an evaluation phase may be performed. The position of objects may then be directly predicted using only the information coming from the wireless network architecture.

[0028] According to an embodiment of the invention, the machine-learning algorithm may be configured to extract the locations of static objects within the indoor environment. Correspondingly, the method may comprise extracting the locations of static objects within the indoor environment. It is understood however that the machine-learning algorithm may also extract the locations of moving / mobile objects.

[0029] For example, the machine learning algorithm may be trained on a predetermined test object arrangement within the indoor environment, such as a passenger cabin. As soon as an object from this test object arrangement is shifted to a different position within the cabin or removed completely from the cabin during operation of the system, the signal propagation data will change compared to the training phase. These changes will even reflect in the signal propagation data if the object is static after the movement. Moreover, additional (static) objects within the cabin that were not considered during the training phase may affect the signal propagation data during operation of the system.

[0030] According to an embodiment of the invention, the signal propagation data may be provided as raw data on the physical layer of the wireless communication infrastructure.

[0031] For example, raw data from the physical layer (in the form of IQ-samples) of the communication layer such as Channel-State-Information (CSI) may be used. Moreover, changes over time of the IQ samples can be utilized. In case multiple wireless access points are used, the collected data may be correlated according to their time of measurement in order to have a common measurement of the environment at a given time.

[0032] According to an embodiment of the invention, the system may further comprise a data aggregation unit in wireless communication with the wireless access points and configured to collect the signal propagation data of the wireless access points and transfer it to the computing element. Each wireless access point may be configured to preprocess the raw data generated at the respective wireless access point and transfer it to the data aggregation unit.

[0033] Hence, in order to treat the data from the different access points efficiently and avoid overloading the communication infrastructure with additional sensing data, the processing may be performed using a distributed approach. This solution enables to easily scale the invention with the number of access points. Each access point may already locally preprocess its own raw data before sending it to the aggregation unit. The aggregation unit will then collect the preprocessed data produced by all the access points in order to make the final prediction of the position.

[0034] The invention will be explained in greater detail with reference to exemplary embodiments depicted in the drawings as appended.

[0035] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate the embodiments of the present invention and together with the description serve to explain the principles of the invention. Other embodiments of the present invention and many of the intended advantages of the present invention will be readily appreciated as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. In the figures, like reference numerals denote like or functionally like components, unless indicated otherwise. Fig. 1 shows a schematic view of a system for object localization in a passenger cabin of an aircraft according to an embodiment of the invention. Fig. 2 shows a schematic flow diagram of a method for object localization using the system of Fig. 1. Fig. 3 shows a schematic side view of an aircraft having a passenger cabin equipped with the system of Fig. 1. Fig. 4 shows a different schematic view of the system of Fig. 1.

[0036] Although specific embodiments are illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention.

[0037] Figures 1 and 4 show schematic views of a system 10 for object localization in an indoor environment 101. Exemplarily, the system 10 is explained in conjunction with a passenger cabin 101 of an aircraft 100 as the indoor environment, e.g. the passenger aircraft 100 depicted in Fig. 3, according to an embodiment of the invention.

[0038] The system 10 comprises a wireless communication infrastructure used to facilitate wireless communication within the passenger cabin 101 via a wireless communication network 2 and comprising a plurality of wireless access points 1 used to provide user devices access to the wireless communication network 2 within the passenger cabin 101. To this end, each wireless access point 1 comprises a standard commercial wireless device operating for instance according to a IEEE 802.11 standard, e.g. IEEE 802.11n or IEEE 802.11ac or similar. This wireless communication infrastructure corresponds to wireless communication systems as they are widely used within vehicle cabins, in particular aircraft cabins, for providing the crew and / or the passengers with wireless computing and / or communication access, e.g. to the Internet, as well as for wireless sensor networks and machine-to-machine communication.

[0039] However, contrary to these well-known systems, the wireless access points 1 are further adapted to acquire signal propagation data of wireless signals 3 transmitted over the wireless communication network 2 along multiple propagation paths. The signal propagation data may particularly comprise channel state information (CSI) and similar quantitative information on the spatial, frequency and temporal variations of the wireless signals 3 within the passenger cabin 101. These data can be used to monitor the indoor area traversed by the wireless signals and to establish if any change in the positions of objects 5 takes place within the passenger cabin 101.

[0040] For example, amplitude and phase of the radio-frequency signals exchanged between the wireless access points 1 may be analyzed to get a fingerprint of the passenger cabin 101 and to detect changes within this environment, which then can be used to predict the position of an object 5 within the passenger cabin 101. Also changes over time of the radio signals due to the movement of objects 5 may be analyzed (e.g. via Doppler shift) in order to track moving objects 5 within the passenger cabin 101. The primary goal is to locate objects 5 passively in the passenger cabin 101 (cf. Figs. 1 and 4) by using a finger-printing technique based on machine learning, aggregation of the data from multiple measurement spots and distributed processing.

[0041] To this end, the wireless access points 1 process the signal propagation data as raw data on the physical layer of the wireless communication infrastructure (e.g. in the form of IQ-samples). The collected signal propagation data is correlated according to their time of measurement. Based on the collected data from the different wireless access points 1, a preprocessing and normalization process is applied in order to extract relevant features. Formatting of the data for processing, using machine learning techniques, is also performed at this step. In the example of Fig. 4, each access point 1 comprises a processing unit 6 for this preprocessing step (cf. further below).

[0042] For the analysis of the acquired signal propagation data, the system 10 further comprises a data aggregation unit 6 in wireless communication with the wireless access points 1 (cf. Fig. 4). The data aggregation unit 6 receives the preprocessed signal propagation data from the wireless access points 1 and forwards it to a central computing element 4 configured to analyze the signal propagation data and extract localization data from the signal propagation data. The computing element 4 may be a processor or similar, which is integrated into a computing framework of the aircraft 100 and which may be coupled to a cabin management system of the aircraft 100, for example. The localization data specify the position, and possibly also the movement, of objects 5 located within the passenger cabin 101.

[0043] The computing element 4 is configured to execute a machine-learning algorithm on the signal propagation data in order to extract the position data based on training data. The training data may be generated, for example, using an object position measurement system within the passenger cabin 101 to detect and track labelled objects 5 during a training phase. At the same time, the system 10 may constantly monitor signal propagation data within the passenger cabin 101. The recorded positional information of these tracked objects 5 may be used in conjunction with the acquired signal propagation data to train the machine-learning algorithm during this test phase.

[0044] Alternatively or additionally, other training variants are possible, as the person of skill will readily realize. Another possible external positioning system can be based on the usage of accelerometers or gyroscopes. In another example, a predetermined test arrangement of test objects 5 may be used to calibrate the signal propagation data in a training phase. The machine-learning algorithm may then be executed on signal propagation data recorded during operation of the system 10.

[0045] Based on such a training phase, the machine-learning algorithm is then able to extract the positions not only of moving, but also of static objects 5 within the passenger cabin 101. For example, the training phase may have been performed on an empty cabin. The machine-learning algorithm now will notice any changes to this empty state of the passenger cabin 101. Thus, any new objects 5 within the passenger cabin 101 may be detected on basis of respective changes within the signal propagation data compared to the training scenario. In a similar vein, the system 10 will be able to distinguish any change in position of objects 5 present in the passenger cabin 101 during the training phase.

[0046] As a result, objects 5 such as forgotten luggage from passengers or safety equipment such as life vests may be detected and possibly tracked without introducing expensive and heavy additional equipment within the passenger cabin 101. On the contrary, an already existing wireless communication infrastructure merely needs to be enhanced, e.g. by means of a simple software update, to be able to record and analyze signal propagation data, which is generated within the cabin in any case.

[0047] Fig. 2 shows a schematic flow diagram of a method for object localization using the system 10 of Figs. 1 and 4. The method M comprises under M1 acquiring signal propagation data of wireless signals 3 transmitted over the wireless communication network 2 along multiple propagation paths. The method M further comprises under M2 analyzing the signal propagation data and extracting localization data from the signal propagation data. This may comprise executing a machine-learning algorithm on the signal propagation data extracting the position data based on training data being generated using at least one of an object position measurement system and a predetermined test object arrangement within the passenger cabin. Specifically, the machine-learning algorithm may extract the locations of static objects 5 within the passenger cabin 101.

[0048] In the foregoing detailed description, various features are grouped together in one or more examples or examples with the purpose of streamlining the disclosure. It is to be understood that the above description is intended to be illustrative, and not restrictive. The invention is defined by the apended claims.List of reference signs

[0049] 1wireless access point 2wireless communication network 3wireless signal 4computing element 5object 6data aggregation unit 7processing unit 10system for object localization 100aircraft 101passenger cabin Mmethod M1-M2method steps

Claims

1. Aircraft (100) having a passenger cabin (101) equipped with a system (10) for object localization, the system (10) comprising: a wireless communication infrastructure adapted to facilitate wireless communication within the passenger cabin (101) via a wireless communication network (2) and comprising at least one wireless access point (1) adapted to provide user devices access to the wireless communication network (2) within the passenger cabin (101), wherein the wireless access point (1) is further adapted to acquire signal propagation data of wireless signals (3) transmitted over the wireless communication network (2) along multiple propagation paths; and a computing element (4) configured to analyze the signal propagation data and extract localization data from the signal propagation data, the localization data specifying the position of objects (5) located within the passenger cabin (101), wherein the computing element (4) is configured to execute a machine-learning algorithm on the signal propagation data, wherein the machine-learning algorithm is configured to extract the position data based on training data being generated using at least one of an object position measurement system and a predetermined test object arrangement within the passenger cabin (101); wherein the wireless communication infrastructure comprises a plurality of wireless access points (1); wherein the signal propagation data comprise correlation data of the wireless access points (1), wherein the correlation data are correlated according to their time of measurement in order to have a common measurement of the passenger cabin (101) at a given time; wherein the signal propagation data are provided as raw data on the physical layer of the wireless communication infrastructure; wherein the system (10) further comprises a data aggregation unit (6) in wireless communication with the wireless access points (1) and configured to collect the signal propagation data of the wireless access points (1) and transfer it to the computing element (4), wherein each wireless access point (1) is configured to preprocess the raw data generated at the respective wireless access point (1) and transfer it to the data aggregation unit (6).

2. System (10) according to claim 1, wherein the signal propagation data comprise channel state information, CSI.

3. Aircraft (100) according to claim 1 or 2, wherein the signal propagation data comprise at least one of spatial, frequency and temporal variations of the wireless signals (3) within the passenger cabin (101).

4. Aircraft (100) according to one of the claims 1 to 3, wherein each wireless access point (1) is a standard commercial wireless device operating according to a IEEE 802.11 standard.

5. Aircraft (100) according to one of the claims 1 to 4, wherein the machine-learning algorithm is configured to extract the locations of static objects (5) located within the passenger cabin (101).

6. Method (M) for object localization in a passenger cabin (101) of an aircraft (100) equipped with a system (10) for object localization according to one of the claims 1 to 5, the method (M) comprising: acquiring (M1), with the wireless access points (1) of the wireless communication infrastructure, signal propagation data of wireless signals (3) transmitted over the wireless communication network (2) along multiple propagation paths; and analyzing (M2), with the computing element (4), the signal propagation data and extracting, with the computing element (4), localization data from the signal propagation data, the localization data specifying the position of objects (5) located within the passenger cabin (101), wherein the computing element (4) executes a machine-learning algorithm on the signal propagation data extracting the position data based on training data being generated using at least one of an object position measurement system and a predetermined test object arrangement within the passenger cabin (101); wherein the signal propagation data comprise correlation data of the wireless access points (1), wherein the correlation data are correlated according to their time of measurement in order to have a common measurement of the passenger cabin (101) at a given time; wherein the signal propagation data are provided as raw data on the physical layer of the wireless communication infrastructure; wherein the data aggregation unit (6) collects the signal propagation data of the wireless access points (1) and transfer it to the computing element (4), wherein each wireless access point (1) preprocesses the raw data generated at the respective wireless access point (1) and transfers it to the data aggregation unit (6).

7. Method (M) according to claim 6, wherein the machine-learning algorithm extracts the locations of static objects (5) located within the passenger cabin (101).