Occupancy Sensing Using Ultra-Wideband

Abstract

To provide an occupation sensing which uses a super-wide band (UWB) key-less infrastructure.SOLUTION: A channel impulse response (CIR) measured value is received from a plurality of UWB transceiver nodes arranged near a plurality of positions. The occupation of each of the plurality of positions is predicted on the basis of a CIR tensor formed from the CIR measured value of each transceiver node by using a classification model.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to occupancy sensing using wireless communications, such as ultra-wideband communications. [Background technology]

[0002] Background of the Invention Vehicle occupancy is an area of ​​increasing interest, both to meet regulatory requirements and to provide a superior user experience. Essentially, vehicles can detect front seat occupancy and prompt seatbelt buckling. Such systems can also provide additional features such as airbag control and improve the user experience with climate and audio control. Vehicle occupancy information is also a building block for effective shared autonomy, including human sensing, shared perception control, and deep personalization for human-centric autonomous driving systems. Summary of the Invention [Means for solving the problem]

[0003] Summary of the Invention In one or more embodiments, a method is provided for occupancy sensing using an ultra-wideband (UWB) keyless infrastructure. Channel impulse response (CIR) measurements are received from a plurality of UWB transceiver nodes positioned proximate a plurality of locations. A classification model is utilized to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the UWB transceiver nodes.

[0004] In one or more embodiments, a system for occupancy sensing using wireless communications is provided, wherein a computing device includes a processor programmed to receive channel impulse response (CIR) measurements from a plurality of wireless transceiver nodes positioned proximate a plurality of locations, and to utilize a classification model to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the wireless transceiver nodes.

[0005] In one or more embodiments, a non-transitory computer-readable medium includes instructions for occupancy sensing using ultra-wideband (UWB), which, when executed by a processor, cause the processor to receive channel impulse response (CIR) measurements from a plurality of UWB transceiver nodes positioned proximate a plurality of locations, and utilize a classification model to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the UWB transceiver nodes. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary system for use in occupancy sensing using an ultra-wideband keyless infrastructure. [Figure 2] FIG. 10 shows a heatmap of the amplitude of the channel impulse response (CIR) measured from multiple CIR blinks. [Figure 3] FIG. 3 is a graphical representation of CIR representing the mean and standard deviation of the heat map shown in FIG. 2. [Figure 4] FIG. 1 illustrates an example of CIR deviation for multiple ultra-wideband (UWB) receivers in an occupancy scenario where the vehicle is empty. [Figure 5] FIG. 10 illustrates another example of CIR deviations for multiple UWB receivers in an occupancy scenario with a passenger in the driver's seat. [Figure 6] FIG. 1 illustrates an exemplary data flow for occupancy sensing using an ultra-wideband keyless infrastructure. [Figure 7] FIG. 10 is a diagram illustrating an example of two misaligned CIRs. [Figure 8] FIG. 8 is a diagram showing an example after the two CIRs in FIG. 7 are matched. [Figure 9] FIG. 1 illustrates a detailed example of an aspect of data flow using a single-input, multiple-output classification model. [Figure 10] FIG. 1 illustrates a detailed example of an aspect of data flow using a multiple-input, multiple-output classification model. [Figure 11] FIG. 10 is a diagram illustrating a detailed example of a data flow mode using a MaskMIMO classification model. [Figure 12] FIG. 10 is a diagram illustrating an example of a data flow incorporating a feedback loop in determining occupancy for each seat. [Figure 13] FIG. 1 illustrates a process for occupancy sensing using a UWB keyless infrastructure. [Figure 14] FIG. 1 illustrates an exemplary computing device for occupancy sensing using a UWB keyless infrastructure. DETAILED DESCRIPTION OF THE INVENTION

[0007] Example Embodiments of the present disclosure are described herein. However, it will be understood that the disclosed embodiments are merely exemplary and that other embodiments may take various alternative forms. The drawings are not necessarily to scale, and some features may be exaggerated or reduced to show details of particular components. Therefore, specific structural and functional details disclosed herein should not be construed as limiting, but merely as representative basic methods for teaching those skilled in the art how to use the embodiments in various ways. As will be understood by those skilled in the art, various features shown and described with reference to any one drawing can be combined with features shown in one or more other drawings to form embodiments not explicitly shown or described. The combinations of features shown provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired for particular applications or implementations.

[0008] FIG. 1 illustrates an example system 100 for use in occupancy sensing using an ultra-wideband keyless infrastructure. The system 100 includes a deployment of multiple ultra-wideband (UWB) transceiver nodes 102 within a vehicle cabin. As shown, the system 100 utilizes eight UWB transceiver nodes 102, although more or fewer may be used. One of the UWB transceiver nodes 102 broadcasts UWB packets, while the other nodes can collect channel impulse response (CIR) measurements. These CIR signals can be provided to a processor 104, which receives the raw CIRs, processes the CIRs to time-align them, and feeds the aligned CIRs into a multiple-input, multiple-output convolutional neural network (CNN) with multitasking masks (referred to herein as MaskMIMO) to perform seat-by-seat occupancy classification. If multiple UWB channels are available, one or more UWB transceiver nodes 102 may transmit in parallel with one or more UWB transceiver nodes 102 listening to each UWB transceiver node 102, thereby allowing measurements on multiple channels to be taken simultaneously.

[0009] The UWB transceiver node 102 is configured to operate using UWB wireless technology. UWB is a popular technology providing high-precision positioning, asset tracking, and access control applications. Due to its accurate ranging capabilities and robustness against relay attacks, automobile manufacturers are upgrading their keyless entry infrastructure to UWB-based systems. In many instances, UWB refers to signals with a bandwidth greater than 500 MHz or 20% of the arithmetic center frequency. In many instances, the UWB frequency range is 3.1 GHz to 10.6 GHz, and the power spectral density (PSD) limit of a UWB transmitter is -41.3 dBm / MHz. The advantage of UWB is its wide bandwidth, which provides much better temporal / spatial resolution than other wireless technologies. Generally, the temporal resolution of wireless sensing is τ = 1 / B, where B is the channel bandwidth. Because UWB has a bandwidth greater than 500 MHz, its temporal resolution can be better than 2 nanoseconds. This is therefore 3×10 8 For electromagnetic waves traveling at speeds of meters per second, this corresponds to a potential spatial resolution of 60 centimeters. Compared to other wireless sensing technologies, UWB offers more granular sensing capabilities, especially in in-vehicle environments where multipath effects are strong. Furthermore, UWB consumes less power, making it more energy efficient and less susceptible to interference than other wireless technologies. Due to its high temporal / spatial resolution, low power consumption, and low interference, UWB is well suited for in-vehicle occupancy sensing.

[0010] The UWB transceiver node 102 can be used as a keyless entry infrastructure to replace passive keyless entry systems that use a combination of low-frequency (LF) and ultra-high-frequency (UHF) channels to measure the proximity of a key fob and check whether it is inside or within a specific range (e.g., 2 m) of a vehicle. One advantage of UWB for keyless entry is that UWB radios transmit explicit timing information defined in the IEEE 802.15.4-2015 UWB standard, which significantly extends the reception time of signals from relay devices at the vehicle compared to legitimate signals from local keys, thereby thwarting relay attacks. Furthermore, the implementation of UWB radios in smartphones eliminates the need for users to carry an extra key fob; instead, they can use a UWB-enabled smartphone to lock, unlock, or start their vehicle.

[0011] The transmitted signal of the UWB transceiver node 102 acting as a transmitter is a series of predetermined symbols in the IEEE 802.15.4 format. These UWB data packets are sometimes referred to as "blinks." These signals may travel multiple paths and arrive at the UWB transceiver node 102 acting as a receiver with different amplitude attenuations and times of flight. The received signal may be compared to the series of known transmitted symbols and the CIR may be calculated as follows:

number

[0012] FIG. 2 illustrates a heatmap 200 of CIR amplitudes measured from multiple CIR blinks. As shown, the Y-axis of heatmap 200 represents the number of CIR blinks, and the X-axis represents the index of the illustrated CIR path number over time. Note, however, that not all CIR paths are shown in heatmap 200. Instead, a truncated window of CIRs is presented; in this example, there are 15 paths before the first path index and 85 paths after. CIR heatmap 200 can be fed into signal processing techniques and machine learning algorithms for various sensing purposes.

[0013] FIG. 3 is a graphical representation 300 of CIR, plotting the heat map shown in FIG. 2 by mean and standard deviation. As shown, the Y-axis represents CIR amplitude, and the X-axis represents the index number of the CIR path shown. This allows for observation of CIR amplitude over time. In addition to amplitude and standard deviation, various features such as peaks / valleys, distance between peaks / valleys, and number of peaks / valleys can also be calculated and fed into machine learning algorithms for various sensing purposes.

[0014] Because the CIR is affected by multipath signal reflections from objects and people inside the vehicle, the deviation, mean, and standard deviation of the CIR in the heat map can be used for vehicle occupancy sensing. Thus, vehicle occupancy monitoring can additionally utilize the keyless infrastructure supported by the UWB transceiver nodes 102 in an orthogonal sensing modality to detect vehicle occupancy, which is useful for supporting regulatory requirements and providing a customized user experience. Because the system 100 leverages existing keyless infrastructure, the system 100 can operate using the aforementioned UWB transceiver nodes 102 installed in vehicles for keyless entry. Furthermore, because the system 100 does not have location-specific requirements, it can utilize existing UWB transceiver nodes 102.

[0015] FIG. 4 shows an example 400 of CIR deviations for multiple UWB receivers for an occupancy scenario in which the vehicle is empty. FIG. 5 shows another example 500 of CIR deviations for multiple UWB receivers for an occupancy scenario in which a passenger is present in the driver's seat. In each of examples 400 and 500, a graphical representation of the mean and standard deviation CIR (e.g., as shown in FIG. 3 ) is provided for a layout of UWB transceiver nodes 102 (e.g., in this example, the layout of the exemplary UWB transceiver nodes 102 of FIG. 1 ). In the example layout, UWB transceiver nodes 102 operating as transmitters are located in the center panel of the vehicle, and UWB transceiver nodes 102 operating as receivers are located in various locations on the vehicle. By way of non-limiting example, these locations include the left front, rearview mirror, right front, front roof, rear roof, center console, left rear, trunk, and right rear of the vehicle.

[0016] It should be noted that while many of the examples herein relate to vehicular environments, these techniques are applicable to other environments or other location applications involving UWB transceiver nodes 102. For example, the techniques described herein can be used to determine the location of an occupant independent of seat position. Another possibility is that the techniques described herein can be used to determine the location of an occupant in a building or other structure in which the UWB transceiver node 102 is installed. A further possibility is that the techniques described herein can be used to determine the location of an occupant in an outdoor environment in which the UWB transceiver node 102 is installed.

[0017] Regardless of the particular environment or location, the CIR deviation of each of the receiver UWB transceiver nodes 102 exhibits different multipath characteristics for each occupancy scenario. For example, the left front and rearview mirror UWB transceiver nodes 102 exhibit high CIR deviations between embodiment 400 and embodiment 500 due to slight human movement when there is an occupant in the driver's seat as shown in Figure 5, while all UWB transceiver nodes exhibit low CIR deviations when the vehicle is empty as shown in Figure 4. The CIR deviations can be fed into machine learning algorithms for vehicle occupancy sensing.

[0018] Because wireless signals have different characteristics from digital images, such as spatial resolution and field of view, CNN models specifically designed for UWB data can be utilized. Using CIR data as input, the system 100 can use a deep learning model with a multiple-input, multiple-output (MIMO) model (e.g., MaskMIMO) with multitasking masks to learn spatial / temporal features from 2D convolutions and seat-specific attention from multitasking masks. MaskMIMO is accurate and robust against unknown vehicle locations and unseen scenarios by learning spatial / temporal features from 2D convolutions and seat-specific occupancy attention from multitasking masks. This model requires less training effort for data collection, signal processing, feature engineering, and model training. Furthermore, the system 100 has low computational cost, making it practical for real-time execution on a processor 104, such as an embedded device with limited resources.

[0019] 6 illustrates an exemplary data flow 600 for occupancy sensing using an ultra-wideband keyless infrastructure. In one embodiment, the data flow 600 may be accomplished by a processor 104 in communication with a UWB transceiver node 102. Generally, the data flow 600 includes a link selector 602, a channel state extractor 604, a signal processing / data pre-processing stage 606, a machine learning component using a classification model 608, and a decision component 610 related to making seat-by-seat occupancy predictions.

[0020] The link selector 602 automatically or manually selects those features that contribute most to the predicted output of the seat-by-seat occupancy prediction for each UWB transceiver node 102. These features may include, for example, one or more of the CIR amplitude and standard deviation, peaks / valleys, distance between peaks / valleys, number of peaks / valleys, etc. of the CIR data.

[0021] Using the selected data, channel condition extraction 604 is performed, which may include, for example, capturing CIR data from each of the UWB transceiver nodes 102 according to the selected characteristics. For example, each receiver UWB transceiver node 102 may collect CIR and transmit the decoded CIR measurements to the processor 104.

[0022] In one embodiment, one of the UWB transceiver nodes 102 can operate as a transmitter, and the UWB transceiver nodes 102 of the remaining nodes can operate as receivers. Continuing with the example UWB transceiver node 102 layout of FIG. 1 , one of the UWB transceiver nodes 102 can be a transmitter, and the other seven UWB transceiver nodes 102 can be receivers. Regardless of the layout or number of UWB transceiver nodes 102, the UWB transceiver node 102 assigned to a transmitter can be changed periodically (e.g., every 30 milliseconds) under the control of the processor 104, for example. One possibility is to change the UWB transceiver node 102 in a round-robin manner, where the current transmitter is changed to a receiver and the next UWB transceiver node 102 in the ordered sequence of UWB transceiver nodes 102 is changed to a transmitter.

[0023] Processing of collected UWB CIR data can include two aspects: a signal processing / data pre-processing stage 606 to convert raw UWB signals into normalized CIR tensors, and a classification model 608 phase that includes prediction of per-seat occupancy from the normalized CIR tensors to yield per-seat occupancy decisions, each of which is described next.

[0024] In the channel condition extractor 604, for UWB, the transmitter and receiver are typically not time-synchronized. Therefore, CIRs measured at different times may be randomly shifted relative to each other. Therefore, an early step in the signal processing / data pre-processing stage 606 may include performing CIR alignment to convert the raw CIRs into a time series of CIRs that represent consistent multipath characteristics.

[0025] 7 shows an example 700 of two misaligned CIRs. As shown, the Y-axis represents CIR amplitude and the X-axis represents the index number of the CIR path shown. In the example 700 shown, the two CIRs are separated by a time interval of approximately 2 seconds.

[0026] 8 shows an example 800 of the two CIRs of FIG. 7 after alignment. In one example, CIR alignment can be performed using a first path index. The first path index can be determined by a leading edge detection algorithm that compares the received power of each path to a threshold calculated from a noise estimate. The CIRs can be aligned by the first path index after removing the time shift.

[0027] Referring again to FIG. 6, upon receiving the CIR, processor 104 may perform CIR alignment (e.g., using the first path index) and convert the raw CIR into a time series of truncated CIRs for each node's paths (e.g., to accumulate a predetermined number of paths for each node, such as 101 paths in a non-limiting example).

[0028] In response to the completion of a predetermined data collection (e.g., in one example, 10 round-robin circles) equivalent to approximately 2.4 seconds, the CIR amplitudes of the UWB transceiver nodes 102 can be concatenated into a four-dimensional CIR tensor according to the grouping of transmitting nodes. This four-dimensional CIR tensor can be used as input for further processing. Continuing with the example of FIG. 1, the size of the four-dimensional CIR tensor can represent eight UWB transceiver nodes 102, seven receivers per round, ten CIR blinks, and 101 CIR paths.

[0029] Continuing with the preprocessing discussion, the 4-dimensional CIR tensors for training, validation, and testing may be normalized. In one embodiment, this can be achieved as follows:

number

number

number

number

[0030]

number

[0031] Once the CIR processing is complete to convert the raw signals from the UWB transceiver node 102 into normalized CIR tensors, the machine learning classification model 608 uses the normalized CIR tensors to predict seat-by-seat occupancy from the normalized CIR tensors.

[0032] A classification model can be designed based on input data and output targets. The output of the classification model includes occupancy predictions for each seat in a car. A possible model architecture is to use a single label for all combinations of different car occupancy scenarios. For example, "0000" represents an empty car, while "1000" means the driver's seat is occupied and the other three seats are empty. In this case, the number of output classes would be very large, requiring 16 possible outputs to cover all combinations of "0" or "1" for the four seat positions. This large number of outputs can increase the complexity of the model. Furthermore, a large number of output classes can create bottlenecks in computation and optimization, making it difficult to train the model.

[0033] The number of output classes can be addressed by utilizing a multi-task learning approach, in which multiple classification tasks are jointly trained. Multi-task learning reduces complexity and improves the generality, scalability, and flexibility of the classification algorithm. First, the output is divided into simpler tasks (e.g., performing a binary classification of vacant or occupied for each seat location (in a non-limiting example, four seating configurations)). This reduces the complexity and computational cost of both the model architecture and the optimization algorithm, making it easier to train the model. Second, because various tasks are related, features learned from each task can improve other tasks. By learning related tasks in parallel with a shared representation, multi-task learning can improve the overall performance of all tasks. Third, multi-task learning is scalable, making it easy to add new tasks as new data becomes available. For example, it is possible to add "dog" to the seat occupancy classification and train the pre-trained model with the additional data. Finally, multi-task learning allows the flexibility to add weights to various tasks for multi-task learning. For example, if occupancy of the driver's seat is a high priority, the model can give the driver's seat a higher weighting.

[0034] The input to the classification algorithm can take the form of a 4D tensor output from the signal preprocessing stage 606 described above. This input format also influences the design choices of the classification algorithm. 4D tensors can use 4D convolution to extract feature maps. However, 4D convolution is highly complex in time and space and may not be natively supported by deep learning frameworks such as TensorFlow and PyTorch. 4D convolution can be replaced by 3D or 2D convolution using decomposition models such as single-input multiple-output (SIMO) or multiple-input multiple-output (MIMO).

[0035] 9 illustrates a detailed example 900 of an embodiment of data flow 600 using a single-input, multiple-output (SIMO) classification model 608. As shown, the 3D CIR tensors of all nodes are concatenated into a single 3D tensor at 902. Using the exemplary arrangement of UWB transceiver nodes 102 of FIG. 1 and the exemplary approach described above, this concatenation can represent eight UWB transceiver nodes 102, seven receivers per round, ten CIR blinks, and 101 CIR paths. Thus, purely for purposes of illustration and not limitation, a data size of 7x10x101 from each of seven receiving UWB transceiver nodes 102 can result in a concatenated data set of 56x10x101.

[0036] Next, 2D convolution can be performed along the temporal and spatial domains. The preprocessed 3D CIR tensors of each node 102 can be fed into two convolutional layers 904 and 908, followed by corresponding activation layers 906 and 910. The convolutional layers can be of different sizes, with the first convolutional layer 904 typically being larger than the second convolutional layer 908. The activation layers 906 and 910 can include features such as a batch normalizer that normalizes the output by subtracting the batch mean and dividing by the batch standard deviation, a rectified linear unit (ReLU) that zeros out negative activations, a max pooler that downsamples the resulting data, and a dropout layer that drops out a random set of activations in the layer by setting them to zero to avoid overfitting. The data can be subjected to a Flatten 912 operation to convert the data into a vector, and then a softmax layer 914 with the same number of nodes as the output (e.g., the number of seat positions) can be used to generate the final probability of seat occupancy.

[0037] For example, purely for purposes of illustration and not limitation, convolutional layer 904 may utilize a 5x5 matrix to provide 256 outputs for a total data size of 256x10x101, activation layer 906 may perform pooling with a 3x3 matrix resulting in a dataset of 256x3x33, convolutional layer 908 may utilize a 3x3 matrix to provide 128 outputs for a total data size of 128x3x33, and activation layer 910 may perform pooling with a 3x3 matrix resulting in a dataset of 128x1x11, which may be flattened into a vector of size 1408x1 from which the final output is determined, for example via softmax.

[0038] FIG. 10 illustrates a detailed example 1000 of an embodiment of a data flow 600 using a multiple-input, multiple-output (MIMO) classification model 608. In this example, each UWB transceiver node 102 uses its own two-dimensional convolutional layer 1002. For example, instead of starting with concatenation, each UWB transceiver node 102 feeds its data into a separate set of convolutional layers 1002, 1006 and activation layers 1004, 1008. Note again that the layer sizes and number of layers may differ from those illustrated. The convolutional outputs of layer 1008 of each UWB transceiver node 102 are concatenated into a single layer at 1010 for multi-task classification, for example, with a Flatten 1012 and a Softmax layer 1014 similar to the Flatten 912 and Softmax layer 914 described above for the SIMO classification model. In MIMO, the neural architecture of each node is similar, but the neural weights of different nodes are different. Thus, in such an approach, these layers 1002, 1004, 1006, 1008 learn spatial and temporal domain features independently for each UWB transceiver node 102.

[0039] For example, purely for purposes of illustration and not limitation, each convolutional layer 1002 may utilize a 5x5 matrix to provide 256 outputs for a total data size of 256x10x101, each activation layer 1004 may perform pooling using a 3x3 matrix resulting in a data set of 256x3x33, each convolutional layer 1006 may utilize a 3x3 matrix to provide 128 outputs for a total data size of 128x3x33, and each activation layer 1008 may perform pooling using a 3x3 matrix resulting in a data set of 128x1x11. These independent data sets may be concatenated into a 128x8x11 data set and flattened into a 11264x1 vector from which the final output is determined, for example, again via softmax.

[0040] Importantly, however, the SIMO and MIMO classification model 608 does not capture correlated spatial features for different UWB transceiver nodes 102 and different seat positions. To address this, a multitasking mask can be added to the MIMO classification model 608 to learn multitasking attention from multiple UWB transceiver nodes 102.

[0041] FIG. 11 illustrates a detailed example 1100 of an embodiment of data flow 600 using Mask MIMO classification model 608. Mask MIMO classification model 608 includes similar aspects as described above with respect to MIMO classification model 608. For example, elements 1102-1112 operate as described above with respect to elements 1002-1012, respectively. After Flattening, the flattened vector is applied to a set of dense layers 1114, one for each seat position. Each dense layer 1114 converts the high-dimensional vector output in 1112 into a low-dimensional object vector. The output of the dense layer 1114 is further applied to a sigmoid activation function to generate a pre-weighted probability output for the occupancy of each corresponding seat position. Importantly, because UWB transceiver nodes 102 are positioned at different locations within the vehicle, different UWB transceiver nodes 102 have different weights for the performance of different seat positions. These pre-weighted outputs are then weighted using a multitasking mask 1116 configured to learn attentional and spatial features for each seat to automatically calculate weights for different UWB transceiver nodes 102 and seat locations.

[0042] In the illustrated embodiment, the multitasking mask 1116 operates on a concatenation 1118 of data from all UWB transceiver nodes 102, similar to the input to the SIMO classification model 608 described above. The concatenated data is then fed to a convolutional layer 1120, an activation layer 1122, another convolutional layer 1124, and another activation layer 1126. The convolutional layers 1120, 1124 may vary in size and number of layers. The activation layers 1122 and 1126 may also include features such as a batch normalizer that normalizes the output by subtracting the batch mean and dividing by the batch standard deviation, a rectified linear unit (ReLU) that zeros out negative activations, a max pooler that downsamples the resulting data, and a dropout layer that drops out a random set of activations in that layer by setting them to zero to avoid overfitting. This data may be flattened into a vector at 1128 and then applied to a set of dense layers 1130, one for each seat position, to generate multi-tasking weights for each seat position using a sigmoid activation function. A final per-seat occupancy prediction is then calculated by multiplier 1132 by multiplying the multi-tasking weights from the multi-tasking mask 1116 with the softmax scores. These results may then be thresholded or otherwise applied to generate a final per-seat occupancy prediction at 1134 (e.g., a binary prediction with a first value such as 1 for occupied and a second value such as 0 for unoccupied).

[0043] For example, purely for purposes of illustration and not limitation, aspects of the MIMO data size may be consistent with those described with respect to the MIMO classification model 608. Further, with respect to the computation of the multitasking mask 1116, also purely for purposes of illustration and not limitation, the concatenation 1118 of the data of the UWB transceiver nodes 102 may result in a data vector of dimensions 56x10x101, the convolutional layer 1120 may utilize a 3x3 matrix to provide 128 outputs for an overall data size of 56x10x128, the activation layer 1122 may perform pooling using a 3x3 matrix resulting in a data set of 18x3x128, the convolutional layer 1124 may utilize a 3x3 matrix to provide 64 outputs for an overall data size of 56x10x64, and the activation layer 1126 may perform pooling using a 3x3 matrix resulting in a data set of 6x1x64. This data can be flattened into a 384x1 vector, from which a density output can be determined as 128x1, which can be provided to a sigmoid function to generate multitasking weights for each seat position. A corresponding dense layer 1114 can then reduce the 11264x1 size flattened concatenation to a more manageable size, such as 128x1, which can then be applied to the sigmoid function for that flow to determine pre-mask-weighted probabilities. These pre-mask-weighted probabilities and multitasking weights can optionally be provided to multiplier 1132.

[0044] Therefore, the MaskMIMO classification model 608 learns both independent and shared features from the multipath characteristics of multiple UWB transceiver nodes 102. The MaskMIMO classification model 608 also utilizes the multitasking mask 1116 to learn spatial features and multitasking attention from UWB transceiver nodes 102 at various locations. This design allows the MaskMIMO classification model 608 to be robust to various unseen scenarios. Because the MaskMIMO classification model 608 is robust to various scenarios, the MaskMIMO classification model 608 does not need to collect a large amount of data for different scenarios. Instead, the MaskMIMO classification model 608 can be trained with only four vehicle positions and provide robustness and high accuracy for various unseen scenarios. Furthermore, unlike other machine learning techniques such as k-nearest neighbors (kNN) and support vector machines (SVM), which typically require feature engineering / selection, the MaskMIMO classification model 608 can automatically learn features and therefore require little human effort for signal processing. Furthermore, the MaskMIMO classification model 608 can use a multi-output CNN model, which allows it to be retrained with new data or new tasks without restarting training from scratch. Finally, the MaskMIMO classification model 608 is computationally inexpensive and can run in real time even on resource-constrained embedded devices.

[0045] 12 shows an example data flow 1200 incorporating a feedback loop 1202 for seat-by-seat occupancy determination, which can further improve the performance of conventional operations. Similar to label data, feedback can be received from a variety of sources, such as manual input (e.g., a human in the loop) to provide ground truth or the use of other sensor modalities (cameras, radar, seat weight sensors, etc.) as data input for training the MaskMIMO classification model 608 for occupancy determination.

[0046] 13 shows a process 1300 for occupancy sensing using a UWB keyless infrastructure. In one embodiment, process 1300 can be implemented using the techniques described in detail herein.

[0047] In operation 1302, channel impulse response (CIR) measurements are received from multiple UWB transceiver nodes 102 positioned near multiple locations. In some examples, the locations are seat locations, but in other examples, the locations may be different from seat locations, such as when the UWB transceiver nodes 102 are designed for use in keyless authentication. In one embodiment, one of the multiple UWB transceiver nodes 102 is periodically reassigned to be a transmitter, and other of the multiple UWB transceiver nodes 102 are periodically reassigned to be receivers. Thus, CIR measurements of data from UWB transceiver nodes 102 operating as transmitters are collected from UWB transceiver nodes 102 operating as receivers.

[0048] In operation 1304, the classification model 608 is used to identify features of the CIR tensor formed from the CIR measurements of each of the UWB transceiver nodes 102 to generate an output for each location. In one example, the classification model 608 may be a single-input multiple-output classification model 608, where the CIR tensors of multiple UWB transceiver nodes 102 may be concatenated into a single three-dimensional tensor for the classification model 608. In another example, the classification model 608 may be a multiple-input multiple-output classification model 608, where data from each of the UWB transceiver nodes 102 may be fed into a separate set of layers, where the outputs of the layers for each of the UWB transceiver nodes are concatenated into a single layer for multi-task classification.

[0049] In yet another example, multiple-input, multiple-output (MIMO) classification of the MaskMIMO classification model 608 is used to identify CIR tensor features for each UWB transceiver node and generate pre-weighted outputs for each seat position. In such an example, the CIR tensor from each UWB transceiver node may be fed to a corresponding set of convolutional and activation layers. The outputs of the corresponding set of layers may then be concatenated, and the concatenated outputs may be flattened for multi-task classification. The flattened concatenated outputs may be applied to a set of dense layers with sigmoid activation functions, one for each seat position, to generate pre-weighted outputs for each seat position. Furthermore, a multi-tasking mask that identifies multi-tasking attention from the CIR tensor may be used to generate multi-tasking weights for each seat position. In one embodiment, the concatenation of the CIR tensors from each UWB transceiver node may be fed to a set of convolutional and activation layers. The layer outputs can be flattened into vectors, and the flattened outputs can be applied to a set of dense layers with sigmoid activation functions, one for each seat position, to generate multi-tasking weights for each seat position. Furthermore, the pre-weighted outputs can be weighted using the multi-tasking weights to generate final outputs that account for correlated spatial features between UWB transceiver nodes. In one embodiment, for each seat position, the pre-weighted output for the seat position can be multiplied by the multi-tasking weight corresponding to the seat position to determine the result.

[0050] For measurements, time synchronization can be performed to generate time-aligned CIR tensors between UWB transceiver nodes. In one embodiment, a first path index is determined by leading edge detection, and the received power of each path in the CIR measurement is compared to a threshold calculated from a noise estimate of the CIR measurement. The CIR measurements are aligned by the first path index to remove time shifts between the CIR measurements. In some examples, the CIR tensor is also normalized.

[0051] In operation 1306, an occupancy is predicted for each of a plurality of positions according to the final output. In one embodiment, the results for each position may be thresholded to provide a binary seat occupancy prediction for each of the positions. In one embodiment, the positions are seat positions.

[0052] FIG. 14 illustrates an exemplary computing device 1400 for occupancy sensing using a UWB keyless infrastructure. The processor 104 may include such a computing device 1400. Operations performed herein, such as those illustrated in FIGS. 1-13, may be performed by such a computing device 1400. The computing device 1400 may include a memory 1402, a processor 1404, and non-volatile storage 1406. The processor 1404 may include one or more devices selected from a high-performance computing (HPC) system including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer-executable instructions resident in the memory 1402. Memory 1402 may include a single memory device or multiple memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. Non-volatile storage 1406 may include one or more persistent data storage devices, such as a hard drive, optical drive, tape drive, non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information.

[0053] The processor 1404 may be configured to load into memory 1402 and execute computer-executable instructions resident in program instructions 1408 on non-volatile storage 1406 and embodying the algorithms and / or procedures of one or more embodiments. The program instructions 1408 may include an operating system and applications. The program instructions 1408 may be compiled or translated from computer programs written using various programming languages ​​and / or technologies, including, but not limited to, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL, alone or in combination. In some examples, machine learning aspects may be implemented using deep learning frameworks such as TensorFlow or PyTorch.

[0054] The computer-executable instructions of the program instructions 1408, when executed by the processor 1404, may cause the computing device 1400 to perform one or more of the algorithms and / or procedures disclosed herein. The non-volatile storage 1406 may also include data 1410 that supports the functions, features, and processing of one or more embodiments described herein.

[0055] The processes, methods, or algorithms disclosed herein can be provided or implemented in a processing device, controller, or computer, which may include any existing programmable or dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored as data and instructions executable by a controller or computer in many forms, including, but not limited to, information permanently stored on non-writable storage media such as ROM devices, and information revocably stored on writable storage media such as floppy disks, magnetic tape, CDs, RAM devices, and other magnetic and optical media. The processes, methods, or algorithms can also be implemented in software-executable objects. Alternatively, the processes, methods, or algorithms can be embodied, in whole or in part, using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0056] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible configurations encompassed by the scope of the claims. The terms used herein are terms of description rather than limitation, and it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. As noted above, features of various embodiments can be combined to form additional embodiments of the present invention not expressly described or shown. While various embodiments have been described as advantageous or preferable over other embodiments or prior art configurations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system attributes that depend on the particular application and implementation. These attributes include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. If any embodiments are described herein as less desirable than other embodiments or prior art configurations with respect to one or more characteristics, these embodiments do not depart from the scope of the present disclosure and may be desirable for particular applications.

[0057] With respect to the processes, systems, methods, heuristics, etc. described herein, the steps of such processes, etc. are described as occurring according to a particular ordered sequence, but it is understood that such processes can be practiced with the described steps being performed in an order other than the order described herein. Furthermore, it is understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of describing particular embodiments and should not be construed as limiting the scope of the claims.

[0058] Accordingly, it should be understood that the above description is illustrative and not limiting. Many embodiments and applications other than the examples provided will become apparent upon reading the above description. The scope should not be determined with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology described herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, this application should be understood as open to modification and variation.

[0059] All terms used in the claims are intended to be given their broadest reasonable interpretation and ordinary meaning as understood by one of ordinary skill in the art described herein, unless expressly indicated to the contrary herein. In particular, the use of singular articles such as "a," "the," "said," etc., should be construed as reciting one or more of the indicated elements, unless a contrary limitation is expressly stated in the claim.

[0060] The Abstract of the Disclosure is provided to enable the reader to quickly grasp the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the meaning or scope of the claims. Furthermore, in the foregoing Examples, it will be appreciated that various features have been grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are specifically recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Examples, with each claim standing on its own as separately claimed subject matter.

[0061] While representative embodiments have been described above, it is not intended that these embodiments describe all possible forms of the present invention. Rather, the words used herein are words of description rather than limitation, and it will be understood that various changes may be made without departing from the spirit and scope of the present invention. Furthermore, features of various embodiments may be combined to form further embodiments of the present invention.

Claims

1. 1. A method for occupancy sensing using ultra-wideband (UWB), comprising: receiving channel impulse response (CIR) measurements from a plurality of UWB transceiver nodes located near a plurality of locations; utilizing a classification model to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the UWB transceiver nodes; Including, The method, further comprising normalizing the CIR tensor before applying the CIR tensor to the classification model.

2. The method comprises: performing time synchronization on the CIR measurements to time-align the CIR tensors between the UWB transceiver nodes; using the temporally aligned CIR tensors as inputs to the classification model; further comprising: The method of claim 1.

3. The method comprises: periodically reassigning one of the plurality of UWB transceiver nodes to a transmitter and another of the plurality of UWB transceiver nodes to a receiver; collecting the CIR measurements of the one of the plurality of UWB transceiver nodes operating as a transmitter from the others of the UWB transceiver nodes operating as receivers; further comprising: The method of claim 1.

4. the classification model is a single-input multiple-output classification model, and the CIR tensors of the multiple UWB transceiver nodes are concatenated into a single three-dimensional tensor as input to the single-input multiple-output classification model. The method of claim 1.

5. the classification model is a multiple-input multiple-output classification model, where each of the UWB transceiver nodes feeds data to a separate set of layers, and the outputs of the layers of each of the UWB transceiver nodes are concatenated into a single layer for multi-task classification. The method of claim 1.

6. the classification model is a multiple-input multiple-output classification model using a multi-task mask, with each of the UWB transceiver nodes feeding data to a separate set of layers; The method comprises: utilizing the multitasking mask to identify multitasking attention from the CIR tensor and generate multitasking weights for each seat position; weighting the output of the layer of each of the UWB transceiver nodes using the multitasking weights to generate a weighted output that takes into account correlated spatial features between the UWB transceiver nodes; further comprising: The method of claim 1.

7. 1. A system for occupancy sensing using wireless communication, the system comprising a computing device, the computing device comprising: receiving channel impulse response (CIR) measurements from a plurality of wireless transceiver nodes positioned proximate a plurality of locations; and utilizing a classification model to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the wireless transceiver nodes. a processor programmed to: The processor is further programmed to normalize the CIR tensor before applying the CIR tensor to the classification model.

8. The processor: performing time synchronization on the CIR measurements to time-align the CIR tensors between the wireless transceiver nodes; Utilizing the temporally aligned CIR tensors as inputs to the classification model It is further programmed to The system of claim 7.

9. The processor: periodically reassigning one of the plurality of wireless transceiver nodes to a transmitter and another of the plurality of wireless transceiver nodes to a receiver on a channel-by-channel basis; collecting the CIR measurements of the one of the plurality of wireless transceiver nodes operating as a transmitter from the other of the wireless transceiver nodes operating as receivers; It is further programmed to The system of claim 7.

10. the classification model is a single-input multiple-output classification model; The processor: concatenating the CIR tensors of the plurality of wireless transceiver nodes into a single three-dimensional tensor; Input the three-dimensional tensor into the single-input multi-output classification model. It is further programmed to The system of claim 7.

11. the classification model is a multi-input multi-output classification model; The processor: feeding data from each of said wireless transceiver nodes to a separate set of layers; Concatenating the outputs of the layers of each of the wireless transceiver nodes into a single layer for multi-task classification. It is further programmed to The system of claim 7.

12. the classification model is a multiple-input multiple-output classification model using a multi-task mask, with each of the UWB transceiver nodes feeding data to a separate set of layers; The processor: utilizing the multitasking mask to identify multitasking attention from the CIR tensor and generate multitasking weights for each seat position; Using the multitasking weights, weighting the output of the layer of each of the UWB transceiver nodes generates a weighted output that takes into account correlated spatial features between the wireless transceiver nodes. It is further programmed to The system of claim 7.

13. 1. A non-transitory computer-readable medium comprising instructions for occupancy sensing using ultra-wideband (UWB), the instructions, when executed by a processor, causing the processor to: receiving channel impulse response (CIR) measurements from a plurality of UWB transceiver nodes positioned proximate a plurality of locations; Utilizing a classification model to predict occupancy of each of the plurality of locations based on a CIR tensor formed from the CIR measurements from each of the UWB transceiver nodes. It is for the purpose of The medium is When executed by the processor, the processor: performing time synchronization on the CIR measurements to time-align the CIR tensors between the UWB transceiver nodes; normalizing the CIR tensor before applying it to the classification model; The normalized and time-aligned CIR tensors are used as inputs to the classification model. The medium further comprises instructions for:

14. The medium is When executed by the processor, the processor: causing one of the plurality of UWB transceiver nodes to be a transmitter and another of the plurality of UWB transceiver nodes to be a receiver on a channel-by-channel basis; collecting the CIR measurements of the one of the plurality of UWB transceiver nodes operating as a transmitter from the others of the UWB transceiver nodes operating as receivers; further comprising instructions for: The medium of claim 13.

15. the classification model is a single-input multiple-output classification model; The medium is When executed by the processor, the processor: concatenating the CIR tensors of the plurality of UWB transceiver nodes into a single three-dimensional tensor; Input the three-dimensional tensor to the single-input, multi-output classification model. further comprising instructions for: The medium of claim 13.

16. the classification model is a multi-input multi-output classification model; The medium is When executed by the processor, the processor: causing data from each of said UWB transceiver nodes to be fed to a separate set of layers; Concatenating the outputs of the layers of each of the UWB transceiver nodes into a single layer for multi-task classification. further comprising instructions for: The medium of claim 13.

17. the classification model is a multiple-input multiple-output classification model using a multi-task mask, with each of the UWB transceiver nodes feeding data to a separate set of layers; The medium is When executed by the processor, the processor: utilizing the multitasking mask to identify multitasking attention from the CIR tensor and generate multitasking weights for each seat position; Using the multitasking weights, the outputs of the layers of each of the UWB transceiver nodes are weighted to generate a weighted output that takes into account correlated spatial features between the UWB transceiver nodes. further comprising instructions for: The medium of claim 13.

Images