Indoor localization based on multiple device sensors

Abstract

An exemplary embodiment of the present disclosure provides an exemplary method that includes obtaining location data associated with first and second computing devices. The exemplary method may include determining an on-user device status for each of the first and second computing devices. The exemplary method may include obtaining inertial measurement unit (IMU) sensor data from each of the first and second computing devices. The exemplary method may include inputting the IMU sensor data from each of the first and second computing devices into a machine-learned model. The exemplary method may include obtaining output data from the machine-learned model that indicates a predicted location. The exemplary method may include comparing the output data that indicates the predicted location with data that indicates a location of the target subzone. The exemplary method may include transmitting data that directs a user interface of the first computing device to provide a content item for display.

Description

[Technical Field]

[0001] The present disclosure relates generally to location determination using data from multiple sensors. More particularly, the present disclosure relates to reconciling data from multiple sensors to determine indoor location determination of a computing device located on a user. [Background technology]

[0002] A computing device may be associated with multiple sensors, and sensor data may be acquired and used to determine the device's localization. Summary of the Invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0004] In one exemplary aspect, the present disclosure provides an exemplary system for indoor localization based on multiple device sensors, including one or more processors and one or more memory devices storing executable instructions for causing the one or more processors to perform operations. In some implementations, the one or more memory devices may include one or more temporary or non-transitory computer-readable media storing executable instructions for causing the one or more processors to perform operations. In the exemplary system, the operations may include acquiring location data associated with a first computing device and a second computing device, the location data indicating that each of the first computing device and the second computing device is at a target location. In the exemplary system, the operations may include determining an on-user device status for each of the first and second computing devices in response to acquiring the location data. In the exemplary system, the operations may include acquiring inertial measurement unit (IMU) sensor data from each of the first and second computing devices in response to determining the on-user device status for each of the first and second computing devices. In an exemplary system, the operations can include inputting IMU sensor data from each of the first and second computing devices into a machine-learned model, the machine-learned model configured to combine the IMU sensor data from each of the first and second computing devices. In an exemplary system, the operations can include obtaining output data from the machine-learned model indicating a predicted position. In an exemplary system, the operations can include comparing the output data indicating the predicted position to data indicating a position of the target subzone.In an exemplary system, the operation includes, in response to obtaining data indicating the co-presence of each of the first and second computing devices within a target subzone of the target location, transmitting data to a user interface of the first computing device instructing it to provide a content item for display.

[0005] In some embodiments of the exemplary system, the on-user device status for each of the first and second computing devices indicates that each of the first and second computing devices is located on a user moving within the target location.

[0006] In some embodiments of the exemplary system, the IMU sensor data includes accelerometer data and gyroscope data.

[0007] In some embodiments of the exemplary system, the machine-learned model is configured to convert the accelerometer and gyroscope data into Cartesian coordinates that indicate absolute position.

[0008] In some embodiments of the exemplary system, the operations include obtaining IMU sensor data from a third computing device.

[0009] In some embodiments of the exemplary system, the first computing device includes a smartphone, the second computing device includes a smartwatch, and the third computing device includes earphones.

[0010] In some embodiments of the exemplary system, the computing is performed on a first computing device.

[0011] In some embodiments of the exemplary system, the IMU sensor data from each of the first and second computing devices is combined using a fusion method.

[0012] In some embodiments of the exemplary system, the machine-learned model includes a neural network.

[0013] In some embodiments of the exemplary system, the machine-learned model includes a fully connected neural network.

[0014] In some embodiments of the exemplary system, the first computing device is a primary computing device and the second computing device is a secondary computing device. In some embodiments of the exemplary system, the first computing device functions as a modem to facilitate transmission of data between the second computing device and a server computing system.

[0015] In an exemplary aspect, the present disclosure provides an exemplary computer-implemented method. The exemplary method includes obtaining location data associated with a first computing device and a second computing device, the location data indicating that each of the first computing device and the second computing device is at a target location. In the exemplary method, the method includes determining an on-user device status for each of the first and second computing devices in response to obtaining the location data. In the exemplary method, the method includes obtaining inertial measurement unit (IMU) sensor data from each of the first and second computing devices in response to determining the on-user device status for each of the first and second computing devices. In the exemplary method, the method includes inputting the IMU sensor data from each of the first and second computing devices into a machine-learned model, the machine-learned model configured to combine the IMU sensor data from each of the first and second computing devices. In the exemplary method, the method includes obtaining output data from the machine-learned model indicative of a predicted location. In the exemplary method, the method includes comparing the output data indicative of the predicted location to data indicative of a location of the target subzone. In an exemplary method, the method includes, in response to obtaining data indicating the co-presence of each of the first and second computing devices within a target subzone of the target location, transmitting data to a user interface of the first computing device instructing it to provide a content item for display.

[0016] In some embodiments of the exemplary method, the on-user device status for each of the first and second computing devices indicates that each of the first and second computing devices is located on a user moving within the target location.

[0017] In some embodiments of the example method, the IMU sensor data includes accelerometer data and gyroscope data.

[0018] In some embodiments of the exemplary method, the machine-learned model is configured to convert the accelerometer data and the gyroscope data into Cartesian coordinates that indicate absolute position.

[0019] In some embodiments of the example method, the method includes obtaining IMU sensor data from a third computing device.

[0020] In some embodiments of the exemplary method, the first computing device includes a smartphone, the second computing device includes a smartwatch, and the third computing device includes earphones.

[0021] In some embodiments of the exemplary method, the computing is performed on a first computing device.

[0022] In some embodiments of the example method, the IMU sensor data from each of the first and second computing devices is combined using a fusion method.

[0023] In some embodiments of the exemplary method, the machine-learned model includes a neural network.

[0024] In some embodiments of the exemplary method, the machine-learned model includes a fully connected neural network.

[0025] In some embodiments of the exemplary method, the first computing device is a primary computing device and the second computing device is a secondary computing device. In some embodiments of the exemplary method, the first computing device functions as a modem to facilitate transmission of data between the second computing device and a server computing system.

[0026] In an exemplary aspect, the present disclosure provides an exemplary transitory or non-transitory computer-readable medium embodied in a computer-readable storage device and storing instructions that, when executed by a processor, cause the processor to perform operations. In the exemplary transitory or non-transitory computer-readable medium, the operations include acquiring location data associated with a first computing device and a second computing device, the location data indicating that each of the first computing device and the second computing device is at a target location. In the exemplary transitory or non-transitory computer-readable medium, the operations include determining an on-user device status for each of the first and second computing devices in response to acquiring the location data. In the exemplary transitory or non-transitory computer-readable medium, the operations include acquiring inertial measurement unit (IMU) sensor data from each of the first and second computing devices in response to determining the on-user device status for each of the first and second computing devices. In an exemplary temporary or non-transitory computer-readable medium, the operations include inputting IMU sensor data from each of the first and second computing devices into a machine-learned model, the machine-learned model being configured to combine the IMU sensor data from each of the first and second computing devices. In the exemplary temporary or non-transitory computer-readable medium, the operations include obtaining output data from the machine-learned model indicating a predicted location. In the exemplary temporary or non-transitory computer-readable medium, the operations include comparing the output data indicating the predicted location with data indicating a location of a target subzone. In the exemplary temporary or non-transitory computer-readable medium, the operations include, in response to obtaining data indicating co-presence of each of the first and second computing devices within a target subzone of the target location, transmitting data to a user interface of the first computing device directing the user interface to provide a content item for display.

[0027] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1] 1 illustrates a block diagram of an exemplary data flow according to an exemplary embodiment of the present disclosure. [Figure 2A] 1 illustrates an exemplary device environment, according to an exemplary embodiment of the present disclosure. [Figure 2B] 1 illustrates an exemplary device environment, according to an exemplary embodiment of the present disclosure. [Figure 3A] 1 illustrates an exemplary device environment and device position trajectory, according to an exemplary embodiment of the present disclosure. [Figure 3B] 1 illustrates an exemplary graphical representation of a device position trajectory, according to an exemplary embodiment of the present disclosure. [Figure 4A] 1 illustrates an exemplary device environment and device position trajectory, according to an exemplary embodiment of the present disclosure. [Figure 4B] 1 illustrates an exemplary graphical representation of a device position trajectory, according to an exemplary embodiment of the present disclosure. [Figure 5] 3 illustrates an exemplary block diagram of data flow according to an exemplary embodiment of the present disclosure. [Figure 6] 3 illustrates an exemplary block diagram of data flow according to an exemplary embodiment of the present disclosure. [Figure 7] 3 illustrates an exemplary block diagram of data flow according to an exemplary embodiment of the present disclosure. [Figure 8] 1 shows a flowchart diagram of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates an example block diagram of an example system for performing indoor localization of a user device based on sensor data, according to an example embodiment of the present disclosure. [Figure 10]FIG. 1 illustrates an example block diagram of an example system for performing indoor localization of a user device based on sensor data, according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] In general, the present disclosure is directed to systems and methods for acquiring sensor data from multiple devices associated with a user, which is processed by a location prediction pipeline to predict the user's indoor location. For example, the systems and methods can provide for more accurate localization of a user within a store by combining sensor data from multiple pseudo-wearable computing devices associated with the user. The prediction pipeline can acquire sensor data acquired from an inertial measurement unit (IMU) sensor. The prediction pipeline can include a body on / off logic component. The prediction pipeline can include an IMU streaming component or a fusion component. The prediction pipeline includes a machine-learned model that generates an output including a predicted user location including three degrees of freedom (e.g., x, y, z coordinates). The predicted user location can be used to perform a content selection and delivery process, where content is provided to the user in response to the user location being within a threshold of a target location (e.g., within one meter of a particular section of the store).

[0030] Indoor localization is challenging due to the trade-off between sensor specifications (e.g., bill of materials (BOM), power, product form factor) and accuracy. While methods exist using ultra-wideband (UWB) anchors, Wi-Fi sensors, quick response (QR) codes, near-field communication (NFC), geofencing, radio frequency identification (RFID), and Bluetooth low energy (BLE) beacons, these methods can be expensive and require the use of bulky form-factor units. This implementation can introduce user friction during installation. Alternatively, using only costly sensorless smartphones can result in reduced accuracy. For example, indoor GPS or traditional three-degree-of-freedom (3DoF) solutions (e.g., measuring device displacement in the x, y, and z axes) from smartphone inertial measurement units (IMUs) can be inherently noisy.

[0031] The present disclosure provides a solution that utilizes sensor data acquired from a multi-device first-party ecosystem associated with a user entering an indoor location (e.g., a retail store). The solution is low-friction and provides improved accuracy. Devices in the first-party ecosystem can include wearable (or pseudo-wearable) devices such as smartphones, earphones, smartwatches, laptop computers, and tablets. Each of these devices can acquire data via motion sensors such as IMUs. In some implementations, the system can convert IMU-like sensor data into three-degree-of-freedom data for each device simultaneously. The system can perform a fusion step that combines the individual sensor data sets, performs a noise reduction process, and estimates a noise-reduced three-degree-of-freedom result.

[0032] As an example, a user may move around a physical location (e.g., a store) with a smartphone and a smartwatch (or multiple devices). Locomotor signals may be synchronized between two (or more) devices. By combining the sensor data obtained from the two devices, a common average may be determined to produce more accurate localization results. The accuracy of the improvement may correlate with the number of devices utilized in the process.

[0033] User location based on user device sensor data can be used to predict or estimate the user's location. The computing system can compare the predicted (or estimated) location with the target subzone. The computing system can determine that the user is within the target subzone. In response to determining that the user is within the target subzone, the computing system can perform a content selection process and provide one or more content items for display via the user device's interface (e.g., a telephone graphical user interface, an earphone speaker, a smartwatch graphical user interface).

[0034] The present disclosure provides many technical effects and advantages. For example, the system can more accurately determine the precise indoor user location (e.g., within one meter). The present disclosure provides more robust data collection and processes for accurately determining the precise location of a user for use in various embodiments. Indoor user location is an important technology area with practical applications in various technical fields. This technical field can include more accurate location determination for identifying individuals to implement safety measures (e.g., individuals who have fallen in their homes, users in emergency situations), integration with Internet of Things (IoT) devices (e.g., within a home or office), or presenting content to a user in response to the user being located within a target subzone. Furthermore, because the system can perform many of these processes on a hub device (e.g., a smartphone), it can reduce bandwidth usage by reducing the amount of data that needs to be communicated off-device (e.g., over a network). Furthermore, processing IMU sensor data is more computationally efficient than processing GPS or other location data, reducing computational resource and bandwidth usage.

[0035] 1 illustrates an exemplary data flow 100 according to an embodiment of the present disclosure. The data flow 100 may include a prediction pipeline 108 that acquires sensor data from devices. For example, the prediction pipeline 108 may acquire first device sensor data 102, second device sensor data 104, and third device sensor data. The first device, second device, and third device may be any type of computing device. For example, the computing device may be a wearable computing device or a pseudo-wearable computing device. By way of example, the device may include a smartphone, a smartwatch, earphones, a fitness tracker, a laptop computer, a tablet, or any other computing device. The computing device may include a sensor. The sensors may include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detects changes in altitude), an optical heart rate sensor (detects heartbeats per minute), an SpO2 monitor (e.g., measures blood oxygen levels), a bioimpedance sensor(s), a proximity sensor (e.g., to conserve battery and activate the display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, an electrodermal sensor, a skin temperature sensor, an accelerometer(s), a gyroscope(s), a magnetometer(s), a global positioning system (GPS), a heart rate sensor(s) (e.g., an electrodermal sensor or a photodiode), a pedometer(s) (e.g., electrical, mechanical, or microelectromechanical), a pressure sensor(s) (e.g., a strain gauge), an optical sensor, an acoustic sensor, or other sensors.

[0036] The prediction pipeline 108 may obtain device sensor data including the first device sensor data 102, the second device sensor data, or the third device sensor data 106. The prediction pipeline may process the device sensor data to determine an output 110. The output 110 may include location prediction data 112. The location prediction data 112 may include a predicted location (e.g., a target location) within the indoor environment. The computing system may process the location prediction data 112 to perform an action. The action may include providing content (e.g., to one of the user device or a third-party device), facilitating the provision of emergency services, or controlling a device (e.g., turning on lights, speakers, a television, or other IoT device).

[0037] An exemplary implementation of the present disclosure is determining a user's indoor location within a target location (e.g., a physical store location, an associated location, a home location, a work location). As shown in FIG. 2A , a user 202 can be located inside the target location 200. The user 202 can carry one or more devices. For example, the devices can include a first device 222 (e.g., a smartphone), a second device 224 (e.g., a smartwatch), and a third device 226 (e.g., earphones). As described herein, the devices can include any computing device (e.g., a smartphone, a smartwatch, earphones, a fitness tracker, a laptop computer, or a tablet). The target location 200 can include a target subzone 210. The target subzone 210 can be associated with a particular portion of the target location (e.g., a section, a sign, a display, or a target subzone). When the user enters the target location 200, a computing system can acquire sensor data from the user devices (e.g., the first device 222, the second device 224, the third device 226). In some implementations, the computing system can determine that the user has entered the target location. This determination can trigger further action, including, for example, obtaining sensor data from a user device associated with the user 202. The user device can be part of a first-party ecosystem of devices that the user 202 brings to the target location 200.

[0038] A user 202 may walk around the target location 200, looking at various items and visiting different departments. As shown in FIG. 2B , the user 202 may enter a target subzone 210. A computing system may obtain sensor data from computing devices associated with the user 202 (e.g., a first device 222, a second device 224, a third device 226). The computing system may determine a predicted location (e.g., an estimated location) of the user 202 (e.g., via the prediction pipeline 108). The computing system may compare the predicted location to the target subzone 210. Based on this comparison, the computing system may determine that the user 202 is located within the target subzone 210.

[0039] 3A shows an example trajectory of a user 302 from a first location 305A to a second location 305B. The user 302 can move within the target location 300. For example, the user 302 can move from the first location 305A (e.g., the entrance to the target location 300) to the second location 305B (e.g., within the target subzone 310). The trajectories shown may include trajectory 312, trajectory 314, trajectory 316, and trajectory 320. The trajectory 312 may include data indicating multiple positions between a start position 322A and an end position 322B of a sensor associated with the user device 322 (e.g., the first device 222). The trajectory 314 may include data indicating multiple positions between a start position 324A and an end position 324B of a sensor associated with the user device 324 (e.g., the second device 224). The trajectory 316 may include data indicating multiple positions between a start position 326A and an end position 326B of a sensor associated with a user device 326 (e.g., the third device 226). The trajectory 320 may be associated with a ground truth trajectory of the user 302. The trajectory 320 may include data indicating multiple positions between a first position 305A and a second position 305B.

[0040] In some implementations, the computing system can convert the IMU sensor data into position data indicating a position having three degrees of freedom (e.g., x, y, z coordinates (Cartesian coordinates)). FIG. 3B shows an example graphical representation of trajectories 312, 314, 316, and 320 in Cartesian coordinates. The computing system can directly combine the IMU sensor data or convert the sensor data from the IMU sensor data into data having three degrees of freedom (e.g., x, y, z coordinates). The computing system can perform fusion of the sensor data (e.g., via a prediction pipeline) to generate an output (e.g., output 110). The output can include position prediction data (e.g., position prediction data 112) or a predicted trajectory (e.g., predicted trajectory 425).

[0041] 4A shows an example of a predicted trajectory 425 and a ground truth trajectory 420 of a user 402 from a first location 405A to a second location 405B. The user 402 may move within the target location 400. For example, the user 402 may move from the first location 405A (e.g., the entrance to the target location 400) to a second location 405B (e.g., within the target subzone 410). The trajectory shown may include the ground truth trajectory 420 and the predicted trajectory 425. The ground truth trajectory 420 may represent a known trajectory of the user 402 from the first location 405A to the second location 405B. The predicted trajectory 425 may represent a predicted (e.g., combined) trajectory of the user (and associated devices) from the first predicted location 415A to the second predicted location 415B. The predicted trajectory 425 may be generated by performing a fusion of trajectories associated with multiple user devices (e.g., trajectories 312, 314, and 316). The predicted trajectory 425 may be determined to be within a threshold difference (e.g., 5%, 1%, 0.5%, 0.05%, or any other error margin) from the ground truth trajectory 420.

[0042] In some implementations, the computing system can convert the IMU sensor data into position data indicating a position having three degrees of freedom (e.g., x, y, z coordinates). FIG. 4B shows an example graphical representation of trajectories 420 and 425. The computing system can directly combine the IMU sensor data or convert the sensor data from the IMU sensor data into data having three degrees of freedom (e.g., x, y, z coordinates). The computing system can perform fusion of the sensor data (e.g., via a prediction pipeline) to generate an output (e.g., output 110). The output can include position prediction data (e.g., position prediction data 112) or a predicted trajectory (e.g., predicted trajectory 425).

[0043] 5 illustrates an example data flow 500 according to an embodiment of the present disclosure. The data flow 500 may include obtaining first device sensor data 502, second device sensor data 504, or third device sensor data 506. The data flow may include inputting the first device sensor data 502, second device sensor data 504, or third device sensor data 506 into a prediction pipeline 508 and obtaining output 510. The output may include location prediction data 532.

[0044] The prediction pipeline 508 may include body on / off logic 512, IMU streaming 514, and a fusion layer 522. The body on / off logic 512 may include determining whether a device is located on a user. The body on / off logic 512 may include a computing system that determines whether each device is located on a user. For example, a user may have multiple associated devices. However, in some examples, a user may not have all devices "on their body." For example, a user may place a smartphone, earphones, or smartwatch on a table (or other surface) at home or in a location. The computing system may perform operations to determine whether a device is located on the user (and whether it is located with other associated user devices).

[0045] In some implementations, the body on / off logic may vary depending on the type of computing device. As an example, the computing system may acquire sensor data associated with one or more computing devices. The sensors may include an ambient light sensor, a motion sensor, a heart rate sensor, a proximity sensor, a light sensor, or any other sensor.

[0046] The device may be a smartphone. Body on / off can be confirmed by determining whether the smartphone is in the user's hand, in the user's pocket, in the user's bag, or otherwise located on the user. By analyzing low-frequency spectral energy from the IMU signature, the smartphone can be determined to be located in the user's hand. If the device is in the user's pocket or bag, ambient light sensor (ALS) data can be analyzed to determine whether the item is in a dark location (e.g., similar to determining whether the head is close to the phone when making a call).

[0047] The device may be a smartwatch. Body on / off can be confirmed by processing photoplethysmography (PPG) sensor data used to measure heart rate. If the direct current (DC) value of reflected PPG is high, the measurement may indicate a reflected signal resulting from an immediate occlusion, such as the skin or subcutaneous components of a human wrist, indicating that the device is placed "on the body."

[0048] The device can be a smart earphone (or a pair of smart earphones) that can be used to measure contact signals to the ear using proximity, light, or acoustic sensors to determine whether the device is physically on or off.

[0049] The computing system may determine that the on-user device status indicates that the device is located on the user within the target location. In response, the computing system may perform an IMU streaming process to obtain data for one or more devices located on the user. In some implementations, the computing system may determine that the on-user device status indicates that the device is not located on the user within the target location. In response, the computing system may exclude the device and associated sensor data from the IMU streaming process.

[0050] IMU streaming 514 may include obtaining IMU sensor data associated with a user in response to determining that a device is on the user (e.g., an on-user device status). By way of example, by processing first device sensor data 502, second device sensor data 504, and third device sensor data 506, the computing system may determine that a first device, a second device, and a third device are located on the user. In response, the computing system may obtain IMU sensor data associated with the devices (e.g., first device IMU sensor data 516, second device IMU sensor data 518, or third device IMU sensor data 520). The computing system may input the IMU sensor data to a fusion layer. Fusion layer 522 may include a system that converts the IMU sensor data into three-degree-of-freedom position data before performing fusion (e.g., as shown in FIG. 7) or that directly performs fusion of the IMU sensor data (e.g., as shown in FIG. 6).

[0051] In some implementations, fusion can occur on a computing device associated with the user (e.g., a first computing device). For example, the computing device can stream data from an originating device to another device (e.g., from a second computing device to a first computing device, or from a user computing device to a server computing device). As an example, the first computing device can serve as a hub for the second computing device (e.g., a phone communicatively connected to a smartwatch and earphones). In some implementations, IMU sensor data can be converted to three degrees of freedom (DOF) to provide more physically meaningful measurements (e.g., for a human to perform quality control checks). In some implementations, IMU sensor data can be fused without conversion to three DOF position data. Thus, in some implementations, a machine-learned model can be trained to learn appropriate fusion subtitles that enable merging of IMU data from multiple sensors / devices without converting the sensor data to three DOF position data.

[0052] 6, the fusion layer 522 may include acquiring IMU streaming 514 data from multiple devices in a data acquisition step 602. In the fusion layer 522, a computing system may convert data from the IMU streaming 514 into position data represented in three degrees of freedom. This may include data associated with a first device 614, a second device 616, and an Nth device 618. The position data for the first device 614 may include X1 data 614A, Y1 data 614B, and Z1 data 614C. The position data for the second device 616 may include X2 data 616A, Y2 data 616B, and Z2 data 616C. The position data for the Nth device 618 may include X1 data 614A, Y1 data 614B, and Z2 data 616C. n Data 618A, Y n Data 618B and Z nData 618C may be included. There may be any number of devices associated with the user. For example, there may be two devices, three devices, four devices, up to N devices.

[0053] The IMU sensor data can be converted into three degrees of freedom data using any means. For example, the IMU sensor data can be converted into a three-degree-of-freedom trajectory. The system can perform double integration to convert acceleration or gyroscope measurements into three degrees of freedom of position (e.g., x, y, z coordinates) or a translational representation of movement through three-dimensional space (e.g., target location). Converting and combining IMU sensor data from multiple devices can enable more accurate indoor user localization by more accurately utilizing IMU measurements using a multi-device polling structure.

[0054] The system acquires data from the data acquisition step 602 and can rearrange the data into each degree of freedom. For example, the computing system can rearrange all X data (X1 data 614A, X2 data 616A, X n data 618A), all Y data (Y1 data 614B, Y2 data 616B, Y n Data 618B), all Z data (Z1 data 614C, Z2 data 616C, Z n The X data 624, the Y data 626, and the Z data 628 may be combined in a fusion step 604. In the fusion step 604, the X data may be fused to form fused X data 624, the Y data may be fused to form fused Y data 626, and the Z data may be fused to form fused Z data 628. A machine-learned model 630 may obtain the fused X data 624, the fused Y data 626, and the fused Z data 628. The machine-learned model 630 may generate an output 510. The output 510 may be a representation of the X F Coordinates 632A, Y F Coordinates 632B and Z F It may include location data including coordinates 632C. (X F ,Y F ,ZF ,) may represent the absolute location of a device associated with a user (e.g., this user) within a target location (e.g., target subzone 210, 310, 410).

[0055] The fusion can be performed in many ways. For example, the fusion can be performed by a machine-learned model. The machine-learned model can be a neural network. In some implementations, the neural network can be a fully connected neural network. The machine-learned model can be trained in an offline process to perform the fusion process. The machine-learned model can be trained in an online process. The machine-learned model can be any machine-learned model described herein. The machine-learned model can be trained using any machine-learned model training process.

[0056] Referring to FIG. 7 , fusion layer 522 may include fusion of IMU sensor data. For example, a computing system may acquire first device IMU sensor data 712, second device IMU sensor data 714, and nth device IMU sensor data 716 (or data from multiple devices). The computing system may input the device IMU sensor data (first device IMU sensor data 712, second device IMU sensor data 714, and nth device IMU sensor data 716) into machine-learned model 720. The fusion may be performed in many ways. For example, the fusion may be performed by a machine-learned model. The machine-learned model may be a neural network. In some implementations, the neural network may be a fully connected neural network. The machine-learned model may be trained in an offline process to perform the fusion process. The machine-learned model may be trained in an online process. The machine-learned model may be any machine-learned model described herein. The machine-learned model may be trained using any machine-learned model training process.

[0057] The machine-learned model 720 can process the IMU sensor data (e.g., first device IMU sensor data 712, second device IMU sensor data 714, and nth device IMU sensor data 716). The computing system can obtain an output 510. The output 510 can be X F Coordinates 732A, Y F Coordinates 732B and Z F It may include location data including coordinates 732C. (X F ,Y F ,Z F ,) may represent the absolute location of a device associated with a user (e.g., this user) within a target location (e.g., target subzone 210, 310, 410).

[0058] 5, the computing system may obtain output 510. Output 510 may include location prediction data 532. Location prediction data 532 may include, for example, a predicted absolute location of a user in an indoor location. For example, the absolute location may be predicted to within one meter of accuracy.

[0059] The absolute position may be determined by combining an identified starting position (e.g., the entrance of the target location) obtained via a sensor (e.g., using GPS, Wi-Fi connection strength). This may be used as a starting reference point for determining the absolute position. An exemplary method for determining the absolute position may include filtering the IMU sensor data. An exemplary method for determining the absolute position may include performing a double integral on the IMU sensor data to convert the IMU sensor data (e.g., accelerometer data, gyroscope data) into a position in three degrees of freedom or a trajectory in three degrees of freedom.

[0060] The computing system can use this output to perform various operations (or actions). For example, the computing system can compare the predicted location of the user (and associated devices) to target subzones (e.g., target subzone 210, target subzone 310, target subzone 410) within a target location (e.g., target location 200, target location 300, target location 400).

[0061] By way of example, the action may include sending instructions to one or more of the user devices associated with the user that cause at least one of the user devices associated with the user to display a content item. For example, the content item may be information, provide a selectable user interface element (e.g., a hyperlink, a button), content associated with a particular portion of a store, or other content relevant to the user's indoor location. In some implementations, the content item may include an advertisement. For example, a target subzone may be associated with a particular item (e.g., a shirt). The computing system may obtain an output indicating the user's predicted location and compare the predicted location to the target subzone. If the user's predicted location is within a threshold of the target subzone, the computing system may transmit data that causes a content item associated with the target subzone (e.g., a message indicating "buy one, get one free").

[0062] In some implementations, the target location can be a location associated with the user's work environment or home environment. The user's work environment or home environment can include multiple IoT devices. The computing system can determine the user's location within the environment and perform an action based on the determined location. As an example, the user can walk within a target subzone associated with a particular smart device (e.g., a light, a television, a refrigerator, a speaker, an oven, etc.), and in response to the user's predicted location being within a threshold of the target subzone, the computing system can perform an action associated with the smart device (and the IoT device). In some implementations, the predicted location can be used in safety-related implementations. For example, the predicted location can indicate a user fall (e.g., if the initial 3DOF position indicates a high z-coordinate position and the final 3DOF position indicates a low z-coordinate position). Additionally or alternatively, the predicted location can be used by emergency services to determine whether the user is within the target location or specifically where the user is located (e.g., in the event of a fire, evacuation, or other emergency).

[0063] In addition to the above, users may be provided with controls that allow them to choose both whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's behavior or the user's current location), as well as whether the user receives content or communications from the server. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, a user's identifying information may be processed so that personally identifiable information about the user cannot be determined, or, if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be determined. Thus, users may have control over what information is collected about them, how that information is used, and what information is provided to them.

[0064] 8 illustrates a flowchart of an example method 800 for performing indoor user location in accordance with an example embodiment of the present disclosure. While FIG. 8 illustrates steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particularly shown order or arrangement. Various steps of method 800 may be omitted, rearranged, combined, or adapted in various ways without departing from the scope of the present disclosure.

[0065] At (802), method 800 may include acquiring location data associated with the first and second computing devices, the location data indicating that the first and second computing devices are at a target location. For example, the computing system may acquire location data associated with the first and second computing devices, the location data indicating that the first and second computing devices are at the target location. As described herein, the location data may be any data indicative of a user location. For example, the user location may be determined using GPS data, cellular tower data, Wi-Fi data, or any other location data. As described herein, the target location may be a physical store location or a location associated with the user (e.g., a home location, a work location). By way of example, acquiring location data indicating that the user and associated devices are located at the target location (e.g., an entrance to the associated location) may be used as a trigger for the system to automatically perform the steps described herein. For example, in response to determining that the user has arrived at the target location (e.g., an associated location, a location of interest), the computing system may acquire IMU sensor data from each device.

[0066] In some implementations, the method includes acquiring position data associated with a third computing device. In some implementations, the method includes acquiring IMU sensor data from the third computing device. As described herein, the described methods can be performed using any number of computing devices. For example, the described methods can be performed using two computing devices, three computing devices, four computing devices, up to N computing devices.

[0067] In some embodiments, the first computing device includes a smartphone, the second computing device includes a smartwatch, and the third computing device includes earphones. In some embodiments, the smartphone may be a primary computing device, and the smartwatch and earphones may be secondary computing devices. In some embodiments, the smartphone, smartwatch, and earphones may be communicatively coupled to each other (e.g., using Bluetooth). For example, the first computing device is a primary computing device and the second computing device is a secondary computing device. As an example, the first computing device may function as a modem to facilitate transmission of data between the second computing device(s) and a server computing system.

[0068] At (804), method 800 may include determining an on-user device status for the first and second computing devices in response to obtaining the location data. For example, the computing system may determine an on-user device status for the first and second computing devices in response to obtaining the location data. As described herein, the on-user device status may indicate that the computing device is located on the user. In some implementations, the on-user device status may indicate that the computing device is not located on the user (e.g., at home or stationary on a surface).

[0069] In some implementations, the on-user device status of the first and second computing devices indicates the first and second computing devices are located on a user moving within the target location. In some implementations, the on-user device status is determined for additional computing devices associated with the user.

[0070] At (806), method 800 may include acquiring inertial measurement unit (IMU) sensor data from the first and second computing devices in response to determining the on-user device status of the first and second computing devices. For example, the computing system may include acquiring inertial measurement unit (IMU) sensor data from the first and second computing devices in response to determining the on-user device status for the first and second computing devices. As described herein, the IMU sensor data includes accelerometer data and gyroscope data. In some implementations, the computing is performed on the first computing device.

[0071] In some implementations, the computing system may determine that the on-user device status indicates that the device is not located on the user, and in response, the computing system may exclude acquisition of sensor data for the device.

[0072] At 808, method 800 may include inputting IMU sensor data from the first and second computing devices into a machine-learned model. For example, a computing system may input the IMU sensor data from the first and second computing devices into the machine-learned model. As described herein, the IMU sensor data from the first and second computing devices may be combined using a fusion method. In some implementations, the machine-learned model is configured to convert the accelerometer data and gyroscope data into Cartesian coordinates indicating absolute position. By way of example, the accelerometer data and gyroscope data may be converted to Cartesian coordinates via double integration. In some implementations, the IMU sensor data may be filtered. In some implementations, the machine-learned model includes a neural network. For example, the neural network may be a fully connected neural network.

[0073] In some implementations, at 808, method 800 may include inputting IMU sensor data from an additional computing device (e.g., a third computing device) into the machine-learned model.

[0074] At 810, method 800 may include obtaining output data from the machine-learned model indicating the predicted location. For example, a computing system may obtain the output data from the machine-learned model indicating the predicted location. As described herein, the output data indicating the predicted location may be expressed in three degrees of freedom (e.g., Cartesian coordinates, x-y-z coordinates). The output data may be in any format that is available and comparable to location data associated with the target subzone.

[0075] At (812), method 800 may include comparing the output data indicative of the predicted location with data indicative of the location of the target sub-zone. For example, a computing system may compare the output data indicative of the predicted location with data indicative of the location of the target sub-zone. As described herein, the target sub-zone may be associated with a particular portion of a store (e.g., a display, section, or sign). The target sub-zone may be associated with a particular portion of a home, office, retail space, or museum.

[0076] At (814), method 800 may include, in response to obtaining data indicating the co-presence of the first and second computing devices within a target subzone of the target location, transmitting data to a user interface of the first computing device directing the user interface of the first computing device to provide a content item for display. For example, in response to obtaining data indicating the co-presence of the first and second computing devices within the target subzone of the target location, the computing system may transmit data to a user interface of the first computing device directing the user interface of the first computing device to provide a content item for display. As described herein, the content item for display may be associated with a third party (e.g., a content provider, an advertiser). By way of example, the content item may be related to the target subzone. For example, the target subzone may be associated with a store section of a particular T-shirt brand. The content item may be associated with additional information about the T-shirt brand, coupons for the shirt, or advertisements related to the shirt. In some implementations, the target subzone may be a portion of a museum. In some implementations, the content item may be associated with a portion of the museum (e.g., a particular historical exhibit, an artwork, etc.). In some implementations, the content item may be a selectable user interface element (e.g., a user can select it to perform an action such as turning on or controlling lights, a television, speakers, or other devices).

[0077] 9 illustrates a block diagram of an example computing system 900 that generates, trains, or uses a machine-learned model to predict the indoor location of a user device associated with a user, according to an example embodiment of the present disclosure. The computing system 900 includes a client computing system 902, a server computing system 904, and a training computing system 906 that are communicatively coupled via a network 908.

[0078] The client computing system 902 may be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop), a game console or controller, an embedded computing device, or any other type of computing device.

[0079] The client computing system 902 includes one or more processors 912 and memory 914. The one or more processors 912 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operatively connected processors. The memory 914 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 914 can store data 916 and instructions 918 that are executed by the processor 912 to cause the client computing system 902 to perform operations.

[0080] In some implementations, the client computing system 902 may store or include one or more machine-learned models 920. For example, the machine-learned models 920 may be or include various machine-learned models, such as neural networks (e.g., fully-connected neural networks, deep neural networks), or other types of machine-learned models, including nonlinear or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, fully-connected neural networks, or other types of neural networks. Some exemplary machine-learned models may utilize attention mechanisms, such as self-attention. For example, some exemplary machine-learned models may include multi-head self-attention models (e.g., Transformer models). Exemplary machine-learned models 920 are described with reference to FIGS. 5, 6, and 7.

[0081] In some implementations, one or more machine-learned models 920 may be received from the server computing system 904 over the network 908, stored in the user computing device memory 914, and then used or implemented by one or more processors 912. In some implementations, the client computing system 902 may implement multiple parallel instances of a single machine-learned model 920 (e.g., to perform fusion actions on IMU sensor data, convert IMU sensor data to three-degree-of-freedom data, or determine a predicted position of a user).

[0082] More specifically, the machine-learned model can obtain sensor data associated with one or more user devices (e.g., first device sensor data 924A, second device sensor data 924B, third device sensor data 924C). The sensor data may be associated with one or more sensors associated with one of the multiple user devices. For example, the sensors may include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detects changes in altitude), an optical heart rate sensor (detects heartbeats per minute), an SpO2 monitor (e.g., measures blood oxygen levels), bioimpedance sensor(s), a proximity sensor (e.g., to conserve battery and activate a display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, an electrodermal sensor, a skin temperature sensor, accelerometer(s), gyroscope(s), magnetometer(s), a global positioning system (GPS), heart rate sensor(s) (e.g., electrodermal sensors or photodiodes), pedometer(s) (e.g., electrical, mechanical, or microelectromechanical), pressure sensor(s) (e.g., strain gauge), an optical sensor, an acoustic sensor, or other sensors. Sensor data may include gravitational acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurement units (e.g., generated using accelerometers or gyroscopes), motion orientation, position (e.g., sensed via satellite), strain, pressure, resistance change (e.g., static or dynamic).

[0083] In some embodiments, the training data 966 may include ground truth data associated with the user's absolute indoor location (or associated trajectory). For example, the ground truth data may be obtained from a database. The database may be populated through experimentation (e.g., conducted in a laboratory or at actual indoor locations). The computing system may retrieve, suggest, or generate one or more content items in response to determining the predicted user location.

[0084] Additionally or alternatively, one or more machine-learned models 926 may be included in or stored and implemented on a server computing system 904 that communicates with the user computing system 902 according to a client-server relationship. For example, the machine-learned models 926 may be implemented by the server computing system 904 as part of a web service (e.g., a content provisioning service, a location determination service, a location identification service, a campaign management service, a content strategy management service). Thus, one or more models 920 may be stored and implemented on the client computing system 902, or one or more models 926 may be stored and implemented on the server computing system 904.

[0085] The client computing system 902 may also include one or more user input components 122 that can receive user input. For example, the user input component 922 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0086] The client computing system may include sensor data from a sensor database 924. The sensor data from the sensor database 924 may include first device sensor data 924A, second device sensor data 924B, or third device sensor data 924C. The first device sensor data 924A may include data obtained via one or more sensors associated with the first device. For example, the first device sensor data 924A may include sensor data from the sensor database 924. The sensor data from the sensor database 924 may include gravitational acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurement unit (e.g., generated using an accelerometer or gyroscope), motion orientation, position (e.g., detected via satellite), strain, pressure, or resistance change (e.g., static or dynamic). The sensor data from the sensor database 924 may be used by the client computing system 902 and processed by machine-learned model(s) (and associated prediction pipelines) to determine a predicted location of the user (and associated user device). The computing system can use the user's predicted location (e.g., and sensor data from the sensor database 924) to send a request to the server computing system 904 for one or more content items (e.g., messages, images, videos, selectable user interface elements, advertisements). The computing system can obtain, generate, or cause the one or more suggested content items to be presented to the user via a user interface of the device (e.g., the user device).

[0087] The server computing system 904 includes one or more processors 932 and memory 934. The one or more processors 932 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller) and can be a single processor or multiple operably connected processors. The memory 934 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 934 can store data 936 and instructions 938 that are executed by the processor 932 to cause the server computing system 904 to perform operations.

[0088] In some implementations, server computing system 904 includes or is implemented by one or more server computing devices. When server computing system 904 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0089] As described above, the server computing system 904 may store or include one or more machine-learned models 926. For example, the machine-learned models 926 may be or include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, fully connected neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine-learned models may utilize attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include multi-head self-attention models (e.g., Transformer models). Exemplary machine-learned models 926 are described with reference to FIGS. 5, 6, and 7.

[0090] The client computing device 902 or the server computing system 904 can train the machine-learned model 920 or 926 through interaction with a training computing system 906 that is communicatively coupled via a network 908. The training computing system 906 can be separate from the server computing system 904 or can be part of the server computing system 904.

[0091] The training computing system 906 includes one or more processors 952 and memory 954. The one or more processors 952 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 954 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 954 can store data 956 and instructions 958 that are executed by the processor 952 to cause the training computing system 906 to perform operations. In some implementations, the training computing system 906 includes or is otherwise implemented with one or more server computing devices (e.g., server computing system 904).

[0092] The training computing system 906 may include a model trainer 960 that trains the machine-learned models 920 or 926 stored on the client computing system 902 or the server computing system 904 using various training or learning techniques, such as backpropagation. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, or various other loss functions. Gradient descent may be used to iteratively update the parameters through several training iterations.

[0093] In some implementations, performing backpropagation may include performing truncated backpropagation through time. The model trainer 960 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0094] In particular, model trainer 960 can train machine-learned model 920 or 926 based on a set of training data 966. Training data 966 can include, for example, ground truth position data. For example, the ground truth position data can be data that has been properly converted from IMU sensor data to three-degree-of-freedom position data, and the IMU sensor data is compared to the ground truth three-degree-of-freedom position data.

[0095] In some implementations, if the user provides consent, training examples may be provided by the client computing system 902. Thus, in such implementations, the machine-learned model 920 provided to the client computing system 902 may be trained by the training computing system 906 based on user-specific data received from the client computing system 902. In some implementations, this process may be referred to as personalizing the model.

[0096] Model Trainer 960 includes computer logic utilized to provide desired functionality. Model Trainer 960 can be implemented in hardware, firmware, or software controlling a general-purpose processor. For example, in some implementations, Model Trainer 960 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, Model Trainer 960 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0097] Network 908 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 908 may occur over any type of wired or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0098] The machine-learned models described herein may be used in a variety of tasks, applications, or use cases.

[0099] In some implementations, the input to the machine-learned model(s) of the present disclosure may be sensor data. The machine-learned model(s) may process the sensor data to generate an output. As an example, the machine-learned model(s) may process the sensor data to generate a recognition output. As another example, the machine-learned model(s) may process the sensor data to generate a prediction output. As another example, the machine-learned model(s) may process the sensor data to generate a classification output. As another example, the machine-learned model(s) may process the sensor data to generate a segmentation output. As another example, the machine-learned model(s) may process the sensor data to generate a visualization output. As another example, the machine-learned model(s) may process the sensor data to generate a diagnostic output. As another example, the machine-learned model(s) may process the sensor data to generate a detection output.

[0100] In some implementations, input to the machine-learned model(s) of the present disclosure may be statistical data. The statistical data may be, represent, or include data computed or calculated from some other data source. The machine-learned model(s) may process the statistical data to generate an output. As an example, the machine-learned model(s) may process the statistical data to generate a recognition output. As another example, the machine-learned model(s) may process the statistical data to generate a prediction output. As another example, the machine-learned model(s) may process the statistical data to generate a classification output. As another example, the machine-learned model(s) may process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) may process the statistical data to generate a visualization output. As another example, the machine-learned model(s) may process the statistical data to generate a diagnostic output.

[0101] In some cases, the machine-learned model(s) may be configured to perform a task that includes encoding input data for reliable or efficient transmission or storage (or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), and the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or data).

[0102] In some implementations, the client computing system 902 can include multiple computing devices. The client devices can communicate directly with the server computing system 904 or the training computing system 906 over a network 908. In some implementations, the client computing system can include multiple computing devices using primary client devices, as shown in FIG.

[0103] FIG. 10 illustrates a block diagram of an exemplary computing system 1000 that generates, trains, or uses a machine-learned model to predict the indoor location of a user device associated with a user, according to an exemplary embodiment of the present disclosure. The computing system 1000 includes a primary computing device 1010, a secondary computing device 1030, and a secondary computing device 1050. In some implementations, the computing devices may be associated with a client computing system (e.g., client computing system 902 shown in FIG. 9). In some implementations, the secondary computing devices 1030 and 1050 may be communicatively coupled to the primary computing device (e.g., over a network (cellular or wireless), via a Bluetooth connection). For example, the secondary computing device 1030 may be tethered to the primary computing device 1010. In some implementations, the primary computing device 1010 may be used as a modem to connect one or more secondary devices (e.g., secondary device 1030 or 1050) to a network (e.g., network 908). As an example, the primary computing device 1010 may be communicatively coupled via a network to a server computing system (e.g., server computing system 904) or a training computing system (e.g., training computing system 906) that is communicatively coupled via network 908.

[0104] The primary computing device 1010 may be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet). The client computing system 902 may be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop), a game console or controller, an embedded computing device, or any other type of computing device.

[0105] The primary computing device 1010 includes one or more processors 1012 and memory 1014. The one or more processors 1012 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 1014 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 1014 can store data 1015 and instructions 1016 that are executed by the processor 1012 to cause the primary computing device 1010 to perform operations.

[0106] In some implementations, the primary computing device 1010 may store or include one or more machine-learned models 1018. For example, the machine-learned models 1018 may be or include various machine-learned models, such as neural networks (e.g., deep neural networks), or other types of machine-learned models, including nonlinear or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine-learned models may utilize attention mechanisms, such as self-attention. For example, some exemplary machine-learned models may include multi-head self-attention models (e.g., Transformer models). Exemplary machine-learned models 1018 are described with reference to FIGS. 5, 6, 7, and 9.

[0107] In some implementations, one or more machine-learned models 1018 may be received from a server computing system (e.g., server computing system 904) over network 908, stored in user computing device memory 1014, and then used or implemented by one or more processors 1012. In some implementations, the primary computing device 1010 may implement multiple parallel instances of a single machine-learned model 1018 (e.g., to perform fusion actions on IMU sensor data, convert IMU sensor data to three-degree-of-freedom data, or determine a predicted position of a user).

[0108] More specifically, the machine-learned model can obtain sensor data associated with one or more user devices (e.g., sensor data 1020, sensor data 1040, or sensor data 1060). The sensor data can be associated with primary computing device 1010, secondary computing device 1030, or secondary computing device 1050. For example, the sensors may include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detects changes in altitude), an optical heart rate sensor (detects heartbeats per minute), an SpO2 monitor (e.g., measures blood oxygen levels), bioimpedance sensor(s), a proximity sensor (e.g., to conserve battery and activate a display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, an electrodermal sensor, a skin temperature sensor, accelerometer(s), gyroscope(s), magnetometer(s), a global positioning system (GPS), heart rate sensor(s) (e.g., electrodermal sensors or photodiodes), pedometer(s) (e.g., electrical, mechanical, or microelectromechanical), pressure sensor(s) (e.g., strain gauge), an optical sensor, an acoustic sensor, or other sensors. Sensor data may include gravitational acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurement units (e.g., generated using accelerometers or gyroscopes), motion orientation, position (e.g., sensed via satellite), strain, pressure, resistance change (e.g., static or dynamic).

[0109] The secondary computing device 1030 may be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop), a game console or controller, an embedded computing device, or any other type of computing device.

[0110] The secondary computing device 1030 includes one or more processors 1032 and memory 1034. The one or more processors 1032 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 1034 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 1034 can store data 1035 and instructions 1036 that are executed by the processor 1032 to cause the primary computing device 1010 to perform operations.

[0111] The secondary computing device 1050 may be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop), a game console or controller, an embedded computing device, or any other type of computing device.

[0112] The secondary computing device 1050 includes one or more processors 1052 and memory 1054. The one or more processors 1052 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 1054 can include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which can be transitory or non-transitory, and combinations thereof. The memory 1054 can store data 1055 and instructions 1056 that are executed by the processor 1052 to cause the primary computing device 1010 to perform operations.

[0113] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as operations performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0114] While the subject matter of the present disclosure has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided for purposes of illustration and not limitation of the present disclosure. Those skilled in the art, once they arrive at the foregoing understanding, will be able to readily create modifications, variations, and equivalents to such embodiments. Accordingly, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet another embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents.

[0115] The steps shown or described are merely exemplary and may be omitted, combined, or performed in an order other than that shown or described. The numbering of the steps as shown is for ease of reference only and does not imply that any particular order is necessary or preferred.

[0116] The functions or steps described herein may be embodied in computer-usable data or computer-executable instructions, executed by one or more computers or other devices to perform one or more functions described herein. Generally, such data or instructions include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular data types when executed by one or more processors of a computer or other data processing device. Computer-executable instructions may be stored on a computer-readable medium, such as a hard disk, optical disk, removable storage medium, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. As will be appreciated, the functionality of such instructions may be combined or distributed as desired. Additionally, functionality may be embodied in whole or in part in firmware or hardware equivalents, e.g., integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Particular data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated as being within the scope of the computer-executable instructions or computer-usable data described herein.

[0117] Although not required, those skilled in the art will appreciate that various aspects described herein may be embodied as a method, system, apparatus, or one or more computer-readable mediums storing computer-executable instructions. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment, an entirely firmware embodiment, or an embodiment combining software, hardware, or firmware aspects in any combination.

[0118] As described herein, the various methods and acts may be operable across one or more computing devices or networks. Functions may be distributed in any manner or may be located on a single computing device (e.g., a server, a client computer, a user device, etc.).

[0119] Aspects of the present disclosure have been described with respect to exemplary embodiments thereof. Numerous other embodiments, modifications, or variations within the scope and spirit of the appended claims may occur to those skilled in the art from a consideration of this disclosure. For example, one skilled in the art may recognize that the steps shown or described can be performed in a sequence other than that listed, or that one or more of the steps shown may be optional or may be combined. The features of any and all of the following claims can be combined or rearranged in any possible manner.

[0120] While the subject matter of the present disclosure has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation and not by way of limitation of the present disclosure. Those skilled in the art, once they arrive at the foregoing understanding, may readily create modifications, variations, or equivalents to such embodiments. Accordingly, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet another embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and / or equivalents.

[0121] Terms are described herein using lists of exemplary elements joined by conjunctions such as "and," "or," or "but." It should be understood that such conjunctions are provided for illustrative purposes only. Lists joined by certain conjunctions such as "or" can refer, for example, to "at least one of" or "any combination of" the exemplary elements listed therein, and "or" should be understood as "and / or" unless otherwise indicated. Additionally, terms such as "based on" should be understood as "based at least in part on."

Claims

1. 1. A computing system comprising: one or more processors; one or more computer-readable media storing executable instructions to cause the one or more processors to perform operations, the operations comprising: obtaining location data associated with a first computing device and a second computing device, the location data indicating that each of the first computing device and the second computing device is at a target location; determining an on-user device status for each of the first and second computing devices in response to obtaining the location data; acquiring inertial measurement unit (IMU) sensor data from each of the first and second computing devices in response to determining an on-user device status for each of the first and second computing devices; inputting the IMU sensor data from each of the first and second computing devices into a machine-learned model, the machine-learned model configured to combine the IMU sensor data from each of the first and second computing devices; obtaining output data from the machine-learned model indicating a predicted location; comparing the output data indicative of the predicted location with data indicative of the location of a target subzone; In response to obtaining data indicating co-presence of each of the first and second computing devices within the target subzone of the target location, transmitting data to a user interface of the first computing device directing the user interface to provide a content item for display; a computing system including:

2. 2. The system of claim 1, wherein the on-user device status for each of the first and second computing devices indicates that each of the first and second computing devices is located on a user moving within the target location.

3. The system of claim 1 or 2, wherein the IMU sensor data includes accelerometer data and gyroscope data.

4. The system of claim 3 , wherein the machine-learned model is configured to convert the accelerometer data and the gyroscope data into Cartesian coordinates representing absolute position.

5. The system of claim 1 , further comprising acquiring IMU sensor data from a third computing device.

6. 6. The system of claim 5, wherein the first computing device comprises a smartphone, the second computing device comprises a smartwatch, and the third computing device comprises earphones.

7. The system of claim 1 , wherein the computing is performed on the first computing device.

8. The system of claim 1 , wherein the IMU sensor data from each of the first and second computing devices is combined using a fusion method.

9. The system of claim 1 , wherein the machine-learned model comprises a neural network.

10. The system of claim 1 , wherein the machine-learned model comprises a fully connected neural network.

11. 1. A computer-implemented method comprising: obtaining location data associated with a first computing device and a second computing device, the location data indicating that each of the first computing device and the second computing device is at a target location; determining an on-user device status for each of the first and second computing devices in response to obtaining the location data; acquiring inertial measurement unit (IMU) sensor data from each of the first and second computing devices in response to determining an on-user device status for each of the first and second computing devices; inputting the IMU sensor data from each of the first and second computing devices into a machine-learned model, the machine-learned model configured to combine the IMU sensor data from each of the first and second computing devices; obtaining output data from the machine-learned model indicating a predicted location; comparing the output data indicative of the predicted location with data indicative of the location of a target subzone; In response to obtaining data indicating co-presence of each of the first and second computing devices within the target subzone of the target location, transmitting data to a user interface of the first computing device directing the user interface to provide a content item for display; A computer-implemented method comprising:

12. 12. The computer-implemented method of claim 11, wherein the on-user device status for each of the first and second computing devices indicates that each of the first and second computing devices is located on a user moving within the target location.

13. 13. The computer-implemented method of claim 11 or 12, wherein the IMU sensor data includes accelerometer data and gyroscope data.

14. 14. The computer-implemented method of claim 13, wherein the machine-learned model is configured to convert the accelerometer data and the gyroscope data into Cartesian coordinates representing absolute position.

15. 15. The computer-implemented method of claim 11, further comprising obtaining IMU sensor data from a third computing device.

16. 16. The computer-implemented method of claim 15, wherein the first computing device comprises a smartphone, the second computing device comprises a smartwatch, and the third computing device comprises earphones.

17. 17. The computer-implemented method of claim 11, wherein the computing is performed on the first computing device.

18. 18. The computer-implemented method of claim 11, wherein the IMU sensor data from each of the first and second computing devices is combined using a fusion method.

19. 19. The computer-implemented method of claim 11, wherein the machine-learned model comprises a neural network.

20. 19. The computer-implemented method of any one of claims 11 to 18, wherein the first device is a primary computing device, the second computing device is a secondary computing device, and the first device functions as a modem to facilitate transmission of data between the second device and a server computing system.

21. A computer readable storage medium comprising instructions embodied in the medium and which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 20.

Images