Indoor positioning based on multiple device sensors

Abstract

An example embodiment of the present disclosure provides an example method that includes obtaining location data associated with a first computing device and a second computing device. An example method may include determining a user device carry-on state for each of a first computing device and a second computing device. An example method may include obtaining inertial measurement unit (IMU) sensor data from each of a first computing device and a second computing device. An example method may include inputting IMU sensor data from each of a first computing device and a second computing device into a machine learning model. An example method may include obtaining output data indicative of a predicted location from a machine learning model. An example method may include comparing output data indicative of a predicted location to data indicative of a location of a target sub-region. An example method may include sending data instructing a user interface of a first computing device to provide a content item for display.

Description

Technical Field

[0001] The present disclosure generally relates to positioning using data from multiple sensors. More particularly, the present disclosure relates to coordinating data from multiple sensors to determine an indoor positioning of a computing device located on a user. Background Art

[0002] The computing device may be associated with a plurality of sensors. Sensor data may be obtained and used to determine the device location. Summary of the invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] In one example aspect, the present disclosure provides an example system for indoor positioning based on multiple device sensors, the example system comprising one or more processors and one or more memory devices, the one or more memory devices storing instructions, the instructions executable to cause the one or more processors to perform operations. In some implementations, the one or more memory devices may include one or more temporary or non-temporary computer-readable media, the one or more temporary or non-temporary computer-readable media storing instructions, the instructions executable to cause the one or more processors to perform operations. In the example system, the operation may include obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are each at a target location. In the example system, the operation may include determining a user device on-user device status of each of the first computing device and the second computing device in response to obtaining the location data. In the example system, the operation may include obtaining inertial measurement unit (IMU) sensor data from each of the first computing device and the second computing device in response to determining an on-user device status of each of the first computing device and the second computing device. In the example system, the operation may include inputting IMU sensor data from each of the first computing device and the second computing device into a machine learning model, wherein the machine learning model is configured to combine the IMU sensor data from each of the first computing device and the second computing device. In the example system, the operation may include obtaining output data indicating a predicted location from the machine learning model. In the example system, the operation may include comparing the output data indicating the predicted location with data indicating the location of the target sub-area. In the example system, the operation may include, in response to obtaining data indicating co-location of each of the first computing device and the second computing device within the target sub-area of ​​the target location, sending data instructing a user interface of the first computing device to provide a content item for display.

[0005] In some embodiments of the example system, the user device on-body status of each of the first computing device and the second computing device indicates that each of the first computing device and the second computing device is located on the body of a user moving within the target location.

[0006] In some embodiments of the example system, the IMU sensor data includes accelerometer data and gyroscope data.

[0007] In some embodiments of the example system, the machine learning model is configured to transform the accelerometer data and the gyroscope data into data Cartesian coordinates indicating absolute position.

[0008] In some embodiments of the example system, the operations include obtaining IMU sensor data from a third computing device.

[0009] In some embodiments of the example system, the first computing device comprises a smart phone, the second computing device comprises a smart watch, and the third computing device comprises earbuds.

[0010] In some embodiments of the example system, the computing is performed on a first computing device.

[0011] In some embodiments of the example system, IMU sensor data from each of the first computing device and the second computing device is combined using a fusion method.

[0012] In some embodiments of the example system, the machine learning model includes a neural network.

[0013] In some embodiments of the example system, the machine learning model includes a fully connected neural network.

[0014] In some embodiments of the example system, the first device is a primary computing device and the second computing device is a secondary computing device. In some embodiments of the example system, the first device functions as a modem to facilitate transmission of data between the second computing device and the server computing system.

[0015] In an example aspect, the present disclosure provides an example computer-implemented method. The example method includes obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are each at a target location. In the example method, the method includes determining a user device carry-on status of each of the first computing device and the second computing device in response to obtaining the location data. In the example method, the method includes obtaining inertial measurement unit (IMU) sensor data from each of the first computing device and the second computing device in response to determining a user device carry-on status of each of the first computing device and the second computing device. In the example method, the method includes inputting the IMU sensor data from each of the first computing device and the second computing device into a machine learning model, wherein the machine learning model is configured to combine the IMU sensor data from each of the first computing device and the second computing device. In the example method, the method includes obtaining output data indicating a predicted location from the machine learning model. In the example method, the method includes comparing the output data indicating the predicted location with data indicating the location of the target sub-area. In the example method, the method includes, in response to obtaining data indicating co-location of each of the first computing device and the second computing device within a target sub-region of a target location, sending data instructing a user interface of the first computing device to provide a content item for display.

[0016] In some embodiments of the example method, the user device on-body status of each of the first computing device and the second computing device indicates that each of the first computing device and the second computing device is located on the body of a user moving within the target location.

[0017] In some embodiments of the example method, the IMU sensor data includes accelerometer data and gyroscope data.

[0018] In some embodiments of the example methods, the machine learning model is configured to transform the accelerometer data and the gyroscope data into data Cartesian coordinates indicating absolute position.

[0019] In some embodiments of the example method, the method includes obtaining IMU sensor data from a third computing device.

[0020] In some embodiments of the example method, the first computing device comprises a smart phone, the second computing device comprises a smart watch, and the third computing device comprises earbuds.

[0021] In some embodiments of the example method, the computing is performed on a first computing device.

[0022] In some embodiments of the example method, IMU sensor data from each of the first computing device and the second computing device is combined using a fusion method.

[0023] In some embodiments of the example methods, the machine learning model includes a neural network.

[0024] In some embodiments of the example methods, the machine learning model includes a fully connected neural network.

[0025] In some embodiments of the example method, the first device is a primary computing device and the second computing device is a secondary computing device. In some embodiments of the example method, the first device functions as a modem to facilitate transmission of data between the second computing device and the server computing system.

[0026] In an example aspect, the present disclosure provides an example transient or non-transitory computer-readable medium, the example transient or non-transitory computer-readable medium being embodied in a computer-readable storage device and storing instructions that, when executed by a processor, cause the processor to perform operations. In the example transient or non-transitory computer-readable medium, the operations include obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are each at a target location. In the example transient or non-transitory computer-readable medium, the operations include determining a user device with-body status of each of the first computing device and the second computing device in response to obtaining the location data. In the example transient or non-transitory computer-readable medium, the operations include obtaining inertial measurement unit (IMU) sensor data from each of the first computing device and the second computing device in response to determining a user device with-body status of each of the first computing device and the second computing device. In the example transient or non-transitory computer-readable medium, the operation includes inputting IMU sensor data from each of the first computing device and the second computing device into a machine learning model, wherein the machine learning model is configured to combine the IMU sensor data from each of the first computing device and the second computing device. In the example transient or non-transitory computer-readable medium, the operation includes obtaining output data indicating a predicted location from the machine learning model. In the example transient or non-transitory computer-readable medium, the operation includes comparing the output data indicating the predicted location with data indicating the location of the target sub-region. In the example transient or non-transitory computer-readable medium, the operation includes, in response to obtaining data indicating co-location of each of the first computing device and the second computing device within the target sub-region of the target location, sending data instructing a user interface of the first computing device to provide a content item for display. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0028] Figure 1 Depicted is an example block diagram of data flow according to an example embodiment of the present disclosure.

[0029] FIG. 2A to FIG. 2B An example device environment is depicted according to an example embodiment of the present disclosure.

[0030] Figure 3A Depicted are example device environments and device location trajectories according to example embodiments of the present disclosure.

[0031] Figure 3B Depicted are example graphical representations of device location trajectories according to example embodiments of the present disclosure.

[0032] Figure 4A Depicted are example device environments and device location trajectories according to example embodiments of the present disclosure.

[0033] Figure 4B Depicted are example graphical representations of device location trajectories according to example embodiments of the present disclosure.

[0034] Figures 5 to 7 Depicted is an example block diagram of data flow according to an example embodiment of the present disclosure.

[0035] Figure 8 A flow chart of an example method according to an example embodiment of the present disclosure is depicted.

[0036] Figures 9 and 10 An example block diagram of an example system for performing indoor positioning of a user device based on sensor data according to an example embodiment of the present disclosure is depicted. DETAILED DESCRIPTION

[0037] In general, the present disclosure relates to systems and methods for obtaining sensor data from multiple devices associated with a user for processing by a location prediction pipeline to predict the user's indoor location. For example, the system and method can more accurately perform positioning of a user within a store by combining sensor data from multiple pseudo-wearable computing devices associated with the user. The prediction pipeline can obtain sensor data obtained from an inertial measurement unit (IMU) sensor. The prediction pipeline can include an on-body / off-body logic component. The prediction pipeline can include an IMU streaming component or a fusion component. The prediction pipeline includes a machine learning model that generates an output including a predicted user location, the predicted user location including three degrees of freedom (e.g., xyz coordinates). The predicted user location can be used to perform a content selection and transmission process that causes content to be supplied to the user in response to the user location being within a threshold of a target location (e.g., within one meter of a particular section of a store).

[0038] Indoor positioning is difficult due to the trade-off between sensor specifications (e.g., broadcast external module (BOM), power, product form factor) and accuracy. Although there are methods that use ultra-wideband (UWB) anchors, Wi-Fi sensors, quick response (QR) codes, near field communication (NFC), geo-fencing, radio frequency identification (RFID), Bluetooth low energy (BLE) beacons, etc., these methods may be costly and require the use of large form factor units. This implementation may be associated with a facility-user friction component. Alternatively, using only a high-cost sensorless smartphone may result in a decrease in accuracy. For example, indoor GPS or conventional three-degree-of-freedom (3DoF) solutions (e.g., measuring xyz device displacement) from a smartphone inertial measurement unit (IMU) may be inherently noisy.

[0039] The present disclosure provides a solution that utilizes sensor data obtained from a multi-device first-party ecosystem that accompanies a user who has entered an indoor location (e.g., a retail store). This solution is low-friction and has improved accuracy. The devices in the first-party ecosystem may include wearable (or pseudo-wearable) devices such as smart phones, earbuds, smart watches, laptops, tablets, etc. Each of these devices can obtain data via an IMU-like motion sensor. In some implementations, the system can simultaneously convert the IMU-like sensor data into three-degree-of-freedom data for each device. The system can perform a fusion step that combines the various sensor data sets and performs a denoising process to estimate the denoised three-degree-of-freedom result.

[0040] For example, a user may be moving within a physical location (e.g., a store) with a smart phone and a smart watch (or more devices). The pedestrian movement signal may be synchronized between the two (or more) devices. By combining the sensor data obtained from the two devices, a common average may be determined to generate a more accurate positioning result. The improved accuracy may be related to the number of devices utilized in the process.

[0041] The positioning of the user based on the user device sensor data can be used to predict or estimate the user's location. The computing system can compare the predicted (or estimated) location with the target sub-region. The computing system can determine that the user is located within the target sub-region. In response to determining that the user is located within the target sub-region, the computing system can perform a content selection process and provide one or more content items for display via an interface of the user device (e.g., a graphical user interface of a phone, a speaker of an earbud, a graphical user interface of a smart watch).

[0042] The present disclosure provides many technical effects and benefits. For example, the system can more accurately determine the precise indoor user location (e.g., within one meter). The present disclosure provides more robust data collection and processing to accurately determine the precise location of the user for various implementations. Indoor user positioning is an important technical field with practical applications in a variety of technical fields. The technical field may include more accurate positioning for locating individuals to implement safety measures (e.g., individuals who fall at home, users in emergency situations), integration with Internet of Things (IoT) devices (e.g., in a home or office), or for displaying content to users in response to users being located in a target sub-area. Additionally, the system can perform many of these processes on a central device (e.g., a smart phone), thereby reducing bandwidth usage by reducing the amount of data that must be communicated outside the device (e.g., via a network). Additionally, processing IMU sensor data can be more computationally efficient than processing GPS or other location data, thereby reducing the use of computing resources and bandwidth.

[0043] Figure 1An example data flow 100 is depicted in accordance with an embodiment of the present disclosure. The data flow 100 may include a prediction pipeline 108 that obtains sensor data from a device. For example, the prediction pipeline 108 may obtain first device sensor data 102, second device sensor data 104, and third device sensor data. The first device, the second device, and the third device may be any kind of computing device. For example, the computing device may be a wearable computing device or a pseudo-wearable computing device. For example, the device may include a smart phone, a smart watch, earbuds, a fitness tracker, a laptop, a tablet, or any other computing device. The computing device may include a sensor. The sensors may include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detects changes in altitude), an optical heart rate sensor (detects heart beats per minute), an SpO2 monitor (e.g., for measuring blood oxygen levels), a bioimpedance sensor, a proximity sensor (e.g., to save battery and wake up the display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, a skin electrode activity sensor, a skin temperature sensor, an accelerometer, a gyroscope, a magnetometer, a global positioning system (GPS), a heart rate sensor (e.g., an electrode sensor or a photodiode), a pedometer (e.g., electrical, mechanical, or microelectromechanical), a pressure sensor (e.g., a strain gauge), an optical sensor, an audio sensor, or other sensors.

[0044] The prediction pipeline 108 may obtain device sensor data, including the first device sensor data 102, the second device sensor data, or the third device sensor data 106. The prediction pipeline may process the device sensor data to determine an output 110. The output 110 may include location prediction data 112. The location prediction data 112 may include a predicted location within an indoor environment (e.g., a target location). The computing system may process the location prediction data 112 to perform an action. The action may include providing content (e.g., to one of the user devices or to a third party device), facilitating the provision of emergency services, or controlling a device (e.g., turning on a light, speaker, TV, or other IoT device).

[0045] An example implementation of the present disclosure is to determine the indoor location of a user in a target location (e.g., a physical store location, a related location, a home location, a work location). Figure 2AAs shown, user 202 may be located inside target location 200. User 202 may carry one or more devices. For example, the devices may include a first device 222 (e.g., a smart phone), a second device 224 (e.g., a smart watch), and a third device 226 (e.g., earbuds). As described herein, the devices may include any computing device (e.g., a smart phone, a smart watch, an earbud, a fitness tracker, a laptop, or a tablet). Target location 200 may include a target sub-area 210. Target sub-area 210 may be associated with a specific portion of the target location (e.g., a section, a sign, a display, or a target sub-area). When the user enters target location 200, the computing system may obtain sensor data from the user device (e.g., first device 222, second device 224, third device 226). In some implementations, the computing system may determine that the user has entered the target location. This determination may trigger further operations, including, for example, obtaining sensor data from a user device associated with user 202. The user device may be part of a first-party ecosystem of devices that user 202 brings into target location 200.

[0046] User 202 can walk around target location 200 to view various items or visit different departments. Figure 2B As depicted, user 202 may walk into target sub-region 210. The computing system may obtain sensor data from computing devices (e.g., first device 222, second device 224, third device 226) associated with user 202. The computing system may determine (e.g., via prediction pipeline 108) a predicted location (e.g., an estimated location) of user 202. The computing system may compare the predicted location to target sub-region 210. Based on the comparison, the computing system may determine that user 202 is located within target sub-region 210.

[0047] Figure 3AAn example trajectory of a user 302 from a first location 305A to a second location 305B is depicted. The user 302 may move in the target location 300. For example, the user 302 may move from the first location 305A (e.g., the entrance of the target location 300) to the second location 305B (e.g., within the target sub-area 310). The depicted trajectory may include a trajectory 312, a trajectory 314, a trajectory 316, and a trajectory 320. The trajectory 312 may include data indicating a plurality of locations between a starting location 322A and an ending location 322B of a sensor associated with a user device 322 (e.g., the first device 222). The trajectory 314 may include data indicating a plurality of locations between a starting location 324A and an ending location 324B of a sensor associated with a user device 324 (e.g., the second device 224). The trajectory 316 may include data indicating a plurality of locations between a starting location 326A and an ending location 326B of a sensor associated with a user device 326 (e.g., the third device 226). Track 320 may be associated with a ground-truth track of user 302. Track 320 may include data indicating a plurality of locations between first location 305A and second location 305B.

[0048] In some implementations, the computing system can convert the IMU sensor data into position data indicating a position with three degrees of freedom (eg, xyz coordinates (Cartesian coordinates)). Figure 3B An example graphical representation of trajectories 312, 314, 316, and 320 in Cartesian coordinates is depicted. The computing system may combine the IMU sensor data directly, or may transform the sensor data from the IMU sensor data into data with three degrees of freedom (e.g., xyz coordinates). The computing system (e.g., via a prediction pipeline) may perform fusion of the sensor data to generate an output (e.g., output 110). The output may include position prediction data (e.g., position prediction data 112) or a predicted trajectory (e.g., predicted trajectory 425).

[0049] Figure 4AAn example of a predicted trajectory 425 and a true value trajectory 420 of a user 402 from a first location 405A to a second location 405B is depicted. The user 402 may move inside the target location 400. For example, the user 402 may move from a first location 405A (e.g., the entrance of the target location 400) to a second location 405B (e.g., within the target sub-area 410). The depicted trajectory may include a true value trajectory 420 and a predicted trajectory 425. The true value trajectory 420 may represent a known trajectory of the user 402 from the first location 405A to the second location 405B. The predicted trajectory 425 may represent a predicted (e.g., combined) trajectory of the user (and associated device) from the first predicted location 415A to the second predicted location 415B. The predicted trajectory 425 may be generated by performing a fusion of trajectories associated with multiple user devices (e.g., trajectories 312, 314, and 316). The predicted trajectory 425 may be determined to be within a threshold difference (eg, an error margin of 5%, 1%, 0.5%, 0.05%, or any other error margin) from the true value trajectory 420 .

[0050] In some implementations, the computing system can convert the IMU sensor data into position data indicating a position with three degrees of freedom (eg, xyz coordinates). Figure 4B An example graphical representation of trajectories 420 and 425 is depicted. The computing system may combine the IMU sensor data directly, or may transform the sensor data from the IMU sensor data into data with three degrees of freedom (e.g., xyz coordinates). The computing system (e.g., via a prediction pipeline) may perform fusion of the sensor data to generate an output (e.g., output 110). The output may include position prediction data (e.g., position prediction data 112) or a predicted trajectory (e.g., predicted trajectory 425).

[0051] Figure 5 An example data flow 500 is depicted in accordance with an implementation of the present disclosure. The data flow 500 may include obtaining first device sensor data 502, second device sensor data 504, or third device sensor data 506. The data flow may include inputting the first device sensor data 502, second device sensor data 504, or third device sensor data 506 into a prediction pipeline 508 and obtaining an output 510. The output may include position prediction data 532.

[0052] The prediction pipeline 508 may include on / off body logic 512, IMU streaming 514, and a fusion layer 522. The on / off body logic 512 may include determining whether a device is on the user. The on / off body logic 512 may include a computing system determining whether a corresponding device is on the user. For example, a user may have multiple associated devices. However, in some instances, the user may not have all devices "on body". For example, the user may leave a smart phone, earbuds, or smart watch at home or on a table (or other surface) within a location. The computing system may perform operations to determine whether a device is on the user (and co-located with other associated user devices).

[0053] In some implementations, the on / off logic may vary depending on the type of computing device. For example, the computing system may obtain sensor data associated with one or more computing devices. The sensors may include ambient light sensors, motion sensors, heart rate sensors, proximity sensors, optical sensors, or any other sensors.

[0054] The device may be a smartphone. Presence / off-presence may be confirmed by determining whether the smartphone is in the user's hand, in the user's pocket, in the user's bag, or otherwise on the user's person. The smartphone may be determined to be in the user's hand by analyzing low frequency spectrum energy from the IMU signature. If the device is in the user's pocket or bag, ambient light sensor (ALS) data may be analyzed to determine if the item is in a dark location (e.g., similar to checking if a phone near the head is on a phone call).

[0055] The device may be a smartwatch. On / off body may be confirmed by processing photoplethysmography (PPG) sensor data for heart rate measurement. If the direct current (DC) value of the reflective PPG is high, the measurement may indicate that the reflected signal is coming from a direct obstruction (such as the skin or subcutaneous components of a person's wrist), which may indicate that the device is "on body".

[0056] The device can be a smart earbud (or a pair of smart earbuds). Proximity sensors, optical sensors, or audio sensors can be used to measure on-ear contact signals to confirm presence / absence.

[0057] The computing system may determine that the user device on-body status indication device is located on the user within the target location. In response, the computing system may perform an IMU streaming process to obtain data of one or more devices located on the user. In some implementations, the computing system may determine that the user device on-body status indication device is not located on the user within the target location. In response, the computing system may exclude the device and associated sensor data from the IMU streaming process.

[0058] IMU streaming 514 may include obtaining IMU sensor data associated with the user in response to determining that the device is on the user (e.g., the user device is on-body). For example, by processing the first device sensor data 502, the second device sensor data 504, and the third device sensor data 506, the computing system may determine that the first device, the second device, and the third device are located on the user. In response, the computing system may obtain IMU sensor data associated with the device (e.g., the first device IMU sensor data 516, the second device IMU sensor data 518, or the third device IMU sensor data 520). The computing system may input the IMU sensor data into a fusion layer. The fusion layer 522 may include the system transforming the IMU sensor data into three-degree-of-freedom position data and then performing fusion (e.g., as Figure 7 ) or directly perform fusion of IMU sensor data (e.g., as depicted in Figure 6 depicted).

[0059] In some implementations, fusion can occur on a computing device associated with a user (e.g., a first computing device). For example, a computing device can stream data from a device origin to another device (e.g., from a second computing device to a first computing device, from a user computing device to a server computing device). For example, a first computing device can be used as a hub for a secondary computing device (e.g., a phone with a communicatively connected smart watch and earbuds). In some implementations, IMU sensor data can be transformed into three degrees of freedom, which provides more physically meaningful measurements (e.g., for a person to perform quality control checks). In some implementations, IMU sensor data can be fused without being transformed into three-degree-of-freedom position data. Therefore, in some implementations, a machine learning model can be trained to learn the correct fused subheadings to allow IMU data from multiple sensors / devices to be merged without transforming the sensor data into three-degree-of-freedom position data.

[0060] Go to Figure 6 , the fusion layer 522 may include obtaining data from the IMU stream 514 from the plurality of devices at the data obtaining step 602. In the fusion layer 522, the computing system may convert the data from the IMU stream 514 into position data represented in three degrees of freedom. This may include data associated with the first device 614, the second device 616, and the Nth device 618. The position data of the first device 614 may include X1 data 614A, Y1 data 614B, and Z1 data 614C. The position data of the second device 616 may include X2 data 616A, Y2 data 616B, and Z2 data 616C. The position data of the Nth device 618 may include X2 data 616A, Y2 data 616B, and Z2 data 616C. n Data 618A, Y n Data 618B and Zn Data 618C. There may be any number of devices associated with the user. For example, there may be two devices, three devices, four devices, up to N devices.

[0061] IMU sensor data may be converted to three degrees of freedom data using any means. For example, IMU sensor data may be converted to a three degrees of freedom trajectory. The system may perform a double integral to convert acceleration or gyroscope measurements to three degrees of freedom of position (e.g., xyz coordinates) or a translational representation of movement through three-dimensional space (e.g., target location). Converting and combining IMU sensor data from multiple devices may allow for more accurate indoor user positioning by using a multi-device polling structure to make the use of IMU measurements more accurate.

[0062] The system of the present invention can obtain data from the data acquisition step 602 and rearrange the data into corresponding degrees of freedom. For example, the computing system can combine all X data (X1 data 614A, X2 data 616A, X n data 618A) are grouped together, and all Y data (Y1 data 614B, Y2 data 616B, Y n data 618B) are grouped together, and all Z data (Z1 data 614C, Z2 data 616C, Z n At the fusion step 604, the X data may be fused to form fused X data 624, the Y data may be fused to form fused Y data 626, and the Z data may be fused to form fused Z data 628. The machine learning model 630 may obtain the fused X data 624, the fused Y data 626, and the fused Z data 628. The machine learning model 630 may generate an output 510. The output 510 may include position data, which includes X F Coordinates 632A, Y F Coordinates 632B and Z F Coordinates 632C. (X F , Y F , Z F ) may represent an absolute position of a device associated with a user (eg, the user) within a target location (eg, a target sub-area 210, 310, 410).

[0063] Fusion can be performed in numerous ways. For example, fusion can be performed by a machine learning model. The machine learning model can be a neural network. In some implementations, the neural network can be a fully connected neural network. The machine learning model can be trained in an offline process to perform the fusion process. The machine learning model can be trained in an online process. The machine learning model can be any machine learning model described herein. The machine learning model can be trained using any machine learning model training process.

[0064] Go to Figure 7 , the fusion layer 522 may include fusion of IMU sensor data. For example, the computing system may obtain the first device IMU sensor data 712, the second device IMU sensor data 714, and the device n IMU sensor data 716 (or data from multiple devices). The computing system may input the device IMU sensor data (the first device IMU sensor data 712, the second device IMU sensor data 714, and the device n IMU sensor data 716) into the machine learning model 720. Fusion can be performed in many ways. For example, fusion can be performed by a machine learning model. The machine learning model can be a neural network. In some implementations, the neural network can be a fully connected neural network. The machine learning model can be trained in an offline process to perform the fusion process. The machine learning model can be trained in an online process. The machine learning model can be any machine learning model described herein. The machine learning model can be trained using any machine learning model training process.

[0065] The machine learning model 720 can process the IMU sensor data (e.g., the first device IMU sensor data 712, the second device IMU sensor data 714, and the device n IMU sensor data 716). The computing system can obtain the output 510. The output 510 can include position data, which includes X F Coordinates 732A, Y F Coordinates 732B and Z F Coordinates 732C. (X F , Y F ,Z F ) may represent an absolute position of a device associated with a user (eg, the user) within a target location (eg, a target sub-area 210, 310, 410).

[0066] Return to Figure 5 , the computing system may obtain output 510. Output 510 may include location prediction data 532. Location prediction data 532 may include, for example, a predicted absolute location of a user in an indoor location. For example, the absolute location may be predicted within an accuracy of one meter.

[0067] The absolute position may be determined by combining an identified starting position (e.g., an entrance to a target location) obtained via a sensor (e.g., GPS, using Wi-Fi connection strength). This may be used as a starting reference point for determining the absolute position. An example method for determining the absolute position may include filtering the IMU sensor data. An example method for determining the absolute position may include performing a double integration on the IMU sensor data to convert the IMU sensor data (e.g., accelerometer data, gyroscope data) into a position in three degrees of freedom or a trajectory in three degrees of freedom.

[0068] The computing system can use the output to perform various operations (or actions). For example, the computing system can compare the predicted location of the user (and associated device) to a target sub-region (e.g., target sub-region 210, target sub-region 310, target sub-region 410) within a target location (e.g., target location 200, target location 300, target location 400).

[0069] For example, the action may include sending an instruction to one or more of the user devices associated with the user, the instruction causing the content item to be displayed on at least one of the user devices associated with the user. For example, the content item may be informational, providing a selectable user interface element (e.g., a hyperlink, a button), content associated with a particular part of a store, or other content associated with the user's indoor location. In some implementations, the content item may include an advertisement. For example, a target sub-area may be associated with a particular item (e.g., a shirt). The computing system may obtain an output indicating the user's predicted location and compare the predicted location with the target sub-area. If the user's predicted location is within a threshold of the target sub-area, the computing system may send data that causes a content item related to the target sub-area (e.g., a message indicating "buy one, get one free").

[0070] In some implementations, the target location may be a location associated with the user's work or home environment. The user's work or home environment may include multiple IoT devices. The computing system may determine the user's location within the environment and perform an action based on the determined location. For example, a user may walk in a target sub-area associated with a specific smart device (e.g., a lamp, TV, refrigerator, speaker, oven, etc.), and in response to the user's predicted location being within a threshold of the target sub-area, the computing system may perform an action associated with the smart device (e.g., IoT device). In some implementations, the predicted location may be used in safety-related implementations. For example, the predicted location may indicate that the user has fallen (e.g., the initial three-degree-of-freedom position indicates a higher z-coordinate position, while the final three-degree-of-freedom position indicates a low z-coordinate position). Additionally or alternatively, the predicted location may be used by emergency services to determine whether the user is within the target location or specifically where the user is located (e.g., in the event of a fire, evacuation, or other emergency).

[0071] In addition to the above description, controls may be provided to the user that allow the user to make choices about whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's activities or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or, in the case of geographic location information, the user's location may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0072] Figure 8 A flow chart of an example method 800 for performing indoor user positioning is depicted according to an example embodiment of the present disclosure. Figure 8 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The various steps of method 800 may be omitted, rearranged in various ways, combined or adjusted without departing from the scope of the present disclosure.

[0073] At (802), method 800 may include obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are at a target location. For example, the computing system may obtain location data associated with the first computing device and the second computing device, the location data indicating that the first computing device and the second computing device are at a target location. As described herein, the location data may be any data indicating a user's location. For example, the user's location may be determined using GPS data, cellular tower data, Wi-Fi data, or any other location data. As described herein, the target location may be a physical store location or a location associated with the user (e.g., a home location, a work location). For example, obtaining location data indicating that the user and the associated device are at a target location (e.g., at an entrance to a relevant location) may be used as a trigger for the system to automatically perform the steps described herein. For example, in response to determining that the user has arrived at a target location (e.g., a relevant location, a location of interest), the computing system may obtain IMU sensor data from the corresponding device.

[0074] In some implementations, the method may include obtaining position data associated with a third computing device. In some implementations, the method includes obtaining IMU sensor data from the third computing device. As described herein, the described method may be performed using any number of computing devices. For example, the described method may be performed using two computing devices, three computing devices, four computing devices, up to N computing devices.

[0075] In some implementations, the first computing device includes a smartphone, the second computing device includes a smartwatch, and the third computing device includes earbuds. In some implementations, the smartphone can be a primary computing device, and the smartwatch and earbuds can be secondary computing devices. In some implementations, the smartphone, smartwatch, and earbuds can be communicatively coupled to each other (e.g., using Bluetooth, etc.). For example, the first computing device is the primary computing device, and the second computing device is the secondary computing device. For example, the first computing device can be used as a modem to facilitate the transmission of data between the second computing device and the server computing system.

[0076] At (804), method 800 may include determining a user device on-body status of the first computing device and the second computing device in response to obtaining the location data. For example, the computing system may determine a user device on-body status of the first computing device and the second computing device in response to obtaining the location data. As described herein, the user device on-body status may indicate that the computing device is on the user. In some implementations, the user device on-body status may indicate that the computing device is not on the user (e.g., at home, fixed to a surface).

[0077] In some implementations, the user device with-body status of the first computing device and the second computing device indicates that the first computing device and the second computing device are located on a user moving within the target location. In some implementations, the user device with-body status of an additional computing device associated with the user is determined.

[0078] At (806), method 800 may include obtaining inertial measurement unit (IMU) sensor data from the first computing device and the second computing device in response to determining a user device body-carrying status of the first computing device and the second computing device. For example, the computing system may obtain inertial measurement unit (IMU) sensor data from the first computing device and the second computing device in response to determining a user device body-carrying status of the first computing device and the second computing device. As described herein, the IMU sensor data includes accelerometer data and gyroscope data. In some implementations, the computing is performed on the first computing device.

[0079] In some implementations, the computing system may determine that the user device on-body status indicates that the device is not on the user. In response, the computing system may exclude obtaining sensor data of the device.

[0080] At (808), method 800 may include inputting IMU sensor data from the first computing device and the second computing device into the machine learning model. For example, the computing system may input IMU sensor data from the first computing device and the second computing device into the machine learning model. As described herein, the IMU sensor data from the first computing device and the second computing device are combined using a fusion method. In some implementations, the machine learning model is configured to transform accelerometer data and gyroscope data into data Cartesian coordinates indicating absolute position. For example, the accelerometer data and the gyroscope data may be converted to Cartesian coordinates by a double integral. In some implementations, the IMU sensor data may be filtered. In some implementations, the machine learning model includes a neural network. For example, the neural network may be a fully connected neural network.

[0081] In some implementations, at (808), method 800 may include inputting IMU sensor data from an additional computing device into the machine learning model (eg, a third computing device).

[0082] At (810), method 800 may include obtaining output data indicating the predicted location from the machine learning model. For example, the computing system may obtain output data indicating the predicted location from the machine learning model. As described herein, the output data indicating the predicted location may be in three degrees of freedom (e.g., Cartesian coordinates, xyz coordinates). The output data may be in any form that may be utilized and compared with location data associated with the target sub-region.

[0083] At (812), method 800 may include comparing the output data indicating the predicted location with data indicating the location of the target sub-area. For example, the computing system may compare the output data indicating the predicted location with data indicating the location of the target sub-area. As described herein, the target sub-area may be associated with a particular portion of a store (e.g., a display, section, or sign). The target sub-area may be associated with a particular portion of a house, office, retail space, or museum.

[0084] At (814), method 800 may include, in response to obtaining data indicating co-location of the first computing device and the second computing device within the target sub-area of ​​the target location, sending data instructing the user interface of the first computing device to provide the content item for display. For example, the computing system may, in response to obtaining data indicating co-location of the first computing device and the second computing device within the target sub-area of ​​the target location, send data instructing the user interface of the first computing device to provide the content item for display. As described herein, the content item for display may be associated with a third party (e.g., a content provider, an advertiser). For example, the content item may be related to the target sub-area. For example, the target sub-area may be associated with a section of a store with a particular brand of T-shirts. The content item may be associated with additional information about the brand of T-shirts, a coupon for the shirt, or an advertisement associated with the shirt. In some implementations, the target sub-area may be part of a museum. In some implementations, the content item may be associated with the part of the museum (e.g., a particular historical display, a work of art, etc.). In some implementations, the content item may be a selectable user interface element (e.g., the user interface element may be selected by a user to perform an action, such as turning on or controlling a light, a TV, a speaker, or other device).

[0085] Fig. 9 A block diagram of an example computing system 900 that generates, trains, or uses a machine learning model to predict an indoor location of a user device associated with a user is depicted according to an example embodiment of the present disclosure. The computing system 900 includes a client computing system 902, a server computing system 904, and a training computing system 906 that are communicatively coupled via a network 908.

[0086] The client computing system 902 can be any type of computing device, such as, for example, a mobile computing device (e.g., a smart phone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop computer), a game console or controller, an embedded computing device, or any other type of computing device.

[0087] The client computing system 902 includes one or more processors 912 and memory 914. The one or more processors 912 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 914 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 914 may store data 916 and instructions 918 that are executed by the processor 912 to cause the client computing system 902 to perform operations.

[0088] In some implementations, the client computing system 902 may store or include one or more machine learning models 920. For example, the machine learning model 920 may be or may otherwise include various machine learning models, such as a neural network (e.g., a fully connected neural network, a deep neural network) or other types of machine learning models, including nonlinear models or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, a fully connected neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Reference Figure 5 , Figure 6 and Figure 7 Discuss example machine learning model 920.

[0089] In some implementations, one or more machine learning models 920 may be received from the server computing system 904 via the network 908, stored in the user computing device memory 914, and then used or otherwise implemented by the one or more processors 912. In some implementations, the client computing system 902 may implement multiple parallel instances of a single machine learning model 920 (e.g., to perform fusion actions on IMU sensor data, transform IMU sensor data into three-degree-of-freedom data, or determine a predicted position of a user).

[0090] More specifically, the machine learning model can obtain sensor data associated with one or more user devices (e.g., first device sensor data 924A, second device sensor data 924B, third device sensor data 924C). The sensor data can be associated with one or more sensors associated with one or more user devices. For example, the sensor can include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detecting changes in altitude), an optical heart rate sensor (detecting heart beats per minute), an SpO2 monitor (e.g., for measuring blood oxygen levels), a bioimpedance sensor, a proximity sensor (e.g., to save battery and wake up the display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, a skin electrical activity sensor, a skin temperature sensor, an accelerometer, a gyroscope, a magnetometer, a global positioning system (GPS), a heart rate sensor (e.g., an electrode sensor or a photodiode), a pedometer (e.g., electrical, mechanical, or micro-electromechanical), a pressure sensor (e.g., a strain gauge), an optical sensor, an audio sensor, or other sensors. Sensor data may include gravitational acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurements (e.g., generated using an accelerometer or gyroscope), motion orientation, position (e.g., detected via a satellite), strain, pressure, change in resistance (e.g., static or dynamic).

[0091] In some embodiments, the training data 966 may include real-value data associated with the absolute position (or associated trajectory) of the indoor user. For example, the real-value data may be obtained from a database. The database may be populated via experiments (e.g., conducted in a laboratory, in a real indoor location). The computing system may retrieve, suggest, or generate one or more content items in response to determining the predicted user location.

[0092] Additionally or alternatively, one or more machine learning models 926 may be included in or otherwise stored and implemented by a server computing system 904 that communicates with the client computing system 902 according to a client-server relationship. For example, the machine learning model 926 may be implemented by the server computing system 904 as part of a web service (e.g., a content provisioning service, a location determination service, a positioning service, a campaign management service, a content policy management service). Thus, one or more machine learning models 920 may be stored and implemented at the client computing system 902, or one or more models 926 may be stored and implemented at the server computing system 904.

[0093] The client computing system 902 may also include one or more user input components 122 that may receive user input. For example, the user input component 922 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0094] The client computing system may include sensor data from a sensor database 924. The sensor data from the sensor database 924 may include first device sensor data 924A, second device sensor data 924B, or third device sensor data 924C. The first device sensor data 924A may include data obtained via one or more sensors associated with the first device. For example, the first device sensor data 924A may include sensor data from the sensor database 924. The sensor data from the sensor database 924 may include gravity acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurement units (e.g., generated using an accelerometer or gyroscope), motion orientation, position (e.g., detected via a satellite), strain, pressure, or resistance change (e.g., static or dynamic). The sensor data from the sensor database 924 may be used by the client computing system 902 to determine the predicted position of the user (and associated user device) through processing by a machine learning model (and associated prediction pipeline). The computing system may use the predicted location of the user (e.g., and sensor data from sensor database 924) to send a request for one or more content items (e.g., messages, images, videos, selectable user interface elements, advertisements) to server computing system 904. The computing system may cause one or more suggested content items to be retrieved, generated, or presented to the user via a user interface of a device (e.g., a user device).

[0095] The server computing system 904 includes one or more processors 932 and memory 934. The one or more processors 932 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 934 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 934 may store data 936 and instructions 938 that are executed by the processor 932 to cause the server computing system 904 to perform operations.

[0096] In some implementations, the server computing system 904 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 904 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0097] As described above, the server computing system 904 may store or otherwise include one or more machine learning models 926. For example, the machine learning model 926 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, fully connected neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Reference Figure 5 , Figure 6 and Figure 7 Discuss example machine learning model 926.

[0098] The client computing system 902 or the server computing system 904 can train the machine learning model 920 or 926 via interaction with a training computing system 906 communicatively coupled through a network 908. The training computing system 906 can be separate from the server computing system 904 or can be part of the server computing system 904.

[0099] The training computing system 906 includes one or more processors 952 and memory 954. The one or more processors 952 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 954 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 954 may store data 956 and instructions 958 that are executed by the processor 952 to cause the training computing system 906 to perform operations. In some implementations, the training computing system 906 includes one or more server computing devices or is otherwise implemented by one or more server computing devices (e.g., server computing system 904).

[0100] The training computing system 906 may include a model trainer 960 that trains a machine learning model 920 or 926 stored at the client computing system 902 or the server computing system 904 using various training or learning techniques (e.g., such as error back propagation). For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, or various other loss functions. Gradient descent techniques may be used to iteratively update parameters in multiple training iterations.

[0101] In some implementations, performing error back propagation may include performing truncated back propagation through time. The model trainer 960 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0102] In particular, the model trainer 960 can train the machine learning model 920 or 926 based on a set of training data 966. The training data 966 can include, for example, true value position data. For example, the true value position data can be data correctly transformed from IMU sensor data into three-degree-of-freedom position data, and the IMU sensor data is compared with the true value three-degree-of-freedom position data.

[0103] In some implementations, if the user has provided consent, the training examples may be provided by the client computing system 902. Thus, in such implementations, the machine learning model 920 provided to the client computing system 902 may be trained by the training computing system 906 based on user-specific data received from the client computing system 902. In some cases, this process may be referred to as personalizing the model.

[0104] The model trainer 960 includes computer logic for providing the desired functionality. The model trainer 960 can be implemented in hardware, firmware, or software that controls a general purpose processor. For example, in some implementations, the model trainer 960 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 960 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium (such as RAM, a hard disk, or an optical or magnetic medium).

[0105] The network 908 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 908 can be conducted via any type of wired or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL).

[0106] The machine learning models described in this specification can be used for a variety of tasks, applications, or use cases.

[0107] In some implementations, the input of the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0108] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. Statistical data may be, represent, or otherwise include data calculated or computed from some other data source. The machine learning model may process statistical data to generate an output. As an example, the machine learning model may process statistical data to generate an identification output. As another example, the machine learning model may process statistical data to generate a prediction output. As another example, the machine learning model may process statistical data to generate a classification output. As another example, the machine learning model may process statistical data to generate a segmentation output. As another example, the machine learning model may process statistical data to generate a visualization output. As another example, the machine learning model may process statistical data to generate a diagnostic output.

[0109] In some cases, the machine learning model can be configured to perform a task that includes encoding input data for reliable or efficient transmission or storage (or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating an embedding for the input data (e.g., input audio or visual data).

[0110] In some implementations, the client computing system 902 may include multiple computing devices. The client device may communicate directly with the server computing system 904 or the training computing system 906 via the network 908. In some implementations, the client computing system may include multiple computing devices including a master client device, such as Fig.10 Depicted.

[0111] Fig.10 A block diagram of an example computing system 1000 is depicted that generates, trains, or uses a machine learning model to predict an indoor location of a user device associated with a user, according to an example embodiment of the present disclosure. The computing system 1000 includes a primary computing device 1010 and secondary computing devices 1030 and 1050. In some implementations, the computing devices may communicate with a client computing system (e.g., Fig. 9 908). In some implementations, the secondary computing devices 1030 and 1050 can be communicatively coupled to the primary computing device (e.g., over a network (cellular or wireless), via a Bluetooth connection). For example, the secondary computing device 1030 can be tethered to the primary computing device 1010. In some implementations, the primary computing device 1010 can be used as a modem to connect one or more secondary devices (e.g., secondary devices 1030 or 1050) to a network (e.g., network 908). For example, the primary computing device 1010 can be communicatively coupled via a network to a server computing system (e.g., server computing system 904) or a training computing system (e.g., training computing system 906) that is communicatively coupled via the network 908.

[0112] The host computing device 1010 can be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet). The client computing system 902 can be any type of computing device, such as, for example, a mobile computing device (e.g., a smartphone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop computer), a game console or controller, an embedded computing device, or any other type of computing device.

[0113] The host computing device 1010 includes one or more processors 1012 and memory 1014. The one or more processors 1012 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 1014 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 1014 may store data 1015 and instructions 1016 that are executed by the processor 1012 to cause the host computing device 1010 to perform operations.

[0114] In some implementations, the main computing device 1010 may store or include one or more machine learning models 1018. For example, the machine learning model 1018 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Reference Figure 5 , Figure 6 , Figure 7 and Fig. 9 Discuss example machine learning model 1018.

[0115] In some implementations, one or more machine learning models 1018 may be received from a server computing system (e.g., server computing system 904) via network 908, stored in user computing device memory 1014, and then used or otherwise implemented by one or more processors 1012. In some implementations, the host computing device 1010 may implement multiple parallel instances of a single machine learning model 1018 (e.g., to perform fusion actions on IMU sensor data, transform IMU sensor data into three-degree-of-freedom data, or determine a predicted position of a user).

[0116] More specifically, the machine learning model may obtain sensor data associated with one or more user devices (e.g., sensor data 1020, sensor data 1040, or sensor data 1060). The sensor data may be associated with the primary computing device 1010, the secondary computing device 1030, or the secondary computing device 1050. For example, the sensor may include an inertial measurement unit (IMU) sensor, an ambient light sensor, an accelerometer (e.g., a 3-axis accelerometer), an altimeter (detecting changes in altitude), an optical heart rate sensor (detecting heart beats per minute), an SpO2 monitor (e.g., for measuring blood oxygen levels), a bioimpedance sensor, a proximity sensor (e.g., to save battery and wake up the display when needed), a compass, a GPS, a gyroscope, a gesture sensor, an ultraviolet (UV) sensor, a magnetometer, a skin electrode activity sensor, a skin temperature sensor, an accelerometer, a gyroscope, a magnetometer, a global positioning system (GPS), a heart rate sensor (e.g., an electrode sensor or a photodiode), a pedometer (e.g., electrical, mechanical, or micro-electromechanical), a pressure sensor (e.g., a strain gauge), an optical sensor, an audio sensor, or other sensors. Sensor data may include gravitational acceleration, linear acceleration, angular acceleration, rotational acceleration, inertial measurements (e.g., generated using an accelerometer or gyroscope), motion orientation, position (e.g., detected via a satellite), strain, pressure, change in resistance (e.g., static or dynamic).

[0117] The secondary computing device 1030 can be any type of computing device, such as, for example, a mobile computing device (e.g., a smart phone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop computer), a game console or controller, an embedded computing device, or any other type of computing device.

[0118] The secondary computing device 1030 includes one or more processors 1032 and memory 1034. The one or more processors 1032 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 1034 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 1034 may store data 1035 and instructions 1036 that are executed by the processor 1032 to cause the primary computing device 1010 to perform operations.

[0119] The secondary computing device 1050 can be any type of computing device, such as, for example, a mobile computing device (e.g., a smart phone or tablet), a wearable computing device, a personal computing device (e.g., a laptop or desktop computer), a game console or controller, an embedded computing device, or any other type of computing device.

[0120] The secondary computing device 1050 includes one or more processors 1052 and memory 1054. The one or more processors 1052 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 1054 may include one or more computer-readable storage media that may be temporary or non-temporary, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 1054 may store data 1055 and instructions 1056 that are executed by the processor 1052 to cause the primary computing device 1010 to perform operations.

[0121] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0122] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily produce changes, modifications, and equivalents of such embodiments. Therefore, the present disclosure does not exclude such modifications, changes, or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

[0123] The steps depicted or described are merely illustrative and may be omitted, combined, or performed in an order different from that depicted or described; the numbering of the depicted steps is merely for ease of reference and does not imply that any particular order is necessary or preferred.

[0124] The functions or steps described herein may be embodied in computer-usable data or computer-executable instructions executed by one or more computers or other devices to perform one or more functions described herein. Generally, such data or instructions include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific data types when executed by one or more processors in a computer or other data processing device. Computer executable instructions may be stored in computer-readable media such as hard disks, optical disks, removable storage media, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. It should be understood that the functions of such instructions can be combined or distributed as needed. In addition, the functions may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Specific data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated to be within the scope of computer-executable instructions and computer-usable data described herein.

[0125] Although not required, it will be appreciated by those of ordinary skill in the art that the various aspects described herein may be embodied as methods, systems, devices, or one or more computer-readable media storing computer-executable instructions. Thus, the various aspects may take the form of all-hardware embodiments, all-software embodiments, all-firmware embodiments, or embodiments combining software, hardware, or firmware aspects in any combination.

[0126] As described herein, various methods and actions may operate across one or more computing devices or networks. The functions may be distributed in any manner, or may be located in a single computing device (eg, server, client computer, user device, etc.).

[0127] Various aspects of the present disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications or variations within the scope and spirit of the appended claims may occur to one of ordinary skill in the art upon review of this disclosure. For example, one of ordinary skill in the art may appreciate that the steps depicted or described may be performed in an order different from that described, or that one or more of the steps shown may be optional or combined. Any and all features in the following claims may be combined or rearranged in any possible manner.

[0128] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. After obtaining an understanding of the foregoing, those skilled in the art can easily produce changes, modifications or equivalents to such embodiments. Therefore, the present disclosure does not exclude such modifications, changes or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, changes or equivalents.

[0129] Terms are described herein using lists of example elements joined by conjunctions such as "and," "or," "but," etc. It should be understood that such conjunctions are provided for explanation purposes only. A list joined by a specific conjunction such as "or," for example, may refer to "at least one of" or "any combination of" the example elements listed therein, where "or" should be understood as "and / or" unless otherwise indicated. Furthermore, terms such as "based on" should be understood as "based, at least in part, on."

Claims

1. A computing system comprising: one or more processors; as well as One or more computer-readable media storing instructions executable to cause the one or more processors to perform operations comprising: obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are each at a target location; determining a user device-with-you status of each of the first computing device and the second computing device in response to obtaining the location data; obtaining inertial measurement unit (IMU) sensor data from each of the first computing device and the second computing device in response to determining a user device presence status of each of the first computing device and the second computing device; inputting the IMU sensor data from each of the first computing device and the second computing device into a machine learning model, wherein the machine learning model is configured to combine the IMU sensor data from each of the first computing device and the second computing device; obtaining output data indicating a predicted position from the machine learning model; comparing the output data indicative of the predicted position with data indicative of the position of the target sub-region; and In response to obtaining data indicating co-location of each of the first computing device and the second computing device within the target sub-region of the target location, sending data instructing a user interface of the first computing device to provide a content item for display.

2. The system of claim 1, wherein the user device on-body status of each of the first computing device and the second computing device indicates that each of the first computing device and the second computing device is located on a user moving within the target location.

3. A system as claimed in any preceding claim, wherein the IMU sensor data comprises accelerometer data and gyroscope data.

4. The system of claim 3, wherein the machine learning model is configured to transform the accelerometer data and the gyroscope data into data Cartesian coordinates indicating absolute position.

5. The system of any preceding claim, further comprising obtaining IMU sensor data from a third computing device.

6. The system of claim 5, wherein the first computing device comprises a smart phone, the second computing device comprises a smart watch, and the third computing device comprises earbuds.

7. A system as claimed in any preceding claim, wherein the calculation is performed on the first computing device.

8. The system of any preceding claim, wherein the IMU sensor data from each of the first computing device and the second computing device is combined using a fusion method.

9. A system as claimed in any preceding claim, wherein the machine learning model comprises a neural network.

10. The system of any preceding claim, wherein the machine learning model comprises a fully connected neural network.

11. A computer-implemented method comprising: obtaining location data associated with a first computing device and a second computing device, the location data indicating that the first computing device and the second computing device are each at a target location; determining a user device-with-you status of each of the first computing device and the second computing device in response to obtaining the location data; obtaining inertial measurement unit (IMU) sensor data from each of the first computing device and the second computing device in response to determining a user device presence status of each of the first computing device and the second computing device; inputting the IMU sensor data from each of the first computing device and the second computing device into a machine learning model, wherein the machine learning model is configured to combine the IMU sensor data from each of the first computing device and the second computing device; obtaining output data indicating a predicted position from the machine learning model; comparing the output data indicative of the predicted position with data indicative of the position of the target sub-region; as well as In response to obtaining data indicating co-location of each of the first computing device and the second computing device within the target sub-region of the target location, sending data instructing a user interface of the first computing device to provide a content item for display.

12. The computer-implemented method of claim 11, wherein the user device on-body status of each of the first computing device and the second computing device indicates that each of the first computing device and the second computing device is located on a user moving within the target location.

13. The computer-implemented method of any one of claims 11 to 12, wherein the IMU sensor data comprises accelerometer data and gyroscope data.

14. The computer-implemented method of claim 13, wherein the machine learning model is configured to transform the accelerometer data and the gyroscope data into data Cartesian coordinates indicating absolute position.

15. The computer-implemented method of any one of claims 11 to 14, further comprising obtaining IMU sensor data from a third computing device.

16. The computer-implemented method of claim 15, wherein the first computing device comprises a smart phone, the second computing device comprises a smart watch, and the third computing device comprises earbuds.

17. The computer-implemented method of any one of claims 11 to 16, wherein the computation is performed on the first computing device.

18. The computer-implemented method of any one of claims 11 to 17, wherein the IMU sensor data from each of the first computing device and the second computing device is combined using a fusion method.

19. A computer-implemented method as described in any one of claims 11 to 18, wherein the machine learning model comprises a neural network.

20. The computer-implemented method of any one of claims 11 to 18, wherein the first device is a primary computing device and the second computing device is a secondary computing device, and wherein the first device functions as a modem to facilitate transmission of data between the second device and a server computing system.

21. A computer readable medium embodied in a computer readable storage device and comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 20.

Images