Query results based on sensor data

Abstract

A system for generating query results based on sensor data may include a processor for detecting sensor data from a plurality of sensors and storing the sensor data as aggregated sensor data. The sensor data may be aggregated based on a time corresponding to the sensor data, wherein the sensor data is stored using a notification-based push operation in response to a change in a value of the sensor data exceeding a predetermined threshold. The processor may also convert the aggregated sensor data into an indexable data format (IDF) and provide low-latency query results to a query based on the aggregated sensor data of the IDF.

Description

Background Art

[0001] The computer device may be coupled to any suitable number of sensors. In some examples, the sensors may detect sensor data indicating various characteristics of the operating environment of the computing device. For example, the sensor data may indicate video sensor data, temperature sensor data, user sensor data, etc. In some embodiments, the application may detect the sensor data and use the sensor data to execute application instructions. Summary of the invention

[0002] A simplified overview is provided below in order to provide a basic understanding of some aspects described herein. This overview is not an exhaustive overview of the claimed subject matter. This overview is neither intended to identify key or important elements of the claimed subject matter nor to delineate the scope of the claimed subject matter. The sole purpose of this overview is to present some concepts of the claimed subject matter in a simplified form as a prelude to a more detailed description presented later.

[0003] Embodiments described herein include a system for generating query results based on sensor data, wherein the system may include a processor for detecting sensor data from a plurality of sensors. The processor may also store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in a value of the sensor data exceeding a predetermined threshold, a notification-based push operation is used to store the sensor data. Additionally, the processor may convert the aggregated sensor data into an indexable data format (IDF), and provide low-latency query results to the query based on the aggregated sensor data in the IDF.

[0004] In another embodiment, a method for generating a query result based on sensor data may include detecting sensor data from a plurality of sensors. The method may also include storing the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in a value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a notification-based push operation. Additionally, the method may include converting the aggregated sensor data into an indexable data format (IDF), and providing a low-latency query result to the query based on the aggregated sensor data in the IDF.

[0005] In another embodiment, one or more computer-readable storage media for generating query results based on sensor data may include multiple instructions, and the multiple instructions cause the processor to detect sensor data from multiple sensors in response to execution of the processor. The multiple instructions may also cause the processor to store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in the value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a notification-based push operation. In addition, the multiple instructions may also cause the processor to convert the aggregated sensor data into an indexable data format (IDF), and provide low-latency query results to the query based on the aggregated sensor data in the IDF.

[0006] The following description and accompanying drawings set forth in detail certain illustrative aspects of the claimed subject matter. However, these aspects are indicative of several of the various ways in which the principles of the invention may be employed, and the claimed subject matter is intended to include all such aspects and their equivalents. Other advantages and novel features of the claimed subject matter will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following detailed description may be better understood by reference to the appended drawings, which contain specific examples of several features of the disclosed subject matter.

[0008] Figure 1 is a block diagram of an example of a computing system that can generate query results based on sensor data;

[0009] Figure 2 is a process flow diagram of an example method for generating query results based on sensor data;

[0010] Figures 3A-3C depicts an example block diagram showing data flow in a system for generating query results based on sensor data;

[0011] Figure 4 is an example data schema for sensor data; and

[0012] Figure 5 is a block diagram of an example computer-readable storage medium that can generate query results based on sensor data. DETAILED DESCRIPTION

[0013] Users can associate with an increasing number of sensors that can be embedded in physical devices, vehicles, household appliances, and other devices. Connecting these devices and enabling them to exchange data can also be referred to as the Internet of Things. Thus, a user can associate with steps recorded by a pedometer, distance traveled detected by a global positioning satellite sensor, speed at which a user is traveling detected by an accelerometer, or heart rate or temperature of a user detected by a fitness wearable sensor, etc. In some embodiments, a digital assistant can detect a search query from a user and provide a response using a web-based search associated with the search query. However, a user's search query can also be satisfied based on sensor data collected by various sensors associated with the user.

[0014] The technology described herein includes a system for generating query results based on sensor data. The system can detect sensor data from multiple sensors and store the sensor data as aggregated sensor data. In some examples, in response to a change in the value of the sensor data exceeding a predetermined threshold, a notification-based push operation is used to store the sensor data. Additionally, the system can convert the aggregated sensor data into an indexable data format (IDF) and provide low-latency query results to the query based on the aggregated sensor data in the indexable data format.

[0015] Thus, the techniques described herein can enable a user to retrieve query results based on sensor data. In some examples, a system for generating query results based on sensor data can limit the amount of sensor data collected based on a predetermined threshold. Thus, the techniques described herein can implement an enhanced method for searching collected sensor data to provide query results based on sensor data for user queries. In some embodiments, a user query can be detected from a digital assistant using a first device, and sensor data can be collected from any number of sensors that can exchange data with a network search engine accessible to the digital assistant. In some examples, query results based on sensor data can include correlating sensor data from multiple different sensors.

[0016] The techniques described herein can reduce the amount of storage used to collect sensor data. The techniques described herein can also reduce the processing time of a processor by searching a smaller data set of sensor data. Additionally, the techniques described herein can reduce the power consumption of a computing device by reducing the time used to perform a search query based on the stored sensor data.

[0017] As a preliminary matter, some of the drawings describe concepts referred to as functions, modules, features, elements, etc. in the context of one or more structural components. The various components shown in the drawings may be implemented in any manner, for example, by software, hardware (e.g., discrete logic components, etc.), firmware, etc., or any combination of these implementations. In one embodiment, the various components may reflect the use of corresponding components in an actual implementation. In other embodiments, any single component shown in the drawings may be implemented by multiple actual components. The depiction of any two or more separate components in the drawings may reflect different functions performed by a single actual component. The following discussion Figure 1 Detailed information is provided regarding different systems that may be used to implement the functionality shown in the figures.

[0018] Other figures describe concepts in the form of flow charts. In this form, some operations are described as constituting different blocks executed in a certain order. Such implementations are exemplary and non-restrictive. Some blocks described herein can be combined together and executed with a single operation, some blocks can be decomposed into multiple component blocks, and some blocks can be executed in an order different from the order shown in this article (including the parallel mode of executing blocks). The frame shown in the flow chart can be implemented by any combination of software, hardware, firmware, etc. or these implementations. As used herein, hardware can include computer systems, discrete logic components (e.g., application specific integrated circuits (ASICs)), etc., and any combination of these components.

[0019] As far as terminology is concerned, the phrase "configured to" encompasses any manner in which any kind of structural component can be constructed to perform the identified operation. The structural component can be configured to perform the operation using software, hardware, firmware, etc., or any combination thereof. For example, the phrase "configured to" can refer to the logical circuit structure of a hardware element for implementing the associated function. The phrase "configured to" can also refer to the logical circuit structure of a hardware element designed by coding to implement the associated function of firmware or software. The term "module" refers to a structural element that can be implemented using any suitable hardware (e.g., processor, etc.), software (e.g., application, etc.), firmware, or any combination of hardware, software, and firmware.

[0020] The term "logic" encompasses any functionality for performing a task. For example, each operation shown in the flowchart corresponds to logic for performing that operation. The operations may be performed using software, hardware, firmware, etc., or any combination thereof.

[0021] As used herein, the terms "component," "system," "client," and the like are intended to refer to a computer-related entity that is hardware, software (e.g., in execution), and / or firmware, or a combination thereof. For example, a component can be a process, an object, an executable, a program, a function, a library, a subroutine, and / or a computer or a combination of software and hardware running on a processor. By way of illustration, both an application running on a server and a server can be components. One or more components can reside within a process, and a component can be localized on one computer and / or distributed between two or more computers.

[0022] In addition, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming techniques and / or engineering techniques for producing software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any tangible computer-readable device or media.

[0023] Computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, and magnetic strips, etc.), optical disks (e.g., compact disks (CDs) and digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., cards, sticks, and key drives, etc.). In contrast, computer-readable media in general (i.e., non-storage media) may additionally include communication media, such as transmission media for wireless signals, etc.

[0024] Figure 1 1 is a block diagram of an example of a computing system that can generate query results based on sensor data. Example system 100 includes computing device 102. Computing device 102 includes processing unit 104, system memory 106, and system bus 108. In some examples, computing device 102 can be a game console, a personal computer (PC), an accessory console, a game controller, and other computing devices. In some examples, computing device 102 can be a node in a cloud network.

[0025] System bus 108 couples system components including, but not limited to, system memory 106 to processing unit 104. Processing unit 104 may be any of a variety of available processors. Dual microprocessors and other multi-processor architectures may also be used as processing unit 104.

[0026] The system bus 108 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and a local bus using any type of available bus architecture known to those of ordinary skill in the art. The system memory 106 includes computer-readable storage media including volatile memory 110 and non-volatile memory 112.

[0027] In some embodiments, a unified extensible firmware interface (UEFI) manager or basic input / output system (BIOS), containing basic routines to transfer information between elements within computer 102, such as during startup, is stored in non-volatile memory 112. By way of illustration and not limitation, non-volatile memory 112 may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.

[0028] Volatile memory 110 includes random access memory (RAM) used as external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (E SDRAM), SynchLink TM DRAM (SLDRAM), Direct RAM (RDRAM), Direct Dynamic RAM (DRDRAM) and Dynamic RAM (RDRAM).

[0029] The computer 102 also includes other computer-readable media, such as removable / non-removable computer storage media, volatile / non-volatile computer storage media. Figure 1 Shown is, for example, disk storage 114. Disk storage 114 includes, but is not limited to, devices such as a disk drive, floppy drive, tape drive, Jaz drive, Zip drive, LS-210 drive, flash memory card, or memory stick.

[0030] Additionally, disk storage device 114 may include storage media alone or in combination with other storage media, including but not limited to optical disk drives, such as compact disk ROM devices (CD-ROM), CD recordable drives (CD-R drives), CD rewritable drives (CD-RW drives), or digital versatile disk ROM drives (DVD-ROM). To facilitate connection of disk storage device 114 to system bus 108, a removable or non-removable interface, such as interface 116, is typically used.

[0031] It should be recognized that Figure 1Software is described that acts as an intermediary between a user in a suitable operating environment 100 and the described basic computer resources. Such software includes an operating system 118. The operating system 118, which may be stored on disk storage 114, is used to control and allocate the resources of the computer 102.

[0032] System applications 120 utilize operating system 118's management of resources through program modules 122 and program data 124 stored in system memory 106 or disk storage 114. It should be appreciated that the disclosed subject matter may be implemented with a variety of operating systems or combinations of operating systems.

[0033] The user enters commands or information into the computer 102 through the input device 126. The input device 126 includes, but is not limited to, a pointing device (e.g., a mouse, a trackball, a stylus, etc.), a keyboard, a microphone, a joystick, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a webcam, any suitable (physical or virtual) dial accessory, etc. In some examples, the input device may include a natural user interface (NUI) device. NUI refers to any interface technology that enables a user to interact with a device in a "natural" way, without artificial constraints imposed by input devices such as a mouse, a keyboard, a remote control, etc. In some examples, the NUI device includes devices that rely on voice recognition, touch and stylus recognition, gesture recognition on the screen and gesture recognition adjacent to the screen, both, air gestures, head and eye tracking, sound and voice, vision, touch, gestures, and machine intelligence. For example, the NUI device may include a touch-sensitive display, sound and voice recognition, intent and target understanding, and motion gesture detection using a depth camera (e.g., a stereo camera system, an infrared camera system, an RGB camera system, and a combination of these camera systems). NUI devices may also include motion gesture detection using accelerometers or gyroscopes, facial recognition, three-dimensional (3D) displays, head, eye and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide a more natural interface. NUI devices may also include technology for sensing brain activity using electric field sensing electrodes. For example, NUI devices may use electroencephalography (EEG) and related methods to detect the electrical activity of the brain. Input device 126 is connected to processing unit 104 via interface port 128 through system bus 108. Interface port 128 includes, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB).

[0034] Output devices 130 use some of the same types of ports as input devices 126. Thus, for example, a USB port may be used to provide input to computer 102 and output information from computer 102 to output devices 130.

[0035] Output adapters 132 are provided to illustrate the presence of some output devices 130, such as monitors, speakers, and printers, among other output devices 130, that are accessible via the adapters. By way of illustration and not limitation, output adapters 132 include video and sound cards that provide a connection unit between output devices 130 and system bus 108. It may be noted that other devices and systems of devices (e.g., remote computing device 134) provide both input and output capabilities.

[0036] The computer 102 may be a server that hosts various software applications in a networked environment using logical connections to one or more remote computers (e.g., remote computing devices 134). The remote computing devices 134 may be client systems configured with a web browser, PC applications, mobile phone applications, etc. The remote computing devices 134 may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, mobile phones, peer devices or other public network nodes, etc., and typically include many or all of the elements described with respect to the computer 102.

[0037] The remote computing device 134 may be logically connected to the computer 102 through a network interface 136, and then connected via a communication connection 138, which may be wireless. The network interface 136 encompasses wireless communication networks, such as local area networks (LANs) and wide area networks (WANs). LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks (e.g., Integrated Services Digital Network (ISDN)) and variations thereon, packet switching networks, and digital subscriber lines (DSL).

[0038] The communication connection 138 refers to the hardware / software employed to connect the network interface 136 to the bus 108. Although the communication connection 138 is shown internal to the computer 102 for clarity of illustration, the communication connection 138 may also be external to the computer 102. For exemplary purposes, the software used for the connection to the network interface 136 may include internal technology and external technology, such as a mobile phone switch, a modem including a conventional telephone grade modem, a cable modem and a DSL modem, an ISDN adapter, and an Ethernet card.

[0039] Computer 102 may further include radio 140. For example, radio 140 may be a wireless local area network radio that may operate one or more wireless frequency bands. For example, radio 140 may operate on an industrial, scientific, and medical (ISM) radio band at 2.4 GHz or 5 GHz. In some examples, radio 140 may operate at any radio frequency on any suitable radio frequency band.

[0040] The computer 102 includes one or more modules 122, such as a sensor data detector 142, a sensor data storage manager 144, a sensor data converter 146, and a query result generator 148. In some embodiments, the sensor data detector 142 can detect sensor data from multiple sensors. The sensors can include weather-based sensors, user-based sensors for monitoring user locations, fitness or health-based sensors for measuring a user's heart rate or temperature, and the like. In some embodiments, the sensor data storage manager 144 can store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on the time corresponding to the sensor data. In some embodiments, in response to a change in the value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a push operation based on a notification. For example, unless the value of the sensor data deviates from the value of the previously detected sensor data by an amount exceeding a predetermined threshold, the detected sensor data can be discarded or deleted. In some embodiments, the sensor data converter 146 can convert the aggregated sensor data into an indexable data format (IDF). In some embodiments, the query result generator 148 can provide low-latency query results to the query based on the aggregated sensor data in the indexable data format.

[0041] It should be understood that Figure 1 The block diagram is not intended to indicate that computing system 102 will include Figure 1 Rather, computing system 102 may include fewer components or Figure 1 104. In addition, any of the functions of the sensor data detector 142, the sensor data storage manager 144, the sensor data converter 146, and the query result generator 148 may be partially or completely implemented in hardware and / or implemented in the processing unit (also referred to herein as the processor) 104. For example, the functions may be implemented using an application specific integrated circuit, implemented in logic implemented in the processing unit 104, or implemented in any other device.

[0042] Figure 2 is a process flow diagram of an example method for generating query results based on sensor data. Figure 1 The method 200 is implemented using a computing system 102 ).

[0043] At block 202, the sensor data detector 142 may detect sensor data from a plurality of sensors. In some embodiments, the plurality of sensors may include a pedometer, a thermometer, a magnetometer, an accelerometer, a barometer, a video sensor, a flow sensor, a pressure sensor, a camera, a humidity sensor, and the like. In some examples, the sensor data may include weather information, image information, and user health information such as heart rate or number of steps walked, and the like. In some embodiments, the sensor data detector 142 may detect that the plurality of sensors are associated with the user requesting the query. For example, the sensor data detector 142 may detect a user identifier associated with the detected sensor data.

[0044] At box 204, the sensor data storage manager 144 may store the sensor data as aggregated sensor data. In some examples, the sensor data may be aggregated based on the time corresponding to the sensor data. For example, the sensor data may be aggregated based on the time period in which the sensor data is detected by the sensor. In some examples, the sensor data may be stored using a notification-based push operation in response to a change in the value of the sensor data exceeding a predetermined threshold. For example, the sensor data storage manager 144 may discard changes in sensor data below a predetermined threshold to reduce storage utilized by the sensor. In some examples, the predetermined threshold is based on the type of sensor that detects the sensor data. For example, the sensor data storage manager 144 may detect a five-degree threshold for a thermometer, which indicates that a temperature change of five degrees or more is to be stored. Otherwise, a smaller temperature change may be discarded or deleted. In some examples, the notification-based push operation may send any changes in the sensor data to be stored to an intermediate storage device. The intermediate storage device may implement offline processing of the sensor data, as discussed in more detail below with respect to FIG. 3.

[0045] At block 206, the sensor data converter 146 may convert the aggregated sensor data into an indexable data format (IDF). In some examples, the indexable data format may include any suitable number of element values, such as a sensor globally unique identifier, a sensor name, a user identifier, a device identifier, a compressed data indicator, a time when a notification for receiving sensor data was generated, a sensor type, a time when the sensor data was recorded, and a data type, etc. In some embodiments, the sensor data converter 146 may store the aggregated sensor data in the indexable data format in an offline storage device.

[0046] At box 208, the query result generator 148 can provide low-latency query results to the query based on the aggregated sensor data in an indexable data format. The query can be detected from a user interface or a digital personal assistant, etc. In some examples, low-latency query results can be generated based on a change in a value exceeding a predetermined threshold using a limited amount of stored sensor data. In some embodiments, the low-latency query results include sensor data related to any suitable characteristics such as location information, user information, device information, etc. In some embodiments, the low-latency query results may include a sensor among multiple sensors that consumed an amount of power exceeding a power threshold during a period of time.

[0047] In some examples, the sensor data is detected by a pedometer, and the query result generator 148 can correlate the sensor data with the location information to indicate the amount of activity performed at multiple locations. For example, the query result generator 148 can group the number of steps taken by the user based on the location of the user when walking.

[0048] In some embodiments, the query result generator 148 can generate a ranking of each sensor in a plurality of sensors based on the query and the sensor data. In some examples, the query result generator 148 can generate a low-latency query result, wherein the low-latency query result includes sensor data organized based on the ranking of the plurality of sensors. For example, the ranking of the sensor may depend on a predetermined reliability value or a user preference assigned to each sensor. In some examples, the query result generator 148 can generate a low-latency query result based on a sensor with a high predetermined reliability value or a sensor selected as a user's preference. In some embodiments, the query result generator 148 can generate a personalized index of sensor data for the user, and the low-latency query result is based on the personalized index. In some examples, the query includes a natural language query detected from a digital personal assistant.

[0049] In one embodiment, Figure 2 The process flow diagram of method 200 is intended to indicate that the blocks of method 200 are to be performed in a particular order. Alternatively, in other embodiments, the blocks of method 200 may be performed in any suitable order and may include any suitable number of blocks of method 200. Furthermore, any number of additional blocks may be included within method 200 depending on the particular application.

[0050] Figures 3A-3C An example block diagram showing data flow in a system for generating query results based on sensor data is depicted. Figure 3ATo begin with, in some embodiments, the system 300 may include any number of sensors 302, such as cameras, thermometers, accelerometers, mobile device sensors, pedometers, automotive-based sensors, robot-based sensors, and the like. In some examples, the sensor 302 may send sensor data to sensor gateways 304, 306, and 308. The sensor gateways 304, 306, and 308 may provide a unified interface between the sensor 302 and the coordinator 310. For example, the sensor gateways 304, 306, and 308 may normalize sensor data detected from sensors that are built using many different platforms with widely varying processing power, energy, and bandwidth capabilities. In some examples, the sensor 302 may have different access interfaces (e.g., radios) for communicating with low-power wireless sensor nodes or serial buses to communicate at high speed and synchronize data transmission with higher-power and higher-bandwidth sensors. In some examples, the sensor 302 may not always be connected to the sensor gateways 304, 306, and 308. Sensor gateways 304 , 306 , and 308 may implement sensor-specific technologies to communicate with each sensor 302 .

[0051] In some embodiments, the coordinator 310 can access the sensor gateways 304, 306, and 308 to obtain sensor data streams to submit data collection requirements, or the coordinator 310 can access sensor characteristics through a standardized web service application programming interface (API). In some examples, each sensor 302 can maintain a separate sensor gateway 306. In some embodiments, the sensor gateways 304, 306, and 308 can implement a sharing strategy defined by a contributor. For example, the sensor gateways 304, 306, and 308 can maintain raw data in a local database for a local application executed by the sensor 302, and the sensor 302 can maintain private data while sending non-private data to the coordinator 310. The data center sensor gateway 304 can be used by sensors 302 that do not maintain their own sensor gateways. In some examples, a single sensor can publish its data to the data center sensor gateway 304 through a web service API.

[0052] In some embodiments, the coordinator 310 can be an access point to the system 300 for applications and sensors 302. The coordinator 310 can include a user manager 312, a sensor manager 314, and an application manager 316. The user manager 312 can implement a user authentication mechanism. In some embodiments, the sensor manager 314 can provide an index of available sensors 302 and the characteristics of the sensors 302. For example, the sensor manager 314 can convert a user-friendly sensor description (e.g., a location boundary, a logical name, or a sensor type) into a physical sensor identifier. The sensor manager 314 can also include an API for the sensor gateways 304, 306, and 308 to manipulate the sensors 302 and the types of sensors 302. For example, the sensor manager 314 can define new sensor types, register new sensors of defined types, modify the characteristics of registered sensors, and delete registered sensors.

[0053] In some embodiments, the application manager 316 can be an access point to shared data for add-ons in the system 300. In some examples, the application manager 316 can manage the sensor gateways 304, 306, and 308. The application manager 316 can also accept sensing queries from add-ons and satisfy the sensing queries based on the available sensors 302. In some embodiments, in order to minimize the load on the sensors 302 or the corresponding sensor gateways 304, 306, and 308, the application manager 316 can attempt to combine requests for common data. The application manager 316 can also cache recently accessed sensor data so that the local cache can serve future queries that do not have strict real-time requirements.

[0054] In some embodiments, the coordinator 310 may send data to data transformers 318, 320, 322, 323, and 324. The data transformers 318, 320, 322, 323, and 324 may convert data semantics through processing. For example, the data transformers 318-324 may extract a headcount from a video stream, perform unit conversion, perform data fusion, and implement data visualization services. In some examples, the transformers 318-324 may perform different tasks. For example, the iconizer data transformer 318 may convert raw sensor readings into icons that represent sensor types with icon shapes and sensor values ​​with icon colors. In some examples, a graphics application may use the output of the iconizer data transformer 318 instead of raw sensor values. In another example, the graph generator data transformer 320 may obtain raw sensor readings and generate a 2D spatial graph. In some embodiments, the notification agent 324 may determine when to send sensor data to the sensor collection application 326.

[0055] In some examples, applications utilize sensor data to execute instructions. Applications 326, 327, and 328 can be interactive applications in which users specify data requirements, such as a user query for the average heart rate of hikers on a particular trail in the last season, etc. Applications 326, 327, and 328 can also include automated applications in back-end enterprise systems that access sensor streams for business processing, such as an inventory management application accessing shopper counts from parking counters, accessing customer behavior from video streams, and correlating shopper counts and customer behavior with sales records. In one example, a sensor map application 328 can visualize sensor data from an iconizer transformer 318 and a map generator transformer 330 on a map representation of a location.

[0056] In some embodiments, the sensor collection application 326 can collect sensor data from any number of sensors 302 and send the sensor data to the intermediate repository 332. In some examples, the sensor collection application 326 can implement a strategy for collecting sensor data that deviates from a previous value by more than a predetermined threshold. For example, if the value of sensor data from a thermometer sensor is at least a certain number of degrees higher or lower than a previously detected value, the sensor collection application 326 can store the sensor data. If the sensor collection application 326 detects sensor data that is below a predetermined threshold, the sensor collection application 326 can discard or delete the sensor data. Thus, the sensor collection application 326 can limit the sensor data collected from each sensor 302 and sent to be stored in the intermediate repository 332. Figure 3B The size of the sensor data in the intermediate repository 332.

[0057] In some embodiments, the predetermined threshold for each sensor 302 may be different. For example, the predetermined threshold may indicate that the number of steps from the pedometer that exceeds the previously detected value is stored in the intermediate repository 332. In another example, if the distance of the new position of the position data from the global positioning system sensor from the previously detected value is greater than a predetermined distance, the predetermined threshold may indicate that the position data will be stored. In yet another example, if the increase or decrease in the number of users detected in the video frame or image compared to the previously detected value exceeds a threshold, the predetermined threshold may indicate that the number of users will be stored. Therefore, the intermediate repository 332 can store sensor data that exceeds the predetermined threshold detected from any suitable number of sensors. Smaller sensor data sets stored in the intermediate repository 332 can achieve faster analysis and limit storage requirements for the system 300. In some examples, smaller sensor data sets can enable the intermediate repository 332 to store data from a larger number of sensors 302.

[0058] In some examples, the processing job 334 can retrieve the sensor data stored in the intermediate repository 332 as part of the offline storage processing 336. The processing job 334 can send the retrieved sensor data to an aggregator module 338 that can aggregate the sensor data based on time information. For example, the sensor data from the sensor 302 stored in the intermediate repository 332 can be aggregated based on a common time frame during which the sensor data was collected. In some embodiments, the aggregator module 338 can aggregate the sensor data based on any suitable fixed or variable time period. For example, the sensor data from the sensor 302 can be aggregated over a larger time period during specific hours of a day or specific days of a week. In some examples, the aggregator module 338 can aggregate the sensor data in smaller time periods during daytime hours when a larger amount of sensor data is collected, and aggregate the sensor data in larger time periods during nighttime hours when a smaller amount of sensor data is collected.

[0059] In some embodiments, aggregator module 338 may send aggregated sensor data to post-processor 340. In some examples, post-processor 340 may transform sensor data aggregated based on time period into IDF 342. IDF data may enable a search engine to perform queries based on aggregated search data within a shorter time period.

[0060] In some embodiments, the IDF data 342 may be sent to an indexing service 344, which includes a feed index 346. The feed index 346 may include a lookup table in which data is stored in a <key, value> format. In some examples, the feed index 346 may create multiple lookup <key, value> pairs based on the sensor data. In some embodiments, the indexing service 344 may retrieve the generated IDF data file 342 and process the IDF data file 342 into a content block that is incorporated into the feed index 346. In some examples, an index as a service (IaaS) environment may retrieve or stream the content blocks generated by the feed index 346 when the content blocks become available. In some examples, the indexing service 344 periodically initiates a merge process. During an index merge on the feed index 346, the index block files are combined into a new complete version of the index.

[0061] In some embodiments, the user may provide the Figure 3CIn some examples, the front door module 350 can detect the user query and send the query to the backend server. In some examples, the query annotation service 352 or the language understanding engine can detect the user query and process the user query to extract information. For example, the query annotation service 352 can perform query parsing or rewriting to extract tokens by parsing the user query. In one example, the user query "My deep sleep percentage last night" can be parsed as [my], [deep sleep], [last night], [percentage]. In some embodiments, the query annotation service 352 can be trained using labeled search engine queries. For example, the query annotation service 352 can detect intent and entity identification based on information such as [me]=>user, [deep sleep]=>information from band, [last night]=>date / time, and [percentage]=>unit.

[0062] In some embodiments, the top level answer aggregator (ATLA) 354 may include a federation and cache server that routes queries to answer services such as web answers 356, entity answers 358, and sensor answers 360. In some examples, the web answers 356, entity answers 358, and sensor answers 360 may independently generate answers or responses to the queries. In some embodiments, the query indexing service 344 includes detecting a structured, formulated query from a query annotation service 352 or a query understanding engine, and searching the index for matching sensor data for sensor answers 360. In some examples, the sensor answer 360 module may send the query to the SA sorter 362, which may create a ranked list of sensors that is arranged in descending order according to the probability that the sensor data from the sensor may include an answer to the query. The SA sorter 362 may also create a matching score between the sensor output and the natural language query to further refine the search results. Additionally, the SA sorter 362 may create a confidence score for the final sensor search results with respect to other user experiences.

[0063] In some embodiments, post-processing module 364 can rank various answers from network answers 356, entity answers 358, and sensor answers 360 based on relevance to the query. Whole page organization (WPO) 366 can aggregate the answers and return the results to device 370 via user experience 368. In some embodiments, the system can use user feedback to improve query results. For example, the system can implement self-learning techniques by refining query results based on user clicks and satisfaction data.

[0064] In some embodiments, the personal digital assistant can detect the user's registered devices. In some examples, the system can perform automatic correlation: deep sleep information is obtained from the user's device (e.g., smart watch, etc.). In some examples, the information is transformed into a structured format, which is then used by the search engine to query the index service 344.

[0065] It should be understood that the block diagram of Figure 3 is not intended to indicate that system 300 is to include all of the components shown in Figure 3. Rather, system 300 may include fewer components or additional components not shown in Figure 3 (e.g., additional applications, additional modules, additional memory devices, additional network interfaces, etc.).

[0066] Figure 4 is an example data schema for sensor data. In some embodiments, data schema 400 may be generated by post-processor 340 of FIG. 3 and may include sensor object 402, band sensor data object 404, heater sensor data object 406, sensor data parent 408, and sensor data collection object 410. In some examples, sensor object 402 may include a sensor identifier or SensorGuid, a sensor name, a time corresponding to a detected event, a user identifier, a device identifier, and a sensor type. In some embodiments, band sensor data object 404 and heater sensor data object 406 may include a heartbeat and a temperature based on sensor object 402, respectively. In some examples, sensor data parent 408 may include band sensor data object 404 and heater sensor data object 406. In some examples, sensor data collection object 410 may include sensor data parent 408 and the time at which a notification with sensor data is sent. It should be understood that Figure 4 The data model may include fewer objects or Figure 4 Additional objects not shown.

[0067] Figure 5 5 is a block diagram of an example computer-readable storage medium that can generate query results based on sensor data. The tangible computer-readable storage medium 500 can be accessed by the processor 502 through the computer bus 504. In addition, the tangible computer-readable storage medium 500 can include code for directing the processor 502 to perform the steps of the present method.

[0068] The various software components discussed herein may be stored on a tangible computer-readable storage medium 500, such as Figure 5. For example, the tangible computer-readable storage medium 500 may include a sensor data detector 506 that can detect sensor data from a plurality of sensors. In some embodiments, the sensor data storage manager 508 may store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in a value of the sensor data exceeding a predetermined threshold, the sensor data is stored in a notification-based push operation. In some embodiments, the sensor data converter 510 may convert the aggregated sensor data into an indexable data format (IDF). In some embodiments, the query result generator 512 may provide low-latency query results to a query based on the aggregated sensor data in an indexable data format.

[0069] It should be understood that any number of tangible computer-readable storage media 500 may be included, depending on the specific application. Figure 5 Additional software components not shown.

[0070] Example 1

[0071] In one embodiment, a system for generating query results based on sensor data may include a processor for detecting sensor data from multiple sensors. The processor may also store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in a value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a notification-based push operation. Additionally, the processor may convert the aggregated sensor data into an indexable data format (IDF) and provide low-latency query results to the query based on the aggregated sensor data in the IDF.

[0072] Alternatively or additionally, the low-latency query result includes sensor data related to the location information. Alternatively or additionally, the predetermined threshold is based on the type of sensor that detects the sensor data. Alternatively or additionally, the type of sensor includes a thermometer, a video sensor, a flow sensor, a pressure sensor, or a humidity sensor. Alternatively or additionally, the low-latency query result includes a sensor among multiple sensors that consumes an amount of power exceeding a power threshold during a period of time. Alternatively or additionally, the sensor data is detected by a pedometer, and wherein the processor is used to correlate the sensor data with the location information, thereby indicating the amount of activity performed at multiple locations. Alternatively or additionally, the processor is used to store the aggregated sensor data in an offline storage device in an IDF. Alternatively or additionally, the processor is used to generate a ranking of each sensor in multiple sensors based on the query and the sensor data. Alternatively or additionally, the low-latency query result includes sensor data organized based on the ranking of multiple sensors. Alternatively or additionally, the processor is used to detect that multiple sensors are associated with the user requesting the query. Alternatively or additionally, the processor is to generate a personalized index of sensor data for the user, and wherein the low-latency query results are based on the personalized index.Alternatively or additionally, the query comprises a natural language query detected from a digital personal assistant.

[0073] Example 2

[0074] In another embodiment, a method for generating a query result based on sensor data may include detecting sensor data from a plurality of sensors. The method may also include storing the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in a value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a notification-based push operation. Additionally, the method may include converting the aggregated sensor data into an indexable data format (IDF), and providing a low-latency query result to the query based on the aggregated sensor data in the IDF.

[0075] Alternatively or additionally, the low-latency query result includes sensor data related to the location information. Alternatively or additionally, the predetermined threshold is based on the type of sensor that detects the sensor data. Alternatively or additionally, the type of sensor includes a thermometer, a video sensor, a flow sensor, a pressure sensor, or a humidity sensor. Alternatively or additionally, the low-latency query result includes a sensor among multiple sensors that consumes an amount of power exceeding a power threshold during a period of time. Alternatively or additionally, the sensor data is detected by a pedometer, and wherein the method includes correlating the sensor data with the location information, thereby indicating the amount of activity performed at multiple locations. Alternatively or additionally, the method includes storing the aggregated sensor data in an offline storage device in an IDF. Alternatively or additionally, the method includes generating a ranking of each of the multiple sensors based on the query and the sensor data. Alternatively or additionally, the low-latency query result includes sensor data organized based on the ranking of multiple sensors. Alternatively or additionally, the method includes detecting that multiple sensors are associated with a user requesting the query. Alternatively or additionally, the method comprises generating a personalized index of the sensor data for the user, and wherein the low-latency query results are based on the personalized index.Alternatively or additionally, the query comprises a natural language query detected from a digital personal assistant.

[0076] Example 3

[0077] In another embodiment, one or more computer-readable storage media for generating query results based on sensor data may include multiple instructions, and the multiple instructions cause the processor to detect sensor data from multiple sensors in response to execution of the processor. The multiple instructions may also cause the processor to store the sensor data as aggregated sensor data, wherein the sensor data is aggregated based on a time corresponding to the sensor data, and wherein, in response to a change in the value of the sensor data exceeding a predetermined threshold, the sensor data is stored using a notification-based push operation. In addition, the multiple instructions may also cause the processor to convert the aggregated sensor data into an indexable data format (IDF), and provide low-latency query results to the query based on the aggregated sensor data in the IDF.

[0078] Alternatively or additionally, the low-latency query result includes sensor data related to the location information. Alternatively or additionally, the predetermined threshold is based on the type of sensor that detects the sensor data. Alternatively or additionally, the type of sensor includes a thermometer, a video sensor, a flow sensor, a pressure sensor, or a humidity sensor. Alternatively or additionally, the low-latency query result includes a sensor among multiple sensors that consumes an amount of power exceeding a power threshold during a period of time. Alternatively or additionally, the sensor data is detected by a pedometer, and wherein the processor is used to correlate the sensor data with the location information, thereby indicating the amount of activity performed at multiple locations. Alternatively or additionally, multiple instructions may also cause the processor to store the aggregated sensor data in an offline storage device in an IDF. Alternatively or additionally, multiple instructions may also cause the processor to generate a ranking of each sensor in multiple sensors based on the query and the sensor data. Alternatively or additionally, the low-latency query result includes sensor data organized based on the ranking of multiple sensors. Alternatively or additionally, multiple instructions may also cause the processor to detect that multiple sensors are associated with the user requesting the query. Alternatively or additionally, the plurality of instructions may further cause the processor to generate a personalized index of the sensor data for the user, and wherein the low-latency query results are based on the personalized index.Alternatively or additionally, the query comprises a natural language query detected from a digital personal assistant.

[0079] In particular and with respect to the various functions performed by the components, devices, circuits, systems, etc. described above, unless otherwise indicated, the terms used to describe such components (including references to "units") are intended to correspond to any component (e.g., functional equivalent) that performs the specified functions of the described components, even if not structurally equivalent to the disclosed structures, that performs the functions in the exemplary aspects of the claimed subject matter shown herein. In this regard, it will also be recognized that the present invention includes systems and computer-readable storage media having computer-executable instructions for performing the actions and events of the various methods of the claimed subject matter.

[0080] There are many ways (e.g., appropriate APIs, toolkits, driver code, operating systems, controls, stand-alone or downloadable software objects, etc.) to implement the claimed subject matter that enables applications and services to use the techniques described herein. The claimed subject matter encompasses uses from the perspective of an API (or other software object) and software or hardware objects that operate according to the techniques set forth herein. Thus, the various implementations of the claimed subject matter described herein can have aspects that are all in hardware, partly in hardware and partly in software, as well as in software.

[0081] The foregoing systems have been described with respect to the interoperation between several components. It will be appreciated that such systems and components may include these components or specified subcomponents, some of the specified components or subcomponents, and additional components, according to various permutations and combinations of the foregoing. Subcomponents may also be implemented as components that are communicatively coupled to other components, rather than being included within a parent component (hierarchy).

[0082] Additionally, it may be noted that one or more components may be combined into a single component providing aggregate functionality, or one or more components may be separated into several separate subcomponents, and any one or more intermediate layers (e.g., management layers) may be provided to communicatively couple to such subcomponents to provide integrated functionality. Any component described herein may also interact with one or more other components not specifically described herein but generally known to those skilled in the art.

[0083] Additionally, while a particular feature of the claimed subject matter may have been disclosed with respect to one of several implementations, such feature may be combined with one or more other features of other implementations, which may be desirable and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "including," "having," "containing," variations thereof, and other similar words are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition word and do not exclude any additional or other elements.

Claims

1. A system for generating query results based on sensor data, comprising: A processor for: detecting sensor data from multiple sensors; In response to a change in the value of the sensor data relative to previously detected sensor data not exceeding a predetermined threshold, deleting the sensor data; storing the sensor data in an intermediate repository utilizing a notification-based push operation in response to a change in the value of the sensor data relative to previously detected sensor data exceeding the predetermined threshold, wherein the stored sensor data in the intermediate repository is non-private data, and wherein private data is maintained in a local database; aggregating the stored sensor data in the intermediate repository based on a time when the sensor data was deleted; Converting the aggregated sensor data into an indexable data format (IDF); and Low latency query results are provided to queries based on the aggregated sensor data at the IDF.

2. The system according to claim 1, wherein: The low-latency query results include sensor data related to location information.

3. The system according to claim 1, wherein: The predetermined threshold is based on a type of sensor detecting the sensor data.

4. The system according to claim 3, wherein: The types of sensors include thermometers, video sensors, flow sensors, pressure sensors or humidity sensors.

5. A system according to claim 1, 2 or 3, wherein: The low-latency query result includes a sensor of the plurality of sensors that consumed an amount of power exceeding a power threshold during a period of time.

6. A system according to claim 1, 2 or 3, wherein: The sensor data is detected by a pedometer, and wherein the processor is configured to correlate the sensor data with location information to indicate an amount of activity performed at a plurality of locations.

7. A system according to claim 1, 2 or 3, wherein: The processor is configured to store the aggregated sensor data in the IDF in an offline storage device.

8. The system according to claim 1, wherein: The processor is configured to generate a ranking of each sensor of the plurality of sensors based on the query and the sensor data.

9. The system according to claim 8, wherein: The low-latency query result includes sensor data organized based on the ranking of the plurality of sensors.

10. The system according to claim 1, wherein: The processor is configured to detect that the plurality of sensors are associated with a user requesting the query.

11. The system according to claim 10, wherein: The processor is configured to generate a personalized index of the sensor data for the user, and wherein the low-latency query result is based on the personalized index.

12. The system according to claim 11, wherein: The query comprises a natural language query detected from a digital personal assistant.

13. A method for generating a query result based on sensor data, comprising: detecting sensor data from multiple sensors; In response to a change in the value of the sensor data relative to previously detected sensor data not exceeding a predetermined threshold, deleting the sensor data; storing the sensor data in an intermediate repository utilizing a notification-based push operation in response to a change in the value of the sensor data relative to previously detected sensor data exceeding the predetermined threshold, wherein the stored sensor data in the intermediate repository is non-private data, and wherein private data is maintained in a local database; aggregating the stored sensor data in the intermediate repository based on a time when the sensor data was deleted; Converting the aggregated sensor data into an indexable data format (IDF); and Low latency query results are provided to queries based on the aggregated sensor data at the IDF.

14. The method according to claim 13, wherein: The low-latency query result includes a sensor of the plurality of sensors that consumed an amount of power exceeding a power threshold during a period of time.

15. One or more computer-readable storage media for generating query results based on sensor data, comprising a plurality of instructions, wherein the plurality of instructions, in response to execution by a processor, causes the processor to perform the following operations: detecting sensor data from multiple sensors; In response to a change in the value of the sensor data relative to previously detected sensor data not exceeding a predetermined threshold, deleting the sensor data; In response to a change in the value of the sensor data relative to previously detected sensor data exceeding the predetermined threshold, storing the sensor data in an intermediate repository using a notification-based push operation, wherein the stored sensor data in the intermediate repository is non-private data, and wherein private data is maintained in a local database; aggregating the stored sensor data in the intermediate repository based on a time when the sensor data was deleted; Converting the aggregated sensor data into an indexable data format (IDF); and Low latency query results are provided to queries based on the aggregated sensor data at the IDF.

Images