Object and location tracking using graphs in multiple graphs

Abstract

A wearable device is configured with various sensing devices that cyclically monitor and collect data about the physical environment around the user to help locate and track real-world objects. The various heterogeneous sensing devices digitize objects and the physical world. Each sensing device is configured with a threshold data change, wherein when the data collected by one or more sensing devices exceeds the threshold, a query is performed on each sensor graph or sensing device. The queried sensor graph data is stored in nodes in a spatial graph, wherein the nodes are connected to each other using edges to create a spatial relationship between the object and the space. The object can be uploaded to an object graph associated with the spatial graph, wherein the object is digitized using each of the available sensors. This digital information can then be used, for example, to locate the object.

Description

Background Art

[0001] Indoor positioning systems (IPS) are used to track the location of objects within a physical environment such as an office building, shopping mall, or warehouse. For example, an IPS may include the use of radio frequency identification devices (RFID) to track objects and triangulate the location of a user based on the signal strength of a computing device to a network device such as a Wi-Fi access point or Bluetooth beacon. Various sensors and other technologies may also be used for object tracking, indoor tracking, such as pedestrian dead reckoning (PDR), cameras, and global positioning systems (GPS). The respective IPS have characteristics that are advantageous for certain scenarios, and also have characteristics that are disadvantageous for other scenarios. Summary of the invention

[0002] The spatial graph instantiated on the wearable device, remote server, or edge computing unit periodically queries all available sensors associated with the sensing device to establish a node for each query, synthesizes the data queried from each sensor, and uses the data to track objects within the physical environment. Multiple sensing devices associated with the wearable device are used within the physical environment and are constantly turned on to collect data of the environment and draw sensor graphs. Sensors and implemented technologies may include thermometers, GPS, accelerometers, inertial measurement units (IMUs), gyroscopes, microphones, cameras, Wi-Fi, etc. Each node on the spatial graph represents a collection of sensor graphs pulled from each sensor when the node is created. Each node is then linked spatially using edges, thereby creating a spatial print chain. The data within the node can be used to track objects within the physical environment. For example, a camera can identify a user's baseball glove, and a microphone can identify the sound of a car key being pressed against it. Baseball gloves and car keys can be recorded in one or more nodes of the spatial graph, respectively, which allows the user to locate each object later when querying the computing device storing the spatial graph.

[0003] There are two scenarios where a wearable device, remote server, or edge computing unit performs spatial printing that associates sensor graph data with newly created nodes. One scenario is when data changes in one or more sensor graphs exceed a threshold level, and the second scenario is when a user uploads an object to the object graph. In both cases, various sensor graphs and sensing devices are queried to create a digital copy of the user's environment and objects.

[0004] The remote server or edge computing unit can be configured to link two different spatial printing chains, which consist of a series of nodes and edges. For example, the first spatial printing chain may have been previously uploaded, and the second spatial printing chain may have been recently scanned and uploaded. The online linker running when the wearable device is online with the edge computing unit or remote server can find the relationship between the two chains and connect the two chains to create a logically traceable series of nodes. For example, when the wearable device is online, operating and transmitting sensor data to the edge computing unit, the edge computing unit can find connections between the spatial printing chains to create an organic connection between the chains (for example, such as by finding connections between nodes of two independent chains). The online linker can operate locally at the wearable device or edge computing unit, remotely at the remote server, or at a combination of components. In addition, if there is no clear connection or relationship between the spatial printing chains, an offline linker can be used. The offline linker can access the previously uploaded spatial printing chain on the remote server and find similarities, relationships, or connections for connecting the two chains.

[0005] Over time, data in a remote server may become obsolete. For example, data may become obsolete when it is outdated, contains a file size that is too large, is irrelevant, or is deemed unnecessary by a user. In these cases, a garbage collector on the remote server may periodically search for and delete such data to save storage capacity.

[0006] Advantageously, the wearable device, edge computing unit, and remote server provide a system in which heterogeneous data is periodically collected for a physical environment and collected into a spatial graph. The spatial graph provides a representation of real-world objects and locations, and the real-world objects are digitized using a collection of various sensor data. In addition, the data can be synthesized to identify objects and make logical connections with respect to their locations in the real world. For example, the remote server can be configured to recognize real-world objects and make inferences about where the objects are located in the physical environment. Nodes and edges provide concepts and spatial relationships related to the physical world, and thereby can provide actual or approximate locations for items when absolute locations are not available.

[0007] In this regard, the current configuration of the indoor positioning system (IPS) solves problems that exist in other technologies such as GPS, which may not be very useful when the user is indoors. When the user is indoors, the configuration of the spatial graph, nodes, and edges can supplement GPS technology or completely replace GPS technology to provide accurate location and object tracking. In addition, this configuration can improve other IPS technologies where some sensor data is sometimes not available, where querying all sensor graphs for corresponding nodes can provide important useful data for object tracking and positioning technology.

[0008] This "Summary of the Invention" is provided to introduce some concepts in a simplified form, which will be further described in the "Detailed Description of the Invention" below. This "Summary of the Invention" is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to implementations that solve any or all of the shortcomings noted in any part of this disclosure. It should be understood that the above-mentioned subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as one or more computer-readable storage media. These and various other features will become clear by reading the following detailed description and viewing the associated drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Illustrative components associated with a wearable device are shown;

[0010] Figure 2 Shows from Figure 1 illustrative sensor information acquired by one or more sensor devices in;

[0011] Figure 3 An illustrative system architecture for a wearable device is shown;

[0012] Figure 4 An illustrative environment in which a wearable device interacts with a remote server and an edge computing unit is shown;

[0013] Figure 5 Schematic diagram showing the physical environment and different types of sensorgrams;

[0014] Figure 6 shows a schematic diagram of a spatial print with a sensorgram triggered by changes in Wi-Fi signal strength;

[0015] Figure 7 A schematic diagram with a spatial graph and nodes is shown;

[0016] Figure 8 A schematic diagram with an object graph is shown;

[0017] Fig. 9 A schematic diagram is shown in which a marked object has changed position;

[0018] Fig.10 A schematic diagram showing an object in which a set of keys are tags in an object graph;

[0019] Fig.11 A schematic diagram is shown in which an online linker links two spatial printing chains;

[0020] Fig.12 A schematic diagram showing the use of an offline linker and a garbage collector;

[0021] Figures 13 to 15 An illustrative process performed by one or more wearable devices, edge computing units, or remote servers is shown;

[0022] Fig.16 is a simplified block diagram of an illustrative wearable device that may be used in part to implement mapping of an environment using a one-dimensional radiation sensor; and

[0023] Fig.17 is a simplified block diagram of an illustrative computer system that may be used in part to implement object and position tracking utilizing graphs in multiple graphs.

[0024] Like reference numerals indicate like elements in the drawings. Unless otherwise noted, the elements are not drawn to scale. DETAILED DESCRIPTION

[0025] Figure 1 Illustrative hardware components associated with wearable device 105 are shown. Wearable device 105 may include one or more of a variety of different computing devices configured to be easily and conveniently worn by a user to perform various functions and provide a beneficial user experience.

[0026] Wearable devices 105 including displays and input systems can utilize various types of user interfaces. Some wearable devices can be operated through voice interaction as well as sensed gestures and / or user activities. Wearable devices can be configured to operate continuously in a non-intrusive manner, and can also support functions and user experiences that rely on explicit user interaction or other user input. Wearable devices can also be configured with communication and networking interfaces to enable them to interact with local and remote users, devices, systems, services, and resources.

[0027] The wearable device 105 is configured to include various sensors, as described below, which can be configured to detect and / or measure motion, light, surface, temperature, humidity, position, altitude, and other parameters describing the user or device environment. Other sensors can be supported on the wearable device, which are directly or indirectly located on the user's body to monitor parameters describing the user's physiological state, such as motion, position or posture of the body and / or body parts, pulse, skin temperature, etc. The sensors can operate in various combinations, so that in some cases, a given descriptive parameter can be derived by a suitable system in the wearable device using data from more than one sensor, or by combining the sensor data with other information available to the device.

[0028] Figure 1The hardware components of the wearable device depicted in are not exhaustive and are used to illustrate various sensors that can generate data graphs and be incorporated into the spatial graphs, as discussed in further detail below. The aggregation of heterogeneous sensors implemented helps develop a complete digital profile for an object or physical environment. The wearable device includes a processor 110, a memory 115, and a plurality of sensing devices (i.e., sensors) 120. However, the wearable device or computing device employed may use more or less Figure 1 The sensing devices listed in. The sensing devices may include a global positioning system (GPS) 125, a Wi-Fi transceiver 130, transceiver 135 , camera 140 , thermometer 145 , magnetometer 150 , microphone 155 , pedometer 160 , accelerometer 165 , gyroscope 170 , inertial measurement unit (IMU) 175 , proximity sensor 180 , barometer 185 , and light sensor 190 .

[0029] Figure 2 Illustrative types of sensory information 205 are shown that are derived by a wearable device or an external computing device (e.g., an edge computing unit or a remote server) based on data acquired from a sensing device 120. For example, by synthesizing data acquired from one or more sensors, it can be determined whether the wearable device is located inside or outside 210. In addition, pedestrian dead reckoning (PDR) 215 principles can be used to acquire measurements using, for example, an IMU (such as a collection of accelerometers, gyroscopes, and magnetometers). PDR measurements can indicate the distance and direction traveled by a user, such as when a GPS device is not available (e.g., the user is indoors). The sensory information can indicate or approximate latitude and longitude 220 using, for example, a GPS device. Object recognition 225 can be performed using a camera, and temperature 230 can be acquired using a thermometer. Other types of sensory information not shown can be acquired using one or more sensing devices 120.

[0030] Figure 3 An illustrative system architecture 300 of a wearable device 105 is shown. As shown, the wearable device 105 can be implemented using components such as glasses, covers, gloves, headbands, headphones, hats, helmets, earplugs, shoes, wristbands, and belts that are worn, and / or placed on the user using accessories such as neckbands, arm / leg bands, lanyards, etc. The wearable device can also be incorporated or embedded in clothing. In a typical implementation, the wearable device can be configured to provide a hands-free and eyes-free experience when the device operates using battery power to promote portability and mobility.

[0031] In simplified form, the architecture is conceptually arranged in layers and includes a hardware layer 315, an operating system (OS) layer 310, and an application layer 305. The hardware layer 315 provides an abstraction of the various hardware used by the wearable device 105 to the layers above it (e.g., network and radio hardware, etc.). Figure 1 As shown, the hardware layer supports processor(s) 110, memory 115, sensor devices 120, and network connectivity components (eg, Wi-Fi transceiver). Although not shown, other components are possible, such as input / output devices, display screens, and the like.

[0032] In this illustrative example, the application layer 305 supports various applications 330, including applications configured as spatial printing 335 as discussed herein. The spatial printing is configured to generate nodes within a spatial graph, pull data from each employed sensor, and associate the pulled data with the generated nodes. When the currently detected data for the corresponding sensing device changes and exceeds a threshold, a node is created within the spatial graph. Each sensor can be configured with its own threshold parameters. Thus, for example, when the PDR data indicates that the user has traveled a threshold of six feet, the wearable device can create a node within the spatial graph and perform a spatial printing, where the data of all sensors is collected and associated with the newly created node. In an additional example, a spatial printing can be performed when a thermometer detects a temperature increase or decrease of five or ten degrees, a camera recognizes an object, a microphone recognizes the sound of an object, and the like. At this point, the characteristics of certain objects can be pre-stored in a memory so that the sensors on the wearable device can then compare and identify these objects.

[0033] Other applications not shown may include a web browser configured to provide connectivity to the World Wide Web, games, etc. Although Figure 3 Only certain applications are described in the , but any number of applications, whether proprietary or developed by a third party, may be used by a wearable device. These applications are typically implemented using locally executed code. However, in some cases, these applications may rely on services and / or remote code execution provided by a remote server or other computing platform (such as a platform supported by a service provider or other cloud-based resource).

[0034] The OS layer 310 supports, among other operations, the management system 320 and operating applications 325, such as operating a space printing application 335 (as indicated by the arrow). The OS layer may interoperate with the application and hardware layers to perform various functions.

[0035] Figure 4An exemplary environment 400 is shown in which a user 405 operates a wearable device 105, and the wearable device is configured to interoperate with external computing devices such as a remote server 420 and an edge computing unit 425 via a network 410. The network may include a local area network, a wide area network, the Internet, and the World Wide Web. In this regard, the edge computing unit may be located on-site at the wearable device, receive the spatial map data, and then forward the data to the remote server. In addition, alternative computing devices such as tablet computers and smart phones (not shown) may be utilized, which are configured similarly to the wearable device. In addition, as shown in FIG. Figure 4 As shown, other users can register in a remote server or edge computing unit, where other users can utilize their own spatial map data, or the spatial map can be combined with one or more other users with the user and the user's consent.

[0036] The remote server and edge computing unit can interoperate with the extensibility client 415 of the spatial printing application 335, where the external computing device can construct the spatial map in whole or in part upon receiving the collected data from the sensors of the wearable device. The wearable device can, for example, collect data, where the extensibility client forwards the data to one of the external computer devices for processing. Thus, any discussion about spatial printing or spatial map construction can be local to the wearable device, external to the remote server or edge computing unit, or a combination thereof.

[0037] Figure 5 A schematic diagram showing a physical environment and different types of sensory graphs 505 is shown. For example, as a user 405 navigates a physical environment (in this example, an office building), various sensory graphs may generate data. Other types of physical environments include stores, houses, etc. A sensory graph may be data acquired directly from a single sensor or may be information acquired from one sensor or a combination of sensors ( Figure 2 ). In this example, the sensory graph is about determination of whether the user is inside or outside, temperature, PDR (e.g., distance traveled), camera (e.g., object recognition), changes in latitude and longitude, and Wi-Fi (e.g., changes in signal strength).

[0038] Figure 6An illustrative spatial printing is shown that occurs due to a change in Wi-Fi signal strength. For example, user 405 navigates an office building and a threshold change in Wi-Fi signal strength may occur. As a result of this threshold change, the wearable device queries each sensor map and / or sensing device in real time, i.e., the data may be queried immediately or virtually immediately when the threshold change is detected. The threshold change in Wi-Fi signal may be a change in decibel (dBm) strength, such as plus or minus -10 dBm. The sensor map may be a current representation of data for a particular sensing device or information of a type derived from a sensing device (e.g., PDR data).

[0039] After a query is executed for each sensor graph, a node is created at that location. Figure 7 335 function has been created. For example, each node may be created when sensor data associated with one or more sensors exceeds a set threshold. Thus, Figure 6 An example of Wi-Fi signal strength triggering a data pull is shown, but other sensor graphs also have threshold levels, such as PDR data indicating that a user has walked a threshold number of feet.

[0040] The spatial printing 335 function may be performed on one or a combination of the wearable device, the remote server, or the edge computing unit. For example, the wearable device may automatically perform a spatial printing in which it queries the sensory data when one of its sensors exceeds a threshold. The wearable device may create a node or forward data to one of the external computing devices to generate a node and spatial graph 715.

[0041] Node 705 is contained within spatial graph 715 and is Figure 7 As shown, the actual location of the data corresponding to the corresponding node is pulled within the physical environment. The spatial graph represents a collection of sensor graphs 505 and is a high-order meta-graph with a deep understanding of how locations and objects are related to each other. By placing multiple sensory information in relationship nodes, the relationship between objects and their environment can be understood.

[0042] The nodes are connected with edges 725, which provide a logical and traceable connection between two or more nodes. Thus, for example, various sensor graph data can be tracked between nodes to help track the user's footsteps so that the data can be used together. In contrast to absolute positioning, this helps track object locations relative to locations, objects, and environmental characteristics. This may be useful for indoor positioning systems (IPS) when certain computing devices are technically insufficient (such as GPS for indoor tracking). For example, if one or more nodes detect a Bluetooth connection from a smartphone when the temperature is 32° in a supermarket, this may indicate that the user was previously close to his smartphone in the frozen food section of the supermarket. This may indicate that the user is searching for a lost smartphone not only in the supermarket but also in the frozen food section of the supermarket.

[0043] Callout 720 shows exemplary data that may be pulled at a given node. In this example, the data indicates that the wearable device is located inside, the temperature is 65°, the PDR indicates that the user is two feet away from the previous node, the user's location is 40° North, 73° West, and the user's Wi-Fi strength is rated at -60dBm. Although the spatial print may query each sensor graph 505, in an alternative embodiment, only the relevant sensor graphs may be queried. For example, once the wearable device determines that the user is inside an office building, temperature data may not be relevant and therefore will not be pulled.

[0044] Figure 7 The context provider 730 can be used with a wearable device or other device that creates a spatial map (such as an edge computing unit or a remote server ( Figure 4 )). A context provider can be an external service that provides information about the user's environment. For example, an external service associated with server 735 can provide information about the office building in which the user is located and map the information. The data can include scheduling information within the building (e.g., a meeting in office 112), the number of people within the environment, details about locations within the environment (e.g., different food areas in a supermarket, restroom locations, locations of specific personal offices), and address and company name information about the environment. In addition, sensory information about the environment can also be retrieved, such as temperature information connected to a heating or cooling system, the location of objects in the environment based on sensory data, etc. This information can supplement the sensory information derived from the sensory device. The information listed in this article that can be obtained from external services is not exhaustive, and any number of services or databases can be used to supplement the spatial map data.

[0045] Figure 8An illustrative environment in which an object graph 805 is used for a user or application developer is shown. In this example, object A (represented by number 810) has been marked or added by a user. The object may have been input by user interaction with a user interface associated with a wearable device or a separate computing device such as a personal computer (PC) or a smart phone (not shown). When an object is marked, the object becomes a node within the object graph 805 and is stored in a memory, which node can be associated with a spatial graph. The marking of the object also triggers a query to each sensing device or sensor graph. The query helps to determine the sensor data that makes up the object, thereby digitizing and creating a blueprint of an object with various heterogeneous sensing properties. For example, a camera can capture an image of an object, and a microphone can record the sound of the object (such as a key); these data are then stored in a memory.

[0046] Fig. 9 An illustrative environment in which object A moves position is shown. In this case, the wearable device can update the position of the object by deleting the edge printed in the previous space ( Figure 8 ), and a new edge is created for the space where the object was last detected ( Fig. 9 ).

[0047] Fig.10 An illustrative environment is shown in which real-world objects are used as objects for tracking. In this example, a set of keys 1005 has been previously entered into the object graph 805, and the user may have queried the wearable device or other computing device to locate his keys within the office. Node 1010 can be associated with the user's previous travels, such as earlier in the day, the day before, etc. When analyzing the spatial graph data, the wearable device can determine that the microphone on the wearable device detects keys ringing at or near the office 106. The wearable device or the user's personal device (e.g., a smart phone) can display the inferred location or node of the key to the user. Other relevant data that can support the inference of the location or node is the room temperature (e.g., the typical temperature of the room or office) and the image that may have been captured (e.g., the physical environment and objects of the office). The user can then begin searching for the travel of his keys near the office 106.

[0048] Because the spatial graph includes multiple nodes and spatial prints of data, the wearable device can utilize a series of nodes and edges or spatial print chains to create an inference and logical path or direction to the key. Therefore, once created, the spatial print chain and spatial graph can generally be traversed. For example, the wearable device can provide directions to the user in the opposite direction of the generated spatial print chain to track the user's route of travel to the location where the key was lost and may be located. Each node in the spatial graph is associated with specific sensory data. Therefore, when the user navigates through the office, these directions can be tracked and provide directions to the location for the user. For example, when the user continues to follow these directions and the wearable device continues to sense subsequent data, the previous data associated with the previously existing nodes can verify that the subsequent data is consistent with the previous data. This allows the user to traverse the nodes in the spatial graph and receive subsequent and verifiable instructions.

[0049] As an example, the wearable device may recognize that the temperature has dropped by five degrees at node 1015. Therefore, the wearable device verifies subsequent sensor data with previous sensor data and may therefore also instruct the user to continue walking straight. Other examples of data on nodes that allow the user to traverse the spatial graph include cameras that recognize specific objects at that location, proximity sensors that detect narrow spaces, light sensors that detect too much or too little light, etc.

[0050] Fig.11 An illustrative environment is shown in which an online linker 1105 links spatial print chains as they are constructed. For example, the spatial map may be stored locally at a wearable device or edge computing unit, or remotely at a remote server when constructed ( Figure 4 ). When the wearable device again enters an online or connected state in which the wearable device will operate and create a spatial print, the existing spatial print chain (e.g., connected nodes and edges) can be searched for connections to the spatial print chain being developed. This allows the spatial print chains to be organically connected as they are encountered and generated. The online linker function can be executed at the wearable device, the edge computing unit, the remote server, or a combination of these components.

[0051] For example, connections can be made based on relationships or similarities between nodes or spatially printed chains. For example, if two nodes or chains are placed next to each other and created at a nearby time, a connection can be made. Fig.11 A first spatial print chain 1120 and a second spatial print chain 1125 are shown. In this example, the node near object B at connection 1115 is a logical connection between the two chains. The similarity of connections allowed between chains can have minimum or maximum threshold requirements, such as a threshold requirement of being created within a certain time period (e.g., five minutes), or being located within a threshold distance of each other (e.g., ten feet).

[0052] Fig.12 A detailed illustrative detailed environment is shown, which shows other objects in the object graph, an offline linker 1205, and a garbage collector 1210. Sometimes, a spatial print chain may not be connected, such as when the distance between individual chains is too large, or when there is no direct link. In this case, the remote server can search for connections between recently uploaded data and previously uploaded data. The offline linker can find, for example, spatial print chains that have nodes that are close to each other in space and include one or more similarities in the spatial graph data. For example, time, location, and other relevant sensor data can help locate logical connections between nodes and edges.

[0053] The garbage collector 1210 can identify and delete data, nodes, edges, etc. that become stale over time. For example, stale data can be identified based on its age (e.g., too old to be useful), size (e.g., the file is too large and takes up too much memory space), usefulness (e.g., a user identified certain types or time frames of data as irrelevant), etc.

[0054] Fig.13 is a flow chart of an illustrative method 1300 in which a computing device associates unique sensory data with a node. Unless otherwise specified, the methods or steps shown in the flow chart and described in the accompanying text are not limited to a particular order or sequence. In addition, certain methods or steps may occur or be performed simultaneously, and not all methods or steps must be performed in a given implementation (depending on the requirements of such implementation), and some methods or steps may be optionally utilized.

[0055] In step 1305, a first set of sensory data is received, wherein the data includes a plurality of sensory graphs, each sensory graph being associated with different sensory information. In step 1310, a first node is created in a spatial graph. In step 1315, the first set of sensory data is associated with the first node.

[0056] Fig.14is a flow chart of an illustrative method 1400 in which a computing device uses a sensor map for a spatial map. In step 1405, threshold levels are assigned for sensor data, wherein the sensor data is derived from the sensor devices. In step 1410, the state of the sensor data associated with each sensor device is monitored. In step 1415, a status log is triggered for each sensor device when one or more of the assigned threshold levels are exceeded. The status log may include, for example, a data pull of current data associated with various sensor devices or sensor maps. In step 1420, each sensor map in the constructed sensor map is collected into a spatial map. The spatial map may, for example, assign the status log of the sensor device to the nodes in the spatial map.

[0057] Fig.15 is a flow chart of an illustrative method 1500 in which a computing device creates a blueprint of a tagged object. In step 1505, a series of nodes and edges are established within a spatial graph, wherein the edges connect the nodes and the nodes are associated with sensor data. In step 1510, objects for tracking are tagged within the spatial graph or an object graph associated with the spatial graph. In step 1515, sensors controllable by the computing device are activated to capture the current sensor data blueprint. For example, each sensing device may be queried, wherein data is pulled from each sensing device. In step 1520, the object is associated with the most recently created node and the sensor data for that node.

[0058] Fig.16 An illustrative architecture 1600 of a device capable of executing the various components described herein to provide current user and device authentication to a network application is shown. Fig.16 The illustrated architecture 1600 illustrates an architecture that may be applicable to a wearable device, a server computer, a mobile phone, a PDA, a smart phone, a desktop computer, a netbook computer, a tablet computer, a GPS device, a game console, and / or a portable computer. The architecture 1600 may be used to perform any aspect of the components presented herein.

[0059] Fig.16The illustrated architecture 1600 includes one or more processors 1602 (e.g., central processing units, graphics processing units, etc.), system memory 1604 (including RAM (random access memory) 1606 and ROM (read only memory) 1608)), and a system bus 1610 that operatively and functionally couples the components in the architecture 1600. A basic input / output system containing basic routines that help to pass information between elements in the architecture 1600, such as during a boot process, is typically stored in the ROM 1608. The architecture 1600 also includes a mass storage device 1612 for storing software code for implementing applications, file systems, and operating systems, or other computer-executed code. The mass storage device 1612 is connected to the processor 1602 through a mass storage controller (not shown) connected to the bus 1610. The mass storage device 1612 and its associated computer-readable storage media provide non-volatile storage for the architecture 1600. Although the descriptions of computer-readable storage media contained herein refer to mass storage devices, such as a hard disk or CD-ROM drive, those skilled in the art will appreciate that computer-readable storage media may be any available storage media that architecture 1600 may access.

[0060] The architecture 1600 also supports a sensor package 1630 that includes one or more sensors or components that are configured to detect parameters describing the environment and / or detect parameters describing the user of the device, or a combination thereof. For example, for a wearable computing device, the sensor can be positioned directly or indirectly on the user's body. The sensor can be configured to operate continuously or periodically, and typically operates in a hands-free and / or eyes-free manner. The architecture also supports a power source and / or battery component (identified collectively by reference numeral 1615). For example, in a wearable device application, one or more batteries or power packs can be rechargeable or replaceable to facilitate portability and mobility.

[0061] By way of example and not limitation, computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer readable media include, but are not limited to, RAM, ROM, EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory or other solid-state storage technology, CD-ROM, DVD, HD-DVD (high-definition DVD), Blu-ray or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the architecture 1600.

[0062] According to various embodiments, the architecture 1600 can operate in a networked environment using logical connections to remote computers through a network. The architecture 1600 can be connected to the network through a network interface unit 1616 connected to the bus 1610. It should be understood that the network interface unit 1616 can also be used to connect to other types of networks and remote computer systems. The architecture 1600 can also include an input / output controller 1618 for receiving and processing input from many other devices, including a keyboard, a mouse, a touch pad, a touch screen, a control device (such as buttons and switches), or an electronic pen ( Fig.16 Similarly, the input / output controller 1618 may provide output to a display screen, user interface, printer, or other type of output device (not shown in FIG. Fig.16 ).

[0063] The architecture 1600 may include a speech recognition unit (not shown) to facilitate user interaction via voice commands, a natural language interface, or by interacting with a personal digital assistant (such as provided by Microsoft Corporation). The architecture 1600 may include a gesture recognition unit (not shown) to facilitate user interaction with a device supporting the architecture through sensed gestures, motions, and / or other sensed inputs.

[0064] It should be understood that the software components described herein, when loaded into the processor 1602 and executed, can convert the processor 1602 and the entire architecture 1600 from a general-purpose computing system to a special-purpose computing system customized to facilitate the functions described herein. The processor 1602 can be composed of any number of transistors or other discrete circuit elements (which can present any number of states individually or collectively). More specifically, the processor 1602 can operate as a finite state machine in response to executable instructions contained in the software modules disclosed herein. These computer executable instructions can convert the processor 1602 by specifying how the processor 1602 converts between states, thereby converting the transistors or other discrete hardware elements that constitute the processor 1602.

[0065] Encoding the software modules proposed herein can also convert the physical structure of the computer storage readable medium proposed herein. In different implementations of this specification, the specific conversion of the physical structure can depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the computer storage readable medium, whether the computer storage readable medium is characterized as primary storage or auxiliary storage, etc. For example, if the computer storage readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer storage readable medium by converting the physical state of the semiconductor memory. For example, the software can convert the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also convert the physical state of these components to store data thereon.

[0066] As another example, the computer readable storage medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when software is encoded therein, the software proposed herein can convert the physical state of the magnetic or optical medium. These conversions can include changing the magnetic properties of a specific location in a given magnetic medium. These conversions can also include changing the physical characteristics or properties of a specific location in a given optical medium to change the optical properties of these locations. Without departing from the scope and spirit of this specification, other conversions of physical media are possible, and the aforementioned examples are provided only to facilitate this discussion.

[0067] In view of the above, it should be understood that many types of physical transformations occur in architecture 1600 in order to store and execute the software components presented herein. It should also be understood that architecture 1600 may include other types of computing devices, including wearable devices, handheld computers, embedded computer systems, smart phones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that architecture 1600 may not include Fig.16 All components shown may include Fig.16 Other components not explicitly shown in the figure may also be used in conjunction with Fig.16 The architecture shown is a completely different architecture.

[0068] Fig.171 is a simplified block diagram of an illustrative computer system 1700, such as a wearable device, with which the object and location tracking of the present invention using the diagrams in the multiple figures can be implemented. Although wearable devices are discussed herein, other computing devices having similar configurations discussed herein may also be used, including smart phones, tablet computing devices, personal computers (PCs), laptop computers, etc. The computer system 1700 includes a processor 1705, a system memory 1711, and a system bus 1714, which couples various system components including the system memory 1711 to the processor 1705. The system bus 1714 can be any of several types of bus structures using any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, or a local bus. The system memory 1711 includes a read-only memory (ROM) 1717 and a random access memory (RAM) 1721. A basic input / output system (BIOS) 1725 (containing basic routines that help transfer information between elements within the computer system 1700, such as during startup) is stored in the ROM 1717. The computer system 1700 may also include a hard disk drive 1728 for reading and writing to an internally disposed hard disk (not shown); a disk drive 1730 for reading or writing to a removable disk 1733 (e.g., a floppy disk); and an optical drive 1738 for reading or writing to a removable optical disk 1743, such as a CD (compact disk), a DVD (digital versatile disk) or other optical media. The hard disk drive 1728, the disk drive 1730 and the optical drive 1738 are connected to the system bus 1714 via a hard disk drive interface 1746, a disk drive interface 1749 and an optical drive interface 1752, respectively. The drives and their associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer system 1700. Although the illustrative example includes a hard disk, a removable disk 1733, and a removable optical disk 1743, in some applications of the object and position tracking of the present invention utilizing the multiple figures, other types of computer-readable storage media that can store data that can be accessed by a computer, such as magnetic tapes, flash memory cards, digital video disks, data cassettes, random access memory (RAM), read-only memory (ROM), etc., may also be used. In addition, as used herein, the term computer-readable storage medium includes one or more instances of a type of medium (e.g., one or more disks, one or more CDs, etc.). For purposes of this specification and claims, the phrase "computer-readable storage medium" and its variants are non-transitory and do not include waves, signals, and / or other transient and / or intangible communication media.

[0069] A plurality of program modules may be stored on a hard disk, disk, optical disk, ROM 1717 or RAM 1721, including an operating system 1755, one or more application programs 1757, other program modules 1760, and program data 1763. A user may enter commands and information into the computer system 1700 through input devices such as a keyboard 1766 and a pointing device 1768 such as a mouse. Other input devices (not shown) may include a microphone, a joystick, a game pad, a satellite dish, a scanner, a trackball, a touch pad, a touch screen, a touch-sensitive device, a voice command module or device, a user motion or user gesture capture device, and the like. These and other input devices are typically connected to the processor 1705 through a serial port interface 1771 coupled to the system bus 1714, but may also be connected through other interfaces, such as a parallel port, a game port, or a universal serial bus (USB). A monitor 1773 or other type of display device is also connected to the system bus 1714 via an interface such as a video adapter 1775. In addition to the monitor 1773, wearable devices and personal computers may typically include other peripheral output devices (not shown), such as speakers and printers. Fig.17 The illustrative example shown also includes a host adapter 1778 , a small computer system interface (SCSI) bus 1783 , and an external storage device 1776 connected to the SCSI bus 1783 .

[0070] The computer system 1700 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer 1788. Remote computer 1788 can be selected as a personal computer, server, router, network PC, peer device or other public network node, and typically includes many or all of the elements described above with respect to computer system 1700, although in the example of FIG. Fig.17 Only a single representative remote memory / storage device 1790 is shown. Fig.17 The devices depicted in 179 include a local area network (LAN) 1793 and a wide area network (WAN) 1795. Such networking environments are commonly deployed in, for example, offices, enterprise-wide computer networks, intranets, and the Internet.

[0071] When used in a LAN networking environment, the computer system 1700 is connected to the local area network 1793 through a network interface or adapter 1796. When used in a WAN networking environment, the computer system 1700 typically includes a broadband modem 1798, a network gateway, or other means for establishing communications over the wide area network 1795 (such as the Internet). The broadband modem 1798 (which may be internal or external) is connected to the system bus 1714 via a serial port interface 1771. In a network environment, program modules related to the computer system 1700, or portions thereof, may be stored in the remote memory storage device 1790. Note that Fig.17 The network connections shown are illustrative and other means of establishing a communications link between the computers may be used depending on the specific requirements of an application of the object and position tracking of the present invention utilizing the various figures of the drawings.

[0072] Various exemplary embodiments of object and position tracking of the present invention utilizing graphs in multiple graphs are now presented by way of illustration and not as an exhaustive list of all embodiments. One example includes a computing device configured to collect and synthesize sensory data to track an object, the computing device comprising: one or more processors; and a memory configured to store a spatial graph and sensory data associated with nodes in the spatial graph, the memory having computer-readable instructions that, when executed by the one or more processors, cause the computing device to: receive a first set of sensory data, wherein the first set includes multiple sensory graphs, each sensory graph being associated with different sensory information; create a first node in the spatial graph; and associate the first set of sensory data with the first node.

[0073] In another example, the computer-readable instructions further cause the computing device to: receive a subsequent set of sensory data; create a subsequent node in the spatial graph; associate the subsequent set of sensory data with the subsequent node; and create an edge in the spatial graph linking the first node to the subsequent node, wherein the first node becomes a previously existing node of the subsequent node. In another example, the computer-readable instructions further cause the computing device to: create a series of nodes and edges to create a spatial print chain; locate an object based on sensory data associated with one or more nodes in the series of nodes; and create directions for a user to traverse the edges and nodes within the spatial print chain, wherein the directions lead the user to the object, wherein the current sensory data and previous sensory data associated with the node are used to create the directions. In another example, a set of sensory data is received when one or more measurements in the sensory graph exceed a threshold. In another example, when at least one of the measurements in the sensory graph exceeds a threshold, each sensory device is queried to generate sensory data for a new node. In another example, the sensory data is received from a computing device or edge computing unit of a user configured to store nodes and sensory data for one or more wearable devices. In another example, the computer-readable instructions further cause the computing device to communicate with an external context provider that provides additional context and sensory information for the physical environment, wherein the context or sensory information includes at least a map of the physical environment, the number of people occupying the space, a scheduled event within the physical environment, details about a room in the physical environment, or sensory information to be associated with a node. In another example, the computer-readable instructions further cause the computing device to construct a spatial print chain using a set of nodes and edges connecting the nodes, wherein each spatial print chain includes a corresponding set of nodes and edges of the spatial print chain itself; and connect two different spatial print chains to form a single spatial print chain. In another example, the two different spatial print chains are connected when the offline computing device establishes an Internet connection and transmits the spatial print chain to the computing device. In another example, the two different spatial print chains are connected when the two chains share a relationship with each other.

[0074] Another example includes a method for generating sensory data for a spatial graph, performed by a local computing device within a physical environment, the method comprising: assigning threshold levels to sensory data developed by each sensory device associated with the local computing device; monitoring the state of the sensory data associated with each sensory device; triggering a state log for each sensory device in the sensory devices when one or more of the assigned threshold levels associated with the corresponding sensory devices are exceeded based on the state monitoring, wherein the local computing device constructs a sensor graph based on the current sensory data for each sensory device; and collecting each sensor graph in the constructed sensor graph into a spatial graph, wherein the spatial graph assigns the state log for the sensor device to a node in the spatial graph.

[0075] In another example, the assigned threshold level for each sensing device indicates a change in sensor state. In another example, the sensing device includes at least one of: a global positioning system (GPS), Wi-Fi, cameras, thermometers, magnetometers, pedometers, accelerometers, gyroscopes, inertial measurement units (IMUs), microphones, proximity sensors, barometers, and light sensors. In another example, the method further includes: receiving input for marking an object using a spatial graph at a user interface (UI) of a local computing device; and in response to the received input, querying a sensing device for sensor data associated with the marked object to generate a data blueprint for the marked object. In another example, the method further includes: receiving subsequent input for locating the marked object; locating an indication at one or more nodes representing the marked object within the spatial graph; and outputting the location of the one or more nodes or the located indication associated with the one or more nodes on the UI. In another example, the indication includes a correspondence between the original sensor data from the time the object was marked and subsequent data in the spatial graph.

[0076] Another example includes one or more hardware-based computer-readable storage devices storing instructions that, when executed by one or more processors disposed in the computing device, cause the computing device to: establish a series of nodes and edges within a spatial graph, wherein the edges connect the nodes and the nodes are associated with sensor data, the sensor data being captured concurrently with the creation of each respective node; mark objects for tracking within the spatial graph or an object graph associated with the spatial graph; query sensors accessible to the computing device to capture a current sensor data blueprint, wherein the sensors include sensors of different types; and associate the object with a most recently created node and the sensor data for the node.

[0077] In another example, the instructions further cause the computing device to: create additional nodes as the wearable device navigates the physical area; detect a new location of the object using sensor data of the created nodes; and when the new location of the object is detected, delete the association of the object with the previous node and create a new association between the object and the new node. In another example, the instructions further cause the computing device to invalidate stale nodes in the spatial graph, where a stale node is a node that is older with respect to time or contains irrelevant sensor data. In another example, the instructions further cause the computing device to receive a request for the location of an object; and use the sensor data blueprint for nodes within the spatial graph to provide directions for traversing nodes within the spatial graph leading to the object.

[0078] The subject matter described above is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the illustrated and described example embodiments and applications, and without departing from the true spirit and scope of the invention as set forth in the appended claims.

Claims

1. A method for generating sensory data for a spatial graph, performed by a local computing device within a physical environment, comprising: assigning a threshold level to the sensory data developed by each sensory device associated with the local computing device; monitoring a status of the sensor data associated with each sensor device; triggering a status log for each of the sensing devices when one or more of the assigned threshold levels associated with the respective sensing devices are exceeded based on status monitoring, wherein the local computing device constructs a sensor map based on current sensor data for each sensing device; as well as Each of the constructed sensor graphs is collected into a spatial graph, wherein the spatial graph assigns the status logs for the sensor devices to nodes in the spatial graph. 2 . The method of claim 1 , wherein the assigned threshold level for each sensing device represents a change in sensor state.

3. The method according to claim 1, wherein the sensing device comprises at least one of the following: a global positioning system (GPS), Wi-Fi, Camera, thermometer, magnetometer, pedometer, accelerometer, gyroscope, inertial measurement unit (IMU), microphone, proximity sensor, barometer, or light sensor.

4. The method according to claim 1, further comprising: receiving, at a user interface (UI) of the local computing device, input for labeling an object using the spatial map; as well as In response to the received input, the sensor device is queried for sensor data associated with the tagged object to generate a data blueprint for the tagged object.

5. The method according to claim 4, further comprising: receiving subsequent input for locating the marked object; locating an indication at one or more nodes representing the marked object within the spatial graph; as well as The one or more nodes or the located indicated locations associated with the one or more nodes are output on the UI. 6 . The method of claim 5 , wherein the indication comprises a correspondence between raw sensor data starting from when the object was marked and subsequent data within the spatial map.

7. One or more hardware-based computer-readable memory devices storing instructions that, when executed by one or more processors disposed in a computing device, cause the computing device to: establishing a series of nodes and edges within the spatial graph, wherein the edges connect the nodes and the nodes are associated with sensor data captured concurrently with the creation of each corresponding node; marking an object for tracking within the spatial map or an object map associated with the spatial map; querying sensors accessible by the computing device to capture a blueprint of current sensor data, wherein the sensors include sensors of different types; as well as The object is associated with a recently created node and the sensor data of the node.

8. The one or more hardware-based computer-readable memory devices of claim 7, wherein the instructions further cause the computing device to: As the wearable device navigates the physical area, additional nodes are created; detecting a new position for the object using the sensor data for the created node; and When the new position for the object is detected, the association of the object with the previous node is deleted and a new association is created between the object and the new node.

9. One or more hardware-based computer-readable memory devices according to claim 7, wherein the instructions also cause the computing device to expire stale nodes in the spatial graph, wherein a stale node is a node that is old with respect to time or contains irrelevant sensor data.

10. The one or more hardware-based computer-readable memory devices of claim 7, wherein the instructions further cause the computing device to: receiving a request for a location of the object; and A blueprint using sensor data for nodes within the spatial graph provides directions for traversing nodes within the spatial graph leading to the object.

Images