Object and location tracking with graph of plurality of graphs

Abstract

Embodiments of the invention relate to object and location tracking with a graph in a plurality of graphs. A wearable device is configured with various sensing devices that cyclically monitor and collect data for a physical environment around a user to assist in locating and tracking real-world objects. Various heterogeneous sensing devices digitize objects and physical world. Each sensing device is configured with a threshold data change, where a query is performed on each sensor map or sensing device when data collected by one or more sensing devices exceeds a threshold. The queried sensor graph data is stored within nodes in the space graph, where the nodes are connected to each other using edges to create a spatial relationship between the object and the space. The object may be uploaded into an object graph associated with the spatial graph, wherein the object is digitized with each of the available sensors. This digital information may then be used, for example, to locate an object.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese invention patent application with the international application date of March 9, 2019, which entered the Chinese national phase on September 15, 2020, with national application number 201980019451.0 and named “Object and Position Tracking Using Images in Multiple Images”. Technical Field

[0003]

[0014] Embodiments of the present application relate to object and position tracking utilizing a graph of graphs. Background Art

[0004] Indoor positioning systems (IPS) are used to track the location of objects within physical environments such as office buildings, shopping malls, or warehouses. For example, an IPS may include using radio frequency identification devices (RFID) to track objects and triangulating the user's location based on the signal strength of a computing device to network devices such as Wi-Fi access points or Bluetooth beacons. Various sensors and other technologies can also be used for object tracking and indoor tracking, such as pedestrian dead reckoning (PDR), cameras, and global positioning systems (GPS). The corresponding IPS has characteristics that are beneficial for some scenarios and also have characteristics that are unfavorable for other scenarios. Summary of the Invention

[0005] A spatial graph instantiated on a wearable device, a remote server, or an edge computing unit periodically queries all available sensors associated with the sensing devices to create a node for each query, synthesizes the data queried from each sensor, and uses this data to track objects within the physical environment. Multiple sensing devices associated with the wearable device are used within the physical environment and are continuously turned on to collect data about the environment and build a sensor graph. Sensors and implemented technologies can include thermometers, GPS, accelerometers, inertial measurement units (IMUs), gyroscopes, microphones, cameras, Wi-Fi, and more. Each node in the spatial graph represents a collection of sensor graphs pulled from each sensor when the node is created. Edges are then used to spatially link each node, creating a spatial graph chain. The data within the nodes can be used to track objects within the physical environment. For example, a camera can identify a user's baseball glove, and a microphone can identify the sound of car keys being pressed against each other. The baseball glove and car keys can each be recorded in one or more nodes of the spatial graph, allowing the user to subsequently locate each object when querying the computing device storing the spatial graph.

[0006] There are two scenarios in which a wearable device, a remote server, or an edge computing unit performs spatial printing, associating sensor graph data with newly created nodes. One scenario occurs when data from one or more sensor graphs changes above a threshold level, and the second scenario occurs when a user uploads an object to the object graph. In both cases, various sensor graphs and sensing devices are queried to create a digital replica of the user's environment and objects.

[0007] A remote server or edge computing unit can be configured to link two different spatial print chains, which consist of a series of nodes and edges. For example, the first spatial print chain may have been previously uploaded, while the second spatial print chain may have been recently scanned and uploaded. An online linker running when the wearable device and the edge computing unit or remote server are online can find the relationship between the two chains and connect the two chains to create a logically traceable series of nodes. For example, when the wearable device is online, operating and transmitting sensor data to the edge computing unit, the edge computing unit can find connections between the spatial print chains to create an organic connection between the chains (for example, such as by finding connections between nodes of two independent chains). The online linker can operate locally at the wearable device or edge computing unit, remotely at the remote server, or at a combination of components. In addition, if there is no clear connection or relationship between the spatial print chains, an offline linker can be used. The offline linker can access the previously uploaded spatial print chains on the remote server and find similarities, relationships, or connections for connecting the two chains.

[0008] Over time, data in remote servers may become obsolete. For example, data may become obsolete when it is outdated, contains excessively large file sizes, is irrelevant, or is deemed unnecessary by users. In these cases, a garbage collector on the remote server can periodically search for and delete such data to conserve storage capacity.

[0009] Advantageously, the wearable device, edge computing unit, and remote server provide a system in which heterogeneous data is periodically collected for a physical environment and aggregated into a spatial graph. The spatial graph provides a representation of real-world objects and locations, and the real-world objects are digitized using a collection of various sensor data. Furthermore, the data can be synthesized to identify objects and logically connect them with respect to their locations in the real world. For example, the remote server can be configured to recognize real-world objects and make inferences about where the objects are located in the physical environment. The nodes and edges provide concepts and spatial relationships related to the physical world, and thus can provide actual or approximate locations for items when absolute locations are not available.

[0010] In this regard, the current configuration of indoor positioning systems (IPS) addresses issues with other technologies, such as GPS, which may not be very useful when the user is indoors. When the user is indoors, the configuration of spatial graphs, nodes, and edges can complement or completely replace GPS technology to provide accurate location and object tracking. Furthermore, this configuration can improve upon other IPS technologies where certain sensor data is sometimes unavailable, as querying all sensor graphs for corresponding nodes can provide important and useful data for object tracking and localization technologies.

[0011] This Summary is provided to introduce some concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the shortcomings noted in any part of this disclosure. It should be understood that the above-described subject matter may be implemented as a computer-controlled device, a computer process, a computing system, or an article of manufacture such as one or more computer-readable storage media. These and various other features will become apparent by reading the following detailed description and examining the associated drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Illustrative components associated with a wearable device are shown;

[0013] Figure 2 Shown from Figure 1 illustrative sensory information acquired by one or more sensory devices in;

[0014] Figure 3 An illustrative system architecture for a wearable device is shown;

[0015] Figure 4 An illustrative environment is shown in which a wearable device interacts with a remote server and an edge computing unit;

[0016] Figure 5 Schematic diagram showing the physical environment and different types of sensorgrams;

[0017] Figure 6 shows a schematic diagram of a spatial print with a sensorgram triggered by changes in Wi-Fi signal strength;

[0018] Figure 7 A schematic diagram with a spatial graph and nodes is shown;

[0019] Figure 8 A schematic diagram with an object graph is shown;

[0020] Figure 9A schematic diagram is shown in which a marked object has changed position;

[0021] Figure 10 A schematic diagram showing an object in which a set of keys are tags in an object graph;

[0022] Figure 11 A schematic diagram is shown in which an online linker links two spatial printing chains;

[0023] Figure 12 A schematic diagram showing the use of an offline linker and garbage collector;

[0024] Figures 13 to 15 An illustrative process performed by one or more wearable devices, edge computing units, or remote servers is shown;

[0025] Figure 16 is a simplified block diagram of an illustrative wearable device that may be used, in part, to implement mapping of an environment using a one-dimensional radiation sensor; and

[0026] Figure 17 is a simplified block diagram of an illustrative computer system that may be used, in part, to implement object and position tracking utilizing graphs in multiple graphs.

[0027] Like reference numerals indicate like elements in the drawings. Unless otherwise noted, elements are not drawn to scale. DETAILED DESCRIPTION

[0028] Figure 1 Shown are illustrative hardware components associated with wearable device 105. Wearable device 105 may include one or more of a variety of different computing devices configured to be easily and conveniently worn by a user to perform various functions and provide a beneficial user experience.

[0029] Wearable devices 105, including displays and input systems, can utilize various types of user interfaces. Some wearable devices can be operated through voice interaction as well as sensed gestures and / or user activity. Wearable devices can be configured to operate continuously in a non-intrusive manner and can also support functions and user experiences that rely on explicit user interaction or other user input. Wearable devices can also be configured with communication and networking interfaces to enable them to interact with local and remote users, devices, systems, services, and resources.

[0030] The wearable device 105 is configured to include various sensors, as described below, which can be configured to detect and / or measure motion, light, surface, temperature, humidity, position, altitude, and other parameters describing the user or device environment. Other sensors can be supported on the wearable device, located directly or indirectly on the user's body, to monitor parameters describing the user's physiological state, such as motion, position or posture of the body and / or body parts, pulse, skin temperature, etc. The sensors can operate in various combinations, so that in some cases, a given descriptive parameter can be derived by an appropriate system in the wearable device using data from more than one sensor, or by combining the sensor data with other information available to the device.

[0031] Figure 1 The hardware components of the wearable device depicted in FIG are not exhaustive and are used to illustrate various sensors that can generate data graphs and be incorporated into the spatial graph, as discussed in further detail below. The aggregation of heterogeneous sensors implemented helps develop a complete digital profile for an object or physical environment. The wearable device includes a processor 110, a memory 115, and a plurality of sensing devices (i.e., sensors) 120. However, the wearable device or computing device employed may use more or less Figure 1 The sensing devices listed in . The sensing devices may include a global positioning system (GPS) 125, a Wi-Fi transceiver 130, transceiver 135 , camera 140 , thermometer 145 , magnetometer 150 , microphone 155 , pedometer 160 , accelerometer 165 , gyroscope 170 , inertial measurement unit (IMU) 175 , proximity sensor 180 , barometer 185 , and light sensor 190 .

[0032] Figure 2 Illustrative types of sensory information 205 are shown that are derived by a wearable device or an external computing device (e.g., an edge computing unit or a remote server) based on data obtained from the sensing devices 120. For example, by synthesizing data obtained from one or more sensors, it can be determined whether the wearable device is located inside or outside 210. In addition, pedestrian dead reckoning (PDR) 215 principles can be used to obtain measurements using, for example, an IMU (such as a collection of accelerometers, gyroscopes, and magnetometers). PDR measurements can indicate the distance and direction traveled by the user, such as when a GPS device is not available (e.g., the user is indoors). The sensory information can indicate or approximate latitude and longitude 220 using, for example, a GPS device. Object recognition 225 can be performed using a camera, and temperature 230 can be obtained using a thermometer. Other types of sensory information not shown can be obtained using one or more sensing devices 120.

[0033] Figure 3An illustrative system architecture 300 for a wearable device 105 is shown. As shown, the wearable device 105 can be implemented using components that are worn, such as glasses, glasses, gloves, headbands, headphones, hats, helmets, earplugs, shoes, wristbands, and belts, and / or can be placed on the user's body using accessories such as neckbands, arm / leg bands, and lanyards. Wearable devices can also be incorporated into or embedded in clothing. In a typical implementation, the wearable device can be configured to provide a hands-free and eyes-free experience while the device operates on battery power to promote portability and mobility.

[0034] In simplified form, the architecture is conceptually arranged in layers and includes a hardware layer 315, an operating system (OS) layer 310, and an application layer 305. The hardware layer 315 provides an abstraction of the various hardware used by the wearable device 105 to the layers above it (e.g., network and radio hardware, etc.). In this illustrative example and as shown Figure 1 As shown, the hardware layer supports processor(s) 110, memory 115, sensory devices 120, and network connectivity components (e.g., Wi-Fi transceiver). Although not shown, other components are possible, such as input / output devices, display screens, etc.

[0035] In this illustrative example, the application layer 305 supports various applications 330, including an application configured as a spatial print 335, as discussed herein. The spatial print is configured to generate nodes within a spatial graph, pull data from each employed sensor, and associate the pulled data with the generated nodes. A node is created within the spatial graph when the currently detected data for a corresponding sensing device changes and exceeds a threshold. Each sensor can be configured with its own threshold parameters. Thus, for example, when PDR data indicates that a user has traveled a threshold of six feet, the wearable device can create a node within the spatial graph and execute a spatial print, where data from all sensors is collected and associated with the newly created node. In additional examples, a spatial print can be executed when a thermometer detects a temperature increase or decrease of five or ten degrees, a camera recognizes an object, a microphone recognizes an object's voice, and so on. In this regard, characteristics of certain objects can be pre-stored in memory so that sensors on the wearable device can subsequently compare and identify these objects.

[0036] Other applications not shown may include a web browser configured to provide connectivity to the World Wide Web, games, etc. Although Figure 3Only certain applications are described in

[15] , but wearable devices can use any number of applications, whether proprietary or developed by third parties. These applications are typically implemented using locally executed code. However, in some cases, these applications may rely on services and / or remote code execution provided by a remote server or other computing platform (such as a platform supported by a service provider or other cloud-based resource).

[0037] The OS layer 310 supports, among other operations, the management system 320 and operating applications 325, such as operating a space printing application 335 (as indicated by the arrow). The OS layer can interoperate with the application and hardware layers to perform various functions.

[0038] Figure 4 An exemplary environment 400 is shown in which a user 405 operates a wearable device 105, and the wearable device is configured to interoperate with external computing devices such as a remote server 420 and an edge computing unit 425 via a network 410. The network may include a local area network, a wide area network, the Internet, and the World Wide Web. In this regard, the edge computing unit may be located on-site at the wearable device, receive spatial map data, and then forward the data to the remote server. Additionally, alternative computing devices such as tablet computers and smart phones (not shown) may be utilized that are similarly configured as the wearable device. Additionally, as Figure 4 As shown, other users can register in a remote server or edge computing unit, where other users can utilize their own spatial map data, or the spatial map can be combined with one or more other users with the user and the user's consent.

[0039] The remote server and edge computing unit can interoperate with the extensibility client 415 of the spatial printing application 335, where the external computing device can construct a spatial map in whole or in part upon receiving collected data from the wearable device's sensors. The wearable device can, for example, collect data, where the extensibility client forwards the data to one of the external computing devices for processing. Thus, any discussion regarding spatial printing or spatial map construction can be local to the wearable device, external to the remote server or edge computing unit, or a combination thereof.

[0040] Figure 5 A schematic diagram illustrating a physical environment and different types of sensory graphs 505 is shown. For example, as a user 405 navigates a physical environment (in this example, an office building), various sensory graphs may generate data. Other types of physical environments include stores, houses, etc. A sensory graph may be data acquired directly from a single sensor or may be information acquired from one sensor or a combination of sensors ( Figure 2In this example, the sensory graph is about determining whether the user is inside or outside, temperature, PDR (e.g., distance traveled), camera (e.g., object recognition), changes in latitude and longitude, and Wi-Fi (e.g., changes in signal strength).

[0041] Figure 6 An illustrative spatial printing is shown that occurs due to a change in Wi-Fi signal strength. For example, user 405 is navigating an office building, and a threshold change in Wi-Fi signal strength may occur. As a result of this threshold change, the wearable device queries each sensor map and / or sensing device in real time, i.e., the data may be queried immediately or virtually immediately upon detecting a threshold change. The threshold change in Wi-Fi signal strength may be a change in decibel (dBm) strength, such as plus or minus -10 dBm. The sensor map may be a current representation of data from a particular sensing device or information of a type derived from the sensing device (e.g., PDR data).

[0042] After a query is executed against each sensor graph, a node is created at that location. Figure 7 335 function has been created. For example, each node may be created when sensor data associated with one or more sensors exceeds a set threshold. Thus, Figure 6 An example of Wi-Fi signal strength triggering a data pull is shown, but other sensor graphs also have threshold levels, such as PDR data indicating that a user has walked a threshold number of feet.

[0043] The spatial printing 335 function can be performed on one or a combination of the wearable device, a remote server, or an edge computing unit. For example, the wearable device can automatically perform a spatial printing when one of its sensors exceeds a threshold, in which it queries the sensory data. The wearable device can create a node or forward the data to one of the external computing devices to generate a node and spatial graph 715.

[0044] Node 705 is contained within spatial graph 715 and is Figure 7 As shown, the actual locations within the physical environment corresponding to the data pulled from the corresponding nodes. The spatial graph represents the collection of sensor graphs 505 and is a high-order metagraph that provides a deep understanding of how locations and objects relate to each other. By placing multiple sensory information in relationship nodes, the relationship between objects and their environment can be understood.

[0045] The nodes are connected with edges 725, which provide a logical and traceable connection between two or more nodes. Thus, for example, various sensor graph data can be tracked between nodes to help track the footsteps of a user so that these data can be used together. In contrast to absolute positioning, this helps to track the location of an object relative to a location, an object, and environmental characteristics. This may be useful for indoor positioning systems (IPS) when certain computing devices are technologically insufficient (such as GPS for indoor tracking). For example, if one or more nodes detect a Bluetooth connection from a smartphone when the temperature is 32° inside a supermarket, this may indicate that the user was previously near his smartphone in the frozen food section of the supermarket. This may indicate that the user is searching for a lost smartphone not only within the supermarket but also in the frozen food section within the supermarket.

[0046] Callout 720 shows exemplary data that can be pulled at a given node. In this example, the data indicates that the wearable device is located inside, the temperature is 65°, the PDR indicates that the user is two feet away from the previous node, the user's location is 40° North, 73° West, and the user's Wi-Fi strength is rated at -60 dBm. Although the spatial print can query every sensor graph 505, in alternative embodiments, only relevant sensor graphs can be queried. For example, if the wearable device determines that the user is inside an office building, temperature data may not be relevant and therefore not be pulled.

[0047] Figure 7 It is shown that the context provider 730 can communicate with a wearable device or other device that creates a spatial map (such as an edge computing unit or a remote server ( Figure 4 )). A context provider can be an external service that provides information about the user's environment. For example, an external service associated with server 735 can provide information about the office building in which the user is located and map the information. The data can include scheduling information within the building (e.g., a meeting in office 112), the number of people within the environment, details about locations within the environment (e.g., different food areas in a supermarket, restroom locations, the location of a particular individual's office), and address and company name information about the environment. In addition, sensory information about the environment can also be retrieved, such as temperature information connected to a heating or cooling system, the location of objects in the environment based on sensory data, etc. This information can supplement the sensory information derived from the sensory devices. The information listed in this document that can be obtained from external services is not exhaustive, and any number of services or databases can be used to supplement the spatial map data.

[0048] Figure 8An illustrative environment is shown in which an object graph 805 is used for a user or application developer. In this example, object A (represented by number 810) has been marked or added by a user. The object may have been input by user interaction with a user interface associated with a wearable device or a separate computing device such as a personal computer (PC) or a smart phone (not shown). When an object is marked, the object becomes a node within the object graph 805 and is stored in memory, which node can be associated with a spatial graph. The marking of the object also triggers a query to each sensing device or sensor graph. The query helps to determine the sensor data that makes up the object, thereby digitizing and creating a blueprint of an object with various heterogeneous sensing properties. For example, a camera can capture an image of an object, while a microphone can record the sound of the object (such as a key); these data are then stored in memory.

[0049] Figure 9 An illustrative environment is shown in which object A moves its position. In this case, the wearable device can update the position of the object by deleting the edge printed in the previous space ( Figure 8 ), and a new edge is created for the space where the object was last detected ( Figure 9 ).

[0050] Figure 10 An illustrative environment is shown in which real-world objects are used as objects for tracking. In this example, a set of keys 1005 has previously been entered into the object graph 805, and the user may have queried a wearable device or other computing device to locate his keys within the office. Node 1010 can be associated with the user's previous travels, such as earlier in the day, the day before, etc. When analyzing the spatial graph data, the wearable device can determine that the microphone on the wearable device detected keys ringing at or near the office 106. The wearable device or the user's personal device (e.g., a smartphone) can display the inferred location or node of the keys to the user. Other relevant data that can support the inferred location or node is room temperature (e.g., the typical temperature of a room or office) and images that may have been captured (e.g., the physical environment and objects of the office). The user can then begin searching for the travel of his keys near the office 106.

[0051] Because the spatial graph comprises multiple nodes and spatial prints of data, the wearable device can utilize a series of nodes and edges, or a chain of spatial prints, to create reasoning and logical paths or directions to the key. Therefore, once created, the spatial print chain and spatial graph can generally be traversed. For example, the wearable device can provide directions to the user in the reverse direction of the generated spatial print chain to track the user's route to the location where the key was lost and may be located. Each node in the spatial graph is associated with specific sensory data. Therefore, as the user navigates the office, these directions can be tracked and provide directions to that location for the user. For example, as the user continues to follow these directions and the wearable device continues to sense subsequent data, the previous data associated with the previously existing node can verify that the subsequent data matches the previous data. This allows the user to traverse the nodes in the spatial graph and receive subsequent, verifiable instructions.

[0052] As an example, the wearable device may recognize that the temperature has dropped by five degrees at node 1015. Therefore, the wearable device verifies subsequent sensor data against previous sensor data and may also instruct the user to continue walking straight. Other examples of data at nodes that allow the user to traverse the spatial graph include cameras that identify specific objects at that location, proximity sensors that detect confined spaces, light sensors that detect too much or too little light, and the like.

[0053] Figure 11 An illustrative environment is shown in which an online linker 1105 links spatial print chains as they are constructed. For example, the spatial map may be stored locally at a wearable device or edge computing unit, or remotely at a remote server ( Figure 4 ). When the wearable device enters an online or connected state again, in which the wearable device will run and create a spatial print, it can search existing spatial print chains (e.g., connected nodes and edges) to find connections with the spatial print chain being developed. This allows spatial print chains to be organically connected as they are encountered and generated. The online linker function can be executed at the wearable device, the edge computing unit, the remote server, or a combination of these components.

[0054] For example, connections can be made based on relationships or similarities between nodes or spatially printed chains. For example, if two nodes or chains are placed next to each other and created at a nearby time, a connection can be created. Figure 11 A first spatial print chain 1120 and a second spatial print chain 1125 are shown. In this example, the node near object B at connection 1115 is a logical connection between the two chains. The similarity of the connections allowed between chains can have minimum or maximum threshold requirements, such as being created within a certain time period (e.g., five minutes) or being located within a threshold distance of each other (e.g., ten feet).

[0055] Figure 12 A detailed illustrative detailed environment is shown, which shows other objects in the object graph, an offline linker 1205, and a garbage collector 1210. Sometimes, spatial print chains may not be connected, such as when the distance between individual chains is too large, or when there is no direct link. In this case, the remote server can search for connections between recently uploaded data and previously uploaded data. The offline linker can find, for example, spatial print chains that have nodes that are spatially close to each other and include one or more similarities in the spatial graph data. For example, time, location, and other relevant sensor data can help locate logical connections between nodes and edges.

[0056] The garbage collector 1210 can identify and delete data, nodes, edges, etc. that become stale over time. For example, stale data can be identified based on its age (e.g., too old to be useful), size (e.g., the file is too large and takes up too much memory space), usefulness (e.g., a user identified certain types or timeframes of data as irrelevant), etc.

[0057] Figure 13 1300 is a flow chart of an illustrative method 1300 in which a computing device associates unique sensory data with a node. Unless otherwise specified, the methods or steps shown in the flow chart and described in the accompanying text are not limited to a particular order or sequence. In addition, certain methods or steps may occur or be performed simultaneously, and not all methods or steps must be performed in a given implementation (depending on the requirements of such implementation), and some methods or steps may be utilized optionally.

[0058] In step 1305, a first set of sensory data is received, wherein the data includes a plurality of sensorgrams, each sensorgram associated with different sensory information. In step 1310, a first node is created in the spatial graph. In step 1315, the first set of sensory data is associated with the first node.

[0059] Figure 14is a flow chart of an illustrative method 1400 in which a computing device uses a sensor graph for a spatial graph. In step 1405, threshold levels are assigned for sensory data, wherein the sensory data is derived from the sensory devices. In step 1410, a state of the sensory data associated with each sensory device is monitored. In step 1415, a state log is triggered for each sensory device when one or more of the assigned threshold levels are exceeded. The state log may include, for example, a data pull of current data associated with various sensor devices or sensor graphs. In step 1420, each sensor graph in the constructed sensor graph is collected into a spatial graph. The spatial graph may, for example, assign a state log of a sensor device to a node in the spatial graph.

[0060] Figure 15 15 is a flow chart of an illustrative method 1500 in which a computing device creates a blueprint of labeled objects. In step 1505, a series of nodes and edges are established within a spatial graph, where edges connect nodes and nodes are associated with sensor data. In step 1510, objects are labeled for tracking within the spatial graph or an object graph associated with the spatial graph. In step 1515, sensors controllable by the computing device are activated to capture the current sensor data blueprint. For example, each sensing device can be queried to pull data from each sensing device. In step 1520, the object is associated with the most recently created node and the sensor data for that node.

[0061] Figure 16 An illustrative architecture 1600 of a device capable of executing the various components described herein to provide current user and device authentication to a network application is shown. Figure 16 The illustrated architecture 1600 illustrates an architecture that may be applicable to a wearable device, a server computer, a mobile phone, a PDA, a smartphone, a desktop computer, a netbook computer, a tablet computer, a GPS device, a game console, and / or a portable computer. The architecture 1600 may be used to implement any aspect of the components presented herein.

[0062] Figure 16The illustrated architecture 1600 includes one or more processors 1602 (e.g., central processing units, graphics processing units, etc.), system memory 1604 (including RAM (random access memory) 1606 and ROM (read-only memory) 1608), and a system bus 1610 that operatively and functionally couples the components of the architecture 1600. A basic input / output system, containing basic routines that help transfer information between elements in the architecture 1600, such as during startup, is typically stored in ROM 1608. The architecture 1600 also includes a mass storage device 1612 for storing software code used to implement applications, file systems, and operating systems, or other computer-executable code. The mass storage device 1612 is connected to the processor 1602 through a mass storage controller (not shown) connected to the bus 1610. The mass storage device 1612 and its associated computer-readable storage media provide non-volatile storage for the architecture 1600. Although the descriptions of computer-readable storage media contained herein refer to mass storage devices, such as hard disks or CD-ROM drives, those skilled in the art will appreciate that computer-readable storage media may be any available storage media that can be accessed by architecture 1600 .

[0063] The architecture 1600 also supports a sensor package 1630 that includes one or more sensors or components that are configured to detect parameters describing the environment and / or detect parameters describing the device user, or a combination thereof. For example, for a wearable computing device, the sensors can be positioned directly or indirectly on the user's body. The sensors can be configured to operate continuously or periodically, and typically operate in a hands-free and / or eyes-free manner. The architecture also supports a power source and / or battery component (collectively identified by reference numeral 1615). For example, in wearable device applications, one or more batteries or power packs can be rechargeable or replaceable to facilitate portability and mobility.

[0064] By way of example, and not limitation, computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable media include, but are not limited to, RAM, ROM, EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory or other solid-state memory technology, CD-ROM, DVD, HD-DVD (high-definition DVD), Blu-ray or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the architecture 1600.

[0065] According to various embodiments, the architecture 1600 can operate in a networked environment using logical connections to remote computers over a network. The architecture 1600 can be connected to the network via a network interface unit 1616 connected to the bus 1610. It should be understood that the network interface unit 1616 can also be used to connect to other types of networks and remote computer systems. The architecture 1600 can also include an input / output controller 1618 for receiving and processing input from many other devices, including a keyboard, mouse, touchpad, touch screen, control devices (such as buttons and switches), or an electronic pen ( Figure 16 Similarly, the input / output controller 1618 may provide output to a display screen, user interface, printer, or other type of output device (not shown in FIG. Figure 16 (not shown in the figure).

[0066] The architecture 1600 may include a speech recognition unit (not shown) to facilitate user interaction via voice commands, a natural language interface, or by interacting with a personal digital assistant (such as provided by Microsoft Corporation). The architecture 1600 may include a gesture recognition unit (not shown) to facilitate user interaction with a device supporting the architecture through sensed gestures, motions, and / or other sensed inputs.

[0067] It should be understood that the software components described herein, when loaded into the processor 1602 and executed, can convert the processor 1602 and the entire architecture 1600 from a general-purpose computing system into a special-purpose computing system customized to facilitate the functions described herein. The processor 1602 can be composed of any number of transistors or other discrete circuit elements (which can individually or collectively exhibit any number of states). More specifically, the processor 1602 can operate as a finite state machine in response to the executable instructions contained in the software modules disclosed herein. These computer-executable instructions can convert the processor 1602 by specifying how the processor 1602 transitions between states, thereby converting the transistors or other discrete hardware elements that constitute the processor 1602.

[0068] Encoding the software modules presented herein may also transform the physical structure of the computer storage readable medium presented herein. In different implementations of this specification, the specific transformation of the physical structure may depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the computer storage readable medium, whether the computer storage readable medium is characterized as primary storage or secondary storage, etc. For example, if the computer storage readable medium is implemented as a semiconductor-based memory, the software disclosed herein may be encoded on the computer storage readable medium by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. The software may also transform the physical state of these components in order to store data thereon.

[0069] As another example, the computer-readable storage medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when software is encoded therein, the software proposed herein can transform the physical state of the magnetic or optical medium. These transformations can include changing the magnetic properties of specific locations within a given magnetic medium. These transformations can also include changing the physical features or characteristics of specific locations within a given optical medium to change the optical properties of these locations. Other transformations of physical media are possible without departing from the scope and spirit of this specification, and the foregoing examples are provided only to facilitate this discussion.

[0070] In view of the above, it should be understood that many types of physical transformations occur in the architecture 1600 in order to store and execute the software components presented herein. It should also be understood that the architecture 1600 may include other types of computing devices, including wearable devices, handheld computers, embedded computer systems, smart phones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the architecture 1600 may not include Figure 16 All components shown may include Figure 16 Other components not explicitly shown in the figure may also be used in conjunction with Figure 16 The architecture shown is a completely different architecture.

[0071] Figure 1717 is a simplified block diagram of an illustrative computer system 1700, such as a wearable device, with which the object and location tracking of the present invention, utilizing the various figures, can be implemented. Although wearable devices are discussed herein, other computing devices with similar configurations as discussed herein may also be used, including smartphones, tablet computing devices, personal computers (PCs), laptop computers, and the like. Computer system 1700 includes a processor 1705, system memory 1711, and a system bus 1714, which couples various system components, including system memory 1711, to processor 1705. System bus 1714 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, or a local bus, using any of a variety of bus architectures. System memory 1711 includes read-only memory (ROM) 1717 and random access memory (RAM) 1721. A basic input / output system (BIOS) 1725, containing the basic routines that help transfer information between elements within computer system 1700, such as during startup, is stored in ROM 1717. Computer system 1700 may also include a hard disk drive 1728 for reading from and writing to an internal hard disk (not shown); a magnetic disk drive 1730 for reading from or writing to a removable magnetic disk 1733 (e.g., a floppy disk); and an optical disk drive 1738 for reading from or writing to a removable optical disk 1743, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or other optical media. Hard disk drive 1728, magnetic disk drive 1730, and optical disk drive 1738 are connected to system bus 1714 via hard disk drive interface 1746, magnetic disk drive interface 1749, and optical drive interface 1752, respectively. The drives and their associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 1700. Although the illustrative example includes a hard disk, a removable disk 1733, and a removable optical disk 1743, in some applications of the object and position tracking of the present invention utilizing the diagrams in the various figures, other types of computer-readable storage media that can store data that can be accessed by a computer, such as magnetic tape, flash memory cards, digital video disks, data cartridges, random access memory (RAM), read-only memory (ROM), etc., may also be used. Additionally, as used herein, the term computer-readable storage medium includes one or more instances of a type of medium (e.g., one or more disks, one or more CDs, etc.). For purposes of this specification and claims, the phrase "computer-readable storage medium" and its variants are non-transitory and do not include waves, signals, and / or other transient and / or intangible communication media.

[0072] A number of program modules may be stored on a hard disk, magnetic disk, optical disk, ROM 1717, or RAM 1721, including an operating system 1755, one or more application programs 1757, other program modules 1760, and program data 1763. A user may enter commands and information into the computer system 1700 through input devices such as a keyboard 1766 and a pointing device 1768 such as a mouse. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, trackball, touch pad, touch screen, touch-sensitive device, voice command module or device, user motion or user gesture capture device, and the like. These and other input devices are typically connected to the processor 1705 via a serial port interface 1771 coupled to the system bus 1714, but may also be connected through other interfaces such as a parallel port, game port, or universal serial bus (USB). A monitor 1773 or other type of display device is also connected to the system bus 1714 via an interface such as a video adapter 1775. In addition to the monitor 1773, wearable devices and personal computers typically may include other peripheral output devices (not shown), such as speakers and printers. Figure 17 The illustrative example shown also includes a host adapter 1778 , a small computer system interface (SCSI) bus 1783 , and external storage devices 1776 connected to the SCSI bus 1783 .

[0073] The computer system 1700 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 1788. The remote computer 1788 can be selected as a personal computer, server, router, network PC, peer device or other public network node and typically includes many or all of the elements described above with respect to the computer system 1700, although in Figure 17 Only a single representative remote memory / storage device 1790 is shown. Figure 17 The devices depicted in 179 include a local area network (LAN) 1793 and a wide area network (WAN) 1795. Such networking environments are commonly deployed in, for example, offices, enterprise-wide computer networks, intranets, and the Internet.

[0074] When used in a LAN networking environment, the computer system 1700 is connected to the local area network 1793 through a network interface or adapter 1796. When used in a WAN networking environment, the computer system 1700 typically includes a broadband modem 1798, a network gateway, or other means for establishing communications over the wide area network 1795, such as the Internet. The broadband modem 1798 (which may be internal or external) is connected to the system bus 1714 via the serial port interface 1771. In a network environment, program modules related to the computer system 1700, or portions thereof, may be stored in the remote memory storage device 1790. Note that Figure 17 The network connections shown are illustrative and other means of establishing a communications link between the computers may be used depending on the particular requirements of the object and position tracking application of the present invention utilizing the various figures.

[0075] Various exemplary embodiments of object and position tracking of the present invention utilizing graphs within multiple graphs are now presented by way of illustration and not as an exhaustive list of all embodiments. One example includes a computing device configured to collect and synthesize sensory data to track an object, the computing device comprising: one or more processors; and a memory configured to store a spatial graph and sensory data associated with nodes in the spatial graph, the memory having computer-readable instructions that, when executed by the one or more processors, cause the computing device to: receive a first set of sensory data, wherein the first set includes a plurality of sensory graphs, each sensory graph associated with different sensory information; create a first node in the spatial graph; and associate the first set of sensory data with the first node.

[0076] In another example, the computer-readable instructions further cause the computing device to: receive a subsequent set of sensory data; create a subsequent node in the spatial graph; associate the subsequent set of sensory data with the subsequent node; and create an edge in the spatial graph linking the first node to the subsequent node, wherein the first node becomes a pre-existing node of the subsequent node. In another example, the computer-readable instructions further cause the computing device to: create a series of nodes and edges to create a spatial print chain; locate an object based on sensory data associated with one or more nodes in the series of nodes; and create directions for a user to traverse the edges and nodes within the spatial print chain, wherein the directions lead the user to the object, wherein current and previous sensory data associated with the nodes are used to create the directions. In another example, the set of sensory data is received when one or more measurements in the sensory graph exceed a threshold. In another example, when at least one of the measurements in the sensory graph exceeds the threshold, each sensory device is queried to generate sensory data for the new node. In another example, the sensory data is received from a user's computing device or edge computing unit configured to store nodes and sensory data for one or more wearable devices. In another example, the computer-readable instructions further cause the computing device to communicate with an external context provider that provides additional context and sensory information for the physical environment, where the context or sensory information includes at least a map of the physical environment, the number of people occupying the space, scheduled events within the physical environment, details about rooms in the physical environment, or sensory information to be associated with a node. In another example, the computer-readable instructions further cause the computing device to construct a spatial print chain using a set of nodes and edges connecting the nodes, where each spatial print chain includes its own set of nodes and edges; and to connect two different spatial print chains to form a single spatial print chain. In another example, the two different spatial print chains are connected when the offline computing device establishes an internet connection and transmits the spatial print chains to the computing device. In another example, the two different spatial print chains are connected when the two chains share a relationship with each other.

[0077] Another example includes a method for generating sensory data for a spatial graph, performed by a local computing device within a physical environment, the method comprising: assigning threshold levels to sensory data developed by each sensory device associated with the local computing device; monitoring the status of the sensory data associated with each sensory device; triggering a status log for each sensory device in the sensory devices when one or more of the assigned threshold levels associated with the corresponding sensory devices are exceeded based on the status monitoring, wherein the local computing device constructs a sensor graph based on the current sensory data for each sensory device; and collecting each sensor graph in the constructed sensor graph into a spatial graph, wherein the spatial graph assigns the status log for the sensor device to a node in the spatial graph.

[0078] In another example, the assigned threshold level for each sensing device represents a change in sensor state. In another example, the sensing device includes at least one of: a global positioning system (GPS), Wi-Fi, cameras, thermometers, magnetometers, pedometers, accelerometers, gyroscopes, inertial measurement units (IMUs), microphones, proximity sensors, barometers, and light sensors. In another example, the method further includes: receiving input at a user interface (UI) of a local computing device for marking an object using a spatial graph; and in response to the received input, querying a sensing device for sensor data associated with the marked object to generate a data blueprint for the marked object. In another example, the method further includes: receiving subsequent input for locating the marked object; locating an indication at one or more nodes representing the marked object within the spatial graph; and outputting the location of the one or more nodes or the located indication associated with the one or more nodes on the UI. In another example, the indication includes a correspondence between the original sensor data from the time the object was marked and subsequent data in the spatial graph.

[0079] Another example includes one or more hardware-based computer-readable storage devices storing instructions that, when executed by one or more processors disposed in a computing device, cause the computing device to: establish a series of nodes and edges within a spatial graph, wherein the edges connect the nodes and the nodes are associated with sensor data, the sensor data being captured concurrently with the creation of each respective node; label objects for tracking within the spatial graph or an object graph associated with the spatial graph; query sensors accessible to the computing device to capture a current sensor data blueprint, wherein the sensors include different types of sensors; and associate the object with a most recently created node and the sensor data for the node.

[0080] In another example, the instructions further cause the computing device to: create additional nodes as the wearable device navigates the physical area; detect a new location of an object using sensor data from the created nodes; and upon detecting the new location of the object, delete the object's association with the previous node and create a new association between the object and the new node. In another example, the instructions further cause the computing device to invalidate stale nodes in the spatial graph, where a stale node is a node that is old with respect to time or contains irrelevant sensor data. In another example, the instructions further cause the computing device to receive a request for the location of an object; and use the sensor data blueprint for nodes within the spatial graph to provide directions for traversing nodes within the spatial graph leading to the object.

[0081] The subject matter described above is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example embodiments and applications shown and described, and without departing from the true spirit and scope of the invention as set forth in the appended claims.

Claims

1. A computing device operating in a physical environment and configured to generate a spatial print chain, comprising: one or more processors; as well as one or more hardware-based memory devices having computer-readable instructions that, when executed by the one or more processors, cause the computing device to: receiving sensory data from one or more sensors operatively coupled to the computing device; creating nodes within a spatial graph using the received sensory data; as well as A location is associated with the created node using the received sensory data. 2 . The computing device of claim 1 , wherein the location associated with the node is utilized using a map of the physical environment. 3 . The computing device of claim 2 , wherein the map of the physical environment is obtained from a remote computing device. 4 . The computing device of claim 2 , wherein the instructions further cause the computing device to associate a physical object in the physical environment with the location. 5 . The computing device of claim 4 , wherein the instructions further cause the computing device to use the physical object's association with the location to provide directions to the physical object or the location of the physical object.

6. The computing device of claim 1, wherein the sensor comprises at least one of: a global positioning system (GPS), Wi-Fi, Camera, thermometer, magnetometer, pedometer, accelerometer, gyroscope, inertial measurement unit (IMU), microphone, proximity sensor, barometer, and light sensor.

7. A method for establishing a spatial printing chain, performed by a local computing device in a physical environment, comprising: Connecting to a network and thereby becoming online; When entering the online state, an existing spatial printing chain is received from a remote computing device, wherein the spatial printing chain includes connected nodes and edges, and sensor data is associated with each node.

8. The method of claim 7, wherein the received spatial print chain is connected to a spatial print chain generated by the local computing device.

9. The method of claim 8, wherein the spatial print chains are interconnected in response to the spatial print chains sharing one or more of temporal, physical, or sensory proximity with each other.

10. The method of claim 7, wherein the nodes of the spatial print chain correspond to locations in the physical environment.

11. The method according to claim 7, further comprising: determining whether one or more nodes or one or more edges in the spatial print chain are stale; as well as The one or more nodes or the one or more edges in the spatial print chain that have been determined to be obsolete are deleted.

12. The method of claim 11, wherein a node or an edge is stale based on an age, size, or usefulness of the corresponding node or edge indicated by a user.

13. One or more hardware-based non-transitory computer-readable memory devices storing instructions that, when executed by one or more processors disposed in a computing device, cause the computing device to: incorporating sensory data from one or more sensors into a spatial map, the spatial map representing aspects of a physical environment; detecting physical objects in the physical environment using the sensory data; and The physical object is associated with a location in the physical environment using the sensory data.

14. The one or more hardware-based non-transitory computer-readable memory devices of claim 13, wherein the instructions when executed further cause the computing device to associate the detected physical objects with a plurality of locations in the physical environment.

15. One or more hardware-based non-transitory computer-readable memory devices according to claim 13, wherein the instructions when executed further cause the computing device to associate the received sensory data with nodes in a spatial graph, and wherein physical locations are associated with the nodes in the spatial graph, and the physical locations are used to identify and associate the locations of the physical objects in the physical environment.

16. One or more hardware-based non-volatile computer-readable memory devices according to claim 13, wherein the instructions executed further cause the computing device to establish a series of nodes and edges within the spatial graph, wherein the edges connect the nodes and the nodes are associated with sensor data, and the sensor data is captured simultaneously with the creation of each corresponding node.

17. The one or more hardware-based non-transitory computer-readable memory devices of claim 13, wherein the instructions when executed further cause the computing device to label the objects in the spatial graph or an object graph associated with the spatial graph.

18. The one or more hardware-based non-transitory computer-readable memory devices of claim 13, wherein the instructions when executed further cause the computing device to detect a new location for the object using the sensor data for the created node.

19. The one or more hardware-based non-transitory computer-readable memory devices of claim 18, wherein the instructions when executed further cause the computing device to: Upon detecting the new position for the object, the association from the object to the previous node is deleted and a new association of the object to the new node is created.

20. The one or more hardware-based non-transitory computer-readable memory devices of claim 18, wherein the instructions when executed further cause the computing device to query sensors accessible by the computing device to capture a blueprint of current sensor data, wherein the sensors include different types of sensors.

Images