3D item-level localization and imaging using radio waves

Abstract

A system and method are disclosed for tracking a user's activity with respect to products within an environment. In one embodiment, a processor detects when a user enters the environment and responsively uniquely identifies the user. The processor determines that the user has interacted with one of a plurality of products within the environment and responsively updates a profile of the user with an indication of the product. The processor determines whether the user is attempting to exit the environment and, in response to determining that the user is attempting to exit the environment, prompts the user to confirm that the user intends to remove the product from the environment based on the updated profile. The processor receives input from the user confirming that the user intends to remove the product from the environment and responsively further updates the profile based on the input.

Description

[Technical field]

[0001] The present disclosure relates to the field of three-dimensional localization and imaging, and specifically applies to the process of monitoring a user and tangible items that interact with the user. [Background technology]

[0002] Related technology systems monitor users and tangible items (e.g., in retail environments) through complex systems of quick response (QR) codes, computer vision, and custom hardware (including placement of both emitters and receivers). However, these systems are expensive and laborious, and do not provide complete imaging of the three-dimensional environment, at least because they are limited to the line of sight of the computer vision sensor.

[0003] Described herein are systems and methods that provide complete imaging of a three-dimensional (3D) environment. Complete imaging is made possible by the implementation of a system that can image through walls and other opaque objects that obstruct the line of sight. Specifically, the disclosed systems and methods include deploying a signal receiver that receives signals emanating from, for example, WiFi access points or cell phones. A processor synthesizes the signals received by the receiver to image items or people, for example, in a retail environment. As items or people move through the environment, the processor detects changes in the image based on changes in signal characteristics, such as absorption, refraction, and reflection, caused by the items or people, such that changes throughout the environment can be monitored and tracked, even when there is an obstruction in the line of sight between the receiver and the items or people.

[0004] In an exemplary embodiment, the processor detects when a user enters the environment and responsively uniquely identifies the user (e.g., indicative of the user's height, clothing, etc., based on signal absorption characteristics by the user). The processor determines that the user is interacting with one of a plurality of products in the environment (e.g., based on determining that the user has taken a product from a shelf based on user and product absorption patterns over time as the user moves around) and responsively updates a profile of the user with an indication of the product (e.g., indicating that the product is in the user's shopping cart or basket). The processor determines whether the user is about to exit the environment and, in response to determining that the user is about to exit the environment, prompts the user (e.g., at a kiosk or using a mobile device application) to confirm that the user intends to take a product from the environment based on the updated profile. The processor receives input from the user confirming that the user intends to take a product from the environment and responsively further updates the profile based on the input. [Brief description of the drawings]

[0005] The disclosed embodiments have other advantages and features that will become more readily apparent from the detailed description, the appended claims, and the accompanying figures (or drawings). A brief introduction of the figures is provided below.

[0006] [Figure 1] 1 illustrates an embodiment of an environment in which people and items are tracked by three-dimensional images obtained from a signal receiver. [Diagram 2] 1 illustrates one embodiment of a system configuration for processing data obtained from a signal receiver. [Diagram 3] 1 illustrates one embodiment of a server used to process data obtained from a signal receiver for tracking people and items. [Figure 4]FIG. 1 is a block diagram illustrating components of an example machine that can read instructions from a machine-readable medium and execute them on a processor (or controller). [Diagram 5] 5 is a flowchart illustrating an example process for processing data obtained from a signal receiver for tracking people and items. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The figures and the following description relate to preferred embodiments for purposes of illustration only. It should be noted from the following discussion that alternative embodiments of the structures and methods disclosed herein are readily discernible as viable alternatives that may be employed without departing from the principles of what is claimed.

[0008] Reference will now be made in detail to several embodiments illustrated in the accompanying figures. It should be noted that, wherever possible, similar or similar reference numbers may be used in the figures to indicate similar or similar functionality. The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0009] Configuration overview An embodiment of the disclosed system, method, and computer readable storage medium including three-dimensional localization and imaging techniques uses radio waves to perform localization and imaging, for example, in a retail environment. FIG. 1 illustrates an embodiment of an environment in which people and items are tracked by way of three-dimensional imaging obtained from a signal receiver. Environment 100 is any environment in which people and / or items are tracked or monitored. Although environment 100 is often described herein as a retail environment, this description is for illustrative purposes only and any environment that includes people and tangible items is within the scope of environment 100.

[0010] The transmitters 116 transmit signals throughout the environment 100. The transmitters 116 may include transmitters located anywhere within the environment 100 or within signal range of the environment 100. The transmitters 116 may be located within or near the environment 100 for purposes of tracking items and / or people within the environment 100. Alternatively, the transmitters 116 are or include transmitters used for purposes other than tracking items and / or people within the environment 100. For example, the transmitters 116 may include transmitters from devices carried by people within the environment 100 (e.g., a smartphone device carried by the person 112), permanent or semi-permanent devices installed within the environment 100 (e.g., a wireless internet router or gateway intended to provide WiFi access to people within the environment 100), and other transmitters that generate signals that are detected by the receivers of the receiver array 114.

[0011] The receiver array 114 includes receivers that detect signals from the transmitters 116. Although the receiver array 114 is depicted as multiple receivers clustered at one location in the environment 100, the receivers may be dispersed throughout the environment 100, outside the environment 100, or near the environment 100. The receivers in the receiver array 114 relay the detected signals to a processor, which processes the data (as described in more detail below) and synthesizes image data and location data therefrom. Optionally, a reference receiver 108 is placed at a known location within the environment 100. The reference receiver 108 is used to calibrate the receivers in the receiver array 114 to simplify and ensure the accuracy of the image and location data. In one embodiment, the reference receiver 108 is calibrated by placing the reference receiver 108 at a known distance from the transmitter 116. In another embodiment, the reference receiver 108 is calibrated by a Received Signal Strength Indicator (RSSI) calculation method used to estimate the distance of the reference receiver 108 from the transmitter 116. The reference receiver 108 may be used as a baseline or correction signal for the transmitter 116 after it has been calibrated. For example, the signal from the transmitter 116 may be divided by the reference signal received by the reference receiver 108 to calibrate and / or standardize the signal from the transmitter 116. The reference receiver 108 may be periodically recalibrated to maintain accuracy as the environment 100 may change over time. The receivers of the receiver array 114 and the reference receiver 108 may be configured to limit detection of signals of a given channel or frequency, or a subpart thereof, or to limit relay for imaging / localization purposes to a processor. The processor is configured to process the waveform in a manner similar to light, and the processor creates a visual mapping based on where and how the signal bands detected by the receivers coalesce. The coalescence of the signal bands depends on how the signals are absorbed, reflected, refracted, or otherwise manipulated by product or person properties (as described below).

[0012] The processor may determine a measure of absorption in the environment 100 by tracking the amount of signal emitted by the transmitter at any time and determining how much of that signal is returned to the transmitter 116 and / or receiver array 114 and / or reference receiver 108 and detected again. Based on these tracked parameters, the processor may determine the absorption rate of any emitted waveform. The processor may infer properties about the medium that any waveform hits based on a known mapping of absorption rates of waveform types to media. For example, the processor may look up a database with entries that correlate if the absorption rate of a waveform is X, then the medium that the waveform hits is a liquid, a solid, has a certain level of density, has certain color attributes, etc.

[0013] The processor may also determine the refractive properties of the medium struck by the waveform by measuring the phase shift, deflection, and other effects on the expected path of the signal transmitted by the transmitter 116 before reaching the reference receiver 108. The processor may determine the geometric properties of the medium struck by a given waveform based on its refractive properties. Such refractive properties may also be indicative of the physical properties of the waveform, as described above for measuring absorption. The processor may make these determinations by observing the refractive properties, such as deflection, phase shift, and referencing the properties, as well as a database that maps waveform properties to geometric and / or physical properties. The processor may also utilize the refractive properties to track the velocities of different objects, such as by using the Doppler effect. For example, if the processor detects a greater phase shift at the receivers of the receiver array 114 than at the transmitters of the transmitter 116, the processor may determine (e.g., based on a database entry) that an individual is running through the environment 100.

[0014] The processor may determine the reflective properties of the medium impinging on the waveform. Reflective properties are characteristics that indicate how a material reflects a particular waveform. These properties may allow the processor to infer color (e.g., some colors reflect the radio waveform while others do not). From the reflective properties, the processor may also infer properties of the material (e.g., a high reflectivity may indicate that the material is metallic, since metals are highly reflective). Additionally, the processor may use the reflectivity to identify anomalies or potential errors. For example, a high level of reflectivity may indicate that someone is attempting to steal a product from the environment 100 and is using a highly reflective material to block the radio waveform and hide their actions.

[0015] The database may relate the absorption, refraction, and reflection properties. Such a database may correlate separately for people and products, with entries indicating that the absorption, refraction, and / or reflection properties detected together for a given waveform indicate a particular property of the person and / or product. Such a database may relate these properties to patterns of one or more signals, where a pattern refers to a combination of properties (e.g., absorption, refraction, and / or reflection properties). In one embodiment, rather than using database correlations, a model (e.g., a machine learning model such as a neural network) may be trained to receive the absorption, refraction, and / or reflection properties as inputs and may output a likely identification or attribute of the object or person based on those inputs.

[0016] 3D localization and imaging of products and customers using signal waves Continuing with the exemplary embodiment in which the environment 100 is a retail environment, a processor of a server performing 3D localization and imaging of the environment 100 may detect a person 112 entering the environment 100 via the entrance 102. The processor may detect the entry of the person 112 by determining that the entity has crossed the plane of the entrance 102 and determining that the entity is human based on the imaging of the entity (e.g., based on absorption, refraction, or reflection characteristics compared to a database). The processor may generate a signature of the person 112 based on an observed signal pattern that corresponds to a uniquely identifiable characteristic of the person 112. For example, the signal pattern for generating the unique signature may depend on signal characteristics such as absorption, refraction, and / or reflection that correspond to the unique characteristics of the person (e.g., height, size, and clothing characteristics detected by the method of imaging the entity). The unique signature is stored in a database in memory (e.g., long-term permanent storage). Additionally, the unique signature may include a unique identifier or may be hashed to generate the unique identifier. Based on this signature, the processor may track the movements of the person 112 as the person 112 moves through the environment 100 while maintaining the knowledge that the tracked person is the person 112 because these characteristics of the person 112 remain constant.

[0017] The processor may further positively identify the person 112 when the person 112 enters the environment 100 through the entrance 102. In one embodiment, the processor generates a signature for the person 112 entering the store. The signature is compared to signatures stored in a database. If a match is found in the database for the signature, profile data corresponding to the person 112 (profile data and its management are described below with reference to FIG. 3) is searched for. In some embodiments, the processor may detect the person 112 as well as a mobile device carried by the person 112. Based on the location of the mobile device or based on unique attributes of the signal itself, the processor may detect that a particular signal is from the mobile device of the person 112. The processor may then determine the identity of the person based on the identification information embedded in the signal. For example, if the mobile device has an app installed designed to communicate with the environment 100 (e.g., an app described above and below herein), the app itself may cause the mobile device to emit a signal that identifies the person 112. In some embodiments, computer vision may be used to determine or verify the identity of the person 112. A user's profile may indicate the address of the user's mobile device.

[0018] If the signature is not known, a record may be created for the person 112 to store the signature, and the signature may be stored in a database entry. The stored signature may be given an identifier so that as additional information corresponding to the person 112 is received, it can be added to the record with that signature. For example, if the signature of the person 112 is unknown to the server, the person 112 is prompted to enter identifying information, which is added to the record with the signature. As another example, the observed behavior of the person 112 (e.g., purchasing habits) may be stored in the record along with the signature (and identifier), regardless of whether the person 112 is anonymous.

[0019] In an exemplary embodiment, input of such identifying information may be a prerequisite for passing through or proceeding after passing through a barrier at the entrance 102 of the environment 100. The prompt may occur on a personal device carried by the person 112 (e.g., by an app required to be installed on the personal device as a prerequisite for entering the environment 100) or may occur at a kiosk at or near the entrance 102. The identifying information may include, for example, information sufficient to issue a bill (or charge) to the person 112 for the products 110 that the person 112 removed from the environment 100. The prompt may be satisfied by the person 112 scanning or displaying a QR code, which is scanned within the environment 100, that provides the identifying information and / or billing information to a processor. In some embodiments, the person 112 is prompted to identify himself / herself at each entry into the environment 100. In other embodiments, once the person 112's signature is recognized by the processor, the person 112 is not required to identify himself / herself at future entries into the environment 100.

[0020] Beyond tracking the identity and movements of the person 112, the processor also tracks the identity and movements of one or more products 110. In some embodiments, the product 110 may be identified based on a unique signal pattern detectable for that object. The signal pattern may be based, in part, on the properties of the object (e.g., absorption, refraction, and / or reflection properties). For example, a physical property of the product 110 may cause the transmitter 116 signal to be absorbed, reflected, refracted, or otherwise coated on the product 110 in a unique manner. The processor may be able to identify a given product based on such unique characteristics.

[0021] The processor may not be able to distinguish between two different products 110 based on imaging the products with their unique characteristics. For example, cans of vegetables such as beans and peas, or boxes of cereal may have identical or nearly identical images and may therefore be erroneously determined by the processor to be the same product. In some cases, this misidentification may be avoided by detecting slight differences in the characteristics of these products. For example, the processor may detect that a signal was transmitted through different cans based on the characteristics of the food, or that the signal was absorbed slightly differently based on different color schemes of the boxes or labels. In other cases, or in addition, this misidentification may be avoided by pre-mapping the location within the environment 100 where each given product 110 is located, and the processor knows which of two similarly imaged products it is based on the location where the product 110 was imaged, along with the unique characteristics that would make that product.

[0022] In some embodiments, the server's processor may distinguish between different products 110 by analyzing the absorption characteristics of features encoded on the packaging of the products 110. For example, coded ink (e.g., a QR code, bar code, serial numbering, or any other form of unique identifier) ​​or other radiopaque marker may be embedded on the product's packaging, and the ink (or radiopaque marker) may be detected by the processor on a given product 110 that has high absorption characteristics of the signal transmitted by the transmitter 116. The processor may detect the ink by including the code itself in the image on the product itself when generating an image of the environment 100, by imaging the product in a manner that factors in the ink. The processor may compare symbols (e.g., words, text, bar codes, QR codes, etc.) to entries in a database where the symbol maps to known products, and determine the identity of the given product based on the comparison.

[0023] As a further embodiment for identifying a product, additional physical characteristics of the product 110 may be determined using additional sensors. For example, the person 112 may be carrying a container (e.g., a shopping basket) equipped with a weight sensor that calculates the weight of a product 110 that is believed to be one of the two or more candidate products when the product is added. A processor may receive the weight and determine the identity of the product based on the weight matching an entry in the database that corresponds to only one of the two or more candidate products (taking into account other factors described herein, such as unique characteristics of the image).

[0024] In a store environment, when a person (e.g., person 112) exits environment 100, a processor determines which of products 110 the person is carrying and updates a record accordingly (as described below with reference to FIGS. 2-3). The processor may determine that person 112 is exiting environment 100 when person 112 enters checkout boundary 106 or when person 112 crosses exit plane 104. In some embodiments, the processor tracks person 112 from the time person 112 enters environment 100 until person 112 exits environment 112, and further tracks which products 110 person 112 has taken during that period. The processor updates a record that person 112 has acquired (e.g., indicating a purchase) each of products 110 taken by person 112 as person exits environment 100.

[0025] In other embodiments, the processor causes a prompt to be displayed to the person 112, the prompt requesting the person 112 to confirm that he or she will obtain or purchase the imaged product 110. The prompt may be generated for display at a kiosk within the checkout boundary 106, or the processor may generate the prompt for display on a mobile device carried by the person 112 in response to determining that the person 112 has entered the checkout boundary 106 (e.g., addressing the prompt to the user's mobile device based on profile information indicating the address of the mobile device). The prompt may include selectable options the person 112 may choose from to confirm the purchase and may include an itemized list of the products taken. Additionally or alternatively, the prompt may include a QR code that the person 112 may scan to confirm that he or she will check out. The environment 100 may have a barrier at the exit 104, and the processor may release the barrier to allow the person 112 to exit the environment 100 through the exit 104 in response to the person 112 confirming that he or she will check out.

[0026] The processor may determine that the person 112 wishes to change the products 110 that he or she wishes to take or purchase. For example, following the embodiment above, the processor may receive input from the person 112 that he or she wishes to take a certain product that was taken from the items to be taken from the environment 100. In another embodiment, the processor may determine that the person 112 does not confirm checking out after entering the checkout boundary 106, but instead exits the checkout boundary 106 and re-enters toward the remainder of the environment 100. In response to detecting that the user has re-entered the remainder of the environment 100, the processor may remove the prompt from any devices for which the processor caused the prompt to be displayed, and may resume monitoring which products 110 are taken or returned by the user 112. Thus, when the processor determines that the person 112 has re-entered the checkout boundary 106, the processor may cause a new prompt to be displayed for the person 112 to use in checking out.

[0027] System Configuration FIG. 2 illustrates one embodiment of a system configuration for processing data obtained from the signal receivers. The receiver array 202 collects received signal data and transmits the data to a computation unit 204. The receiver array 202 has the same functionality as the receiver array 114 described above with reference to FIG. 1. In one embodiment, the computation unit 204 is on the same side of the network 206 as the receiver array 202, and the computation unit 204 combines the data received from each receiver of the receiver array 202 before transmitting the combined data to the server 208. The combined data may include raw data used by the processor of the server 208 to generate an image of the environment 100, or alternatively may be the image itself. In an alternative embodiment, some or all of the computation unit 204 is implemented within the server 208 (e.g., as the aforementioned processor of the server 208), in which case raw data from the receiver array 202 or partially combined data is transmitted to the server 208 over the network 206. Although only one computation unit 204 is depicted, this is for convenience and multiple computation units may be used in the environment of FIG. 2.

[0028] Network 206 may be any network, such as a local area network of environment 100, an ad-hoc network, the Internet, etc. Raw or processed signal data or image data may be transmitted over network 206 for processing by a processor in server 208. Server 208 may generate images of environment 100 and people and products therein, and may also use memory to track data regarding those people and products. Components of server 208 are described below with reference to Figures 3-4.

[0029] Server Components FIG. 3 illustrates one embodiment of an exemplary server used to process data obtained from a signal receiver for tracking a person or item. FIG. 3 depicts a server 300, which may have the functionality of the server described with reference to FIG. 1 and may also have the functionality of server 208 of FIG. 2. Server 300 may include one or more processors 302, which process data in the manner described above with reference to FIG. 1, wherever a processor of the server is mentioned. Server 300 may also include other computing components, for example, as described with reference to FIG. 4. Server 300 also includes one or more memories 304, which may be any known form of memory (e.g., random access memory, etc., as further described below with reference to FIG. 4). Although only one server is depicted with one processor block and one memory block, the functionality of server 300 may be distributed across multiple servers, each of which may have multiple processors or memory blocks to perform the functions described herein. As further described herein, server 300 includes one or more modules. A module may include program code (comprised of one or more instructions) corresponding to a particular function of the module. The modules may be stored in the memory 304. The modules may be executable by the server 302.

[0030] Additionally, the server 300 may include various modules. A user profile module 306 is used by the processor 302 to manage user profile data. The user profile data may be stored within the server 300 or may be accessed (e.g., with the transceiver 310) via a database external to the server 300. The user profile data includes a user's identification information, including a detectable unique characteristic, such as a unique digital signature of the user, as determined by imaging the environment 100 as described above. The user profile data may further include other identification information of the user, such as information entered by the user into a kiosk or app, as described above. Additionally, the user profile data may include payment information (e.g., credit card information or payment account information) that the processor 302 may use to bill a given user for any products 110 purchased by the user. The processor 302 may invoke the user profile module 306 to perform any updates or retrievals of the user profile information in any manner disclosed herein.

[0031] The server 300 may also include an inventory module 308. The inventory module 308 tracks information related to the products 110 in the environment 100. The information may be inherent characteristics of the products 110 that are trackable through the imaging process described above, or may include additional information described above (e.g., symbol information to identify a given product). The inventory module 308 may be used to update inventory data (e.g., stored in the memory 304 of the server 300, or in a remote database) to indicate a purchase by a given user, and user profile data corresponding to the given user may also be updated by the processor 302 (e.g., using the user profile module 306) to indicate the purchase.

[0032] The server 300 may also include a signature generation module 312. The processor 302 may invoke the signature generation module 312 to generate a signature in any manner disclosed herein. The server 300 may further include a location tracking module 314. The location tracking module 314 is configured to track the location of one or more people (e.g., people 112) and / or one or more products (e.g., products 110) within an environment (e.g., environment 100). The location tracking module 314 may invoke the user profile module 306 to update a profile of a given person with the path of travel of the given person through the environment 100. Additionally, the server 300 may include a signal calibration module 316, which may execute a protocol to calibrate the transmitter 116 using the reference receiver 108 in any manner described herein.

[0033] Computer configuration FIG. 4 is a block diagram illustrating components of an exemplary machine that can read instructions from a machine-readable medium and execute them on a processor (or controller) (e.g., server 208, or a client device or kiosk, as described above with reference to FIG. 1). In particular, FIG. 4 illustrates a machine diagram in the exemplary form of a computer system 400 on which program code (e.g., software) may be executed to cause the machine to perform any one or more of the methodologies discussed herein. The program code may be comprised of instructions 424 executable by one or more processors 402. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked arrangement, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0034] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch, a bridge, or any machine capable of executing instructions 424 (sequential or otherwise) that specify actions to be performed by that machine. Additionally, although only a single machine is illustrated, the term "machine" shall be deemed to include any collection of machines that individually or collectively execute instructions 124 to perform any one or more of the methodologies discussed herein.

[0035] The illustrated computer system 400 includes a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual interface 410. The visual interface may include software drivers that allow a user interface to be displayed on a screen (or display). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, or the like. For ease of discussion, the visual interface may be described as a screen. The visual interface 410 may include or interface with a touch-enabled screen. Computer system 400 may also include an alphanumeric input device 412 (e.g., a keyboard or touch screen keyboard), a cursor control device 414 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a storage device 416, a signal generating device 418 (e.g., a speaker), and a network interface device 420, which are also configured to communicate via bus 408.

[0036] The storage device 416 includes a machine-readable medium 422 having stored thereon instructions 424 (e.g., software) that embody any one or more of the methodologies or functions described herein. The instructions 424 (e.g., software) may also reside, completely or at least partially, within the main memory 404 or within the processor 402 (e.g., within a processor's cache memory) during execution by the computer system 400, with the main memory 404 and the processor 402 also constituting machine-readable media. The instructions 424 (e.g., software) may also be transmitted or received over a network 426 via a network interface device 420.

[0037] While machine-readable medium 422 is shown to be a single medium in the exemplary embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions (e.g., instructions 424). The term "machine-readable medium" also includes any medium capable of storing instructions (e.g., instructions 424) for execution by a machine, causing the machine to perform any one or more of the methodologies disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

[0038] Exemplary process of 3D localization and imaging of products and customers FIG. 5 is a flow chart illustrating an exemplary process for processing data obtained from a signal receiver for tracking a person or item. Elements of process 500 may be performed using a server (e.g., server 300) or may each be performed by various modules described herein. Process 500 begins with a processor (e.g., processor 302 of server 300) detecting 502 that a user (e.g., user 112) has entered an environment (e.g., environment 100). In response to detecting that the user has entered the environment, the processor uniquely identifies 504 the user (e.g., using signature generation module 312 to generate a unique signature and using user profile module 306 to map the unique signature to a known user profile or generate a new user profile if the user is unknown). The processor determines 506 whether the user has interacted with multiple products (e.g., products of product 110). In response to determining that the user interacts with the product, the processor updates 508 the user's profile with an indication of the product (e.g., determining an indication of the product using the inventory module 308 and applying the indication to the user profile using the user profile module 306).

[0039] The processor determines 510 if the user is about to exit the environment (e.g., using the location tracking module 314 to determine if the user 112 is approaching the exit 104). In response to determining that the user is about to exit the environment, the processor sends a prompt 512 to the user (e.g., pushes a prompt to the user's mobile device) to confirm that the user is about to remove the product from the environment based on the updated profile. The processor receives 514 input from the user confirming that the user is about to remove the product from the environment and further updates 516 the profile based on the input (e.g., attributes the sale of the product to the user profile). The processor may further adjust the inventory of the product 110 using the inventory module 308 to indicate that the product has been removed from the environment 100.

[0040] Other configuration notes In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and there is no requirement that the operations be performed in the order illustrated. In the illustrated configurations, structures and functions presented as separate components may be implemented as combined structures or components. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter of this specification.

[0041] Certain embodiments are described herein as including logic or multiple components, modules, or mechanisms. A module may constitute either a software module (e.g., code embodied on a machine-readable medium or in a transmission signal) or a hardware module. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a particular manner. In an exemplary embodiment, one or more computer systems (e.g., stand-alone, client or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or processors) may be configured by software (e.g., an application or application portions) as a hardware module that operates to perform certain operations as described herein.

[0042] In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may be comprised of dedicated circuitry or logic that is permanently configured to perform certain operations (e.g., as a special purpose processor, such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC)) or may be comprised of programmable logic or circuitry that is temporarily configured by software to perform certain operations (e.g., as contained in a general purpose processor or other programmable processor). It will be appreciated that the decision to implement a hardware module mechanically, in dedicated permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) is determined by cost and time considerations.

[0043] Thus, the term "hardware module" should be understood to encompass a tangible entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform a particular operation as described herein. As used herein, a "hardware-implemented module" refers to a hardware module. Given an embodiment in which the hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, if the hardware modules consist of a general-purpose processor configured with software, the general-purpose processor may be configured as different hardware modules at different times. The software may, for example, configure the processor to configure a particular hardware module at one time and a different hardware module at another time.

[0044] Hardware modules can provide information to and receive information from other hardware modules. Thus, the described hardware modules may be considered to be communicatively coupled. When multiple such hardware modules are simultaneously present, communication may be achieved by transmission of signals connecting the hardware modules (e.g., via appropriate circuits and buses). In embodiments in which multiple hardware modules are configured or instantiated at different times, communication between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a communicatively connected memory device. At a later date, another hardware module may access the memory device and retrieve and process the stored output. Additionally, hardware modules may initiate communication with input or output devices and operate on resources (e.g., collections of information).

[0045] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the associated operations. Such processors, whether temporarily or permanently configured, may constitute processor-implemented modules that operate to perform one or more operations or functions. Modules referred to herein may, in some example embodiments, constitute processor-implemented modules.

[0046] Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented hardware modules. Performance of certain of the operations may not only reside within one machine, but may also be distributed among one or more processors located across multiple machines. In some exemplary embodiments, the processor or processors may be located in a single location (e.g., in a home environment, an office environment, or as a server farm), while in other embodiments, the processors may be distributed across multiple locations.

[0047] The one or more processors may operate to support execution of the associated operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least a portion of the operations may be performed by a group of computers (as examples of machines that include the processors) and these operations may be accessible via a network (e.g., the Internet) and / or one or more suitable interfaces (e.g., application program interfaces (APIs)).

[0048] The performance of a particular operation may reside not only within one machine, but may also be distributed across multiple machines and one or more processors. In some exemplary embodiments, one or more processors or processor-implemented modules may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other exemplary embodiments, one or more processors or processor-implemented modules may be distributed across multiple geographic locations.

[0049] Some portions of this specification are presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals in a machine's memory (e.g., a computer's memory). These algorithms or symbolic representations are examples of techniques used by those skilled in the data processing arts to convey the substance of their work to others skilled in the art. As used herein, an "algorithm" refers to a self-consistent sequence of operations or similar processes leading to a desired result. In this context, algorithms and operations involve physical manipulations of physical quantities. Such quantities may take the form, though not necessarily, of electrical, magnetic, or optical signals that may be stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is sometimes convenient, primarily for reasons of common usage, to refer to such signals using terms such as data, content, bits, values, elements, symbols, characters, terms, numbers, numbering, or the like. However, these terms are merely convenient labels and must be associated with the appropriate physical quantities.

[0050] Unless otherwise indicated, discussions herein of words such as "processing," "computing," "calculation," "determining," "presenting," "displaying," and the like may refer to machine (e.g., computer) operations or processes that manipulate or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0051] As used herein, any reference to "one embodiment" or "one embodiment" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0052] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more elements cooperate or interact with each other even if they are not in direct contact with each other. The present embodiments are not limited in this context.

[0053] As used herein, the terms "comprises," "comprising," "including," "including," "having," "having," or other variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, article, or apparatus consisting of a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" means an inclusive or, not an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), or both A and B are true (or present).

[0054] Furthermore, the use of "a" or "an" has been employed to describe elements and components of the embodiments of this specification. This is done merely for convenience and to give a general sense of the invention. This specification should be read to include one or at least one, unless otherwise clear, and the singular also includes the plural.

[0055] After reading this disclosure, those skilled in the art will appreciate further alternative structural and functional designs for systems and processes for monitoring users and products in an environment using non-line-of-sight signals in accordance with the principles disclosed herein. Thus, while specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined by the appended claims.

Claims

1. 1. A computer system implemented method comprising: The user has entered the environment. receiving wireless network signals detected by a receiver array, the receiver array continuously monitoring the wireless network signals; determining a measure of absorption of a predetermined waveform of the wireless network signal; determining that the predetermined waveform is indicative of a human based on inputting the absorption measurements into a data structure and receiving an output from the data structure; and detecting the in response to detecting the user entering the environment, uniquely identifying the user based on a signal pattern of the wireless network signals corresponding to at least two uniquely identifiable physical characteristics of the user; determining if the user has interacted with a product in the environment based on an absorption characteristic of the product with respect to the wireless network signal; updating a profile of the user with an indication of the product in response to determining that the user has interacted with the product; determining if the user is attempting to exit the environment; in response to determining that the user is attempting to leave the environment, prompting the user to confirm that the user is attempting to remove the product from the environment based on the updated profile; receiving an input from the user confirming that the user wishes to remove the product from the environment; further updating the profile based on the input; and A method for providing the above.

2. Uniquely identifying the user includes: detecting a signal pattern corresponding to a characteristic of the user; generating a signature based on the signal pattern; determining whether the user is known by determining whether the unique signature matches a stored signature in a database entry that maps the signature to a profile of the user; in response to determining that the user is known, attributing an identifier retrieved from the profile to the user; The method of claim 1 , comprising:

3. Prompting the user to confirm that the user is removing the product from the environment based on the updated profile includes: determining an address of the user's mobile device from the profile; and using the address to push a notification to the mobile device, the notification including a selectable option for the user to confirm that they are removing the product from the environment; and The method of claim 2 comprising:

4. In response to the user determining that the user is unknown, adding said unique signature to said entry in said database; associating an identifier with said entry in said database; uniquely identifying the user using the identifier; The method of claim 2 , further comprising:

5. updating the profile with the indication of the product; identifying, for each of the products, patterns in the wireless network signals that are indicative of characteristics of objects encountered by the wireless network signals in the environment; determining whether the pattern corresponds to a known pattern in one or more entries of a database mapping products to known patterns, the entries of the database indicating at least one of an absorption characteristic, a reflection characteristic, and a refractive characteristic; in response to determining that the pattern matches an entry in the database, identifying the indication of the product based on the entry; The method of claim 1 , comprising:

6. determining that the pattern matches two or more of the entries in the database; in response to determining that the pattern matches two or more of the entries in the database, querying a sensor for a weight of the product; determining whether one of two or more of the entries in the database indicates a weight that matches the weight of the product; in response to determining that one of two or more of the entries in the database indicates a weight that matches a weight of the product, identifying the indicia of the product based on the one of the two or more entries; The method of claim 5 further comprising:

7. The pattern is an absorption pattern, and identifying the absorption pattern of the wireless network signal in the environment for each of the products includes: detecting radiopaque markers embedded in the article based on locations of the article having high signal absorption; Determining whether the absorption pattern corresponds to a known pattern in an entry of a database mapping products to known absorption patterns includes: determining whether an entry in the database maps the product to the detected radiopaque marker. The method according to claim 5.

8. A non-transitory computer-readable storage medium containing stored instructions executable by a processor, the instructions comprising: The user has entered the environment. receiving wireless network signals detected by a receiver array, the receiver array continuously monitoring the wireless network signals; determining a measure of absorption of a predetermined waveform of the wireless network signal; determining that the predetermined waveform is indicative of a human based on inputting the absorption measurements into a data structure and receiving an output from the data structure; and in response to detecting the user entering the environment, instructions for uniquely identifying the user based on a signal pattern of the wireless network signals corresponding to at least two uniquely identifiable physical characteristics of the user; instructions for determining if the user has interacted with a product of a plurality of products in the environment; instructions for updating a profile of the user with an indication of the product in response to determining that the user has interacted with the product based on an absorption characteristic of the product with respect to the wireless network signals; instructions for determining if the user is attempting to exit the environment; instructions, in response to determining that the user is attempting to leave the environment, to prompt the user to confirm that the user is attempting to remove the product from the environment based on the updated profile; and receiving an input from the user confirming that the user wishes to remove the product from the environment; and instructions for further updating the profile based on the input; and A non-transitory computer-readable storage medium comprising:

9. The instructions to uniquely identify the user include: instructions for detecting a signal pattern corresponding to a characteristic of the user; instructions for generating a signature based on the signal pattern; instructions for determining if the user is known by determining if the unique signature matches a stored signature in a database entry that maps the signature to a profile of the user; instructions for attributing an identifier retrieved from the profile to the user in response to determining that the user is known; 9. The non-transitory computer-readable storage medium of claim 8, comprising:

10. The instructions for prompting the user to confirm that the user is removing the product from the environment based on the updated profile include: instructions for determining an address of a mobile device of the user from the profile; instructions for pushing a notification to the mobile device using the address, the notification including a selectable option for the user to confirm that the user intends to remove the product from the environment; and 10. The non-transitory computer-readable storage medium of claim 9, comprising:

11. The instruction: In response to the user determining that the user is unknown, instructions for adding said unique signature to said entry in said database; instructions for associating an identifier with the entry in the database; instructions for uniquely identifying the user using the identifier; 10. The non-transitory computer-readable storage medium of claim 9, further comprising:

12. The instructions for updating the profile with the indication of the product include: instructions for identifying, for each of the products, patterns in the wireless network signals indicative of characteristics of objects encountered by the wireless network signals in the environment; instructions for determining whether the pattern corresponds to a known pattern in one or more entries of a database mapping products to known absorption patterns; instructions, in response to determining that the pattern matches an entry in the database, to identify the indication of the product based on the entry; 9. The non-transitory computer-readable storage medium of claim 8, comprising:

13. The instruction: instructions for determining whether the pattern matches two or more of the entries in the database; instructions for querying a sensor for a weight of the product in response to determining that the pattern matches two or more of the entries in the database; instructions for determining whether one of two or more of the entries in the database indicates a weight that matches the weight of the product; instructions, in response to determining that one of two or more of the entries in the database indicates a weight that matches the weight of the product, for identifying the indicia of the product based on one of the two or more entries; 13. The non-transitory computer-readable storage medium of claim 12, further comprising:

14. The pattern is an absorption pattern, and the instructions for identifying the absorption pattern of the wireless network signal in the environment for each of the products include: detecting radiopaque markers embedded in the article based on locations of the article having high signal absorption; The instructions for determining whether the absorption pattern corresponds to a known pattern in an entry of a database mapping products to known absorption patterns include: The non-transitory computer readable storage medium of claim 12 , comprising instructions for determining whether an entry in the database maps the product to the detected radiopaque marker.

15. 1. A system comprising: Using sensors, the system detects when a user enters the environment. receiving wireless network signals detected by a receiver array, the receiver array continuously monitoring the wireless network signals; determining a measure of absorption of a predetermined waveform of the wireless network signal; determining that the predetermined waveform is indicative of a human based on inputting the absorption measurements into a data structure and receiving an output from the data structure; Detected by in response to detecting the user entering the environment, uniquely identifying the user using a signature generation module based on a signal pattern of the wireless network signals corresponding to at least two uniquely identifiable physical characteristics of the user; determining, using an inventory module, whether the user has interacted with a product among a plurality of products in the environment based on an absorption characteristic of the product with respect to the wireless network signal; responsive to determining that the user has interacted with the product, updating a profile of the user with an indication of the product using a user profile module; Using a location tracking information module, determine if the user is exiting the environment; In response to determining that the user is attempting to leave the environment, prompting the user to confirm that the user is attempting to remove the product from the environment based on the updated profile; receiving input from the user confirming that the user wishes to remove the product from the environment; further updating the profile using the user profile module based on the input.

23. A system comprising: a processor configured to:

16. When the processor causes the user to be uniquely identified, detecting a signal pattern corresponding to a characteristic of the user; generating a signature based on the signal pattern; determining whether the user is known by determining whether the unique signature matches a signature stored in a database entry corresponding to the profile of the user; in response to determining that the user is known, attributing an identifier retrieved from the profile to the user. The system of claim 15 further configured to:

17. When the processor determines based on the updated profile that the user is about to be prompted to confirm that the user is about to remove the product from the environment, determining an address of the user's mobile device from the profile; using the address to push a notification to the mobile device, the notification including a selectable option for the user to confirm that they are removing the product from the environment; The system of claim 16 further configured to:

18. The processor, In response to the user determining that the user is unknown, adding said unique signature to said entry in said database; Associating an identifier with the entry in the database; using said identifier to uniquely identify said user; The system of claim 16 further configured to:

19. When the processor causes the profile to be updated with the indication of the product, identifying, for each of the products, patterns in the wireless network signals that are indicative of characteristics of objects encountered by the wireless network signals in the environment; determining whether the pattern corresponds to a known pattern in one or more entries of a database mapping products to known patterns, the entries of the database indicating at least one of an absorption characteristic, a reflection characteristic, and a refractive characteristic; in response to determining that the pattern matches an entry in the database, identifying the indication of the product based on the entry. The system of claim 15 further configured to:

20. The processor, determining that the pattern matches two or more of the entries in the database; in response to determining that the pattern matches two or more of the entries in the database, querying a sensor for a weight of the product; determining whether one of two or more of the entries in the database indicates a weight that matches the weight of the product; in response to determining that one of two or more of the entries in the database indicates a weight that matches the weight of the product, identifying the indication of the product based on the one of the two or more entries.

20. The system of claim 19, further configured to:

Images