Visual attention-based disclosure using inferred head pose and product position

Abstract

Systems (312) and methods (400) are provided for passively collecting contextual data associated with a user's in-store shopping. In this regard, one or more processors (202, 204) may receive sensor data corresponding to the orientation of each respective earplug (240, 242) from each earplug of a pair of earplugs (238) worn by a user. Based on the received sensor data, the one or more processors (202, 204) may determine a gaze direction of the user as the user travels within the store and one or more products viewed by the user as the user travels within the store by comparing the gaze direction of the user with a product location of the stored product using a content map (128) of the store, wherein the one or more products are contextual data.

Description

Background Art

[0001] Contextual advertising is a form of personalized advertising in which advertisements ("ads") are selected based on contextual data associated with the individual to whom the advertisements are directed. The context typically includes the individual's location, search history, browsing history, or other such information associated with the individual. For example, if an individual is browsing for a pair of sneakers in an online store, a contextual advertisement for a particular sneaker or brand of sneakers may be displayed in the user's browser. In another example, if an individual submits a search query for a particular car model, contextual advertisements for competing car models may be returned to the individual. Contextual advertisements are generally more relevant and interesting to the individual receiving them than general advertisements directed to a large audience, and generally result in more successful ad performance.

[0002] In-store shopping provides additional opportunities to collect contextual data for contextual advertising. For example, the products that an individual views while moving through a store may be contextual data that indicates the types of products that the individual is interested in. However, in order to collect this contextual data, the individual may need to take proactive action. For example, an individual may perform a visual search for products captured by their smartphone camera while shopping in a store. Products within the visual search may provide contextual data to the store or other stores, which may be utilized to provide contextual advertising. For example, in response to a visual search for a particular brand of laundry detergent, a store may provide contextual advertising for competing brands of laundry detergent. Summary of the invention

[0003] The technology described herein relates to passively associating a user's gaze with in-store content for determining contextual data. One aspect of the present disclosure relates to a method for passively collecting contextual data, the method comprising: receiving, by one or more processors, sensor data corresponding to the orientation of each respective earbud from each earbud of a pair of earbuds worn by a user; determining, by the one or more processors, a gaze direction of the user based on the received sensor data as the user moves within a facility; and determining, by the one or more processors, one or more products viewed by the user while moving within the facility by comparing the gaze direction of the user with product locations where the products are stored using a content map of the facility, wherein the one or more products are contextual data.

[0004] In some examples, location data of a user as the user travels within the facility is received by one or more processors. In some cases, the location data includes GPS data, indoor navigation system data, and / or location data contained in sensor data.

[0005] In some examples, comparing the gaze direction of the user to the product locations includes: for each gaze direction: determining from the position data the location of the user when the gaze direction was determined; and determining whether a ray from the user's location in the gaze direction intersects a product location in the product locations. In some examples, the one or more products are products stored at the product locations where the corresponding ray intersects.

[0006] In some examples, a product location includes sub-product locations.

[0007] In some examples, the sensor data from each earbud includes data captured by one or more accelerometers and gyroscopes.

[0008] In some instances, the one or more processors receive a signal from an initiating device, wherein the signal initiates the collection of sensor data by a pair of earbuds. In some examples, the one or more processors receive data from an RFID tag indicating that a user has entered a facility; and initiate the collection of sensor data by a pair of earbuds.

[0009] In some examples, for each gaze direction, one or more processors determine a length of time that the user maintains the corresponding gaze direction.

[0010] In some examples, one or more processors are within at least one earbud of a pair of earbuds.

[0011] In some instances, one or more processors are within a user device.

[0012] Another aspect of the present disclosure relates to a system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: receive sensor data corresponding to the orientation of each corresponding earbud from each earbud in a pair of earbuds worn by a user; determine a gaze direction of the user based on the received sensor data as the user moves within a facility; and determine one or more products viewed by the user while moving within the facility by comparing the gaze direction of the user with product locations of stored products using a content map of the facility, wherein the one or more products are contextual data.

[0013] In some examples, the one or more processors also receive location data of the user as the user travels within the facility.

[0014] In some examples, the location data includes GPS data, indoor navigation system data, and / or location data contained within sensor data.

[0015] In some examples, comparing the user's gaze direction to the product locations includes: for each gaze direction: determining from the position data the user's position when the gaze direction was determined; and determining whether a ray from the user's position in the gaze direction intersects the product location in the product locations.

[0016] In some examples, the one or more products are products stored at product locations intersected by corresponding rays, where the product locations include child product locations.

[0017] In some instances, the sensor data from each earbud includes data captured by one or more accelerometers and gyroscopes.

[0018] In some examples, the system also includes the pair of earplugs.

[0019] Another aspect of the present disclosure relates to a computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] A more complete understanding of the subject matter of the present technology and its various advantages may be realized by referring to the following detailed description with reference to the accompanying drawings, in which:

[0021] Figure 1 An example diagram illustrating a content graph in accordance with aspects of the present disclosure.

[0022] Figure 2 is a schematic diagram illustrating an example pair of earbuds according to aspects of the present disclosure.

[0023] Figure 3 is a block diagram of an example system including an example earbud pair according to aspects of the present disclosure.

[0024] Figure 4 is a flow chart illustrating a method for generating contextual data by determining which products a user viewed during a shopping experience according to aspects of the present disclosure.

[0025] Figure 5 A user according to aspects of the present disclosure is shown entering a store.

[0026] Fig. 6A is a schematic diagram showing a front view of a pair of example earplugs worn by the same user according to aspects of the present disclosure

[0027] Figure 6B is a schematic diagram showing a top view of a pair of example earbuds worn by the same user according to aspects of the present disclosure

[0028] Figure 7 User position and rays corresponding to gaze direction are shown according to aspects of the present disclosure.

[0029] Figure 8 Another user position and light rays corresponding to a gaze direction according to aspects of the present disclosure are shown. DETAILED DESCRIPTION

[0030] Overview

[0031] The technology described herein relates to passively collecting contextual data associated with a user's in-store shopping experience or travel within another facility for contextual advertising, planning product layout and placement, store design, and the like. For example, as a user moves within a store, the user's location and gaze direction can be inferred and compared to the store's content map. The content map may include product locations corresponding to where the products are located within the store. By comparing the user's inferred gaze direction and location with the store's content map, a determination of what products the user looked at during their in-store shopping experience can be determined. Based on the determination of the products the user looked at, contextual data corresponding to the user's in-store shopping experience can be generated.

[0032] As described in more detail herein, sensor data provided by sensors in a pair of earplugs or other such wearables worn by a user can be used to determine the user's gaze direction. For example, each earplug in a pair of earplugs can include an inertial measurement unit (IMU) that provides sensor data, the sensor data including accelerometer and gyroscope data, and the corresponding reference frame of each earplug can be determined from the accelerometer and gyroscope data. Each earplug is worn in the corresponding ear of the user and can be considered to be attached to the same rigid frame-the user's head. In this regard, the earplug is "head-locked" in the sense that the movement of the earplug and the movement of the user's head are closely related. In addition, given the corresponding reference frame of each earplug and its position in the ear on the opposite side of the user's head, the orientation of the user's head can be inferred from the sensor data. The head orientation can be used as a proxy for understanding where the user is looking. In other words, the user's head orientation can provide an indication of the direction of the user's gaze focus. Therefore, by analyzing the movement of the earplug using the IMU, the orientation of the user's head can be inferred during the user's visit to the store, thereby estimating the user's gaze direction during the visit to the store.

[0033] The user's location throughout the store may also be determined. For example, the user's device (such as a smart phone) may include a GPS or other such positioning system. GPS may be used to provide location data corresponding to the user's location as the user moves through the store. Since stores are typically located indoors, GPS signals may be weak or non-existent and may result in lost or inaccurate location data. To address this issue, the location data may include location data contained within sensor data provided by an IMU. The location data may be used to detect the user's location within the store. Additionally or alternatively, the store may include an indoor navigation system capable of providing location data corresponding to the user. Such an indoor navigation system may improve the accuracy of the location data when compared to typical location data provided by a GPS system and location data provided by an IMU.

[0034] When a user enters or approaches a store entrance, analysis of the user's location and gaze direction may begin. For example, when a user approaches an entryway of a store, a sensor such as an RFID tag or a Bluetooth device may trigger data collection to begin. Data collection may include collecting sensor data to determine the user's gaze direction and / or location. Data collection may also include collecting location data from a positioning system and / or location data contained within sensor data.

[0035] The user's location and gaze direction may be compared to a content map of the store to identify which products the user viewed while within the store. In this regard, the content map may define the layout of the store and the product locations within the store. Thus, by comparing the product locations on the content map with the user's location and gaze direction during a visit to the store, it may be determined which products the user viewed within the store. For example, an intersection ray result may be determined by generating a ray from the user's location in the direction of the determined gaze direction. Products stored at product locations that intersect the ray may be determined as products viewed by the user.

[0036] In some instances, a determination of what product a user is looking at can be made in real time. In this regard, the user's location and gaze direction can be continuously determined and compared to the product locations on the content map to identify which product, if any, the user is currently looking at.

[0037] In some examples, the amount of time a user views a product can also be determined by identifying the length of time the user's gaze is directed toward a product location associated with the product. The viewed product and the length of time the user views the product can be considered contextual data.

[0038] The amount of time that a user views a product and the spatial distribution around the product location in the content graph can be used to generate an attention score indicating that the user is interested in the product at the product location. In this regard, a machine learning model, such as a neural network, can be trained to generate an attention score based on the spatial distribution around the product locations viewed by the user and the duration that the user views these locations.

[0039] Content Map

[0040] The content graph discretizes product locations of different types of products, such as physical merchandise or materials (e.g., promotional displays, signs, advertisements, etc.), throughout a facility, such as a store, warehouse, market, or other such location where products are arranged. For clarity and ease of reference, "product" as used herein may be considered to be any physical item or material. For example, a content graph for a clothing store may include discretized product locations for jeans, shirts, socks, etc. Each discretized product location may include sub-product locations to distinguish between product locations for different brands, sizes, models, versions, etc. of the product. For example, the product location for jeans may be subdivided into sub-product locations for each brand of jeans offered by the store. Furthermore, although the examples described herein relate to stores, any other facility may be used.

[0041] Sub-product locations may provide a higher resolution analysis of the products that users are viewing during their visit to a store. For example, using the techniques described herein, if a store has a content map that has a "jeans" product location that is broken down into sub-product locations for each brand of jeans offered by the store, the store can learn which jeans brand attracts the most views, and conversely, which jeans brand does not attract views. In situations where the same or similar types of products are stored in different areas of a store, it may not be necessary to break down product locations within a content map, as the spatial distance between the same or similar products may be sufficient to allow for proper identification using the techniques described herein.

[0042] Based on the information collected and determined, such as the product(s) viewed by the user, user location data, gaze direction, etc., the store can better manage inventory, display / product location, targeted advertising, and provide more targeted offers. For example, if a brand of jeans does not attract the user's interest, the store can provide an advertisement or offer for the brand of jeans to change the inventory. The content map can be created by the store owner, employee, or any other third party (such as a content map creator). The content map can be created using content map creation software or any other software that can define the layout of product locations within a store. The product map can be updated when product locations are changed, new products are added, products are removed from the store, etc.

[0043] Figure 1 1, which shows the location of products within the store 102. As shown, the content map 128 shows the shelf units that hold the products, including shelf 1 122, shelf 2 124, and shelf 3 126. The content map 128 also includes aisles that users can walk through to view the shelves, including aisle 1 104, aisle 2 106, aisle 3 108, aisle 4 110, and aisle 5 112. The content map 128 also includes an entryway, i.e., a doorway 114, through which users can enter the store 102.

[0044] During creation of the content map 128, products may be associated with each shelf and / or a portion of each shelf to generate product locations. Continuing with the example of the clothing store, shelf 1 122 may be associated with shirts to create product location 136, and shelf 2 124 may be associated with pants to create product location 134. Shelf 3 126 may be split, with each portion being assigned a different product. For example, socks may be associated with product location 130, and suits may be associated with product location 132. Thus, the content map 128 includes four product locations, including product location 136 for shirts (covering all of shelf 1 122), product location 134 for pants (covering all of shelf 2 124), product location 130 for socks (covering the products in the lower half of shelf 3 126), and product location 132 for suits (covering the upper half of shelf 3 126).

[0045] like Figure 1 As further shown, product location 134 for pants on shelf 2 124 may be divided into sub-product locations, including sub-product location 120 and sub-product location 138. Each sub-product location may be associated with a particular brand of pants, a size of pants, or any other distinguishing feature or identifier. Similarly, product location 132 covering shelf 3 126 may be divided into sub-product locations, including sub-product location 116 and sub-product location 118. In addition, sub-product location 116 and sub-product location 118 may be associated with a distinguishing feature or identifier, such as suit size or color.

[0046] Although not shown, the content map may include spatial data corresponding to the relative position and / or distance of the product location from the entryway. For example, the spatial data of the product location and sub-product location in the content map 128 relative to the doorway 114 may be stored with or in association with the content map 128.

[0047] Although the content map 128 shows a discretization of product locations on a shelf, the content map may discretize product locations anywhere within the store, including products on tables or other structures, within containers or display cases, etc. In some cases, a product location may correspond to one or more products located on a store floor. Furthermore, the content map may include any number of product locations, entry points, aisles, etc.

[0048] Example System

[0049] Figure 2 2 is a functional block diagram of a pair of earbuds 238, including a left earbud 240 and a right earbud 242 in which the techniques described herein may be implemented. It should not be considered to limit the scope of the present disclosure or the usefulness of the features described herein. For example, and as Figure 2 As shown, the left earbud 240 may include one or more processors 202, memory 206, and other components typically found in general-purpose computing devices. The right earbud 242 may similarly include one or more processors 204, memory 208, and other components typically found in general-purpose computing devices. In some examples, the processors 202 and 204 and the memories 206 and 208 may be part of a microcontroller unit ("MCU").

[0050] The memory 206, 208 may store information accessible by the one or more processors 202 and 204, including instructions 210, 212 that may be executed by the one or more processors. The memory 206, 208 may also include data 214, 216 that may be retrieved, manipulated, stored, etc. by the processors 202 and 204. The memory may be of any type capable of storing information accessible by the processor, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. The memory may be considered to be non-transitory memory.

[0051] Instructions 206, 212 may be any set of instructions executed directly by one or more processors (such as machine code) or any set of instructions executed indirectly, such as a script. In this regard, the terms "instructions", "application(s)", "step(s)", and "program(s)" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled. The functions, methods, and routines of instructions are explained in more detail below. For example, instructions may include a store application.

[0052] Data 214, 216 may be retrieved, stored, modified, etc. by one or more processors 202, 204 according to instructions 210, 212. For example, although the subject matter described herein is not limited to any particular data structure, the data may be stored in a computer register, in a relational database as a table with many different fields and records, or in an XML document. The data may also be formatted in any computing device readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, the data may include any information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (such as at other network locations), or information used by functions to calculate relevant data. The data may also include sensor data and location data, as described herein.

[0053] The one or more processors 202, 204 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a dedicated component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, the pair of earbuds 238 may include dedicated hardware components to perform specific computing processes faster or more efficiently, such as processing machine learning algorithms, processing audio, etc.

[0054] although Figure 2 The processors, memory, and other elements of a pair of earbuds 238 are functionally shown as being within the same block, but the processors, memory, and other elements may actually include multiple processors, computers, computing devices, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a housing that is different from the housings of the left earbud 240 and the right earbud 242. Thus, references to a processor, computer, computing device, or memory will be understood to include references to a collection of processors, computers, computing devices, or memories that may or may not operate in parallel.

[0055] like Figure 2 As further shown in the figure, the left earbud 240 and the right earbud 242 can each include one or more input / output devices 218, 220, respectively. The input / output devices 218, 220 can include, for example, one or more speakers, transducers or other audio outputs, user displays, tactile interfaces or other tactile feedback that provide non-visual and non-auditory information to the user, keypads, keyboards, mice, touch screens, etc. For example, the output devices 218, 220 can be used to play music, emit audio for navigation or other guidance, for multimedia files, for voice calls, for translation of speech, etc.

[0056] One or both of the left earbud 240 and the right earbud 242 may include one or more sensors, such as sensors 222, 224, respectively. For example, sensors 222, 224 may include motion sensors, such as one or more IMUs including inertial measurement units (IMUs) 226, 228. According to some examples, IMU 226 and IMU 228 may each include one or more gyroscopes, such as 3-axis gyroscopes. The gyroscopes may be configured to measure angular motion around the x-axis, y-axis, and z-axis, such as angular velocity and rotation angle. IMU 226, 228 may also include one or more accelerometers, such as 3-axis accelerometers. The accelerometers may be configured to measure linear motion along the x-, y-, and z-axes, such as linear acceleration. The accelerometers may be configured to measure linear motion along the x-, y-, and z-axes, such as linear acceleration.

[0057] To obtain and send information from and to each other and to other remote devices, the left earbud 240 and the right earbud 242 can each include a communication module, such as a communication module 230, 232, respectively. The communication modules can implement wireless network connections, wireless ad hoc connections, and / or wired connections. Via the communication modules 230, 232, a pair of earbuds 238 can establish a communication link, such as a wireless link. The communication modules 230, 232 can be configured to support communication via cellular, LTE, 4G, WiFi, GPS, and other networking architectures. The communication modules 230, 232 can be configured to support Bluetooth LE, near field communication, and non-networked wireless arrangements. The communication modules 230, 232 may support wired connections such as USB, micro USB, USB Type-C, or other connectors, for example to receive data and / or power from a laptop, tablet, smartphone, or other device.

[0058] Each earbud in a pair of earbuds 238 may include one or more internal clocks 234, 236. The internal clocks may provide timing information that may be used for time measurement of apps and other programs run by the computing device and / or earbuds described herein, as well as basic operations performed by computing devices, sensors, input / output, GPS, communication systems, and the like. In some instances, each earbud may include multiple clocks. For example, the clocks 234, 236 may include one or more clocks in a Bluetooth module or chipset, an IMU clock, and an MCU clock. In some instances, a clock may be configured as a "master clock," and a second clock, such as a clock in another earbud other than the master clock, may be configured as an "estimated clock" that is synchronized with the master clock.

[0059] The left earplug 240 can be configured to be worn in the left ear of the user, and the right earplug 242 can be configured to be worn in the right ear of the user. The left earplug 240 and the right earplug 242 can be wireless because they do not need a wired connection to use. For example, the wireless earplug can receive signals wirelessly such as from a music player, a phone or other device to perform multiple functions, such as generating outputs, communicating with each other and / or communicating with other devices, being charged, etc. A pair of earplugs 238 including the left earplug 240 and the right earplug 242 can be truly wireless because they do not need a wired connection to communicate with each other. Each earplug can be configured to have physical features, such as ear tips, which allow the earplug to fit firmly and comfortably in the left and right ears of the user, respectively. In some cases, the earplug can be worn in or on the user's ear. For example, the earplug can be a closed earphone or an open earphone that covers or partially covers the user's ear.

[0060] Using the communication modules 230, 232, a pair of earbuds 238 can communicate with other devices in the system via a network, as described herein with respect to Figure 3 described.

[0061] Figure 3 306 and 308. FIG. 307 is a schematic diagram illustrating an example system 312 in which the features described herein may be implemented. The system 312 may include a pair of earbuds 238 (including a left earbud 240 and a right earbud 242), computing devices 302, 304, and 306, and a storage system 310. As shown, the pair of earbuds 238, computing devices 302, 304, and 306, and storage system 310 may each be located at a different node of a network 308 and may be able to communicate directly and indirectly with other nodes of the network 308. Although in FIG. Figure 3 Only a few computing devices are depicted in , but it should be understood that a typical system may include a large number of connected computing devices, with each different computing device at a different node of the network 308.

[0062] The network 308 and intermediate nodes described herein can be interconnected using various protocols and systems, such that the network can be part of the Internet, the World Wide Web, a specific intranet, a wide area network, or a local network. The network can utilize standard communication protocols such as Ethernet, WiFi, and HTTP, one or more company-specific protocols, and various combinations of the foregoing. Although certain advantages are obtained when sending or receiving information as described above, other aspects of the subject matter described herein are not limited to any particular information transmission method.

[0063] Network slicing enables customization of the capacity and capabilities of the network for different services such as geo-location and route planning, sensor monitoring, computer vision, etc. Edge data center processing and local data center processing augment central data center processing to allocate 5G, 6G and future network resources to user devices (including smartphones and wearables) as well as sensors and other wirelessly connected devices.

[0064] Not only can ground network devices support geolocation, route planning, location identification, sensor monitoring, computer vision, etc., but non-ground network devices such as drones, satellites, and balloons can enable 5G, 6G, and future wireless communications in additional environments such as oceans, rural areas, and other locations that experience insufficient base station coverage.

[0065] 5G, 6G and future wireless networks enable fine range sensing and sub-meter precision positioning as support for geolocation, route planning, location identification, sensor monitoring, computer vision, etc. Utilizing massive bandwidth and wireless resource (time, frequency, space) sharing, these wireless networks enable simultaneous communication and sensing capabilities to support radar applications in user devices (such as smart displays, smartphones, wearables, etc.) as well as sensors and other wirelessly connected devices.

[0066] Each of the computing devices 302, 304, and 306 can be configured similarly to the earbuds in the pair of earbuds 238, with one or more processors, memory, and instructions as described above. For example, the computing devices 302 and 306 can each be a client device intended for use by a user (such as user 314 or 316), and have all the components typically used in conjunction with an individual computing device, such as a central processing unit (CPU), memory (e.g., RAM and internal hard drive) to store data and instructions, user input and / or output, sensors, communication modules, positioning systems, clocks, etc. For example, the communication modules of the computing devices 302, 306 can similarly include one or more antennas for sending and / or receiving signals (such as signals and / or GPS). As another example, the computing devices 302, 306 can have the same and / or different types of user input and / or output as the earbuds in the pair of earbuds 238, such as a screen or touch screen for displaying text, images, videos, etc. The computing user devices 302, 306 may be referred to herein as "user devices" and / or "mobile devices."

[0067] As yet another example, computing device 304 may be a server computer and may have all components typically used in conjunction with a server computer, such as a processor and memory to store data and instructions.

[0068] Computing devices 302 and 306 may each comprise a full-scale individual computing device, or may alternatively comprise a mobile computing device capable of wirelessly exchanging data with a server such as server 304 over a network such as the Internet. For example, computing device 302 may be a server such as Figure 3 306 may be a mobile device such as a mobile phone as shown, or some other mobile device such as a wireless-enabled PDA. As another example, computing device 306 may be a Figure 3 The laptop computer shown, or some other computing device such as a desktop computer, a tablet computer, or other smart device capable of obtaining information via a communication link. In other examples (not shown), the system 312 may additionally or alternatively include a wearable device such as a smart watch, a head-mounted device, etc.

[0069] As with the memories 206, 208, the storage system 310 may be any type of computerized storage device capable of storing information accessible by one or more of the pair of earbuds 238 and the computing devices 302, 34, 306, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. Additionally, the storage system 310 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. The storage system 310 may be connected to the computing device via 308, such as Figure 3 As shown, and / or may be directly connected to a pair of earbuds 238 and / or any of the computing devices 302, 304, 306. The storage system 310 may store data such as location data and sensor data.

[0070] Example Method

[0071] Figure 4 A flowchart 400 is shown of a method for generating contextual data by determining products viewed by a user (such as user 314) while traveling through a store (such as store 102). Although flowchart 400 depicts a particular sequence of operations, the sequence may be changed without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of an example device or system that implements the routine may perform functions substantially simultaneously or in a particular order.

[0072] The method may include triggering data collection when a user enters or approaches a store, as shown in block 402. For example, and as Figure 5As shown, the door 114 of the store 102 can have an initiator device 502 that can trigger data collection when a user, such as user 314, enters the store. The initiator device 502 can be a Bluetooth device, an RFID tag, or another such device that can communicate or otherwise provide an indication to the computing device and / or earbuds to begin data collection.

[0073] like Figure 5 As shown, user 314 can pass through doorway 114 from the outside of store 506 to the inside of store 504, as shown by arrow 508. When user 314 passes through doorway 114, initiator device 502 can provide a signal or otherwise trigger data collection, including collecting sensor data and / or location data. For example, initiator device 502 can be an RFID tag that can be detected by a communication module, such as communication module 230 and / or communication module 232 in left earbud 240 and right earbud 242, respectively, worn by user 314. Alternatively or additionally, the communication module that detects the RFID tag can be within the user's device, such as user device 302 carried by user 314.

[0074] In yet another example, the initiator device 502 may be a Bluetooth device configured to detect a communication module within an earbud and / or a user device, such as a pair of earbuds 238 and the user device 302. Upon detecting the communication module, the initiator device 502 may send a signal to the communication module to initiate the collection of sensor data and location data.

[0075] When the communication module receives a signal from the initiator device 502 and / or the communication module detects the initiator device 502, data collection can be initiated. For example, if the communication module in a pair of earplugs 238 receives a signal or detects the initiator device 502, the IMU 226, 228 can start collecting sensor data. The collection of sensor data can be controlled by the processor 202, 204. In some cases, a pair of earplugs 238 can communicate with the user's device 302, indicating that data collection has or will start. Then, the application running on the user device 302 can start collecting location data from GPS and / or establish communication with the store's indoor navigation system to collect location data. Alternatively or additionally, in the case where the earplugs contain GPS or other such hardware, the earplugs can collect location data. Alternatively or additionally, the location data can be determined from the location data in the sensor data. The application can be part of the operating system of the user device 302, an independent application and / or a web application. For example, the application can be an application provided by the store that the user is entering.

[0076] In another example, where the user device 302 receives a signal or otherwise detects the initiator device 502, the user device may send an initiation signal to a pair of earbuds 238. In response to the initiation signal, the pair of earbuds may begin collecting sensor data. As explained herein, the IMU 226, 228 may begin collecting sensor data, including accelerometer and gyroscope data, from which the respective reference frame and position of each earbud may be determined, as described herein. Data collection may continue for a portion of or the entirety of the user's visit to the store. Data collected during data collection may be time stamped.

[0077] Once data collection begins, or after data collection is complete, the sensor data can be used to determine the direction of a user's gaze as they move through or have moved through the store, e.g. Figure 4 In this regard, and as Fig. 6A and Figure 6B As shown, when worn by user 314, left earplug 240 and right earplug 242 are configured to have fixed positions relative to the user's left ear 602 and right ear 604, respectively, and relative to each other. In this way, left earplug 240 and right earplug 242 can be considered to be attached to the same rigid body - the user's head 606 - and move with the same rigid body. The movement of the user's head 606 can be defined relative to the coordinate system (frame) of the user's head 606. For example, the coordinate system can be defined by various axes passing through the user's head 606, such as a vertical axis ("z axis"), a first horizontal axis ("x axis"), and a second horizontal axis ("y axis"). The x-axis, y-axis, and / or z-axis may intersect at point 608.

[0078] like Fig. 6AAs shown in the front view of the user 314 in FIG. 1 , the left earbud 240 and the right earbud 242 may each have a corresponding coordinate system (indicated by the dashed lines). The coordinate system of each of the left earbud 240 and the right earbud 242 may be defined by the IMU therein. For example, the coordinate system may be defined by the axes of a 3-axis accelerometer and / or the axes of a 3-axis gyroscope. In some instances, the accelerometer and gyroscope in the IMU may have the same set of axes so that both measure motion relative to the same set of axes. For example, as shown, the coordinate system of the left earbud 240 can be defined by an x' axis, a y' axis, and a z' axis, and the coordinate system of the right earbud 242 can be defined by an x" axis, a y" axis, and a z" axis. As further shown, depending on how the left earbud 240 and the right earbud 242 are worn and other factors such as the anatomy of the ear, the coordinate system of the left earbud 240 and the coordinate system of the right earbud 242 may not be aligned with each other or with the coordinate system of the user's head 606. However, the movement of the pair of earbuds 238 - including angular rotation and linear motion that can be detected by the gyroscope and accelerometer, respectively - can be used to determine the orientation of the user's head 606, and therefore determine the user's gaze direction, which can be considered to be consistent with the user's head 606. Figure 6B The y-axis is aligned in .

[0079] By beginning to analyze the sensor data of the user when entering the store at time t=0, the user's gaze direction relative to the content map can be determined. In this regard, when individuals enter a store, they typically walk in a straight line and look straight ahead. Therefore, when the user enters the store, their gaze direction and / or direction of travel may be perpendicular to the entryway. Since their initial orientation is known when the user enters the store, the orientation of the content map, location data, and sensor data can be aligned. Therefore, by tracking the gaze direction throughout the store from time t=0 to t=x, the user's gaze direction throughout the store can be determined relative to the content map at any time, where x is the time at which the user's gaze direction is determined. The determination of the user's gaze direction can be performed by one or more processors and / or computing devices (such as user device 302) of a pair of earbuds 238. In an instance where the computing device performs the determination of the user's gaze direction, the sensor data can be sent from a pair of earbuds 238 to another computing device and / or storage system 310.

[0080] Once data collection begins, or after data collection is complete, location data can be used to determine the location of users as they move or have moved through a store, such as Figure 4As shown in box 406 of . As described herein, the location data can correspond to data provided by GPS, an indoor navigation system, or a sensor (such as an IMU). As with determining the user's gaze direction, determining the user's location throughout the store can be determined relative to the content map at any time. In an instance where the user's location is determined using location data, the user's location can be performed by tracking the user's location throughout the store from time t=0 to t=x, where x is the time at which the user's location is determined. The determination of the user's location can be performed by one or more processors and / or computing devices (such as user device 302) of a pair of earbuds 238.

[0081] Although the foregoing examples describe determining the gaze direction and position of a user at a single time, the gaze direction and position of the user may be determined any number of times. For example, the gaze direction and position of the user may be determined for every second, every fraction of a second, every few seconds, etc. that the user is within the store. For example, starting at time t=0, the gaze direction and position of the user may be determined for every second that the user is within the store.

[0082] Based on the content map, gaze direction, and location of the user at a particular time, intersecting rays can be determined to indicate the products viewed by the user, e.g. Figure 4 The determination of intersecting rays may be performed by a computing device such as server 304 or user device 302. Figure 7 1 shows the position of user 314 at t=0 and a ray along the direction of the gaze of user 314 overlaid on content map 128. As shown, the position of the user is shown by point 702, and ray 704 has a width representing the viewing area of ​​the user when looking along the determined gaze direction. Ray 704 is perpendicular to entrance 114. Figure 7 As further shown in , the ray in the user's gaze direction does not intersect any product location. Therefore, at time t=0, it can be determined that the user 314 is not looking at any product.

[0083] Figure 8The position and light along the gaze direction of user 314 at t=70 are shown, where 70 can be any time value, such as seconds, minutes, etc. In this regard, at time t=70, user 314 is at position 802, which is between shelf 2 124 and shelf 3 126 in aisle 3 108. Light 804 in the user's gaze direction at time t=70 intersects sub-product position 118 on shelf 3 126. Therefore, at time t=70, the viewer can be considered to have viewed the product including the suit at sub-product position 118. In the case where the light overlaps two products or sub-product positions at different distances, the closest product or sub-product position can be selected as the product or sub-product position that the user is viewing. For example, if the user is located in aisle 2 106 and the light corresponding to the user's gaze direction overlaps sub-product position 138 and sub-product position 118, sub-product position 138 can be considered to be the product / sub-product position being viewed.

[0084] In some cases, the amount of time a user views a product location or a sub-product location may be determined. Figure 4 As shown in block 410 of , contextual data may be generated based on the viewed products and in some instances based on the length of time the user views the products. In this regard, the contextual data may include the products viewed by the user and the length of time the user views the products.

[0085] Reference again Figure 8 , light 804 has a width corresponding to the user's viewing area. Based on the location where the light intersects the product location or sub-product location, it can be indicated whether the user is focused on the product at that location. However, gaze directions for light that overlaps two product or sub-product locations or points to the edge of a product or sub-product location may not produce accurate results. To address this issue, the machine model can be trained using the values ​​of the area where the light intersects the product and sub-product locations and the amount of time the user keeps their gaze direction on that area to generate an estimate of whether the user is looking at the product at that product or sub-product location. The value can indicate whether the light intersects the edge or center of the product or sub-product location.

[0086] For example, a machine learning model, such as a neural network, may be trained to generate an attention score indicating the likelihood that a user is viewing a product or sub-product location based on the value and duration that the user is viewing a particular product or sub-product location.

[0087] In addition to the above description, controls may be provided to the user that allow the user to make choices about whether and when the systems, applications, programs, or features described herein may enable the collection of user information (e.g., information about the user's shopping experience, the user's preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that private identifiable information is removed. For example, the user's identity may be processed so that private identifiable information of the user cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0088] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of examples described herein and clauses phrased as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended only to illustrate one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for passively collecting contextual data, the method comprising: receiving, by one or more processors, sensor data corresponding to the orientation of each respective earbud from each earbud of a pair of earbuds worn by a user; determining, by one or more processors, a gaze direction of a user based on received sensor data as the user moves within the facility; as well as One or more products viewed by a user while traveling within the facility are determined by one or more processors by comparing a gaze direction of the user to product locations where the products are stored using a content map of the facility, wherein the one or more products are contextual data.

2. The method according to claim 1, further comprising: As the user travels within the facility, location data for the user is received by one or more processors.

3. The method according to claim 2, wherein: The location data includes GPS data, indoor navigation system data, and / or location data contained in sensor data.

4. The method according to claim 2 or 3, wherein: Comparing the user's gaze direction to the product's location includes: For each look direction: determining from the position data the position of the user when the gaze direction is determined; and A determination is made as to whether a ray from the user's location along the gaze direction intersects a product location among the product locations.

5. The method according to claim 4, wherein: The one or more products are products stored at product locations where corresponding rays intersect.

6. The method according to claim 4 or 5, wherein: The product location includes sub-product locations.

7. A method according to any preceding claim, wherein: Sensor data from each earbud includes data captured by one or more accelerometers and gyroscopes.

8. The method of any preceding claim, further comprising receiving, by one or more processors, a signal from an initiating device, wherein: The signal initiates the collection of sensor data by the pair of earbuds.

9. The method according to any preceding claim, further comprising: receiving, by one or more processors, data from an RFID tag indicating that a user has entered a facility; as well as Initiate collection of sensor data by the pair of earbuds.

10. The method of any preceding claim, further comprising, for each gaze direction: Determine how long the user maintains their gaze direction.

11. A method according to any preceding claim, wherein: One or more processors are within at least one earbud of the pair of earbuds.

12. A method according to any preceding claim, wherein: One or more processors are within the user equipment.

13. A method according to any preceding claim, wherein: The facility is a store.

14. A system comprising: one or more processors; as well as A memory storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving sensor data corresponding to the orientation of each respective earbud from each earbud of a pair of earbuds worn by a user; determining a gaze direction of a user as the user travels within the facility based on the received sensor data; as well as One or more products viewed by the user while traveling within the facility are determined by comparing the user's gaze direction to product locations where the products are stored using a content map of the facility, wherein the one or more products are contextual data.

15. The system of claim 14, wherein: The one or more processors also receive location data of the user as the user travels within the facility.

16. The system of claim 15, wherein: The location data includes GPS data, indoor navigation system data, and / or location data contained in sensor data.

17. The system according to claim 15 or 16, wherein: Comparing the user's gaze direction to the product's location includes: For each look direction: determining from the position data the position of the user when the gaze direction is determined; and A determination is made as to whether a ray from the user's location along the gaze direction intersects a product location among the product locations.

18. The system of claim 14, wherein: The one or more products are products stored at product locations intersected by corresponding rays, wherein the product locations include sub-product locations.

19. A system according to any one of claims 14 to 18, wherein: Sensor data from each earbud includes data captured by one or more accelerometers and gyroscopes.

20. A system according to any one of claims 14 to 19, wherein: The system also includes the pair of earplugs.

21. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 13.

Images