Investigation-based location of electronic shelf label (ESL) devices

Abstract

This disclosure provides systems, methods, and devices for an electronic shelf label (ESL) system that supports positioning of an ESL device. In one aspect, a method includes sending, via a wireless network associated with an environment, a command to each ESL device in a group of ESL devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving visual data collected by the camera for the group during the flash interval from the mobile device via the wireless network; detecting, within the visual data, one or more flashes emitted within the flash interval by each ESL device in the group; and determining a location of each ESL device in the group within the environment based on one or more flashes emitted by each ESL device in the visual data. Other aspects and features are also claimed and described.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 489,598, filed October 18, 2023, entitled “SURVEY-BASED LOCATION OF ELECTRONIC SHELF LABEL (ESL) DEVICES,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to electronic shelf label (ESL) systems, and more specifically to methods and systems for locating ESL devices of ESL systems deployed in indoor environments. Background Technology

[0004] Retail stores typically use paper labels to display information about products displayed on shelves, such as price, discount rates, unit cost, and country of origin. Using such paper labels for price display has limitations. For example, when product information or location on the shelf changes, retailers must generate new paper labels and discard the old ones. This increases maintenance costs for both supply chain and employee labor. Furthermore, from an environmental perspective, replacing labels wastes raw materials such as paper, negatively impacting environmental protection. Moreover, human error is prone to occur, such as mislabeling shelves or products or forgetting to remove temporary price changes from certain shelves, which can lead to shopper frustration.

[0005] Electronic shelf label (ESL) devices are electronic devices used to display prices or other relevant information about items on retail store shelves, replacing paper labels. ESL devices are attached to the front edge of retail shelves and use display devices such as liquid crystal displays (LCDs) or e-ink displays to show various information. ESL devices can be programmed with new product information whenever information about a product or its location changes. Therefore, the same electronic shelf label can be reused.

[0006] Individual devices or nodes in an ESL system can be equipped with Bluetooth Low Energy (BLE) radio components, which can be used to track the location of BLE-enabled devices based on the known locations of nearby ESL devices within an indoor environment, such as a retail environment associated with a retail store or distribution warehouse. However, an ESL system may include hundreds or thousands of ESL devices deployed across different shelves throughout the environment, and determining the location of each ESL device can be a labor-intensive and time-consuming process. The techniques described herein can also provide knowledge of ESL locations, which can be used to link physical and virtual stores (also known as shelf maps) to enable operations such as automated verification of shelf maps. Summary of the Invention

[0007] The following summary outlines some aspects of this disclosure to provide a basic understanding of the techniques discussed. This summary is not an exhaustive overview of all the intended features of this disclosure, nor is it intended to identify key or essential elements of all aspects of this disclosure, nor to define the scope of any or all aspects of this disclosure. The sole purpose of this summary is to present, in a general form, some concepts of one or more aspects of this disclosure as a prelude to the more detailed description that follows.

[0008] Electronic shelf label (ESL) devices deployed within an environment (such as a retail environment associated with a retail store or distribution warehouse) of an ESL system can be used to provide different kinds of information and services to various users of the ESL system, such as shoppers or store staff. For example, an ESL device operating on a wireless network as part of an ESL system can support indoor positioning services to identify the corresponding location of the ESL device within the environment (e.g., on a shelf in a display case within a retail store). As another example, an ESL system can support location services to identify the location of users (or user equipment) and / or other devices or objects within the environment.

[0009] As another example, an ESL system can be used to provide survey techniques for determining the location of ESL devices deployed within an environment based on information collected by a mobile device during an environmental survey. Knowing the location of the ESL devices allows the ESL system to provide more accurate location services, for example, by using the ESL location as a reference point to help determine a user's location in the environment with greater accuracy. In some implementations, the mobile device used for environmental surveys can act as a survey device operated by a human surveyor (e.g., a store worker) or an autonomous robot to collect visual data and other information related to ESL devices near the mobile device as the mobile device (and the user or robot) moves through the environment during the survey. For example, the mobile device may be equipped with a digital camera that captures images and / or videos (e.g., as a sequence of image / video frames) representing groups of different ESL devices appearing at different intervals within the camera's field of view during the survey. Additionally, the device may be equipped with wireless radio components for collecting radio frequency (RF) measurements of beacon signals received from the group of ESL devices at each interval.

[0010] In some implementations, RF measurements collected by mobile devices can be used by the ESL system's server to organize the collection of video data and additional RF data from groups of ESL devices during a survey. For example, the server can broadcast or send commands to each ESL device in an identified group during the survey to fire a flash (e.g., using a light source from an ESL device) within a specified time slot (e.g., 10 ms to 20 ms) during a short period of time within a flash interval associated with that group. As will be described in further detail below, visual data and any RF measurements collected by mobile devices for groups of ESL devices during the survey can be processed by the ESL system's server to distinguish which ESL device fired a flash detected in each image frame captured by the mobile device's camera during the survey (e.g., the identity of the ESL device in the video data). The processed data can be combined with information derived from structure of motion recovery (SFM) techniques applied to the video data to determine the position of each ESL device in the group relative to the camera's pose (e.g., localization and / or orientation) in a three-dimensional (3D) coordinate space representing the environment.

[0011] The example implementation provides a system and method for locating ESL devices in an environment based at least in part on visual data acquired by a camera on a mobile device during an environmental survey.

[0012] In one aspect of this disclosure, a method includes: sending a command from a server via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes during a flash interval associated with the group; receiving visual data collected by a camera for the group from a mobile device via the wireless network; detecting the one or more flashes emitted by each ESL device in the group during the flash interval within the visual data; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0013] In an additional aspect of this disclosure, an apparatus includes at least one processor and memory coupled to the at least one processor. The at least one processor is configured to perform operations including: sending a command via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving visual data collected by a camera for the group during the flash interval from a mobile device via the wireless network; detecting the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0014] In an additional aspect of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. These operations include: sending a command via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving, via the wireless network, visual data collected by a camera for the group during the flash interval from a mobile device; detecting within the visual data the one or more flashes emitted by each ESL device in the group within the flash interval; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0015] In another aspect of this disclosure, an Electronic Shelf Label (ESL) system includes: a group of Electronic Shelf Label (ESL) devices; and a server including a memory and at least one processor coupled to the memory. The at least one processor is configured to perform operations including: sending a command via a wireless network associated with an environment to each ESL device in the group of ESL devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving visual data collected by a camera for the group during the flash interval from a mobile device via the wireless network; detecting the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0016] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily utilized as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and manner of operation) and the associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a limitation of the definitions in the claims.

[0017] Devices, networks, and systems can be configured to communicate via one or more portions of the electromagnetic spectrum. This disclosure refers to certain communication technologies, such as Bluetooth or Wi-Fi, to describe certain aspects. However, this description is not intended to be limited to any particular technology or application, and one or more aspects described with reference to one technology may be understood to be applicable to another technology. Furthermore, it should be understood that, in operation, wireless communication networks adapted according to the concepts herein may operate using any combination of licensed or unlicensed spectrum, depending on load and availability. Therefore, it will be apparent to those skilled in the art that the systems, apparatuses, and methods described herein can be applied to other communication systems and applications besides the specific examples provided.

[0018] For example, the specific implementation described can be implemented in any device, system, or network capable of transmitting and receiving RF signals according to any of the wireless communication standards, including any of the IEEE 802.11 standards, IEEE 802.15.1 Bluetooth, etc. ®Standard, Bluetooth Low Energy (BLE), Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Global System for Mobile Communications (GSM), GSM / General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Terrestrial Trunking Radio (TETRA), Wideband CDMA (W-CDMA), Evolved Data Optimized (EV-DO), 1×EV-DO, EV-DO Revision A, EV-DO Revision B, High-Speed ​​Packet Access (HSPA), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved High-Speed ​​Packet Access (HSPA+), Long Term Evolution (LTE), AMPS, 5G New Radio (5G NR), 6G, or other known signals used for communication within wireless networks, cellular networks, or Internet of Things (IoT) networks (such as systems utilizing 3G, 4G, 5G, or 6G technologies or further implementations thereof).

[0019] In various specific implementations, technologies and devices can be used in wireless communication networks such as Code Division Multiple Access (CDMA) networks, Time Division Multiple Access (TDMA) networks, Frequency Division Multiple Access (FDMA) networks, Orthogonal FDMA (OFDMA) networks, Single Carrier FDMA (SC-FDMA) networks, LTE networks, GSM networks, fifth-generation (5G) or new radio (NR) networks (sometimes referred to as "5G NR" networks, systems, or devices), and other communication networks. As described herein, the terms "network" and "system" are used interchangeably and can refer to a collection of devices capable of communicating with each other via one or more communication technologies.

[0020] While aspects and implementations are described herein by way of example, those skilled in the art will understand that additional implementations and use cases may arise in many different arrangements and scenarios. The innovations described herein can be implemented across many different platform types, devices, systems, shapes, sizes, or package arrangements. For example, implementations or uses may be achieved via integrated chip implementations or other devices based on non-modular components (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail or purchasing devices, medical devices, AI-enabled devices, etc.).

[0021] The scope of implementations can range from chip-level or modular components to non-modular, non-chip-level implementations, and further to aggregated, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more of the described aspects. In some settings, devices incorporating the described aspects and features may also include additional components and features for implementing and practicing the claimed and described aspects. The innovations described herein are expected to be practiced in a wide variety of implementations of different sizes, shapes, or constructions, including both large and small devices, chip-level components, multi-component systems (e.g., radio frequency (RF) chains, communication interfaces, processors), distributed arrangements, end-user equipment, etc.

[0022] In the following description, numerous specific details (such as examples of specific components, circuits, and processes) are set forth to provide a thorough understanding of this disclosure. As used herein, the term "coupled" means a direct connection or a connection via one or more intermediate components or circuits. Furthermore, specific terminology is set forth in the following description and for purposes of explanation to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that practicing the teachings disclosed herein may not require these specific details. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the teachings of this disclosure.

[0023] Certain portions of the following detailed description are presented using other symbolic representations of procedures, logic blocks, processes, and data bit operations within computer memory. In this disclosure, procedures, logic blocks, processes, etc., are conceived as a self-consistent sequence of steps or instructions that produce a desired result. These steps are those that require physical operations on physical quantities. Although not strictly necessary, these physical quantities typically take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated within a computer system.

[0024] In the accompanying drawings, a single block can be described as performing one or more functions. The one or more functions performed by this block may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps are described below in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this disclosure. Additionally, the example device may include components other than those shown, including well-known components such as processors, memory, etc.

[0025] Unless otherwise specifically stated, it will be apparent from the following discussion that, throughout this application, the use of terms such as “access,” “receive,” “transmit,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” “set,” and “generate” refers to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the registers, memories, or other such information storage, transmission, or display devices of the computer system.

[0026] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (such as a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of this disclosure. Although the term "device" is used in the following description and examples to describe various aspects of this disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus can include a device or part of a device for performing the described operations.

[0027] As used herein (including the claims), the term "or" in a list of two or more items means that any one of the listed items may be used alone, or any combination of two or more listed items may be used. For example, if a device is described as containing components A, B, or C, then the device may contain: A alone; B alone; C alone; a combination of A and B; a combination of A and C; a combination of B and C; or a combination of A, B, and C.

[0028] Furthermore, as used herein (including the claims), the word “or” in a list of items beginning with “at least one of” indicates a separate list such that a list such as “at least one of A, B or C” refers to A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these items.

[0029] Additionally, as used herein, the term “substantially” is defined as being largely but not necessarily entirely what is specified (and includes what is specified; for example, substantially 90 degrees includes 90 degrees and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any specific implementation of the disclosure, the term “substantially” may be used in place of the specified content within “[percentage]”, where percentage includes 0.1%, 1%, 5%, or 10%.

[0030] Additionally, as used herein, relative terms, unless otherwise specified, can be understood as a quantity relative to a reference. For example, terms such as “higher” or “lower” or “more” or “less” can be understood as a threshold amount higher, lower, more, or less than a reference value. Attached Figure Description

[0031] A further understanding of the nature and advantages of this disclosure can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals. Furthermore, various components of the same type can be distinguished by adding a dash after the reference numerals and a second reference numeral for differentiation between similar components. If only the first reference numeral is used in the specification, the description applies to any one of the similar components having the same first reference numeral, regardless of the second reference numerals.

[0032] Figure 1A This is a block diagram illustrating a perspective view of an example electronic shelf label (ESL) system in a retail environment according to some embodiments of this disclosure.

[0033] Figure 1B It is based on some implementation schemes of this disclosure. Figure 1A A top-view block diagram of the ESL system.

[0034] Figure 1C These are examples of some implementation schemes according to this disclosure. Figure 1A and Figure 1B A diagram illustrating an example ESL device in an ESL system.

[0035] Figure 2A This is a perspective view of a display shelf including an ESL device for displaying information about products available for selection by a user in a retail environment, according to some embodiments of this disclosure.

[0036] Figure 2B Based on some implementation schemes of this disclosure and Figure 2A A top view of the sample product area corresponding to the aisle between two display shelves within a retail environment.

[0037] Figure 3 This is a timing diagram illustrating time-division multiplexing for communicating with multiple ESL devices according to some embodiments of this disclosure.

[0038] Figure 4 This is a block diagram illustrating an example configuration of an ESL device according to some embodiments of this disclosure.

[0039] Figure 5 This is an illustration of an example of an environmental survey using a mobile device for collecting information for locating ESL devices in an environment, according to some embodiments of this disclosure.

[0040] Figure 6 This is a block diagram of an example server for locating ESL devices in an environment based at least in part on visual data acquired by a camera of a mobile device during an environmental survey, according to some embodiments of this disclosure.

[0041] Figure 7 This is a flowchart of an example method for locating ESL devices in an environment based at least in part on visual data acquired by a camera of a mobile device during an environmental survey, according to some embodiments of this disclosure.

[0042] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0043] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of this disclosure. Rather, the detailed description includes specific details for providing a thorough understanding of the subject matter of the invention. It will be apparent to those skilled in the art that these specific details are not necessary in every situation, and in some cases, well-known structures and components are shown in block diagram form for clarity of presentation.

[0044] This disclosure provides systems, apparatus, methods, and computer-readable media for the indoor positioning of devices, such as ESL devices in an environment, supporting electronic shelf labeling (ESL) systems. Specific implementations of the subject matter described in this disclosure may be implemented to achieve one or more of the potential advantages or benefits listed below. In some aspects, this disclosure provides survey techniques for locating ESL devices, which may be particularly beneficial in retail environments. For example, the location of an ESL device determined using the disclosed measurement techniques can be combined with information related to the layout of the retail environment (e.g., as provided by a shelf map) to create a detailed map of the environment with its unique characteristics and landmarks, such as the specific arrangement of products, shelves, display racks, and other fixtures within the interior space. The ESL system can combine information from the detailed map with signal measurements from different ESL locations to more accurately determine the user's location within the environment and provide enhanced navigation or other location-based services. The terms "environment" and "retail environment" are used interchangeably herein to broadly and inclusively refer to any physical space or layout in which a business operates (e.g., within a "physical" retail store or distribution warehouse) (e.g., to provide products or services to customers). In the example implementation, the shelf map provides a virtual layout of the store, and the shelf map can supplement physical information about the ESL location determined according to the techniques described herein.

[0045] In some implementations, the ESL system may utilize the environment's wireless network and network infrastructure to provide a service for locating ESL devices within the environment based on information collected by a mobile device during an environmental survey. In this context, the mobile device may act as a surveying device operated by a human surveyor (e.g., a store worker) or an autonomous robot (e.g., an autonomous guided vehicle (AGV) or autonomous mobile robot (AMR)) to collect visual data and other information related to nearby ESL devices as it moves through the environment. For example, the mobile device may be equipped with a digital camera that collects sequences of images and / or video frames of ESL devices appearing within the camera's field of view during the survey. Additionally, the mobile device may be equipped with a wireless radio component (e.g., a Bluetooth Low Energy (BLE) radio component) for collecting radio frequency (RF) measurements of beacon signals (e.g., BLE beacons) received from each ESL device. In some implementations, ESL beacons including an ESL identifier (ID) or other information identifying a particular ESL device may be received by the mobile device simultaneously or approximately simultaneously with the capture of images or video frames of the ESL device.

[0046] In some implementations, the ESL system's server may identify a group of ESL devices near a mobile device based on RF measurements received from the mobile device as it moves through the environment. "Nearby" of a mobile device may include a radius or area surrounding the mobile device's current location, within which ESL or other devices close to the mobile device may be located. In some implementations, the size of the radius or surrounding area may be predefined. Alternatively, the size of the area may vary depending on the wireless signaling capabilities supported by the mobile device or the specific wireless technology. For example, devices "near" a mobile device may be limited to those within the mobile device's wireless signaling range and therefore close enough to exchange wireless signals with the mobile device.

[0047] In some implementations, the group of ESL devices identified by the server may include an initial set of ESL devices whose RF measurements (e.g., Received Signal Strength Indicator (RSSI) values) are the strongest and therefore most likely to be within the field of view of the mobile device's camera. The server may also identify additional ESL devices near the mobile device based on additional RF measurements received from each ESL device in this initial set. These additional RF measurements may be based on beacon signals received by each initial ESL device from one or more of its neighboring ESL devices. Such ESL-to-ESL measurements allow the server to identify any neighboring ESL devices whose initial RF measurements received from the mobile device are very weak or completely absent, for example, due to physical obstacles or other environmental conditions affecting signal propagation between the mobile device and neighboring ESL devices. The combination of ESL-to-ESL measurements with RF measurements from the mobile device allows the server to build a dynamic blueprint or chart of nearby ESL devices in the vicinity of the mobile (surveyed) device's current location within the environment (e.g., within a radius or surrounding area). Such measurements may also be used by the server to predict which ESL devices are likely to be near the mobile device along the way and to update the chart accordingly as the mobile device's location within the environment changes during the survey. This may help improve the server's tolerance for any system latency that may occur during the investigation.

[0048] In some implementations, a server can initiate a survey of a group of ESL devices identified as being near a mobile device (e.g., based on their respective RSSI values) by instructing each ESL device in the group to emit a flash using a light source (such as a light-emitting diode (LED)) during a designated time slot (or sub-interval) in the survey period. In some implementations, the “light” emitted by the ESL devices can be invisible to the naked eye and can include, for example, infrared light, ultraviolet light, or other forms of light with wavelengths shorter or longer than visible light. The designated time slot can be a short duration of time (e.g., 10 milliseconds (ms) to 20 ms) allocated to each ESL device within a designated time interval (or “flash interval”) for that group during the survey period. When each ESL device emits a flash during its designated time slot in the survey period, the mobile device’s camera can capture corresponding image or video frames, including the flash emitted by the ESL device.

[0049] As described above, ESL beacons, including information identifying a particular ESL device, can also be received (and measured) by the mobile device simultaneously or approximately simultaneously (e.g., during a specified time slot) with the capture of image / video frames from the flashes of the ESL device by the mobile device's camera. In some implementations, this information can be used by the ESL system's server during post-processing of survey data to distinguish which ESL device emitted the flashes detected in each image frame captured by the mobile device's camera during the survey. If the information received from the mobile device indicates that multiple ESL beacons were received from different ESL devices simultaneously with (or during the same time period) the flashes or corresponding image frames were captured by the camera, the server can use RF measurements from the mobile device to select the ESL device that produced the strongest ESL beacon. In some implementations, the visual data received by the server from the mobile device may include information derived from any image processing performed at the mobile device, for example, after the images were captured by the mobile device's camera. Such information may include, for example, information related to the pixel location of the flashes within the captured image frames and the timing (or corresponding image frames) of the flashes captured on the mobile device. In some implementations, each ESL device may include an ultrasonic transmitter that emits ultrasonic chirps (in addition to flashing lights and / or ESL beacons) that can be detected by the microphone of a mobile device to further distinguish the ESL device from other devices.

[0050] As an alternative to using ESL beacons or RF signals to distinguish flashes emitted by different ESL devices, the server can synchronize the ESL devices to flash multiple times relative to a common reference time at the start of the flash interval. For example, the server can instruct each ESL device in a group to perform an initial synchronized flash (or simultaneous flash) at the common reference time, and then flash individually after a specified offset from the reference time or after a specified delay following the initial synchronized flash. This allows the server to add groups of ESL devices to a common periodic advertising (PA) column corresponding to the flash interval specified for that group during the survey. During the post-survey processing phase, synchronized flashes detected by the server in visual data (e.g., sequences of image / video frames) received from mobile devices can serve as markers indicating the common reference time at the start of the PA column (and the start time of the flash interval associated with the group of ESL devices). Furthermore, each individual flash detected between consecutive frames (or from one frame to the next) can serve as a marker indicating the time slot (or offset relative to the common reference time) specified for a particular ESL device in the group based on its position on the PA column (e.g., based on the order in which it was added to the PA column). In some implementations, flashes emitted by each ESL device at points within a corresponding image frame can be detected by applying one or more object detection algorithms or image filters to the image frame. For example, an image filter can be applied to the image frame to detect flashes emitted by each ESL device based on a sudden change in pixel intensity for a expected duration (e.g., 10 to 20 milliseconds) and / or a sudden change in the color of light associated with the light source of the ESL device.

[0051] In some implementations, the server may apply motion reconstructive techniques to post-process visual data and any associated RF information to determine the trajectories of mobile devices and investigators through the environment during the investigation. In this context, individual ESL flashes detected by the server in the processed visual data (e.g., using object detection and / or other image processing techniques) can serve as uniquely identifiable scale-invariant feature transform (SIFT) features, which can enhance other SIFT features if they do not supersede those features identified by the server from the visual data, for example, using more conventional motion reconstructive techniques.

[0052] The output of such post-processing may include, for example, a point cloud with labeled locations of ESL devices (e.g., xyz coordinates in a three-dimensional (3D) coordinate space labeled as a motion reconstruction map representing the environment in which the ESL device is located or a specific area thereof). For example, points detected within an image frame by flashes emitted by each ESL device can be used to map the location of the ESL device to a set of coordinates in 3D coordinate space. This set of coordinates for each ESL device can be determined using motion reconstruction analysis of the visual data (or the corresponding image frame). The point cloud can then be visualized onto a map of the environment (such as a retail environment like a store or distribution warehouse), which can then be used to provide indoor positioning and navigation services to users (e.g., retail store customers), as described in further detail below.

[0053] Figure 1A This is a perspective block diagram illustrating an example ESL system 100 in a retail environment according to some embodiments of this disclosure. Figure 1A As shown, the ESL system 100 can be deployed across multiple aisles of display shelves 112A to 112H within a retail environment 110. The retail environment 110 may include, for example, a retail floor space of a retail store. As will be described in further detail below, each of the display shelves 112A to 112H may include one or more shelves to which multiple ESL devices are attached. The ESL devices can be used to display pricing and / or other information related to products located on the shelves. Each ESL device may include one or more wireless radio components for transmitting and receiving information via a wireless network associated with the retail store. Such radio components may support any of a variety of wireless communication standards and technologies. Examples of such technologies include, but are not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), 5G NR, Ultra Wideband (UWB), and other RF technologies.

[0054] The following text will refer to Figure 1B In further detail, cameras may be mounted on or near the ends of display shelves 112A to 112H to capture different views of aisles and / or other paths in the area surrounding display shelves 112A to 112H. Additionally, over-the-top (OTT) cameras 102A to 102D (or collectively referred to as “OTT cameras 102”) may be positioned at different over-the-top locations throughout the retail store to capture a larger area of ​​the retail environment 110. In some specific implementations, OTT cameras 102 may be part of a separate surveillance system for the retail store for security and monitoring purposes.

[0055] In some implementations, images and / or videos captured by OTT camera 102 and / or by shelf cameras on display shelves 112A to 112H may be used as Visual Channel Status Information (CSI) to reduce (if not resolvable) any errors or blurriness detected in the positioning of BLE device 124 within retail environment 110. BLE device 124 may be a mobile device of a user (e.g., a customer or employee of a retail store). BLE device 124 may be equipped with a camera for capturing visual data from nearby ESL devices. BLE device 124 may also be equipped with a BLE radio component for receiving low-power beacon signals transmitted by ESL devices attached to display shelves 112A to 112H. In some specific implementations, visual data and radio frequency (RF) measurements of signals received by the BLE radio component of BLE device 124 from nearby ESL devices may be provided to a management server of ESL system 100 for monitoring or tracking the positioning of BLE device 124 (and the user) as they move through different areas of environment 110. In some cases, visual information captured by OTT camera 102 can be used to supplement or replace visual data captured by the camera of BLE device 124, as will be described in further detail below. In some embodiments, the camera may be configured as part of an inventory monitoring system such that visual data identifying empty shelves can be correlated with the physical location of the ESL device to determine out-of-stock products.

[0056] Figure 1B It is based on some implementation schemes of this disclosure. Figure 1A A top-view block diagram of the ESL system 100. (See diagram below.) Figure 1B As shown, the ESL system 100 may also include a management server 122 integrated with or coupled to the gateway node 120. The management server 122 may include at least one processor coupled to memory, wherein the at least one processor is configured to execute computer program code stored on a non-transitory computer-readable medium to cause the management server 122 to perform operations related to the management and control of ESL devices 108A to 108D, access points (APs) 106A and 106B, gateway node 120, and / or other components of the ESL system 100. In some embodiments, the management server 122 may perform operations for locating ESL devices (including ESL devices 108A to 108D) in environment 110 based at least in part on visual data acquired by the camera of the BLE device 124 during an investigation of environment 110, as will be referenced below. Figures 5 to 7 To describe in further detail.

[0057] Gateway node 120 can communicate with access points (APs) 106A and 106B. Although Figure 1BThe example shows only two APs, but ESL system 100 may include fewer or more APs. APs 106A and 106B can communicate with gateway node 120 via a first communication network (wired or wireless communication network). APs 106A and 106B can also communicate with ESL devices 108A through 108D via a second communication network. In some embodiments, the first and second communication networks can be different networks. For example, the first communication network (e.g., between AP 106A and gateway node 120) can be a Wi-Fi network, and the second communication network (e.g., between AP 106A and ESL device 108A) can be a Bluetooth network. In some specific implementations, APs 106A and 106B can be part of the wireless infrastructure of retail environment 110 supporting one or more wireless technologies (e.g., Wi-Fi, Bluetooth, BLE, and / or UWB). ESL system 100 can utilize this infrastructure to implement the survey-based ESL positioning technology disclosed herein.

[0058] In some implementations, each of APs 106A and 106B may communicate with a subset of ESL devices 108A through 108D located in a geographic area assigned to that AP, for example, based on the AP's location relative to each ESL device and / or the AP's signal coverage area. For example, a retail environment 110 may be divided into two equal-sized geographic areas, each covering approximately half of the total retail environment 110. AP 106A may be assigned to geographic area 110A, and AP 106B may be assigned to geographic area 110B. Therefore, AP 106A may communicate with ESL devices 108A and 108B located in geographic area 110A, and AP 106B may communicate with ESL devices 108C and 108D located in geographic area 110B.

[0059] Bluetooth technology provides a secure way to connect and exchange information between electronic devices such as smartphones, other cellular phones, headsets, earphones, smartwatches, laptops, wearable devices, and / or shelf tags. Bluetooth communication may include establishing wireless personal area networks (PANs) (also known as “self-organizing” or “peer-to-peer” networks). These self-organizing networks are often called “piconet”. Each device can belong to multiple piconet. Multiple interconnected piconet can be called a distributed network. A distributed network is formed when members of the first piconet choose to participate in a second piconet. Figure 1B In the example, ESL devices 108A and 108C can be located in a piconet along with AP 106A.

[0060] Because many services offered via Bluetooth may expose private data or allow connected parties to control connected devices, Bluetooth networks may require devices to first establish a "trust relationship" before being allowed to communicate private data with each other. This trust relationship can be established using a process called "pairing," in which a bond is formed between the two devices. This bond allows the devices to communicate with each other in the future without further authentication. Therefore, Figure 1B ESL device 108A can be paired with AP 106A in this manner. The pairing process can be automatically triggered whenever the device is powered on or moves within a certain distance of another Bluetooth device. Pairing information associated with current and previously established pairings can be stored in a Paired Device List (PDL) in the memory of the Bluetooth devices (such as ESL device 108A and / or AP 106A). This pairing information may include a name field, an address field, a link key field, and other similar fields (such as "profile" type) used for authenticating the device or establishing a Bluetooth communication link. When, for example, a power outage causes ESL system 100 to reset, the pairing information allows ESL device 108A to automatically reconnect to AP 106A.

[0061] Bluetooth “profiles” describe the general behavior of Bluetooth-enabled devices communicating with other Bluetooth devices. For example, the Hands-free Profile (HFP) describes how a Bluetooth device (such as a smartphone) can make and receive calls for another Bluetooth device, and the Advanced Audio Distribution Profile (A2DP) describes how stereo-quality audio can be streamed from a first Bluetooth device (such as a smartphone) to another Bluetooth device (such as an earphone). ESL devices 108A to 108D can be configured using ESL profiles that conform to, for example, the Electronic Shelf Label Specification v1.0 dated March 28, 2023, and the Bluetooth Core Specification v5.4 (“Bluetooth Core Specification”) dated January 31, 2023, and / or subsequent versions of such specifications released thereafter (which are incorporated herein by reference). ESL profiles can specify how AP 106A can use one or more ESL services exposed by ESL device 108A.

[0062] In some implementations, management server 122 may include or be coupled to a database that stores and manages product information about products displayed in retail stores or distribution warehouses. The database may also store information used by management server 122 to manage the operation of ESL system 100 and its various components, including ESL devices 108A to 108D and APs 106A and 106B. Such information may include, for example, the location and other relevant information of the various components of ESL system 100, including each of ESL devices 108A to 108D and APs 106A and 106B. In some specific implementations, the database may be part of an enterprise resource planning (ERP) system for managing and tracking inventory (e.g., products), various components of ESL system 100, and other equipment located in the retail environment 110. The ERP inventory management system may be implemented separately from or as part of ESL system 100.

[0063] In some implementations, management server 122 may generate command messages and send them to ESL devices 108A to 108D via the first communication network. The command messages can be used by management server 122 to perform various functions within ESL system 100, such as synchronizing, updating, and changing product information displayed on ESL devices 108A to 108D. Management server 122 may obtain product information from a product database provided for ESL devices 108A to 108D. For example, management server 122 may access the database of the aforementioned ERP system for identification information associated with ESL devices 108A to 108D, which is related to the product information displayed on the corresponding device in ESL devices 108A to 108D.

[0064] Command messages (e.g., product information change messages or management information retrieval messages) created by management server 122 can be transmitted to gateway node 120 as part of a packet suitable for use with gateway node 120. Furthermore, management server 122 can use the communication scheme to receive acceptance messages transmitted from gateway node 120. In some cases, the acceptance message may be converted into a message format suitable for reception by management server 122 before being transmitted by gateway node 120.

[0065] In some implementations, command messages sent by management server 122 via gateway node 120 may include commands directed to different devices in ESL system 100 for conducting a survey of retail environment 110, thereby collecting data for locating ESL devices in environment 110. Examples of such commands may include, but are not limited to, commands instructing groups of BLE devices 124 and / or ESL devices (including one or more of ESL devices 108A to 108D) identified as being near BLE device 124 during the survey to report RF measurements of beacon signals received from nearby ESL devices. The following will discuss... Figures 5 to 7 To describe in further detail, visual data collected by the camera of BLE device 124 during the investigation, as well as RF measurements of ESL beacon signals received from BLE device 124 and / or ESL devices, can be processed by management server 122 using motion structure technology, and the processed information can be used to locate each ESL device (including each of ESL devices 108A to 108D) in the retail environment 110.

[0066] Although only one gateway node 120 is shown in ESL system 100, several such gateway nodes may exist that communicate with management server 122. Each gateway node 120 analyzes the data received from management server 122 to confirm whether there is a message or data to be transmitted to ESL device 108A, and then transmits the confirmed message or data to the corresponding ESL device 108A. Gateway node 120 may configure messages to be transmitted to ESL device 108A into packets according to a suitable communication scheme, and transmit packets with commands to AP 106A to send the packets to ESL device 108A. In addition, gateway node 120 may pass reception acknowledgment messages received from ESL device 108A via AP 106A to management server 122.

[0067] Each of the ESL devices 108A to 108D may include a display, such as a liquid crystal display (LCD) or an electronic ink display, for displaying data about products located on the shelf to which the corresponding ESL device is attached. Information displayed by each of the ESL devices 108A to 108D may be received from the gateway node 120. The ESL devices 108A to 108D may change pricing information or be activated or deactivated while communicating with the gateway node 120. The store manager may transmit commands to the management server 122 regarding the synchronization of products with designated ESL devices and / or commands for correcting information about products assigned to specific ESL devices. Figure 1CAn example of an ESL device 108B is shown, which has a display device for displaying product information such as product descriptions, product images, product prices, product barcodes, product ratings, product stock units (SKUs) and / or product links (e.g., URLs or QR codes)).

[0068] In some implementations, each of the ESL devices 108A to 108D may be configured using memory, a microprocessor, and other components (such as wireless radio components and light sources) for performing various operations related to the survey-based ESL location technologies disclosed herein. Reference will be made below. Figure 4 Examples of configuring such ESL devices are described in more detail.

[0069] In some implementations, visual information obtained from shelf cameras 104A to 104D and / or OTT camera 102 can be used to enhance the indoor positioning and product tracking capabilities of ESL system 100. For example, shelf cameras 104A to 104D may be positioned in the field of view of one or more shelves among display shelves 112A to 112H. Shelf cameras 104A to 104D can be used to assist in tracking inventory levels and / or identifying items picked by users in the environment. Furthermore, OTT camera 102, which may be part of a separate monitoring system for the retail environment 110, may be positioned in different fields of view having a large area captured by the retail environment 110. Object recognition models or systems may be applied to image frames or other visual information received from cameras 102A to 102D and / or 104A to 104D to determine the presence or count of objects and / or people in the field of view of each camera.

[0070] OTT cameras 102A to 102D can be used to support the determination of the location of BLE device 124 and / or other devices within retail environment 110. As described above, BLE device 124 can be a mobile device carried by a user as they move through retail environment 110. When a user moves through retail environment 110 with BLE device 124, BLE device 124 can communicate with ESL devices 108A to 108D, for example, by receiving identification information from ESL devices 108A to 108D. In some embodiments, BLE device 124 may also be equipped with a camera for capturing ESL devices appearing in the camera's field of view as the user moves through different areas of retail environment 110. The location of ESL devices 108A to 108D can be determined by identifying the location of each ESL device within a camera image frame captured by BLE device 124 when BLE device 124 receives a signal and / or by the strength of the signal received from ESL devices 108A to 108D when BLE device 124 captures an image frame during an investigation of retail environment 110, as will be described in further detail below.

[0071] As described above, the retail environment 110 associated with ESL system 100 may include multiple ESL devices (including ESL devices 108A to 108D) organized on shelves in the display racks located throughout the environment 110. Figure 2A An example illustration of such a display shelf with ESL equipment arrangement is shown in the image.

[0072] Figure 2A This is a perspective view of a display rack 112A having ESL devices arranged on various shelves according to some embodiments of the present disclosure. The display rack 112A may include multiple shelves 202A to 202D at different vertical levels from the floor. ESL devices may be attached to shelves 202A to 202D. For example, ESL device 108A may be attached to shelf 202A to display information about products stored on shelf 202A near ESL device 108A.

[0073] ESL devices can provide information to shoppers or store employees operating in the environment, such as providing information about products and / or assisting in determining the location of products or users. Figure 2B This is a top view of a retail environment with user-accessible electronic shelf label (ESL) equipment according to some embodiments of this disclosure. Figure 2B As shown, the display shelf 112A may include densely deployed ESL devices 208 (including... Figure 2A ESL equipment 108A), wherein adjacent ESL equipment may be evenly spaced along shelf 202A. Additional ESL equipment may be similarly positioned along other shelves of display shelf 112A and along the shelves of display shelf 112B.

[0074] A user pushing a shopping cart 212 through the aisle between display shelves 112A and 112B can use ESL device 208 to determine the location of a specific product. For example, this is done in conjunction with the user's shopping cart 212 and / or mobile device (e.g., ...). Figure 1A and Figure 1B The wireless device associated with the BLE device 124 can use RF measurements of radio signals broadcast by the ESL device 208 (and / or other nearby ESL devices) to assess the current location of the shopping cart 212 (and the user) in the environment and guide the user to the desired product location 210 on the appropriate shelf of the display shelf 112A.

[0075] In some implementations, the shopping cart 212 may also be equipped with a camera or any image-capturing device to capture visual information relating to the area of ​​the retail environment in which the shopping cart is located. Visual information captured by the camera on the cart and / or other nearby cameras (such as the camera on the user's mobile device (e.g., BLE device 124)) can be used during an environmental survey to detect one or more features associated with ESL devices in the surrounding environment of the retail environment (e.g., using object detection algorithms or models). The detected features can then be used to determine the location of the ESL devices within the retail environment and map that location to a virtual layout (e.g., a shelf map) and / or a real-world layout of the retail environment. The location of the shopping cart 212 can then be evaluated, at least in part, based on the known locations of the ESL devices. For example, information from shelf cameras (e.g., Figure 1B Shelf-mounted cameras 104A to 104D) and / or OTT cameras (e.g., Figure 1B Visual information from the OTT cameras (102A to 102D) can be used to narrow down the location of the shopping cart 212 and / or the user's location to a specific area of ​​the environment corresponding to the camera's field of view. This may in turn help improve the accuracy of location assessment.

[0076] In some implementations, the AP and the ESL devices of the ESL system (such as...) Figure 1B Communication between AP106A of ESL system 100 and ESL devices 108A to 108D can be performed according to a Time Division Multiple Access (TDMA) scheme, such as... Figure 3 exemplified. Figure 3 This is a timing diagram illustrating time-division multiplexing for communication between an AP and various ESL devices within an ESL system, according to some embodiments of this disclosure. For example... Figure 3 As shown, each ESL device in the AP and ESL systems can be assigned to one of various time slots corresponding to different time slots for sending or receiving information. It should be understood that a similar TDMA scheme can be used for communication between different ESL devices.

[0077] In an ESL system with a large number of ESL devices, the ESL devices can be configured to communicate with APs in different groups, where each group may correspond to a different time frame. For example, Figure 3A first group (ESL1 to ESL5) of ESL devices can be configured to communicate with the AP during a first time frame 310, and a second group (ESL6 to ESL10) of ESL devices can be configured to communicate with the AP during a second time frame 320. The first time frame 310 and the second time frame 320 can alternate during the operation of the wireless network. In this example, the AP can send or broadcast information received by the first group of ESL devices during a first time period (or time slot) 302 in the first time frame 310. The ESL devices in the first group can send information to the AP during subsequent time periods / time slots in the time frame 310. For example, in time frame 310, the first ESL device (ESL1) can send information received by the AP during time period 304A, while the other ESL devices (ESL2 to ESL5) in the group send information during time periods 304B to 304E, respectively. Similarly, the AP can send information received by the second group of ESL devices during time period 306 in the second time frame 320. The ESL devices (ESL6 to ESL10) in this second group can transmit information received by the AP during the corresponding time periods 308A to 308E in time frame 320.

[0078] ESL devices may include components configured together to provide some or all of the functionality described in this disclosure and / or to provide additional functionality. Figure 4 This is a block diagram illustrating an example ESL device 400 according to some embodiments of the present disclosure. For example, the ESL device 400 can be used to implement the above-referenced... Figures 1A to 2B Each of the ESL devices (including ESL devices 108A to 108D) of the described ESL system 100. ESL device 400 may include a low-power microcontroller 410. Although the functionality of the ESL device may be configured by the microcontroller 410 in embodiments of this disclosure, any single processor or combination of processors (e.g., at least one processor) may be used to perform the functionality described according to embodiments of this disclosure.

[0079] Microcontroller 410 may include memory 416. Memory 416 may store computer program code that causes microprocessor 414 to perform some or all of the functionalities described in the various embodiments of this disclosure. Although shown as part of microcontroller 410, memory 416 may be located internally or externally to microcontroller 410. Microcontroller 410 may also include one or more wireless radio components 412. Wireless radio components 412 may include, for example, Bluetooth wireless radio components, including a front end coupled to antenna 408 for transmitting and receiving radio frequency (RF) signals at one or more frequencies in one or more frequency bands. In some embodiments, microcontroller 410 is a system-on-a-chip (SoC), in which two or more components, including wireless radio component 412, microprocessor 414, and / or memory 416, are included in a single semiconductor package. In some embodiments, two or more components may be included on a single semiconductor die.

[0080] ESL device 400 may include I / O devices such as light-emitting diodes (LEDs) 402 and electronic displays 404. LED 402 may include one or more light-emitting diodes (LEDs), or other light sources configured to flash one or more colors of light. LED 402 may be triggered based on commands received from management server 122 via gateway node 120 to flash for a specific duration (e.g., 10 milliseconds to 20 milliseconds) and / or flash in a specific color. For example, LED 402 may flash to draw a user's attention to a specific location on a shelf. In some implementations, groups of ESL devices may be instructed to flash according to a specified color sequence, where each ESL device flashes at least two different colors rapidly and continuously. Using such color sequences can help reduce false alarms because such flashing is highly unlikely to occur accidentally. Furthermore, using different color sequences for flashing by different groups of ESL devices (e.g., one group flashes a series of two or more colors in a specific order, and another group flashes in the opposite order) can at least double the number of different LED flashes (and corresponding ESL devices) that can be detected within a flash interval (e.g., one second). For example, extending this concept to a flash sequence with at least three different colors could result in an eight-fold increase in detection capability. The electronic display 404 could, for example, be an electronic ink display configured to output product information.

[0081] ESL device 400 may be coupled to battery 406 or other power sources to power operations performed by ESL device 400, such as operating wireless radio components 412, LED 402, electronic display 404, memory 416, and / or microprocessor 414. Battery 406 allows ESL device 400 to be placed in locations where a constant power supply is difficult to achieve. Therefore, to enable a single battery charge to provide a long usage period (e.g., lasting longer than several years), ESL device 408 may be configured to reduce power consumption during periods when frequent commands are not expected. For example, ESL device 400 may operate using a wake-up communication scheme. That is, ESL device 400 wakes up at predetermined time intervals to determine if data is waiting to be received. When no data is waiting, power to ESL device 400 is turned off until the next wake-up period to reduce power consumption. When data to be received or transmitted is available, ESL device 400 wakes up to perform communication operations.

[0082] In some implementations, the data received by ESL device 400 may include timing information for synchronizing LEDs 402 of ESL device 400 to emit flashes at specified times during a flash interval specified for a group of ESL devices including ESL device 400. The timing information may include, for example, a first time period at the start of a flash interval (or at another reference point in a flash interval before or after a flash by a single ESL device) when all ESL devices in the group will emit a first flash simultaneously, and a second time period (sub-interval) during the first flash when each ESL device in the group will emit a second flash individually. In this example, the first time period may correspond to a common reference time for the group of ESL devices to emit the first flash, thus marking the start of a periodic advertising column for the group of ESL devices. The second time period may correspond to a specific timing offset or delay specified for each ESL device in the group to emit the second flash after the first flash at the reference time, thus marking the location of each ESL device on the periodic advertising column. Each ESL device in the ESL device may be configured to emit the first and second flashes in a specific color using its corresponding LED or other light source and / or to emit the first and second flashes for a specific duration (e.g., 10 ms or other short duration). In some implementations, each ESL device in the group can also be configured to send a beacon signal simultaneously or approximately simultaneously with the second flash.

[0083] The timing (and color) information of flashes emitted during flash intervals / sub-intervals (and any corresponding beacon signal transmissions) may be included as part of a command transmitted by management server 122 to each ESL device in the group for adding the ESL device group to the periodic advertising column, for example, using the Periodic Advertising Synchronization Delivery (PAST) function of the Bluetooth Core Specification described above for periodic advertising via Bluetooth Low Energy, with the aim of collecting data during a survey in a retail environment, as will be referred to below. Figure 5 To describe in further detail.

[0084] Please note the references above. Figures 1A to 4 The described one or more blocks (or operations) may be referenced below. Figures 5 to 7 This refers to a combination of one or more blocks (or operations) described above. For discussion purposes, reference will be made below to the descriptions above. Figures 1A to 4 ESL systems 100 and / or ESL equipment and other components in retail environments 110 are described. Figures 5 to 7 Examples are provided. However, it should be understood that the survey-based ESL positioning techniques disclosed are not intended to be limiting, and such techniques can be applied to any of a variety of ESL systems and environments.

[0085] Figure 5 This is a diagram illustrating an example of an environmental survey 500 using a mobile device for collecting information for locating ESL devices in an environment, according to some embodiments of this disclosure. This environment may correspond to, for example, as described above. Figure 1B Retail environment 110 or its area, such as a row of shelves 112A or an aisle between shelves 112A and 112B. Figure 5 As shown in the example, mobile devices can use Figure 1A and Figure 1B This is implemented using BLE devices 124, as described above. As described above, each ESL device in this example can use... Figure 4 This is achieved through the configuration of the ESL device 400.

[0086] In this example, the BLE device 124 can serve as a survey device operated by a human investigator 501 (e.g., a retail store worker) carrying the BLE device 124 to collect information related to a group of ESL devices located near the BLE device 124 as the investigator 500 moves through the environment. In some embodiments, the BLE device 124 may be operated or carried by an autonomous robot moving through the environment for the same purpose. The group of ESL devices may include, for example, ESL device 108A and one or more adjacent ESL devices (or “ESLs”) 508 positioned along a shelf 112A.

[0087] During the survey of 500, each ESL device in the group could access the corresponding ESL system's server (e.g., by the group of ESL devices) on the server. Figure 1B The management server 122 (as described above) emits one or more flashes within a specified time or period of the specified flash interval. For example, ESL device 108A can use its LEDs (e.g., Figure 4 An LED 402 (as described above) or other light source emits a flash 502 for a specified duration (e.g., 10 ms to 20 ms) at a specified time during the flash interval. The specified time of flash 502 may be based on a specific offset or delay assigned to ESL device 108A, for example, relative to the start of the flash interval and according to its positioning on a periodic advertising column associated with a group of ESL devices, as described above. In some embodiments, ESL device 108A may also transmit a BLE beacon signal (or “BLE beacon”) 504 at the same time as flash 502.

[0088] BLE device 124 may be equipped with a digital camera having a field of view 510, which captures visual data (e.g., image or video frames) of the flashes 502 emitted by ESL device 108A as investigator 501 moves through the environment with BLE device 124. BLE device 124 may also be equipped with a wireless radio component for collecting RF measurements of the BLE beacon signal 504 while capturing visual data. When ESL 508 flashes its LED individually and appears within the camera's field of view 510, BLE device 124 may also capture visual data (e.g., sequences of image frames) of these ESLs, along with RF measurements transmitted by their respective BLE beacons at corresponding times (or sub-intervals) during specified flash intervals. BLE beacons transmitted by each ESL device in the group (including BLE beacon 504 transmitted by ESL device 108A) may include identification information about the ESL device from which the beacon originated or from which it transmitted the beacon.

[0089] The flashes emitted by each ESL device (including flash 502 emitted by ESL device 108A) can indicate the position of that ESL device within a corresponding image frame captured by the camera of BLE device 124 for that ESL device. In some embodiments, the image frames captured for each ESL device in a sequence of image frames captured by the camera of BLE device 124 for the group of ESL devices may include a timestamp indicating the time of occurrence of the flash emitted by the ESL device relative to the start time of the flash interval associated with the group. Information captured by BLE device 124 for each ESL device in the group during survey 500 (including visual data and RF measurements) can be provided to the server of the ESL system for post-processing.

[0090] In some implementations, the server may use motion reconstruction techniques to post-process the survey information to determine the position and / or pose (e.g., localization and orientation) of the camera within each image or video frame. The camera's position and / or pose may be represented as xyz coordinates in a three-dimensional (3D) coordinate space, for example, representing a motion reconstruction map of the environment (or a corresponding area thereof). The camera position / pose may be combined with ESL positions (such as those identified by BLE beacons captured at corresponding times) indicated by visual data captured for each ESL device (e.g., the position of a flash within an image frame) to determine the coordinates of the ESL device's position in the 3D coordinate space. In some implementations, visual data captured by the cameras of BLE device 124 for at least a portion of the environment may be captured by one or more OTT cameras (e.g., [missing information]) associated with a monitoring system or inventory monitoring system of the environment. Figure 1B Visual information captured within the field of view of the OTT camera 102 can supplement or replace it. For example, given the pose of each OTT camera, visual data from multiple OTT cameras can be used to detect flashes of all ESL devices corresponding to their respective fields of view. In some implementations, each ESL device can be identified in real time using visual data and / or RF measurements obtained during the survey, prior to determining the location of the ESL devices during post-processing. By separating the processing of identification information from the relatively more complex data processing associated with location determination, the ESL system can focus the remaining survey on any ESL devices in the environment that are still undetected or unidentified. This, in turn, allows the ESL system to appropriately allocate flash intervals and time slots only for undetected ESLs, and thus complete the survey in a more efficient manner.

[0091] Figure 6 It is based, in part, on some embodiments of this disclosure, by mobile devices (e.g., Figure 5 A block diagram of an example server 600 is shown, illustrating how a camera (of BLE device 124, as described above) acquires visual data during an environmental survey to locate ESL devices in the environment. Server 600 can be used to implement applications such as those deployed in environments (e.g., Figure 1A and Figure 1B The management server of the ESL system within the retail environment 110 (as described above) (e.g., Figure 1B The management server 122 of the ESL system 100 (as described above). Figure 6As shown, server 600 includes network interface 610, ESL locator 620, and data visualization device 630. ESL locator 620 may include survey manager 622 and post-processor 624. As will be described in further detail below, survey manager 622 can be used to orchestrate the actions of ESL devices for data collection by survey device 640 during a survey of the environment (or its associated area), and post-processor 624 can be used to post-process the data collected by survey device 640 during the survey.

[0092] In some embodiments, the survey device 640 may be a mobile device equipped with a camera for collecting visual data (e.g., sequences of images and / or video frames) associated with an ESL device that appears within the camera's field of view during the survey. Figure 1A and Figure 1B The BLE device 124 (as described above) is described above. Additionally, the survey device 640 may be equipped with a wireless radio component (e.g., a Bluetooth Low Energy (BLE) radio component) for collecting radio frequency (RF) measurements of beacon signals (e.g., BLE beacons) received from ESL devices near the survey device 640. In some embodiments, ESL beacons including an ESL identifier (ID) or other information identifying a particular ESL device may be received by the survey device 640 simultaneously or approximately simultaneously with the capture of images or video frames of the ESL device by the camera of the survey device 640.

[0093] Server 600 and its components or sub-components (including Survey Manager 622 and Post-processor 624 of ESL Locator 620) can communicate via network interface 610 with Survey Device 640 and each ESL device in the environment's ESL devices through a communication network. Such a network can be any network or combination of networks carrying data communication according to any of a variety of communication standards. The network can include, but is not limited to, wired or wireless networks. The network can also include local area networks, medium area networks, or wide area networks (such as the Internet). The network and network interface 610 can support any of a variety of networking protocols and technologies as desired in a particular implementation.

[0094] In some implementations, the survey manager 622 may use RF measurements of ESL beacons received from the survey device 640 via the network interface 610 to identify groups of ESL devices near the survey device 640. The groups identified by the survey manager 622 may include, for example, an initial set of ESL devices whose RF measurements (e.g., Received Signal Strength Indicator (RSSI) values) are the strongest and therefore most likely to be within the field of view of the camera at the survey device 640. The survey manager 622 may also identify additional ESL devices near the survey device 640 based on additional RF measurements received from each ESL device in this initial set. These additional RF measurements may be based on beacon signals received by each initial ESL device from one or more of its neighboring ESL devices. Such ESL-to-ESL measurements allow the survey manager 622 to identify any neighboring ESL devices whose initial RF measurements received from the survey device 640 are too weak or unavailable, for example, due to physical obstacles or other environmental conditions affecting signal propagation between the survey device 640 and the neighboring ESL device.

[0095] In some implementations, the survey manager 622 may initiate a survey of a group of ESL devices, as identified by RF measurements received from the survey device 640 and ESL-to-ESL measurements from neighboring ESL devices, as described above. The survey manager 622 may then send a command to each ESL device in the group to use the light source of the ESL device (e.g., LED 402 of ESL device 400, as described above) at a specified time or time slot corresponding to a different sub-interval of the flash interval specified for that group during the survey. Figure 4 (As shown) emits a flash.

[0096] When each ESL device emits a flash during its designated time slot in the survey period, the camera of the survey device 640 can capture corresponding image frames or video frames of the ESL device, including flashes emitted by its LEDs or light sources. As described above, the designated time slot for each ESL device can be a short duration (e.g., 10 to 20 milliseconds) during a designated flash interval. Keeping the duration of each time slot (or sub-interval) relatively short allows for the capture of a larger number of ESL devices and their corresponding flashes in the visual data. For example, flashing an ESL LED for only 10 milliseconds allows for the capture of flashes from 100 different ESL devices within a flash interval lasting one second. Additionally, as described above, the timing information of each ESL device can be used to synchronize the timing of individual flashes emitted by the group of ESL devices according to a designated offset or delay relative to a common reference time at the start of the designated flash interval in that group. For example, the offset assigned to each ESL device in the group can be incremented according to the order in which the ESL devices are indicated to flash, where the ESL device assessed as closest to the survey device 640 is indicated first, followed by the second closest, and so on. The order of ESL devices can be determined based on the relative strength of ESL beacons received by survey device 640 (where the nearest ESL device produces the strongest ESL beacon), as indicated by the initial RF measurement received by survey manager 622 from survey device 640 and any ESL-to-ESL measurements of neighboring ESL devices received from one or more ESL devices, as described above.

[0097] In some implementations, the survey manager 622 may identify different groups of ESL devices deployed in different areas of the environment, and may collect different sequences of image frames for each group as the survey device 640 moves through the first and second areas during a survey. For example, a first sequence of image frames may be collected by the camera of the survey device 640 during a first flash interval associated with the first group, and a second sequence of image frames may be collected by the camera during a second flash interval associated with the second group. In some specific implementations, a single flash interval may be specified for the first and second groups, and each ESL device in these groups may be assigned a different time slot to fire a flash within the same flash interval. For example, if each group includes three ESL devices, the three ESL devices in the first group may be assigned time slots 1, 2, and 3 within the flash interval, respectively, and the three ESL devices in the second group may be assigned time slots 4, 5, and 6 within the flash interval, respectively.

[0098] To distinguish common reference points in the visual data (e.g., a sequence of image frames) captured by the survey device 640 during post-processing operations to be performed by the post-processor 624, the survey manager 622 may instruct all ESL devices in the group to simultaneously perform synchronized flashes (e.g., by simultaneously flashing their respective LEDs at a common reference time). Individual flashes subsequently emitted by each ESL device at a specified offset can then be used to distinguish individual ESL devices from one another based on the order in which the ESL devices in the group are instructed, as described above. In some embodiments, each ESL device may also be configured to send a beacon (e.g., similar to its identification information, such as a unique ESLID corresponding to the ESL device) at the same time (or time period) specified for the flash. Figure 5 (BLE beacon 504, as described above). The ESL beacon and ID information received by the survey device 640 at a specified time can be used as an alternative to using a common reference time and timing offset to distinguish individual ESL device flashes in the visual data collected by the camera of the survey device 640.

[0099] In some implementations, the survey manager 622 can vary the frequency of the light emitted by the LEDs of each ESL device to produce light of different colors or wavelengths. For example, in addition to visible light, the ESL devices can also produce infrared, ultraviolet, or other forms of light at wavelengths invisible to the naked eye to avoid any visual interference in the environment. Each color option available for the individual flashes produced by the ESL devices can be treated as a separate channel by the survey manager 622 to improve system capacity based on the number of ESL devices that can be identified during each flash interval (e.g., lasting one second). By using multiple LEDs, the capacity can be doubled for each additional LED. The additional capacity can optionally be used to reduce the timing requirements of a fixed number of LEDs. Furthermore, using different colored LEDs for ESL devices installed in known separate locations on shelves or along display shelf tracks allows the flashes produced by such LEDs to be seen in the same image captured by the camera of the survey device 640, and thereby provides scale measurements that can be used to assist in the post-processing of survey data performed by the post-processor 624, as described below.

[0100] In some implementations, post-processor 624 can perform survey post-processing by applying Structure of Motion (SFM) techniques to analyze visual data and other information collected and reported by survey device 640 during the survey. For example, post-processor 624 can analyze each image or video frame to detect unique features within each frame. Examples of such features include, but are not limited to, points, corners, edges, or other distinguishable patterns common across multiple frames. In some specific implementations, post-processor 624 can use Scale Invariant Feature Transform (SIFT) to detect and match local features from one frame to the next. A flash emitted by an ESL device can be a uniquely identifiable point source of the corresponding ESL device within a frame processed by post-processor 624. Therefore, an ESL flash can serve as a reliable (or highly reliable) feature that can be used to enhance or replace features derived from other images, such as SIFT features. Reducing the number of such features has the potential to significantly reduce the computational requirements of the structure of motion processing performed by post-processor 624 while preserving its functionality and performance.

[0101] In some implementations, the post-processor 624 may determine the position and / or pose (e.g., localization and / or orientation) of each ESL device in the first group relative to a camera in the environment based on a flash detected for the ESL device in visual data (e.g., within a corresponding image frame captured by a camera of the survey device 640).

[0102] The output of post-processing performed by post-processor 624 may include, for example, a point cloud with labeled locations of ESL devices (e.g., labeled as xyz coordinates in a three-dimensional (3D) coordinate space representing the environment in which the ESL devices are located or a specific area thereof). Data visualizer 630 can then be used to visualize the point cloud onto a map of the environment. As described above, this environment may be a retail environment associated with a store or distribution warehouse. Therefore, a visualized map including the locations of ESL devices, such as a map applied to a store, can be used to provide indoor positioning and navigation services to users (e.g., retail store customers), such as for locating desired products on shelves in a specific area of ​​a retail environment, as referenced above. Figure 2A and Figure 2B As described.

[0103] Figure 7 This is a flowchart of an example method 700 for locating ESL devices in an environment based at least in part on visual data acquired by a camera of a mobile device during an environmental survey, according to some embodiments of this disclosure. For discussion purposes, the methods described above will be used. Figure 1B ESL system 100 and / or as described above Figure 6Method 700 is described using server 600. However, method 700 is not intended to be limiting. For example, method 700, which includes the operations described in the boxes below, can be managed by management server 122 of ESL system 100 as described above, or by the server 600 described above. Figure 6 The server 600 components are executed.

[0104] Method 700 begins at box 702, which includes an environment (e.g., Figure 1B Radio frequency (RF) measurements were taken during the investigation of the retail environment (110) of receiving wireless signals from mobile devices via wireless networks from ESL devices deployed in the environment.

[0105] At box 704, a group of ESL devices near the mobile device is identified based on the RF measurements received at box 702.

[0106] At box 706, a command is sent via wireless network to each ESL device in the group to fire a flash at a time (or time period) specified for that ESL device during the flash interval associated with the group.

[0107] At frame 708, visual data collected by the mobile device's camera for the group during flash intervals is received from the mobile device via a wireless network.

[0108] Method 700 then proceeds to box 710, which includes detecting within visual data a flash emitted by each ESL device in the group at a specified time during the flash interval.

[0109] At box 712, the position of each ESL device in the group relative to the camera in the environment is determined based on the flashes detected for the ESL device in the visual data.

[0110] In some implementations, different groups of ESL devices may be deployed in different areas of the environment. For example, the group of ESL devices whose location is determined at box 712 may be a first group of ESL devices deployed in a first area of ​​the environment. The visual data received for this first group at box 708 above may include a first sequence of image frames collected by a camera of a mobile device during flash intervals associated with the first group. As the mobile device moves through the first and second areas during the survey, additional visual data (including a second sequence of image frames) may be received (at box 708) by a camera for a second group of ESL devices deployed in a second area of ​​the environment. The second sequence of image frames may be collected by a camera during different flash intervals associated with the second group. The operations at boxes 710 and 712 may then be repeated to determine the location of each ESL device in the second group within the environment based on the corresponding flashes detected for the ESL devices in the additional visual data (or the second sequence of image frames).

[0111] In one or more aspects, the technology used to support ESL operations may include additional aspects, such as any single aspect or any combination of aspects described below or in conjunction with one or more other processes or devices described elsewhere in this document.

[0112] In a first aspect, a method includes: a server sending a command via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; the server receiving visual data collected by a camera for the group from a mobile device via the wireless network during the flash interval; the server detecting the one or more flashes emitted by each ESL device in the group within the flash interval in the visual data; and the server determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0113] In a second aspect, in conjunction with the first aspect, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the detection further includes: receiving, via the wireless network, the ESL beacon transmitted by each ESL device in the group during the flash interval from the mobile device by the server; and, for each flash detected in the visual data, identifying different ESL devices in the group based on the identification information included in the ESL beacon transmitted during the time period corresponding to the flash.

[0114] In a third aspect, in conjunction with one or more of the first or second aspects, wherein each ESL device in the group is located near the mobile device during the investigation of the environment, and wherein the method further includes: receiving by the server, via the wireless network, radio frequency (RF) measurements of wireless signals received by the mobile device from a subset of the ESL devices deployed in the environment by the mobile device during the investigation; and identifying each ESL device in the group by the server based on the RF measurements received for the subset.

[0115] In a fourth aspect, in conjunction with one or more of the first to third aspects, the identifier further includes: receiving by the server additional RF measurements of beacons sent by one or more neighboring ESL devices from at least one ESL device in the group; and identifying the group of ESL devices near the mobile device by the server based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

[0116] In a fifth aspect, in conjunction with one or more of the first to fourth aspects, the one or more flashes include a first flash emitted simultaneously by all the ESL devices in the group at the start of the flash interval and a second flash emitted individually by each ESL device in the group after a specified delay from the start of the flash interval, wherein the method further includes: detecting within the visual data the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; determining a common reference time corresponding to the start of the flash interval of the group based on the first flash detected within the visual data; and identifying each ESL device in the group based on the specified delay corresponding to the ESL device and the timing of the second flash detected within the visual data relative to the common reference time.

[0117] In the sixth aspect, in conjunction with one or more of the first to fifth aspects, the visual data includes image frames collected by the camera for each ESL device in the group during the survey of the environment.

[0118] In a seventh aspect, in conjunction with one or more of the first to sixth aspects, different groups of ESL devices are deployed in different areas of the environment, wherein the group of ESL devices is a first group of ESL devices deployed in a first area of ​​the environment, wherein a second group of ESL devices is deployed in a second area of ​​the environment, wherein the visual data includes a first sequence and a second sequence of image frames collected by the camera for the respective first group and second group as the mobile device moves through the first area and the second area during the survey, wherein the first sequence is collected by the camera during the flash interval associated with the first group, and wherein the second sequence is collected by the camera during different flash intervals associated with the second group.

[0119] In the eighth aspect, in conjunction with one or more of the first to seventh aspects, the position of each ESL device in the group is represented as a set of coordinates relative to the pose of the camera in a three-dimensional (3D) coordinate space representing the environment.

[0120] In the ninth aspect, in conjunction with one or more of the first to eighth aspects, this set of coordinates for each ESL device is determined by motion recovery structure analysis using the visual data.

[0121] In a tenth aspect, in conjunction with one or more of the first to ninth aspects, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group within the flash interval, wherein the visual data includes a sequence of image frames collected by the camera for the group within the flash interval, wherein each image frame in the sequence corresponds to a different ESL device in the group, and wherein determining the location of each ESL device includes: applying an image filter to detect the flash emitted by the ESL device at a point within the corresponding image frame; and mapping the point within the corresponding image frame to the set of coordinates in the 3D coordinate space.

[0122] In an eleventh aspect, an apparatus includes: a memory storing processor-readable code; and at least one processor coupled to the memory, wherein the at least one processor is configured to execute the processor-readable code to cause the at least one processor to perform operations including: sending a command via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving visual data collected by a camera for the group during the flash interval from a mobile device via the wireless network; detecting the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0123] In a twelfth aspect, in conjunction with the eleventh aspect, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further includes: receiving, via the wireless network, the ESL beacon transmitted by each ESL device in the group during the flash interval from the mobile device; and, for each flash detected in the visual data, identifying different ESL devices in the group based on the identification information included in the ESL beacon transmitted during the time period corresponding to the flash.

[0124] In the thirteenth aspect, in conjunction with one or more of the eleventh or twelfth aspects, wherein each ESL device in the group is located near the mobile device during the investigation of the environment, wherein the operation further includes: receiving, during the investigation, radio frequency (RF) measurements from the mobile device via the wireless network of wireless signals received by the mobile device from a subset of the ESL devices deployed in the environment; and identifying each ESL device in the group based on the RF measurements received for the subset.

[0125] In the fourteenth aspect, in conjunction with one or more of aspects eleven to thirteen, the operation for identification further includes: receiving additional RF measurements of beacons sent by one or more neighboring ESL devices from at least one ESL device in the group; and identifying the group of ESL devices near the mobile device based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

[0126] In the fifteenth aspect, in conjunction with one or more of aspects eleven to fourteen, the one or more flashes include a first flash emitted simultaneously by all the ESL devices in the group at the start of the flash interval and a second flash emitted individually by each ESL device in the group after a specified delay from the start of the flash interval, wherein the operation further includes: detecting within the visual data the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; determining a common reference time corresponding to the start of the flash interval for the group based on the first flash detected within the visual data; and identifying each ESL device in the group based on the specified delay corresponding to the ESL device and the timing of the second flash detected within the visual data relative to the common reference time.

[0127] In the sixteenth aspect, in conjunction with one or more of aspects eleven through fifteen, the visual data includes image frames collected by the camera for each ESL device in the group during the survey of the environment.

[0128] In the seventeenth aspect, in conjunction with one or more of aspects eleven to sixteen, different groups of ESL devices are deployed in different areas of the environment, wherein the group of ESL devices is a first group of ESL devices deployed in a first area of ​​the environment, wherein a second group of ESL devices is deployed in a second area of ​​the environment, wherein the visual data includes a first sequence and a second sequence of image frames collected by the camera for the respective first group and second group as the mobile device moves through the first area and the second area during the survey, wherein the first sequence is collected by the camera during the flash interval associated with the first group, and wherein the second sequence is collected by the camera during different flash intervals associated with the second group.

[0129] In the eighteenth aspect, in conjunction with one or more of aspects eleven through seventeen, the position of each ESL device in the group is represented as a set of coordinates relative to the pose of the camera in a three-dimensional (3D) coordinate space representing the environment.

[0130] In the nineteenth aspect, in conjunction with one or more of aspects eleven through eighteen, the set of coordinates for each ESL device is determined by motion recovery structure analysis using the visual data.

[0131] In the twentieth aspect, in conjunction with one or more of aspects eleven to nineteen, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group within the flash interval, wherein the visual data includes a sequence of image frames collected by the camera for the group within the flash interval, wherein each image frame in the sequence corresponds to a different ESL device in the group, and wherein the operation for determining the location of each ESL device includes: applying an image filter to detect the flash emitted by the ESL device at a point within the corresponding image frame; and mapping the point within the corresponding image frame to the set of coordinates in the 3D coordinate space.

[0132] In a twenty-first aspect, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: sending a command via a wireless network associated with an environment to each ESL device in a group of electronic shelf label (ESL) devices deployed in the environment to emit one or more flashes within a flash interval associated with the group; receiving via the wireless network visual data collected by a camera for the group during the flash interval from a mobile device; detecting within the visual data the one or more flashes emitted by each ESL device in the group within the flash interval; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0133] In a twenty-second aspect, in conjunction with the twenty-first aspect, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further includes: receiving, via the wireless network, the ESL beacon transmitted by each ESL device in the group during the flash interval from the mobile device; and, for each flash detected in the visual data, identifying different ESL devices in the group based on the identification information included in the ESL beacon transmitted during the time period corresponding to the flash.

[0134] In the twentieth aspect, in combination with one or more of the twentieth or twentieth aspects, wherein each ESL device in the group is located near the mobile device during the investigation of the environment, and wherein the operation further includes: receiving, during the investigation, radio frequency (RF) measurements from the mobile device via the wireless network of wireless signals received by the mobile device from a subset of the ESL devices deployed in the environment; and identifying each ESL device in the group based on the RF measurements received for the subset.

[0135] In the 24th aspect, in combination with one or more of aspects 21 to 23, the operations for identification further include: receiving additional RF measurements of beacons sent by one or more neighboring ESL devices from at least one ESL device in the group; and identifying the group of ESL devices near the mobile device based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

[0136] In the 25th aspect, in combination with one or more of aspects 21 to 24, the one or more flashes include a first flash emitted simultaneously by all the ESL devices in the group at the start of the flash interval and a second flash emitted individually by each ESL device in the group after a specified delay from the start of the flash interval, and wherein the operation further includes: detecting within the visual data the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; determining a common reference time corresponding to the start of the flash interval for the group based on the first flash detected within the visual data; and identifying each ESL device in the group based on the specified delay corresponding to the ESL device and the timing of the second flash detected within the visual data relative to the common reference time.

[0137] In a twenty-sixth aspect, an Electronic Shelf Label (ESL) system includes: a group of ESL devices deployed within an environment; and a server including a memory and at least one processor coupled to the memory, wherein the at least one processor is configured to perform operations including: sending a command to each ESL device in the group via a wireless network associated with the environment to emit one or more flashes within a flash interval associated with the group; receiving visual data collected by a camera for the group during the flash interval from a mobile device via the wireless network; detecting the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; and determining the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

[0138] In a twentieth aspect, in conjunction with the twentieth aspect, the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further includes: receiving, via the wireless network, the ESL beacon transmitted by each ESL device in the group during the flash interval from the mobile device; and, for each flash detected in the visual data, identifying different ESL devices in the group based on the identification information included in the ESL beacon transmitted during the time period corresponding to the flash.

[0139] In the twentieth aspect, in conjunction with the twentieth aspect, wherein each ESL device in the group is located near the mobile device during the investigation of the environment, wherein the operation further includes: receiving, during the investigation, radio frequency (RF) measurements from the mobile device via the wireless network of wireless signals received by the mobile device from a subset of the ESL devices deployed in the environment; and identifying each ESL device in the group based on the RF measurements received for the subset.

[0140] In the twentieth aspect, in combination with one or more of the twentieth or twentieth aspects, the operation further includes: receiving additional RF measurements of beacons transmitted by one or more neighboring ESL devices from at least one ESL device in the group; and identifying the group of ESL devices near the mobile device based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

[0141] In the thirtieth aspect, in conjunction with one or more of aspects 26 to 29, the one or more flashes include a first flash emitted simultaneously by all the ESL devices in the group at the start of the flash interval and a second flash emitted individually by each ESL device in the group after a specified delay from the start of the flash interval, and wherein the operation further includes: detecting within the visual data the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; determining a common reference time corresponding to the start of the flash interval for the group based on the first flash detected within the visual data; and identifying each ESL device in the group based on the specified delay corresponding to the ESL device and the timing of the second flash detected within the visual data relative to the common reference time.

[0142] The components, functional blocks, and modules described herein with respect to the accompanying figures include processors, electronic devices, hardware devices, electronic components, logic circuits, memory, software code, firmware code, and so on, or any combination thereof. Software should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, and / or functions, regardless of whether it is referred to as software, firmware, middleware, microcode, hardware description languages, or other terms. Furthermore, the features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

[0143] Those skilled in the art will further understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be construed as departing from the scope of this disclosure. Those skilled in the art will also readily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and that components, methods, or interactions of various aspects of this disclosure may be combined or performed in ways other than those illustrated and described herein.

[0144] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been broadly described in terms of functionality and illustrated in the aforementioned exemplary components, blocks, modules, circuits, and processes. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0145] Hardware and data processing means for implementing the various exemplary logic, logic blocks, modules, and circuits described herein can be implemented or executed using general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some embodiments, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some embodiments, specific processes and methods may be performed by circuitry specific to a given function.

[0146] In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuits, computer software, firmware, including the structures disclosed in this specification and their structural equivalents or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0147] If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted through a computer-readable medium. The processes of the methods or algorithms disclosed herein can be implemented in a processor-executable software module that can reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can be implemented to transfer a computer program from one location to another. Storage media can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible to a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically reproduce data, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, the operation of a method or algorithm may reside as a set of code and instructions or any combination of code and instructions on a machine-readable medium and a computer-readable medium that may be incorporated into a computer program product.

[0148] Various modifications to the specific embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are to be granted the broadest scope consistent with this disclosure, the principles disclosed herein, and the novel features.

[0149] Certain features described in this specification in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even originally claimed in this way, one or more features from the claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0150] Similarly, although operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the indicated specific order or sequential order, or to perform all illustrated operations to achieve the desired result. Furthermore, the figures may schematically depict one or more example processes in the form of flowcharts. However, other operations not depicted may be combined with the schematically illustrated example processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any illustrated operation. In some contexts, multitasking and parallel processing are advantageous. Moreover, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other embodiments also fall within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

[0151] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, the method comprising: The server sends commands via a wireless network associated with the environment to each ESL device in a group of electronic shelf label (ESL) devices deployed within the environment to emit one or more flashes within a flash interval associated with the group; The server receives visual data collected by the camera for the group during the flash interval from the mobile device via the wireless network; The server detects, within the visual data, one or more flashes emitted by each ESL device in the group within the flash interval; as well as The server determines the location of each ESL device in the group within the environment based on the one or more flashes emitted by each ESL device in the visual data.

2. The method of claim 1, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the detection further includes: The server receives the ESL beacon sent by each ESL device in the group during the flash interval from the mobile device via the wireless network; as well as For each flash detected in the visual data, different ESL devices in the group are identified based on the identification information included in the ESL beacons transmitted during the time period corresponding to the flash.

3. The method of claim 1, wherein each ESL device in the group is located near the mobile device during the environmental survey, and wherein the method further comprises: During the investigation, the server receives radio frequency (RF) measurements from the mobile device via the wireless network, which in turn receive wireless signals from a subset of the ESL devices deployed within the environment. as well as Each ESL device in the group is identified by the server based on the RF measurements received for the subset.

4. The method of claim 3, wherein the identifier further comprises: The server receives additional RF measurements of beacons sent by one or more neighboring ESL devices from at least one ESL device in the group; as well as The server identifies the group of ESL devices near the mobile device based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

5. The method of claim 1, wherein the one or more flashes comprise a first flash simultaneously emitted by all the ESL devices in the group at a reference point of the flash interval and a second flash individually emitted by a single ESL device in the group at a specified difference from the reference point of the flash interval, and wherein the method further comprises: Within the visual data, detect the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; A common reference time corresponding to the reference point of the flash interval of the group is determined based on the first flash detected within the visual data; as well as Each ESL device in the group is identified based on a specified difference corresponding to each ESL device and the timing of a second flash detected within the visual data corresponding to each ESL device relative to the common reference time.

6. The method of claim 1, wherein the visual data comprises image frames collected by the camera for each ESL device in the group during a survey of the environment.

7. The method of claim 6, wherein different groups of ESL devices are deployed in different areas of the environment, wherein the group of ESL devices is a first group of ESL devices deployed in a first area of ​​the environment, wherein a second group of ESL devices is deployed in a second area of ​​the environment, wherein the visual data includes a first sequence and a second sequence of image frames collected by the camera for the respective first group and second group as the mobile device moves through the first area and the second area during the survey, and wherein the first sequence is collected by the camera during the flash interval associated with the first group, and wherein the second sequence is collected by the camera during different flash intervals associated with the second group.

8. The method of claim 1, wherein the position of each ESL device in the group is represented as a set of coordinates relative to the pose of the camera in a three-dimensional (3D) coordinate space representing the environment.

9. The method of claim 8, wherein the set of coordinates for each ESL device is determined using motion reconstruction analysis of the visual data.

10. The method of claim 9, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during the flash interval, wherein the visual data includes a sequence of image frames collected by the camera for the group during the flash interval, wherein each image frame in the sequence corresponds to a different ESL device in the group, and wherein determining the location of each ESL device includes: An image filter is applied to detect the flash emitted by the ESL device at a point within the corresponding image frame; as well as Map the points within the corresponding image frame to the set of coordinates in the 3D coordinate space.

11. An apparatus comprising: Memory, the memory storing processor-readable code; and At least one processor coupled to the memory, the at least one processor being configured to execute processor-readable code to cause the at least one processor to perform operations including: Commands are sent via an environment-associated wireless network to each ESL device in a group of electronic shelf label (ESL) devices deployed within the environment to emit one or more flashes within a flash interval associated with the group; Receive visual data collected by the camera for the group during the flash interval from the mobile device via the wireless network; Detect the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; as well as The location of each ESL device in the group within the environment is determined based on the one or more flashes emitted by each ESL device in the visual data.

12. The apparatus of claim 11, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period assigned to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further comprises: Receive from the mobile device via the wireless network the ESL beacon transmitted by each ESL device in the group during the flash interval; as well as For each flash detected in the visual data, different ESL devices in the group are identified based on the identification information included in the ESL beacons transmitted during the time period corresponding to the flash.

13. The apparatus of claim 11, wherein each ESL device in the group is located near the mobile device during the environmental survey, and wherein the operation further comprises: During the investigation, radio frequency (RF) measurements were performed on the mobile device via the wireless network, receiving wireless signals from a subset of the ESL devices deployed within the environment. as well as Each ESL device in the group is identified based on the RF measurements received for the subset.

14. The apparatus of claim 13, wherein the operation for identification further comprises: Receive additional RF measurements of beacons transmitted by one or more neighboring ESL devices from at least one ESL device in the group; as well as The group of ESL devices near the mobile device is identified based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

15. The apparatus of claim 11, wherein the one or more flashes comprise a first flash simultaneously emitted by all of the ESL devices in the group at a reference point of the flash interval and a second flash individually emitted by a single ESL device in the group at a specified difference from the reference point of the flash interval, and wherein the processor is configured to perform the following further operations: Within the visual data, detect the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; A common reference time corresponding to the reference point of the flash interval of the group is determined based on the first flash detected within the visual data; as well as Each ESL device in the group is identified based on a specified difference corresponding to each ESL device and the timing of a second flash detected within the visual data corresponding to each ESL device relative to the common reference time.

16. The apparatus of claim 11, wherein the visual data comprises image frames collected by the camera for each ESL device in the group during a survey of the environment.

17. The apparatus of claim 16, wherein different groups of ESL devices are deployed in different areas of the environment, wherein the group of ESL devices is a first group of ESL devices deployed in a first area of ​​the environment, wherein a second group of ESL devices is deployed in a second area of ​​the environment, wherein the visual data includes a first sequence and a second sequence of image frames collected by the camera for the respective first group and second group as the mobile device moves through the first area and the second area during the survey, and wherein the first sequence is collected by the camera during the flash interval associated with the first group, and wherein the second sequence is collected by the camera during different flash intervals associated with the second group.

18. The apparatus of claim 11, wherein the position of each ESL device in the group is represented as a set of coordinates relative to the pose of the camera in a three-dimensional (3D) coordinate space representing the environment.

19. The apparatus of claim 18, wherein the set of coordinates for each ESL device is determined using motion reconstruction analysis of the visual data.

20. The apparatus of claim 19, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during the flash interval, wherein the visual data includes a sequence of image frames collected by the camera for the group during the flash interval, wherein each image frame in the sequence corresponds to a different ESL device in the group, and wherein the operation for determining the position of each ESL device includes: An image filter is applied to detect the flash emitted by the ESL device at a point within the corresponding image frame; as well as Map the points within the corresponding image frame to the set of coordinates in the 3D coordinate space.

21. A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform operations including: Commands are sent via an environment-associated wireless network to each ESL device in a group of electronic shelf label (ESL) devices deployed within the environment to emit one or more flashes within a flash interval associated with the group; Receive visual data collected by the camera for the group during the flash interval from the mobile device via the wireless network; Detect the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; as well as The location of each ESL device in the group within the environment is determined based on the one or more flashes emitted by each ESL device in the visual data.

22. The non-transitory computer-readable medium of claim 21, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period allocated to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further comprises: Receive from the mobile device via the wireless network the ESL beacon transmitted by each ESL device in the group during the flash interval; as well as For each flash detected in the visual data, different ESL devices in the group are identified based on the identification information included in the ESL beacons transmitted during the time period corresponding to the flash.

23. The non-transitory computer-readable medium of claim 21, wherein each ESL device in the group is located near the mobile device during the investigation of the environment, and wherein the operation further comprises: During the investigation, radio frequency (RF) measurements were performed on the mobile device via the wireless network, receiving wireless signals from a subset of the ESL devices deployed within the environment. as well as Each ESL device in the group is identified based on the RF measurements received for the subset.

24. The non-transitory computer-readable medium of claim 23, wherein the operation for identification further comprises: Receive additional RF measurements of beacons transmitted by one or more neighboring ESL devices from at least one ESL device in the group; as well as The group of ESL devices near the mobile device is identified based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

25. The non-transitory computer-readable medium of claim 21, wherein the one or more flashes comprise a first flash simultaneously emitted by all of the ESL devices in the group at a reference point of the flash interval and a second flash individually emitted by a single ESL device in the group at a specified difference from the reference point of the flash interval, and wherein the operation further comprises: Within the visual data, detect the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; A common reference time corresponding to the reference point of the flash interval of the group is determined based on the first flash detected within the visual data; as well as Each ESL device in the group is identified based on a specified difference corresponding to each ESL device and the timing of a second flash detected within the visual data corresponding to each ESL device relative to the common reference time.

26. An electronic shelf label (ESL) system, the electronic shelf label (ESL) system comprising: A group of ESL devices, wherein the group of ESL devices is deployed within an environment; and A server, the server including memory and at least one processor coupled to the memory, wherein the at least one processor is configured to perform operations including: Commands are sent to each ESL device in the group via a wireless network associated with the environment to emit one or more flashes within a flash interval associated with the group; Receive visual data collected by the camera for the group during the flash interval from the mobile device via the wireless network; Detect the one or more flashes emitted by each ESL device in the group within the flash interval within the visual data; as well as The location of each ESL device in the group within the environment is determined based on the one or more flashes emitted by each ESL device in the visual data.

27. The ESL system of claim 26, wherein the one or more flashes detected in the visual data include flashes emitted by each ESL device in the group during a time period allocated to the ESL device within the flash interval, wherein the command further instructs each ESL device in the group to transmit an ESL beacon including identification information of the ESL device during the time period, and wherein the operation further includes: Receive from the mobile device via the wireless network the ESL beacon transmitted by each ESL device in the group during the flash interval; as well as For each flash detected in the visual data, different ESL devices in the group are identified based on the identification information included in the ESL beacons transmitted during the time period corresponding to the flash.

28. The ESL system of claim 26, wherein each ESL device in the group is located near the mobile device during the environmental survey, and wherein the operation further comprises: During the investigation, radio frequency (RF) measurements were performed on the mobile device via the wireless network, receiving wireless signals from a subset of the ESL devices deployed within the environment. as well as Each ESL device in the group is identified based on the RF measurements received for the subset.

29. The ESL system of claim 28, wherein the operation for identification further comprises: Receive additional RF measurements of beacons transmitted by one or more neighboring ESL devices from at least one ESL device in the group; as well as The group of ESL devices near the mobile device is identified based on the RF measurements received from the mobile device and the additional RF measurements received from the at least one ESL device.

30. The ESL system of claim 26, wherein the one or more flashes comprise a first flash simultaneously emitted by all the ESL devices in the group at a reference point of the flash interval and a second flash individually emitted by a single ESL device in the group at a specified difference from the reference point of the flash interval, and wherein the operation further comprises: Within the visual data, detect the first flash emitted by all the ESL devices in the group and the second flash emitted by each ESL device in the group after the first flash; A common reference time corresponding to the reference point of the flash interval of the group is determined based on the first flash detected within the visual data; as well as Each ESL device in the group is identified based on a specified difference corresponding to each ESL device and the timing of a second flash detected within the visual data corresponding to each ESL device relative to the common reference time.

Images