Systems for monitoring a shelf, devices and methods thereof
The system uses sensor arrays and neural networks to monitor shelf activity and customer behavior, addressing the cost and scalability challenges of camera-based systems by leveraging photovoltaic cells and light guides for efficient data collection and processing, enabling real-time insights for restocking and personalized marketing.
Patent Information
- Application Number
- PCT/AU2024/050547
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-24
- Publication Date
- 2024-11-28
AI Technical Summary
Physical store retailers face high costs and scalability issues with deploying camera-based computer vision systems for monitoring customer activity at every shelf, limiting their ability to implement automatic product restocking, marketing metric generation, and personalized shopping experiences.
A system comprising an array of sensors positioned on shelves, coupled with a neural network system, uses local luminosity changes to monitor merchandise and footfall, leveraging photovoltaic cells and light guides for efficient data collection and processing, eliminating the need for cameras.
Enables cost-effective, real-time monitoring of merchandise movements and customer activity, providing insights for restocking and personalized marketing without the infrastructure and maintenance costs associated with traditional camera systems.
Smart Images

Figure AU2024050547_28112024_PF_FP_ABST
Abstract
Description
[0001] Systems for Monitoring a Shelf, Devices and Methods Thereof
[0002] Technical Field
[0003] The present invention relates, in general terms, to systems for monitoring a shelf, devices and methods thereof. In particular, the system monitors merchandise on a shelf and / or footfall adjacent to the shelf.
[0004] Background
[0005] Presently, physical store retailers are limited in their ability to deploy camera-based computer vision systems to monitor customer activity at every shelf in their stores due to the high costs of the infrastructure required. As a result, these retailers cannot benefit from the advantages offered by vision-based shelf monitoring related applications such as: (1) automatic detection of on-shelf availability of products that may need restocking; (2) automatic generation of marketing metrics such as dwell time at shelf locations in front of specific products; and (3) automatic real-time personalization of the physical shopping experience while a customer walks the aisles. State-of-the-art approaches to these applications are especially difficult to cost effectively scale-up to large format stores because they are based on expensive RGB or RGB-D cameras that need to be installed and maintained throughout the ceilings and / or shelves of stores.
[0006] It would be desirable to overcome or ameliorate at least one of the above-described problems.
[0007] Summary
[0008] The present invention provides a system for monitoring merchandise on a shelf and / or footfall adjacent to the shelf, comprising: a) an array of sensors positionable on a front edge of the shelf; and b) a neural network system coupled to the array of sensors; wherein each sensor in the array of sensors is configured to detect a local luminosity and when the local luminosity changes, transmit a signal to the neural network system; wherein the neural network system is configured to process the signal in order to monitor merchandise on the shelf and / or footfall adjacent to the shelf; wherein each sensor comprises a light guide positioned at least on a front facing surface of the sensor and configured to direct the local luminosity to the sensor. In some embodiments, the array of sensors comprises at least one photovoltaic cell, wherein the light guide is positioned on a front facing surface of the sensor and configured to direct the local luminosity to the photovoltaic cell.
[0009] In some embodiments, the array of sensors comprises at least one solar-powered electronic shelf label, wherein the solar-powered electronic shelf label comprises a photovoltaic cell, electronics and display panel housed within a frame, and a light guide film positioned at least on a front face of the frame and configured to direct light to the photovoltaic cell.
[0010] In some embodiments, the photovoltaic cell is positioned at a side of the frame or behind the display panel.
[0011] In some embodiments, the light guide is configured to be selective to light with an incident angle 0° to 90° about relative to a normal line of the light guide.
[0012] In some embodiments, the light guide is configured to be selective to light with a wavelength of about 300 nm to about 800 nm.
[0013] In some embodiments, the light guide is characterised by a tapered cross section with a thick end and a thin end, wherein the thick end is adjacent to the photovoltaic cell.
[0014] In some embodiments, the light guide is a bar characterised by a square cross section with a height and / or width of about 10 mm to about 60 mm.
[0015] In some embodiments, the light guide is made from a material selected from polydimethylsiloxane (PDMS), polycarbonate, polyester, acrylic such as poly(methyl methacrylate), polyethylene terephthalate (PET), polyimide (PI), polyethersulfone (PES), their derivatives and combinations thereof.
[0016] In some embodiments, the sensor further comprises at least one light transflector or reflector disposed within the light guide, and a light source adjacent to the light guide such that light from the light source travels through the light guide and is reflected at the transflector or reflector for displaying a virtual image to a user.
[0017] In some embodiments, the sensors are either arranged periodically on the shelf or throughout the store, or randomly positioned on the shelf or throughout the store.
[0018] In some embodiments, the signal is an electrical signal.
[0019] In some embodiments, the signal is current and / or voltage.
[0020] In some embodiments, the neural network system compiles the signal from each sensor in the array of sensors and converts them into a heat map.
[0021] In some embodiments, the heat map is time stamped.
[0022] In some embodiments, a plurality of time stamped heat maps are concatenated into a 3D matrix.
[0023] In some embodiments, the neural network system comprises one or more processors and one or more memory resources for storing a machine learning model that, when executed by the one or more processors, causes the neural network system to: a) process the signals into convolutional layers, and b) utilising the convolutional layers to monitor merchandise on a shelf and / or footfall adjacent to the shelf.
[0024] In some embodiments, the neural network system is calibrated using information derived from a planogram.
[0025] In some embodiments, the neural network system is calibrated using information derived from a calibration device.
[0026] In some embodiments, the neural network system is trained using a model.
[0027] In some embodiments, the neural network system comprises a 3D convolutional neural network or a recurrent neural network.
[0028] The present invention also provides a sensor, comprising: a) a frame having a front face; b) a photovoltaic cell housed within the frame; and c) a light guide film positioned at least at or adjacent the front face of the frame and configured to direct light to the photovoltaic cell. The present invention also provides a calibration device, comprising an array of light sources, wherein each of the light source is configured to independently emit light of a specific wavelength, intensity and / or direction.
[0029] Brief description of the drawings
[0030] Embodiments of the present invention will now be described, by way of non-limiting example, with reference to the drawings in which:
[0031] Figure 1 shows two examples of the relative positioning of the light guide film and the solar cell.
[0032] Figure 2 shows a blow up image of a solar-powered electronic shelf label and its positioning on a shelf.
[0033] Figure 3 shows the interaction of light with the light guide film and solar cell.
[0034] Figure 4 shows solar-powered electronic shelf labels and its conversion to time series of photovoltaic conversion sensor array
[0035] Figure 5 shows 3D Convolutional Neural Network (3D-CNN) with time series of photovoltaic conversion sensor array input
[0036] Figure 6 shows a fully Connected Neural Network with photovoltaic conversion sensor array input
[0037] Figure 7 shows solar-powered electronic shelf label sensor array with spatial positioning calibration device
[0038] Figure 8 shows a store heat map of model-inferred customer activity based on photovoltaic conversion sensor input
[0039] Figure 9 shows another embodiment of a shelf comprising a system for monitoring merchandise.
[0040] Figure 10 shows an embodiment of the sensor.
[0041] Detailed description
[0042] The inventors envisioned that the use of cameras can be totally avoided in these shelfmonitoring applications. In this regard, the present invention is predicated on relying on sensors to achieve results similar to computer vision-based retail applications. The sensors may be incorporated into electronic price labels already on store shelves. Additionally, sensors can be low cost, which translates to a large cost saving. This in turn facilitates the implementation of this technology. In particular, the sensors can be coupled to deep learning to achieve results similar to computer vision-based retail applications. The sensors may be photovoltaic conversion sensor cells incorporated into electronic shelf labels, enabling them to be cost effective and therefore practical for scalable deployment. In particular, the electronic shelf labels may display a price or information about a displayed merchandise, while the sensor detect a change of light in a space directly or proximal to a front of the sensor, which may indicate a product being removed from the shelf. Alternatively, is solar powered electronic shelf labels are already in use at the store, the solar powered electronic shelf label may be modified such that it comprises the sensor. By arraying sensors at various positions on the shelf, a map of light changes may be obtained, which may provide real time information relating to customer's preference and product re-stocking. As the use of cameras such as RGB or RGB-D cameras is avoided, the disadvantage relating to the use of cameras such as cost inefficiency in terms of unit price, installation, power, and maintenance is also avoided. Additionally, setting up multiple cameras in a large format store can be impractical as deployment to monitor every shelf is not proportionally scalable.
[0043] The present invention thus allows for product management by continuous monitoring of product movements on / off shelves. The sensor may also serves as a product display, thus providing consumers with information relating to the product without blocking the visuals of the consumers.
[0044] As used herein, "device" refers to a thing or entity made or adapted for a particular purpose, such as a piece of mechanical or electronic equipment. The device can be manually operated, or can computer implemented with instructions from a software.
[0045] As used herein, "system" refers to one or more devices configured with or interacting with each other based on a set of rules. The set of rules can be provided by a software and / or process protocols. In this sense, a system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole. A system, surrounded and influenced by its environment, is described by its boundaries, structure and purpose and expressed in its functioning.
[0046] As used herein, "method" refers to a particular procedure for accomplishing or approaching something. Accordingly, and with reference to this invention, when a particular set of rules is selected, the system can provide a method for automatically detecting and quantifying bacterial cells.
[0047] The present invention provides a system for monitoring changes on a shelf. For example, the change may be merchandise removal from a shelf. The change may be merchandise addition to the shelf. The present invention also provides a system for monitoring footfall adjacent to a shelf. The shelf can be a retail shelf, which is part of a display cabinet in a store. The system comprises sensors. The sensors may form an array of sensors, which may be two-dimensional (r x c) (matrix representing a front facing surface of a shelf as shown in Figure 7) or three-dimensional (r x c x h) (front facing surface of shelves distributed over a layout of a store; Figure 8). In this regard, a matrix of sensors may be deployed on the shelves of an aisle. The sensors may be positioned adjacent to a merchandise label or may be part of the merchandise label. Alternatively, the sensors may form part of the front facing strip, the front facing strip positioned at a front edge of the shelf for displaying information to the user (Figure 9). This allows for better mapping of merchandise movement on the shelf. If monitoring is desired for multiple shelves across the layout of the store, a 3D array of sensors may be deployed to provide coverage for the whole store area. The sensors may be positioned at equidistance relative to each other. This allows for better mapping of the store.
[0048] The system also comprises a neural network system. The neural network system is coupled to the array or matrix of sensors. For example, the neural network system is electrically coupled to the array or matrix of sensors. A neural network (also called an artificial neural network) is an adaptive system that learns by using interconnected nodes or neurons in a layered structure that resembles a human brain. A neural network can learn from data; it can be trained to recognize patterns, classify data, and forecast future events.
[0049] Each of the sensors is configured to detect a local luminosity or light proximate to the sensor. As used herein, "luminosity" is a measure of radiated electromagnetic energy (light) per unit time of a space, environment or object. The local luminosity thus refers to an intensity of light around a sensor. This takes into account the relative intensity of the light at the sensor, for example, due to the placement of a shelf relative to an overhead light. The light sensed by the sensor may thus be light from the overhead light, reflected light from the floor, shelf, or signage, or light (LED) from lighted display. Luminosity may be quantified. Luminance is the intensity of the emitted light from a surface per unit area in a particular direction. It provides the amount of light that passes through, is reflected, and falls within a given solid angle.
[0050] The detection may be a continuous detection of local luminosity. When the local luminosity changes, the change in local luminosity is converted into an electrical signal which is transmitted to the neural network system. The signal may be processed by a computer algorithm of the neural network system in order to monitor movement in the vicinity of the shelf and / or change on the shelf. For example, when a consumer reaches out to grab a product, a shadow casted by overhead light falls on the sensor, which changes the local luminosity. This is detected and processed as a movement of product away from the shelf. Additionally, the signal may be time-stamped, and collectively, when signals from a particular sensor are tracked over time, the changes in signals may be processed by a computer algorithm of the neural network system in order to monitor movement in the vicinity of the shelf and / or changes on the shelf over time.
[0051] In other embodiments, each sensor in the array of sensors is configured to detect a change in local luminosity.
[0052] In this way, the system is able to track merchandise movement relative to a shelf. The system may also provide real time information on merchandise tracking, thus providing information on when to re-stock. When the system is deployed over the whole store, it may provide information regarding regions of the store with higher footfall and hence product placement regions with greater visibility to customers.
[0053] The sensors may be an electronic shelf label, or housed in an electronic shelf label. The electronic shelf label further comprises electronics and a display panel housed within a frame. The electronics and display panel works together to provide the electronic label. The electronic label may be powered by a battery, or may be electrically connected to a power source.
[0054] The sensors may comprise photovoltaic cells or solar cells. In some embodiments, each sensor comprises at least one photovoltaic cell. When light shines on a photovoltaic cell, the light may be converted into a current and / or voltage via semiconductor materials that exhibit photovoltaic effect. This allows the local luminosity to be quantified. The sensors may be a solar-powered electronic shelf label. The solar-powered electronic shelf label comprise a photovoltaic cell, electronics and a display panel housed within a frame. The electronics and display panel works together to provide the electronic label.
[0055] When the system is deployed in a store, its deployment may be relatively easy if solar- powered electronic shelf labels are already deployed on retail shelves. For example, solar or photovoltaic cells used for powering electronic shelf labels may be configured or adapted to act as sensors. Solar-powered electronic shelf labels are devices which offer digital price information to customers. These labels use photovoltaic conversion from embedded solar cells to power themselves based on the ambient light in the store.
[0056] Without wanting to be bound by theory, the inventors have found that these devices may be made more sensitive to local luminosity changes, which are created when customers are present in front of or moving past them.
[0057] In this sense, no additional sensors other than what is already available on retail shelves may be required. Besides using these solar cells for powering the electronic shelf labels, the solar cells may also be used as sensors where the converted current / voltage is captured as a time series of values; i.e. the converted voltage / current values are time stamped.
[0058] When solar-powered electronic shelf labels are used, the sensors which also act as shelf labels may be positioned at various positions as required. Of course, additional sensor may be added to improve the coverage that is provided by the already present solar- powered electronic shelf labels. In this regard, the array of sensors may comprise at least one solar-powered electronic shelf label.
[0059] The sensor may comprise a light guide configured to direct light to the sensor. The light guide may be a film. For example, a transparent light guide film may be placed adjacent to the solar cell and over a planar surface of the solar cell to collect light from ambient light and guide it into solar cell. Alternatively, an end of the light guide may be placed adjacent to the solar cell and over a planar surface of the solar cell to collect light from ambient light and guide it into solar cell. In both case, the light guide is positioned substantially on a front facing or light-collecting surface of the sensor. For example, a light-capturing surface of the solar cell is positioned adjacent to the light guide so that light may be directed to the solar cell causing a photovoltaic effect. When the sensor is a solar-powered electronic shelf label, a light guide may be positioned at least on a front face of the frame and configured to direct light to the photovoltaic cell. In this way, the light collection efficiency may increase, which increases sensitivity to ambient light changes across the area of deployed sensors. It was found that these sensors may be more sensitive to local luminosity which changes when customers are present in front of them or walk pass them. Additionally, as the light guide is transparent and flexible, it can be used as a transparent surface and not restrict the viewing angle or view of a customer.
[0060] Figure 1 shows how the light guide may be positioned to direct light to the photovoltaic cell. The photovoltatic cell may be positioned at an end of the light guide, or adjacent to a planar surface of the light guide.
[0061] Figure 2 shows how solar-powered electronic shelf labels may be positioned on a shelf, and an exploded general representation of the solar-powered electronic shelf label. The solar-powered electronic shelf label comprises a photovoltaic cell housed within a frame. In this representation, the photovoltaic cell is positioned at a top side of the frame. The solar-powered electronic shelf label also comprises a display panel and electronics, which together perform the function of displaying the merchandise name, code and / or price. The light guide is positioned at least on a front face of the frame. The light guide may be positioned adjacent to the display panel such that it forms an external front facing of the solar-powered electronic shelf label.
[0062] The positioning of the light guide at a front facing allows the photovoltaic cell to be placed within the mechanical housing (such as at a rear facing) or at an internal side of the frame. When the photovoltaic cell is positioned at a side of the display panel, the light guide may be positioned on a front surface of the display panel and configured to direct light beam towards the photovoltaic cell (i.e. by redirecting light normal to the electronic shelf label towards a side of the label) . When the photovoltaic cell is positioned at a rear surface of the shelf label, the light guide may have U-shaped configuration such that it is positioned on the front surface of the shelf label, at least a side of the shelf label, and rear surface of the shelf label (i.e. the light guide wraps around the display panel and electronics in order to direct light to the photovoltaic cell). Light from the environment is thus directed to the photovoltaic cell via the light guide by way of total internal reflection.
[0063] This can help reduce the size of the solar-powered electronic shelf label and allows the fonts on the electronic display to be bigger. The light guide comprises a polymer which is capable of transmitting light. The transmittance of a material is the proportion of the incident light that moves all the way through to the other side; i.e. the effectiveness in transmitting light. The light guide is light transmitting in that it is capable of transmitting at least 90% of light from a light source, or at least 85%, at least 80%, at least 75%, at least 70%, at least 65% or at least 60%.
[0064] Alternatively, the light guide may have a transparency of more than about 90%. Transparency (or transmission of visible light) is characterized by light transmittance, i.e. the measured percentage of incident light transmitted through a material. The higher the transmittance, the higher the transparency. In other embodiments, the transparency is more than about 85%, about 80%, about 75%, about 70%, about 65% or about 60%.
[0065] The light guide is configured to be selective to light with a wavelength of about 300 nm to about 800 nm. Alternatively, the light guide is capable of transmitting light in the visible light spectrum; i.e. wavelength of about 380 nm to 740 nm. To this end, the light guide is capable of transmitting at least the primary colours (red, yellow, blue). The light guide may be capable of transmitting monochromatic light (red, orange, yellow, green, cyan, blue, violet). The light guide may be also capable of transmitting mixed colours, for example, pink and magenta.
[0066] The light guide directs light to the sensor by total internal reflection. Total internal reflection is the phenomenon in which waves arriving at the interface (boundary) from one medium to another are not refracted into the "external" medium, but completely reflected back into the "internal" medium. It occurs when the "internal" medium has a higher wave speed (i.e., lower refractive index) than the "external", and the waves are incident at a sufficiently oblique angle on the interface. This allows the light guide to capture light for a wider range and direct it to the sensor, thus improving the detection.
[0067] The light guide may be a flexible film. The light guide may be curved as shown in Figure 1. As light is transmitted via total internal reflection within the light guide film, loss of light is minimised even when the light guide is curved. This allows for greater installation flexibility in how the photovoltaic cell may be positioned relative to the electronic display. The light guide may have a flexural modulus of less than about 4 GPa. This provides the light guide with an acceptable flexibility for conforming to a morphology of a sensor. For example, the surface of the sensor may have a curved morphology. The light guide may flexed to conform to the curved morphology of the surface, or the light guide may be curved so that it may be positioned in close contact with the surface having a curved morphology.
[0068] The flexibility of the light guide may be characterised by a stiffness value and / or a hardness value. The stiffness may be a bending stiffness. For example, when polydimethylsiloxane (PDMS) is used, the stiffness can be from about 0.5 MPa to about 8 MPa.
[0069] The light guide may have a refractive index of about 1.4 to about 1.6. For example, if PDMS is used to form the light guide, the light guide can have a refractive index of about 1.4.
[0070] The refractive index (n) of a material is a dimensionless number that describes how fast light travels through the material. It is defined as n= c / v, where c is the speed of light in vacuum and v is the phase velocity of light in the medium. The refractive index determines how much the path of light is bent, or refracted, when entering a material.
[0071] The light guide can be made from a material selected from polydimethylsiloxane (PDMS), polycarbonate, polyester, acrylic such as poly(methyl methacrylate), polyethylene terephthalate (PET), polyimide (PI), polyethersulfone (PES), their derivatives and combinations thereof. Alternatively, glass may be used.
[0072] The light guide may have a cylindrical structure, or a rectangular structure, or a polygonal structure along the light path. The light guide may have a circular cross section, or a squarish, rectangular, triangular or polygonal cross section. The light guide may have a cross sectional thickness of about 1 pm to about 10 mm, about 1 pm to about 1 mm, about 10 pm to about 1 mm, about 100 pm to about 1 mm, or about 200 pm to about 1 mm.
[0073] The light guide may have a tapered cross section (Figure 3). The light guide may have a thick end and a thin end. For example, the light guide may have a wedge shape, wherein a thick end of the wedge may be tapered, bevelled or chamfered. The thick end may be adjacent to the photovoltaic cell. The thin end may be distal from the photovoltaic cell. This directionality acts to focus the incident light towards the photovoltaic cell from the light guide via total internal reflection. It also increases the surface of contact of the light guide film with the photovoltaic cell, thus improving its efficiency.
[0074] As shown in Figure 10, in some embodiments, the light guide is a bar structure with a squarish cross section. The light guide may have a height and / or width of about 10 mm to about 100 mm, about 10 mm to about 90 mm, about 10 mm to about 80 mm, about 10 mm to about 70 mm, about 10 mm to about 60 mm, about 10 mm to about 50 mm, about 10 mm to about 40 mm, about 10 mm to about 30 mm, or about 10 mm to about 20 mm.
[0075] In some embodiments, the light guide is configured to be selective to light with an incident angle 0° to 90° about relative to a normal line of the light guide. The incident angle may be from 10° to 90° or from 20° to 90°.
[0076] The light guide may comprise at least one light transflector or reflector. In this regard, the light transflector or reflector is disposed within the light guide. The term "transflector" is used to refer to an optical component that reflects light incident on one of its surfaces at a particular reflected exit angle, and / or at the same time, the optical component transmits rays incident on another surface at a particular transmitted exit angle. "Reflector" is a subset of "transflector" with reflection set to 100% and transmission set to 0%. The light transflector or reflector may be approximate to an end or side of the light guide, or disposed / embedded within the light guide. The light transflector or reflector may be positioned to reflect light to the sensor. For example, the light transflector or reflector may be disposed at an angle relative to a light path of the light such that the light is reflected towards the sensor, or along the longitudinal sides of the light guide in order to improve total internal reflection towards the sensor.
[0077] For example, the light transflector or reflector can be disposed at an angle of about 5° to about 80° relative to the light path. Preferably, the angle of about 5° to about 60° relative to the light path.
[0078] By disposing the light transflector or reflector within the polymer guide, or preferably interposed within the polymer guide, leakage of the transmitted light beam within the polymer guide is minimised. In this sense, the polymer guide is in contact with an anterior surface and posterior surface of the light transflector or reflector. The light transflector or reflector may act to minimise leakage of the reflected light away from the sensor, and / or prevent / minimise stray light from interfering with the image received at the viewer's eye. This may improve the sensitivity of the system.
[0079] The light transflector or reflector may be a homogenous surface or may comprise an array of nanostructures and / or microstructures. It was found that this allows for transparency of the light reflector. The thickness, width and size of the nanostructures and / or microstructures can be varied to tune reflectivity and the light transmitted.
[0080] For the light transflector or reflector to sufficiently be able to reflect light, the light reflector can be of a certain thickness. For example, the light reflector can have a thickness of about 3 nm to about 800 pm. In other embodiments, the light transflector or reflector has a thickness of about 3 nm to about 700 nm, about 3 nm to about 600 nm, about 3 nm to about 500 nm, about 3 nm to about 400 nm, about 3 nm to about 300 nm, about 3 nm to about 200 nm, or about 3 nm to about 100 nm.
[0081] When the light transflector or reflector is formed as a homogenous surface, it may be selected from a metal-polymer boundary, an air-polymer boundary, a material-polymer boundary or a combination thereof. The boundary is formed from an interface of the polymer and a metal, air, material or a combination thereof. The material and air can have a lower refractive index than the polymer. The metal can be a dielectric metal. These materials can reflect light travelling in the light path towards the sensor.
[0082] Without wanting to be bound by theory, based on Snell's law and total internal reflection (npDMs sin (Qc) = nair sin (90)), when the light enters the polymer (PDMS refractive index = 1.43) from air to light reflector, the light reflects due to the total internal reflection. Hence, by controlling the thickness of the light reflectors from a few microns to tens of microns, it is possible to control the reflected light intensity to make the light guide film transparent to the viewer. This allows the electronic label to be visible to the viewer, while at the same time focus light towards the photovoltaic cell. Hence the total reflection TF = f(s)*f(d), in which f(s) is total internal reflection and f(d) is the depth or thickness of the light reflector.
[0083] At least one light transflector or reflector may be disposed within the light guide and / or light path. For example, a plurality of light transflector or reflectors can be positioned in parallel or in series or in an array within the light path. The plurality of light transflector or reflectors can be positioned at various angles independent of each other. This allows more light to be reflected to the sensor. Further, a plurality of light transflector or reflectors advantageously allows for the light reflectors to be each smaller in size, and can improve the transparency of the light guide. Since each light transflector or reflector may be partially reflecting, it is possible for a viewer to see through the light guide and view the electronic display. The light leakage and image distortion, especially while bending along the frame is reduced (increased sharpness) when a plurality of light reflectors are used. Here, the eyestrain is reduced due to the thin planar structure and field of view / eye box has increased due to the multiple transflector or reflector structures.
[0084] The light transflector or reflector and / or the nanostructures and / or microstructures may be made from a metal. The metal may be a dielectric metal. For example, the metal may be selected from gold, silver, aluminium, chromium, copper, nickel, platinum or their combination thereof.
[0085] The reflectance of a metal layer can be calculated from the index of refraction n and the extinction coefficient k of the metal. The reflectance (R) of a metal surface in air is given by:
[0086] When the light transflector or reflector and / or each of the nanostructures and / or microstructures is a metal-polymer boundary, the metal can have a thickness of about 3 nm to about 80 nm. Preferably, the thickness is about 4 nm to about 60 nm. Depending on the thickness of the metal layer, the amount of light that is reflected can be varied, and at the same time the transparency of the image guide can be varied.
[0087] A material or air having a lower refractive index than the light guide can be used as a light reflector. This is based on the understanding of Snell's law and total internal reflection that light in an optically denser medium can be reflected off an interface with a optically less dense medium if the angle at which the light makes with the normal of the interface is more than the critical angle. The critical angle is the angle of incident in an optically denser medium for which the angle of refraction is 90°.
[0088] Accordingly, the light transflector or reflector and / or each of the nanostructures and / or microstructures can be an air-polymer boundary or a material-polymer boundary. The material can have a refractive index of less than about 1.4. In other embodiments, the material is polytetrafluoroethylene (PTFE). Alternatively, when air is used, at least one air filled gap (air pixel, air filled nanostructure and / or microstructure) is formed within the light guide.
[0089] When the light transflector or reflector and / or each of the nanostructures and / or microstructures is a material having a refractive index of less than about 1.4, the material can have a thickness of about 10 nm to about 200 pm. Preferably, the thickness is about 20 nm to about 10 pm. Depending on the thickness of the material, the amount of light that is reflected can be varied, and at the same time the transparency of the light guide can be varied.
[0090] When the light transflector or reflector is an array of nanostructures and / or microstructures, the array of structures may be separated from each other by a pitch such that the array of structures form an array of pixels. For example, the light reflector may comprise an array of 50 pm square microstructures with 20 pm pitches. To this end, the reflection or transmission is controlled by varying the pitch of nanostructures and / or microstructures (pixels), size and thickness. This gives more degree of freedom for reflection and see through transmission.
[0091] The nanostructures and / or microstructures each independently can have a thickness of about 4 nm to about 500 pm, about 4 nm to about 300 pm, about 4 nm to about 100 pm, about 4 nm to about 50 pm, or about 4 nm to about 10 pm. The nanostructures and / or microstructures can be made of a metal and have a thickness of about 4 nm to about 50 nm. The nanostructures and / or microstructures can be made of a material of lower refractive index compared to the light guide and have a thickness of about 4 nm to about 20 pm.
[0092] The nanostructures and / or microstructures can each independently have a size or width of about 50 nm to about 1000 pm, about 50 nm to about 700 pm, about 50 nm to about 500 pm, about 50 nm to about 300 pm, about 50 nm to about 100 pm, about 50 nm to about 70 pm, about 50 nm to about 50 pm, or about 50 nm to about 10 pm. For example, nanostructures and / or microstructures can made of gold and have a size of about 50 pm.
[0093] The pitch between the nanostructures and / or microstructures can each independently be about 50 nm to about 500 pm, about 50 nm to about 400 pm, about 50 nm to about 300 pm, about 50 nm to about 200 pm, about 50 nm to about 100 pm, about 50 nm to about 70 pm, about 50 nm to about 50 pm, or about 50 nm to about 10 pm. For example, pitch can be about 20 pm.
[0094] The nanostructures and / or microstructures can have each independently have a different shape. For example, the shape can be square, rectangle, triangle, circular, hexagonal or polygonal. The above array of nanostructures and / or microstructures can be arranged in different configurations. For example, the array can be a square array, hexagonal array, triangular array, rectangular array. The boundaries of the array can be undefined or not straight such that a random array is arranged.
[0095] When the light transflector or reflectors comprises of nanostructures and / or microstructures (pixilation), the reflectance (R) of a metal surface in air is given by:
[0096] (n - l)2+ fc2
[0097] R (%) = fpttch f structure size f structure thickness
[0098] (n + l)2+ fc2X 100 wherein 'f' denotes 'function of'.
[0099] Similarly for air-filled pixels, the reflectance (T) is given by (%) — fsX / pitch X f structure size f structure thickness X 100
[0100] Wherein fs is a function of the total internal reflection.
[0101] Accordingly, in some embodiments, the at least one light transflector or reflector is provided by a single air filled gap, an array of micro or nano-size air filled gaps or an array of metal micro or nano-sized structures. The metal can be gold or silver.
[0102] When the light transflector or reflector is positioned at an angle relative to the light path, each of the nanostructures and / or microstructures can be independently positioned at an angle relative to the light reflector. In this sense, each of the nanostructures and / or microstructures is positioned at various angles relative to the light path. For example, when the light reflector is positioned at 45° relative to the light path, each of the nanostructures and / or microstructures can be independently positioned at an angle of 0° to about 40° offset from the plane of the light reflector. Accordingly, each nanostructures and / or microstructures can independently be about 5° to about 85° relative to the light path. Preferably, the angle is of about 5° to about 80°, or about 15° to about 60° relative to the light path. The light guide may further comprise a cladding layer. The cladding layer envelopes or encapsulates the light guide where appropriate. The cladding layer can comprise one or more layers of materials of lower refractive index than the light guide which has a higher refractive index. The cladding layer is preferentially advantageous as it allows light to be confined to the light guide by total internal reflection at the boundary between the two. To this end, light propagation in the cladding layer is suppressed and light leakage can be eliminated or at least reduced. This further minimises light loss to the photovoltaic cell.
[0103] The cladding layer can be selected from a metal, a material having a lower refractive index than the light guide, or a combination thereof. When the cladding layer is a metal, the metal can be selected from gold, silver, aluminium, chromium, copper, nickel, or platinum. When the cladding layer is a material having a lower refractive index than the light guide, the material can be selected from air, or polytetrafluoroethylene (PTFE). In some embodiments, the cladding layer comprises a dielectric material or a dielectric mirror (a dielectric stack). A dielectric mirror, also known as a Bragg mirror, is a type of mirror composed of multiple thin layers of dielectric material. Some dielectric materials, but are not limited to, are magnesium fluoride, silicon dioxide, tantalum pentoxide, zinc sulfide, and titanium dioxide.
[0104] The cladding layer can be about 50 nm to about 200 nm thick. Alternatively, it can be less than about 150 nm, about 120 nm, about 100 nm, about 80 nm, about 50 nm, about 40 nm, about 30 nm, about 20 nm or about 10 nm.
[0105] The light guide may further comprise a buffer layer where appropriate. A buffer layer can be used to envelope or encapsulate the light guide for the purpose of providing mechanical isolation, protection from physical damage and ease of identification. The buffer layer may take the form of a miniature conduit, containing but not connected to the light guiding apparatus (loose buffer). Alternatively, the buffer layer can be in intimate contact with the light guide (tight buffer).
[0106] The buffer layer can be applied to the light guide by methods such as spraying, dipping, extrusion and electrostatic methods. Materials used to create buffers can include fluoropolymers such as polyvinylidene fluoride (Kynar), polytetrafluoroethylene (Teflon), silicon dioxide or polyurethane. The light guide may further comprise a lens. The lens can be a single lens or an array or lenses. The lens can be positioned within the light guide, or between the light guide and the light reflector.
[0107] As mentioned above, ambient light may be directed "inwards" to a photovoltaic cell via the light guide, or light guide and reflector. Alternatively, light may also be directed "outwards" in the opposite direction. In this setup, the light transflector or reflector may also be configured to display a virtual image to the user. In this regard, as shown in Figure 9, the light guide acts as a see through optical display and in combination with the reflector, directs a light image from a light source along a light path through the light guide, and reflected at the transflector or reflector towards the user's eyes. As the light guide and the transflector or reflector are transparent, a see through effect is achieved. This allows product information and price to be displayed without visually hindering the merchandise on the shelf. In addition to a static display, an animated film may also be displayed.
[0108] With this setup, the viewer's eye does not have to be positioned at a specific location to be able to read the electronic display. The light transflector or reflector may also alter the focal length of the light exiting the light guide such that a virtual image may be formed at an appropriate distance from the shelf, and which is thus visible to the user.
[0109] The light source may be hidden within a covered portion of the shelf. For example, the light source may be within the shelf. The light source can be used to project a visible light to the light guide, and thus to a user's eyes. In some embodiment, the light source is an organic light-emitting diode display. In other embodiments, the light source is a light-emitting diode display, a micro light-emitting diode display, a MEMS controlled laser projector, a liquid crystal on silicon projector, a thin film transistor liquid crystal display, or a digital light processing projector. For example, the micro-display unit can be integrated with processors, electronics and rechargeable mini-batteries for feeding images. In some embodiments, the light source is selected from a MEMS powered beam scanning engine, a linear array of lasers, LEDs, or a combination thereof.
[0110] For example, the processor may be a T972 / T950x2 Quad-core A55, up to 1.9GHz; Mali- G31 GPU, with a RAM of 2GB, storage of 8GB and installed with an Android operating software. The combination of light guide and transflector or reflector may provide a resolution of 120 (H) x 3840 (V) pixels. The image may be viewed at about 70° to about 90° from all directions (top, bottom, left, right) relative to the front facing surface of the light guide.
[0111] For example, the light source can be a linear array of light sources (lasers or LEDs) (N numbers) placed (by printing light-emitting components on a strip shaped circuit board, or by one single light source working in conjunction with couple-in light guide component or by MEMS powered beam scanning engine). The light source creates the specific color from the beginning at pixel level by combination of colors from red, green and blue (RGB), white coloured or cyan, magenta and yellow (CMY). The size of the light source is smaller than a reflector (or greater than reflector and pitch in some cases). The pitch of the linear light source can be the same as the light reflector's pitch. When the linear light source is turned on using the control electronics, the light is guided through the light guiding apparatus and each light reflector then acts as an illuminated pixel with light partially reflecting towards a viewer.
[0112] The light source can be constantly ON or switched on in different patterns in synchronous with the light valve using the control electronics.
[0113] In some embodiments, the light source further comprises a brightness enhancement film (BEF), duel brightness enhancement film (DBEF), a light diffuser, or their combination thereof.
[0114] In some embodiments, an optical coupling means for coupling light from light source to the light guide is present. This further improves the amount of light which is transmitted to the light guide.
[0115] Figure 9 and 10 shows an example of the monitoring system comprising a virtual image generator. The sensor 900 may comprise a light guide formed as a bar with a square or rectangular cross section. A plurality of sensors may be positioned at a front facing end of a shelf, thus forming an array of sensors. Figure 9 shows the sensors positioned at a front edge of and on the shelf. Alternatively, the sensors may be positioned at a front edge of and hanging from a bottom side of the shelf. This allows for a top down display. Each sensor 900 may comprise a light guide 902 of about 1200 mm in length, about 41 mm in height and about 42 mm in width. The light guide 902 may be positioned on side of a frame or support 904 of similar length, or adjacent to the front face of the frame 904. The light guide 902 may comprise of a first transparent material 902-1 and a second transparent material 902-2. The first transparent material 902-1 and the second transparent material 902-2 may have different refractive index, such that ambient light may be directed inwards to photovoltaic cell(s) (not shown) within the frame 904, or light from a light source (not shown) may be directed outwards towards a user. Alternatively, the first transparent material 902-1 and the second transparent material 902-2 may be the same material. A transflector or reflector 906 may be positioned at the interface of the first transparent material 902-1 and the second transparent material 902-2. As shown the transflector or reflector 906 is disposed at an angle of about 45° relative to a light path. The transflector or reflector 906 may be formed as a homogenous surface and comprise nanostructures and / or microstructures such that the transparency of the light guide 902 is substantially maintained. When the nanostructures and / or microstructures are arrayed, the array of structures may be separated from each other by a pitch such that the array of structures form a pixel array. This may be about 120 (H) x 3840 (V) pixels. The frame 904 may house within itself thereof other components of the sensor 900. For example, photovoltaic cell(s), a light source, processor may be housed within the support 904.
[0116] The light guide may be fabricated using 3D printing techniques or cut from a polymer block. If light transflectors or reflectors are incorporated into the light guide, the light transflectors or reflectors may be fabricated on a first end of a subunit, following which the light transflectors or reflectors may be incorporated into the light guide by joining another end of a subunit to the first end.
[0117] As an example, 3D printed mould with prism cut coupling side can be used as a starting material. The 3D printed mould has large surface roughness due to layer by layer printing, especially on the walls. A flexible smooth plastic film is used to cover the inner side of the 3D printed mould to make the surface walls smooth. PDMS is poured and cured to make the flexible polymer guide. The polymer guide is provided with 3 prism cuts, into which 5 to 20 nm metal (gold) is deposited on one side of the cut structures. The polymer guide with prism cuts are aligned and connected together using oxygen plasma such that the deposited gold structures are embedded within the polymer guide. The area not covered with gold will attach strongly due to PDMS-PDMS interactions.
[0118] For example, a gold film of around 40 nm can be deposited using metal evaporation only on the waveguide part (on one face) leaving the air-filled display part and four edges (FE) of the PDMS layer using a metallic mask. This gold spacer film will prevent light leakage from the individual waveguide film while bending. Here, the light will be coupled out using the nanothick metal layer into eyes and same time the nanoscale gold is transparent to see the outside world.
[0119] As another example, the method above can be used for forming light reflectors that consist of air. In this embodiment, instead of depositing gold, PDMS can be etched on one side of the cut prism structures such that trenches are formed. When the polymer guide with prism cuts are aligned and connected together using oxygen plasma, the air structures are embedded within the polymer guide.
[0120] For example, the air-filled light transflectors or reflectors can be fabricated using Nanoscribe Photonic Professional GT2 (GT2). The 3D nanoscribe is a 'high resolution printer' that combines the technique of 2 photon polymerisation (2PP) with a traditional 3D printing workflow used for additive manufacturing. It offers a one-step process to fabricate almost any arbitrary complex 3D shapes and objects in 3D with nanoscale resolution over a large area. It is also possible to achieve steep slopes, sharp edges, smooth surfaces, and even complex 3D designs. The nanoscribe has a resolution close to ~100 nm, over an area of 100 x 100 mm2, maximum height of up to 10 mm and with writing speeds of up to 1 mm3 in 10 seconds. A single polymer master for the air-filled reflecting pixels and waveguide can be made with registration marks on corners for alignment (for example, a single film of 500 pm thick). The engineering of PDMS can be carried out to make refractive index uniform across the film, fast curing and for making surface smooth. The engineered PDMS can be poured for curing to get the single waveguide layer embedded with the reflecting pixels. The film will then be peeled off and kept on a clean silicon wafer. This step will be repeated to build multiple layers.
[0121] The light collected by the sensor may be used to provide information about the merchandise on the shelf and footfall adjacent to the shelf. Through the use of sensors in combination with deep neural networks, the system can capture the voltaic signals and infer a series of low-resolution "shadow images" of customers; i.e. heat map showing an activity in front of the shelf or location in the store. In this regard, the sensor array may detect a change in light and transmit this change in light as a voltage and / or current to the neural network system. At a single time point, each signal from a sensor may be compiled into a heatmap. These "shadow images" may be inferred from 2D matrix or 3D array of voltage / current values that represent changes in light intensity. The heatmaps may be time stamped and concatenated into a 3D matrix / array, which then may be input to a spatial-temporal deep neural network software algorithm to detect customer activity, such as make predictions of customer presence, pose and dwell time. Positioned in their natural form as shelf labels, they effectively can serve as a low-resolution "camera" capable of detecting customers in the aisle as they stand in front or walk by. This information may further be stored on a server for review of trends and patterns. Because the shelf labels (sensors) themselves are providing this information, they also offer the precise stock-keeping unit (SKU) information of the products that the customer is present in front of, eliminating the need of visual production recognition. Lastly, customer privacy is preserved as personally identifiable features are not captured.
[0122] Figure 4 shows an example of how the system works. The signals from the sensors may be compiled into a 2D heatmap, where each pixel value contains a normalization of the converted voltage such that darker pixels may represent objects as in: . where F = (V, K.) are converted voitaae values and p{is the Ith normalized pixel, and = 2 Volatage(sliced_cellj)
[0123] A 3D concatenation of heatmaps may be made by summing up the 2D heatmaps. Each individual voltage value on the heatmap may be used as additional input in the image; each "pixel" may be pooled from an array of 10 different voltage values, each of which is mapped to a sliced cell present on a label. Therefore, the resulting image can be thought of as having 10 channels, which can provide richer information in the deep learning model input downstream.
[0124] As seen in Figure 4, the sub-pixilation built on top of the sliced solar cells works in conjunction with the sensors. Given by the increased number of pixels (divided from one whole solar cell as one pixel to sub-sliced panels), the ability of any given single device (from the placement of multiple devices in array) to individually identify the behaviour of objects can be further enhanced as discussed in the higher resolution implementation.
[0125] Considering time as an additional dimension, the resulting data, either 3D in the lower resolution implementation or now 4D in the higher resolution implementation, can be used to make predictions of customer presence, pose and dwell time. And because the shelf labels themselves are providing this information, they also offer the precise SKU information of the products that the customer is present in front of, eliminating the need of visual production recognition. Lastly, as no real images and personally identifiable features are being captured, it naturally enables customer privacy. In some embodiments, the neural network system comprises one or more processors and one or more memory resources for storing a machine learning model that, when executed by the one or more processors, causes the neural network system to: a) process the signals into convolutional layers, and b) utilising the convolutional layers to monitor merchandise on a shelf and / or footfall adjacent to the shelf.
[0126] Visual Recognition
[0127] As discussed above, the sensors may capture movement as a 2D matrix (W x H), or when in a higher resolution, a 3D matrix (W x H x C, where C represents 10 channels). For the purposes of simplifying the description of this disclosure to convey the core concepts, the lower resolution 2D-image version will be referred to.
[0128] The method of detecting customers and recognizing actions such as walking by or taking items from a shelf may be enabled by using spatial-temporal deep neural networks. That is, spatial sensing data is captured over time. The spatial data can be summarized into a frame (pixel image / heat map). A frame is created by a central server which is continually receiving sensor data and slicing in time using received time stamps and arranging the resultant 2D matrix in its memory. The server maintains a frame cache of these 2D matrices and periodically concatenates them together into a 3D matrix where each successive frame is a later moment in time as seen in Figure 4. Therefore, the matrix dimensions are the Width (W), the max number of labels across the shelf, by Height (H), the max number of layers of shelves containing labels, by Time (T), the max number of layers of frames in the server's frame cache.
[0129] The 3D matrix of input data is then fed into one or more spatial-temporal deep neural networks trained to provide predictions (Y) such as detecting the presence and position of likely customers as well as their aforementioned actions as seen in Figure 5. One such implementation of spatial-temporal deep neural network is a 3D convolutional neural network (3D-CNN). In such a model, features along dimensions in both time and space are extracted enabling recognition across frames using 3D convolutional layers. By using both space and time the actions of movement such as walking by shelves or taking items from shelves become detectable. Besides 3D-CNN's, in other implementations, different spatial-temporal deep neural networks are also compatible with this invention, such as using CNN-RNN (Recurrent Neural Network) architectures.
[0130] Matrix Transformation of Sensor Array Data Different forms of deep neural networks may be utilised to make predictions of customer activity near retail shelves. In a basic implementation, the signal output of each shelf label can be connected to each neuron in a fully connected neural network as seen in Figure 6. In this implementation, the advantage is that the input data does not need to include spatial information; that is, the shelf labels can be arranged in any order and at various distances from one another. It is then left up to the network to infer spatial information from the training data.
[0131] A superior implementation in terms of performance may be obtained by providing the spatial information in the data itself, by using a neural network architecture that considers the spatial structure of the shelf labels by its design, such as a Convolutional Neural Network as shown in Figure 5. The downside of this implementation is that more work is involved in providing the spatial information.
[0132] For example, spatial information can be provided in two approaches. One, the utilization of shelf planograms. These are the known product layouts on shelves such that the shelf label positions are either explicitly provided or can be inferred. Planograms provided in a consistent format can be parsed automatically by software algorithms whereby the spatial information of the shelf labels and stored in a database. This database can be used at runtime by a central server receiving the photovoltaic conversion sensor array data which can then be mapped spatially into a 2D matrix. This provides a solution to the problem of "gaps" where the shelf labels are not positioned consistently. That is, an equidistant pixel projection of a label's voltage signal from "shelf space" to "image space" leaves gaps in the image where labels do not exist. The solution is to first consider a uniform spatial distribution of labels according to the maximums in width and height provided into a matrix M, initialized to 0. Each existent label projects to a value in M. Each non-existent label value is now averaged by the pixels surrounding it. This can be achieved using a 3x3 averaging filter, represented mathematically as:
[0133] In some embodiments, the neural network system is calibrated using information derived from a planogram.
[0134] Sensor Array Spatial Positioning Calibration In some practical situations it may not be ideal to only depend upon a planogram to inform the spatial layout of the shelf labels. The reason is that a planogram is an idealized layout, and store staff or customers may have adjusted the layout such that it no longer conforms to specification. In this regard, calibration may be required, in 3D, of a given store's model of spatial layout of shelf labels regardless of a planogram existing or not.
[0135] To that end, a calibration device for sensor array spatial positioning may be used. The calibration device may be implemented in many ways, but in one implementation as seen in Figure 7, which is portable with an internal rechargeable battery and on-board minicomputer. The calibration device may be wheeled, rod shaped, with a height matching that of the tallest store shelves. It can be operated either manually, by store staff moving it through the store aisles, or in other implementations the device may be operated autonomously.
[0136] The device directionally emits unique patterns of light at different heights which may be detected by the sensor and logged. The device position as it moves throughout the store is also logged utilizing indoor-positioning system (IPS) techniques such as those using trilateration and triangulation algorithms with Wi-Fi radio receiver signals.
[0137] Together, these logs can map the spatial positioning of the sensor arrays for transforming signal output into image input. This map can be overlayed on a store blueprint to bring further business benefits as described further in the heatmap section below.
[0138] Other spatial positioning calibration implementations may be used to calibrate the sensors; the specialized light calibration device concept provided is only one such approach. Other calibration devices such as using the radio receivers or other sensors on each shelf label to perform spatial positioning may also work. Additionally, other calibration devices may be a mobile phone application, because each e-tag display can be programmed to show a specifically designed pattern, which can be captured by the phone camera, and used in conjunction with software algorithms to perform the spatial calibration.
[0139] In some embodiments, the neural network system is calibrated using information derived from a calibration device. Accordingly, the present disclosure also provides a calibration device, comprising an array of light source, wherein each of the light source is configured to independently emit light of a specific wavelength, intensity and / or direction.
[0140] In some embodiments, the array of light source is a linear array of light source. In other embodiments, the array of light source is configured to project light at different directions.
[0141] Learning and Model Selection
[0142] Once deployed in a store environment, the system may use trained deep neural network models to perform predictions. Methods of training and selection of the models to use at runtime are described.
[0143] In one implementation, the training approach utilized is supervised learning of the previously described deep neural networks in the "Visual Recognition" section, leverage photovoltaic conversion sensor array "images" in a time series grouped in short "clips", from lab simulated arrangements. These images and clips are labelled by a data annotation team which marks them for the target scenarios, including customer presence and activities such as taking items from the shelf in front of them to walking by. The annotation team will perform this labelling task by observing through a computer program, the sensor output data in low-resolution images and clips, and high- resolution "ground truth" images and clips taken by a video camera positioned in a similar location. Based on comparing the sensor data and the ground truth, the staff will decide the label, mark it in the program, which is then saved in a database that will constitute the model training data for supervised learning.
[0144] The training data collection phase in the lab environment involves using a multitude of arranging of shelving and shelf labels to simulate real-world environments. These arrangements are informed by actual planograms used by retailers in their stores. Important metadata added to the training data is a data structure used to capture the arrangement of the shelf labels used. This metadata is critical information for model selection at runtime.
[0145] When deployed in the actual store environment, the photovoltaic conversion sensor array spatial arrangement is mapped to the closest lab simulated arrangement to select the best matched model for runtime inferencing. To find the closest match model, whereby a similarity distance metric is minimized between the real world spatial arrangements of labels for a particular shelf in a store, to the simulated spatial arrangements of labels for a particular shelf captured in a lab, this invention offers a solution in one implementation as follows. As described in the "Matrix Transformation of Sensor Array Data" section, a 2D matrix referred to as an "image" is generated even when there are "gaps" in the sensor array data. These gaps can be filled in using averaging or other convolutional operations. But for the purpose of searching to find the closest matching model, we can reformulate this image as a binary matrix, where the elements of the matrix are represented by "1" whereby it is mapped to an existent shelf label, and "0" whereby it is a gap; that is, a non-existent shelf label. The matrices generated from a store may be of different dimensions than the matrices generated from the lab. In one implementation, to enable efficient matching, matrices of equal dimensions are generated by zero-padding where necessary. Once in the same shape, similarity measures of various algorithms can be used to efficiently find the closest match between store and lab, such as using Otsuka- Ochiai coefficients, which is particularly well-suited for determining the similarity of binary matrices. Once the most similar lab matrix is found, the images from the store are upscaled or downscaled to match the training data as per the metadata, before being provided as input to the associated models.
[0146] Consider the case where the distance between store and lab is too distant as determine by a threshold. To that end, a method of graceful degradation with regards to model selection based on the density and arrangement of electronic shelf labels may be used. That is, if an area is too sparsely arranged with labels as per a threshold, the "image" it would capture would be too low resolution for more sophisticated models; so, a simpler model may be selected, e.g., a simple person presence binary classification model vs. an activity recognition model.
[0147] In some embodiments, the neural network system is trained using a model.
[0148] In some embodiments, the neural network system comprises a 3D convolutional neural network or a recurrent neural network.
[0149] Customer Store Shelf
[0150] As described in the "Sensory Array Spatial Positioning Calibration" section, each solar- powered electronic shelf label will have its spatial position known not just within the shelf, but in the store as well. Using this data, one valuable business application of this invention is creating a store "heatmap" based on customer activity utilizing the predictions of the deep learned models processing photovoltaic conversion sensor array data. These predictions can then be visualized by projecting them over a blueprint of the store layout. For example, for each shelf location, the areas where the model inferred various customer activities can be aggregated then colorized over the map. For example, areas where there are relatively high products being taken or customer dwelling, may be most red, and gradient scale to blue where there is least activity such as only customers walking by. This essentially produces a "hot" and "cold" map as seen in Figure 8, which can be utilized by the retailer to analyze better positioning of products, in-aisle "features", and / or generating marketing data to share with product manufacturers to test if different packaging designs can entice more customer activity.
[0151] On-shelf Availability Application
[0152] Another one of the most valuable business applications for retailers is determining on- shelf availability of products on their shelves. Today, this is largely achieved by having store staff periodically walk the aisles to observe if restocking is needed. To leverage store staff as shelf monitors is an inefficient use of human resources especially of large- format store retailers. A more efficient method is to have automatic monitoring of on- shelf availability, and store staff need only be notified of such events to have a specifically targeted review of a particular shelf, not all shelves in the store. Many automatic solutions to this problem have been proposed such as having traditional RGB or RGB-D cameras be installed to face shelves or aisle sections and use computer vision algorithms to determine likelihood of on-shelf availability. The problem with these solutions is the cost of the cameras and related infrastructure, making it impractical for many retailers, especially of large-format sizes, to deploy and maintain.
[0153] A significantly more cost-effective solution is provided herein. By utilizing the solar cells already existing in solar-powered electronic shelf labels effectively as sensors as per the photovoltaic conversion process, in conjunction with spatial-temporal deep neural networks server-side, inference of customer shopping actions at specific shelves in front of specific products can effectively be achieved. This inference offers probable candidates for out-of-stock replenishment. On-shelf availability of products alerts can then be generated based on this data with notifications provided for store staff in near real-time on their mobile devices. ic Inference Demographic inference is also possible utilizing photovoltaic conversion sensor array data of solar-powered electronic shelf labels in conjunction with spatial-temporal deep neural networks and store mapping data.
[0154] The method for demographic classification of retail customers uses tracking. That is, the shopping behaviour of customers in front of specific shelves and SKUs during their shopping journey throughout the store can tracked and inferred from. By utilizing shopping action inference of a customer in front of specific products, then capturing the continual movement as customers walk by other shelf labels, effectively tracking them to other products, where they may further shop, can be utilized to effectively create a temporary customer profile.
[0155] For example, if a customer shops in front of products known to be used by a particular age or gender, then moves to another shelf with products known to be used by that demographic, the probability that the customer is associated with that demographic becomes higher.
[0156] Once a customer profile is inferred based on their prior journey, a business application is to dynamically display different graphics on the shelf labels for various business scenarios. That is, the labels of the shelves they may be dwelling at may dynamically change to appear more personalized to the shopper based on understanding their profile. For example, specific deals or wording may be changed to facilitate more customer engagement with certain products.
[0157] The present disclosure also provides a sensor, comprising: a) a frame having a front face; b) a photovoltaic cell housed within the frame; and c) a light guide film positioned at least at or adjacent to the front face of the frame and configured to direct light to the photovoltaic cell.
[0158] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.
[0159] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.
[0160] Throughout this specification and the claims which follow, unless the context requires otherwise, the phrase "consisting essentially of", and variations such as "consists essentially of" will be understood to indicate that the recited element(s) is / are essential i.e. necessary elements of the invention. The phrase allows for the presence of other non-recited elements which do not materially affect the characteristics of the invention but excludes additional unspecified elements which would affect the basic and novel characteristics of the method defined.
[0161] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.
Claims
Claims1. A system for monitoring merchandise on a shelf and / or footfall adjacent to the shelf, comprising: a) an array of sensors positionable on a front edge of the shelf; and b) a neural network system coupled to the array of sensors; wherein each sensor in the array of sensors is configured to detect a local luminosity and when the local luminosity changes, transmit a signal to the neural network system; wherein the neural network system is configured to process the signal in order to monitor merchandise on the shelf and / or footfall adjacent to the shelf; wherein each sensor comprises a light guide positioned at least on a front facing surface of the sensor and configured to direct the local luminosity to the sensor.
2. The system according to claim 1, wherein the array of sensors comprises at least one photovoltaic cell, wherein the light guide is positioned on a front facing surface of the sensor and configured to direct the local luminosity to the photovoltaic cell.
3. The system according to claim 1 or 2, wherein the array of sensors comprises at least one solar-powered electronic shelf label, wherein the solar-powered electronic shelf label comprises a photovoltaic cell, electronics and display panel housed within a frame, and a light guide film positioned at least on a front face of the frame and configured to direct light to the photovoltaic cell.
4. The system according to claim 2 or 3, wherein the photovoltaic cell is positioned at a side of the frame or behind the display panel.
5. The system according to any one of claims 1 to 4, wherein the light guide is configured to be selective to light with an incident angle 0° to 90° about relative to a normal line of the light guide.
6. The system according to any one of claims 1 to 5, wherein the light guide is configured to be selective to light with a wavelength of about 300 nm to about 800 nm.
7. The system according to any one of claims 1 to 6, wherein the light guide is characterised by a tapered cross section with a thick end and a thin end, wherein the thick end is adjacent to the photovoltaic cell.
8. The system according to any one of claims 1 to 6, wherein the light guide is characterised by a square cross section with a height and / or width of about 10 mm to about 60 mm.
9. The system according to claim 8, wherein the sensor further comprises at least one light transflector or reflector disposed within the light guide, and a light source adjacent to the light guide such that light from the light source travels through the light guide and is reflected at the transflector or reflector for displaying a virtual image to a user.
10. The system according to any one of claims 1 to 9, wherein the light guide is made from a material selected from polydimethylsiloxane (PDMS), polycarbonate, polyester, acrylic such as poly(methyl methacrylate), polyethylene terephthalate (PET), polyimide (PI), polyethersulfone (PES), their derivatives and combinations thereof.
11. The system according to any one of claims 1 to 10, wherein the sensors are either arranged periodically on the shelf or throughout the store, or randomly positioned on the shelf or throughout the store.
12. The system according to any one of claims 1 to 11, wherein the signal is an electrical signal.
13. The system according to any one of claims 1 to 12, wherein the signal is current and / or voltage.
14. The system according to any one of claims 1 to 13, wherein the neural network system compiles the signal from each sensor in the array of sensors and converts them into a heat map.
15. The system according to claim 14, wherein the heat map is time stamped.
16. The system according to claim 15, wherein a plurality of time stamped heat maps are concatenated into a 3D matrix.
17. The system according to any one of claims 1 to 16, wherein the neural network system comprises one or more processors and one or more memory resources for storing a machine learning model that, when executed by the one or more processors,causes the neural network system to: a) process the signals into convolutional layers, and b) utilising the convolutional layers to monitor merchandise on a shelf and / or footfall adjacent to the shelf.
18. The system according to any one of claims 1 to 17, wherein the neural network system is calibrated using information derived from a planogram.
19. The system according to any one of claims 1 to 18, wherein the neural network system is calibrated using information derived from a calibration device.
20. The system according to any one of claims 1 to 19, wherein the neural network system is trained using a model.
21. The system according to any one of claims 1 to 20, wherein the neural network system comprises a 3D convolutional neural network or a recurrent neural network.
22. A sensor, comprising: a) a frame having a front face; b) a photovoltaic cell housed within the frame; and c) a light guide film positioned at least at or adjacent to the front face of the frame and configured to direct light to the photovoltaic cell.
23. A calibration device, comprising an array of light sources, wherein each of the light source is configured to independently emit light of a specific wavelength, intensity and / or direction.
Citation Information
Patent Citations
Electronic shelf label device and electronic shelf label system
JP2009112622A
Low aspect ratio concentrator photovoltaic module with improved light transmission and reflective properties
US20070227582A1
Merchandise inventory data collection for shelf systems using light sensors
US20170220987A1
Calibrating apparatus and method
US20180255630A1
Sensor bar shelf monitor
US20200334835A1
