Intelligent movable closing system for cooling cabinet

Abstract

The invention relates to an intelligent movable closure system for a cooling cabinet. Disclosed herein is an intelligent movable closure system that allows access to the interior from the exterior of a cabinet. In some embodiments, the movable structure may be coupled with a cabinet interior that is adapted to remain at a temperature that is colder than an external cabinet ambient temperature, for example, to store perishable consumer goods in a retail environment. In particular, the intelligent movable closure system may include an electronic component that displays a graphical rendering corresponding to one or more products stored in the retail product container. In addition, the intelligent movable closed system may include a controller that determines an inventory state of the product in the closed system and controls graphics rendering based on the determined inventory state.

Description

[0001] This application is a divisional application of an application with an application date of January 13, 2021, application number 202180011384.5, and invention name “Intelligent movable closure system for cooling cabinets”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] First, this application is a partial continuation application of U.S. patent application serial number 16 / 741,581 filed on January 13, 2020, which is a partial continuation application claiming the priority application date benefit of international patent application serial number PCT / US2019 / 066869 filed on December 17, 2019 (the international patent application is published as WO2020 / 131881), which is a partial continuation application claiming the priority application date benefit of U.S. patent application serial number 16 / 222,643 filed on December 17, 2018 (the U.S. patent application was published on April 25, 2019 as U.S. patent application publication number US2019 / 0122263). Secondly, this application is a part-continuation application claiming the benefit of the priority filing date of U.S. patent application serial number 15 / 931,971 filed on May 14, 2020, which is a part-continuation application claiming the benefit of the priority filing date of U.S. patent application serial number 15 / 931,971 filed on January 13, 2020, which is a part-continuation application claiming the benefit of the priority filing date of U.S. patent application serial number 16 / 741,581 filed on January 13, 2020, which is a part-continuation application claiming the benefit of the priority filing date of international patent application serial number PCT / US2019 / 066869 filed on December 17, 2019 (published as U.S. patent application publication number US2020 / 131881), which is a part-continuation application claiming the benefit of the priority filing date of U.S. patent application serial number 16 / 222,643 filed on December 17, 2018 (which U.S. patent application was published on April 25, 2019 as U.S. patent application publication number US2020 / 131881). No. 2019 / 0122263), filed on January 13, 2020; and, U.S. patent application No. 16 / 741,581, filed on January 13, 2020, is also a continuation-in-part application claiming the benefit of the priority filing date of international patent application PCT / US2018 / 046103, filed on August 9, 2018 (which international patent application was published as WO2019 / 032893 on February 14, 2019), which is a continuation-in-part application claiming the benefit of the priority filing date of U.S. patent application serial number 15 / 674,198, filed on August 10, 2017; and, U.S. patent application No. 16 / 741,581, filed on January 13, 2020, is also a continuation-in-part application claiming the benefit of the priority filing date of U.S. patent application serial number 15 / 674,198, filed on August 10, 2017. Also, all of the aforementioned applications are incorporated herein by reference in their entirety. Technical Field

[0004] Disclosed herein is a smart removable enclosure system that allows access to an interior from the exterior of a cabinet. In some embodiments, the removable structure can be coupled to a cabinet interior that is adapted to be maintained at a cooler temperature than an exterior cabinet ambient temperature, such as to store perishable consumer goods in a retail environment. In particular, the smart removable enclosure system can include an electronic component that displays a graphical rendering corresponding to one or more products stored in a retail product case. Background Art

[0005] Although more and more people are purchasing items online, physical retail sales (e.g., delivered through cooling systems) are still significant and are actually growing in certain market segments such as grocery stores, drug stores, and convenience stores. More than 75% of purchases in physical stores are impulse purchases. Studies have shown that impulse decisions are highly influenced by in-store messaging and that most in-store impulse purchases occur at retail product aisles, such as refrigerators and freezers. Therefore, disruption of impulse shopping behavior at retail product aisles could present a huge value creation opportunity for both consumer packaged goods brands and retailers. Summary of the invention

[0006] The following overview presents a simplified overview of certain features. This overview is not an extensive overview and is not intended to identify key or critical elements. A system of one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which software, firmware, hardware, or a combination thereof causes the system to perform actions in operation. One or more computer programs may be configured to perform specific operations or actions by including instructions, which, when executed by a data processing device, cause the device to perform actions. An overall aspect includes a smart removable closed system configured to be installed in a retail (or non-retail) environment, which needs to be adapted to be maintained at a cabinet (or other enclosure) at a temperature that is colder (or warmer) than the external cabinet ambient temperature. In some instances, the system includes an internal storage volume of a retail product container, which stores perishable items for purchase in a retail environment, wherein a removable door prohibits viewing the internal storage volume. The system also includes a display device attached to the movable door, wherein the display device is configured to display at least a planogram of the interior storage volume, wherein the planogram may include a message or graphic indicating that one or more products are out of stock. The system may also include an optical sensor, such as a camera, mounted on the movable door to capture one or more images of the interior storage volume of the retail product case. The system may also include a controller communicatively coupled to the display device. The system also includes a data storage area that may store an input planogram mapping of the interior storage volume and a description of a plurality of products, wherein the description may include a shape, color, and / or size of one or more of the plurality of products. In some instances, the system may also include one or more processors proximate to the retail product case. The system also includes a memory storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform one or more steps disclosed in the present application. For example, the method steps may include: post-processing one or more images captured by the optical sensor into a composite image; determining the inventory status of a product among multiple products in a retail product shelf based on the composite image, the input planogram map, and the descriptions of multiple products; determining an advertisement to be displayed on a display device based on the determined inventory status, wherein when the inventory status of the product is out of stock, the advertisement may include an out of stock indicator; and sending an advertisement to a controller for display on the display device. One or more other steps (or different steps) may be performed by the aforementioned method performed by the system. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0007] Some embodiments may include one or more of the following features. A smart movable enclosure system, wherein the input planogram mapping may include: an indication of a product and an indication of a designated location of the product. Determining the inventory status of the product may include determining that the product is not detected at the designated location based on the composite image. In some instances, the inwardly facing optical sensor may be configured to capture multiple images during the operation of the movable door, wherein the operation may include opening the movable door and closing the movable door. The system may include an inventory tracker, which, when executed by one or more processors, causes the one or more processors to transmit a message to a computing device indicating that the product is not in the retail product case. Determining the inventory status of the product may include identifying the stored products in the retail product case based on the composite image, and further determining at least one selected from the following: the location of the stored product, the shape of the stored product, the color of the stored product, the size of the stored product, and a combination thereof. Determining the inventory status of the product may include determining that the product exists in the retail product case based on at least one selected from the following: determining that the location of the stored product corresponds to the designated location; determining that the shape of the stored product corresponds to the shape of the product in the description; determining that the color of the stored product corresponds to the color of the product in the description; determining that the size of the stored product corresponds to the size of the product in the description; and a combination thereof. Determining the inventory status of the product may include determining that another product is present at a specified location in a retail product case based on at least one selected from the following: determining that the shape of the other product does not correspond to the shape of the product in the description; determining that the color of the other product does not correspond to the color of the product in the description; or determining that the size of the other product does not correspond to the size of the product in the description. Determining the inventory status of the product may include determining the number of products in the retail product case. The intelligent movable closure system may include receiving an input planogram mapping from a computing device. In some instances, the data storage area may be located in a retail environment and capture one or more images based on the operation of the movable door (e.g., opening, closing, etc.). In some instances, the advertisement may include an illustration associated with the product. The advertisement may include a planogram that may include illustrations associated with multiple products. In some instances, the advertisement may include illustrations associated with different products. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0008] One general aspect includes a method for generating advertising content on a display screen that is attached to a removable door that is attached to a retail product case that may include an interior storage volume. The method may also include determining a planogram map of the interior storage volume and a description of a plurality of products, wherein the description may include a shape, color, and / or size of each of the plurality of products. The method also includes post-processing one or more images captured by an optical sensor mounted on the removable door (e.g., mounted on, in, or around the removable door) into a composite image. The method also includes determining an inventory status of a product in the retail product case from a plurality of products based on (i) the composite image, (ii) the planogram map, and (iii) the description of the plurality of products. The method may also include determining an advertisement to be displayed on the display screen based on the determined inventory status. The method also includes sending the advertisement for display on the display screen. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0009] One general aspect also includes a computer platform for a cooling system, which may include a retail product case having an internal storage volume. The computer platform also includes one or more processors. The platform also includes a memory storing computer executable instructions that, when executed by the one or more processors, cause the one or more processors to: determine a planogram mapping of the internal storage volume and a description of a plurality of products, wherein the description may include a shape, color, and / or size of each of the plurality of products; post-process one or more images captured by an optical sensor mounted on a movable door into a composite image; and determine an inventory status of a product of the plurality of products in the retail product case based on the composite image, the input planogram mapping, and the description of the plurality of products; determine an advertisement to be displayed on a display screen based on the determined inventory status; and send the advertisement for display on the display screen. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0010] Embodiments may include one or more of the following features. A computing platform for a cooling system may include a retail product cabinet having an internal storage volume, a removable door that blocks viewing of the internal storage volume, a display attached to the removable door, and an optical sensor attached to the removable door. In some instances, the display is configured to display a planogram of at least the internal storage volume. The planogram mapping may include one or more of the following: an indication of a product; and / or an indication of a designated location of a product. Determining the inventory status of a product may include one or more of the following: calculating a confidence score based on a composite image and a description of a plurality of products; and / or determining that a product is out of stock in the internal storage volume when the confidence score is below a threshold. Embodiments of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium. These and other features and advantages are described in more detail below.

[0011] The following overview presents a simplified overview of certain features. This overview is not an extensive overview and is not intended to identify key or critical elements. In one embodiment disclosed herein, a computing platform includes one or more retail product containers and a controller that generates different advertisements (ads) on a display of one or more retail product containers in a cooling system. Each of the retail product containers may include an internal storage space for storing one or more retail products, a display (display screen) on its door, and a customer detector that detects the presence of one or more customers near the retail product container. The controller communicates with each of the retail product containers via a communication channel (e.g., a wireless channel), for example, to display advertisements for viewing by one or more customers. For example, the controller may be configured to direct the retail product container to display a full-screen advertisement on its display. The display of the full-screen advertisement may be a default process, or may be performed when the customer detector does not detect motion (e.g., motion of a customer) or a human form. When motion (e.g., motion of a customer) or a human form is detected within a predetermined distance, the controller may then direct the retail product container to terminate the full-screen advertisement and display a planogram of the products located in the internal storage volume.

[0012] According to another aspect, a customer detector of a retail product cabinet may include one or more cameras and proximity sensors. The one or more cameras and proximity sensors may be located on a door front of the retail product cabinet.

[0013] According to another aspect, when a customer is detected close enough to the door of the cabinet, an invitation indication can be displayed on a display of the cabinet. For example, a quick response (QR) code can be generated at the center of the planogram. In response to the invitation, if the detected customer desires to opt-in, the customer can scan the QR code via a wireless device. When the customer opts in, the computing platform can interact with the customer via a mobile application. In some instances, instead of or in addition to the QR code, an image other than a QR code (e.g., an image with a machine-detectable watermark), a Bluetooth (or other short-range wireless protocol) beacon, or other mechanism can be provided to indicate the invitation to the customer.

[0014] According to another aspect, when a customer opts in in response to an invitation displayed on a display of the case, a personalized advertisement may be displayed based on the customer information obtained through the mobile application. The personalized advertisement may indicate customized pricing for one or more products stored at the retail product case.

[0015] According to another aspect, when the customer does not opt-in in response to the invitation, the computer platform can continue to generate a planogram with inserted advertisements (e.g., banner ads, spot deals, and hot spots) based on generalized information (e.g., weather conditions, weather forecasts, and current or upcoming events).

[0016] According to another aspect, the computing platform may download a set of advertisements associated with a retail product case (e.g., from a cloud computing service). In addition, the computing platform may generate one of the downloaded advertisements on a display of the retail product case. The set of downloaded advertisements may be based on the type of products stored at the retail product case. The type of product may be determined from a case identification or from inventory information provided by the retail product case.

[0017] According to another aspect, the computing platform may determine an advertisement to present on a display of a retail product case based on a rule set that may include weather conditions near the case, the date, and / or characteristics about customers near the case. For example, the advertisement presented may be selected because the customer stood by a beverage case for 20 seconds, it was 98 degrees outside, and the date was just before the Fourth of July.

[0018] According to another aspect, the computing platform can determine a measure of effectiveness of an advertisement presented on a display of a retail product case. For example, the computing platform can detect whether a customer is within a determined distance of the retail product case and / or whether more than a predetermined number of customers are viewing the presented advertisement.

[0019] According to another aspect, the computing platform can generate advertisements targeted to a product level.For example, a customer detector can detect that a customer is viewing a particular product displayed in a planogram at a retail product shelf.

[0020] According to another aspect, an advertisement displayed at a first retail product cabinet can promote products stored at a second retail product cabinet. For example, the computing platform can predict based on customer information that a customer will shop at a second retail product cabinet after shopping at a first retail product cabinet. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The foregoing summary, as well as the following detailed description of exemplary embodiments, will be better understood when read in conjunction with the accompanying drawings, which are included by way of example and not as limitations of the claimed invention.

[0022] Figure 1A , Figure 1B and Figure 1C (collectively referred to as "FIG. 1") illustrates an embodiment of a computing platform supporting one or more retail product cases in accordance with one or more embodiments.

[0023] Figure 2 The front of a door of a retail product cabinet is shown according to an embodiment.

[0024] Figure 3 The back side of a door of a retail product cabinet is shown according to an embodiment.

[0025] Figure 4 A retail product case is shown according to an embodiment.

[0026] Figure 5A and Figure 5B (collectively referred to as "FIG. 5") illustrates a portion of a door in a retail product cabinet according to one or more embodiments; specifically, Figure 5A shows a cross-section of a portion of a door of a retail product cabinet, and Figure 5B A portion of a door of a retail product cabinet is shown according to an embodiment.

[0027] Figure 6 An example placement of a camera on a door of a retail product case is shown in accordance with an embodiment.

[0028] Figure 7 Different views of a retail product case are shown with the door open, according to an embodiment.

[0029] Figure 8 Different types of advertisements that may be displayed on a display screen of a retail product case are shown according to an embodiment.

[0030] Fig. 9 A planogram mapping of the interior storage volume of a retail product cabinet is shown along with descriptions of different products in accordance with an embodiment.

[0031] Fig.10Different advertisements that may be displayed on a display screen of a retail product shelf based on a determined product inventory status are shown according to an embodiment.

[0032] Fig.11 An advertising computer system supporting one or more retail product cases is shown according to an embodiment.

[0033] Fig.12 An edge computing device supporting one or more retail product shelves is shown according to an embodiment.

[0034] Fig.13 A flow chart for configuring an advertisement system according to an embodiment is shown.

[0035] Fig.14 A flow chart for generating an advertisement on a screen of a retail product shelf is shown according to an embodiment.

[0036] Fig.15 A flow diagram for displaying an advertisement selected from a rule set is shown according to an embodiment.

[0037] Fig.16 A flow chart for determining a level of effectiveness of a displayed advertisement is shown in accordance with an embodiment.

[0038] Fig.17 A flow chart for generating advertisements for a product level according to an embodiment is shown.

[0039] Fig.18 A flow chart is shown for displaying advertisements at a first retail product case for products stored at a second retail product case, according to an embodiment.

[0040] Fig.19 A flow diagram is shown for determining and displaying advertisements in a display screen associated with a retail product case, according to an embodiment.

[0041] Fig. 20 A flow diagram is shown for determining and displaying advertisements in a display screen associated with a retail product case, according to an embodiment. DETAILED DESCRIPTION

[0042] According to one aspect of some of the embodiments, a smart movable closure system is disclosed. In some embodiments, the system may include an electronic component that captures images and / or other media (e.g., audio, video, infrared data, or other media) inside or outside the cabinet. In addition, in some embodiments, the system may also include an electronic component that displays, in particular, a graphical rendering or other visual output corresponding to one or more products stored in the cabinet. The cabinet may be adapted to be maintained at a temperature cooler than the external cabinet ambient temperature, for example to store perishable consumer goods in a retail environment. As such, a retail product container can provide a system with a user interface in a retail environment.

[0043] The computing platform generates displayed content at the retail product cabinet of the cooling system based on information about customers near the retail product cabinet, one or more products stored at the retail product cabinet, and generalized information outside the retail product cabinet. According to one aspect of some embodiments, the computing platform can generate selected advertisements (ads) at one or more retail product cabinets. The retail product cabinet includes an internal storage space for storing one or more products, a display on the door of the cabinet, and a customer detector for detecting the presence of customers near the retail product cabinet. According to an embodiment, when it is detected that a customer is within a predetermined distance from the retail product cabinet, an invitation indicator is displayed on the display of the cabinet. When a customer chooses to join, personalized advertisements can be displayed based on customer information that can be obtained through a mobile application. Personalized advertisements can also include customized pricing for one or more products stored at the retail product cabinet. According to an embodiment, the displayed advertisement can be based on the inventory status of the products stored in the retail product cabinet. For example, the display of the cabinet can display an out-of-stock indicator based on detecting that there are no more products in the cabinet.

[0044] Refrigerator is probably one of the most neglected, least promoted, least technically complex products, and may be one of the most operationally challenging departments of a retail store. Traditional retail product cabinets (e.g., refrigerators, freezers, vending machines, etc.) may include glass panels or doors through which customers can view the products stored in the retail product cabinets and the products are available for purchase. However, the stored products may not be neatly presented, and the stored products may be blocked by other products, making it difficult for customers to easily see which products are available for purchase. In addition, store employees may often need to maintain the appearance of the products stored therein, track inventory, or install and replace promotional labels and other displays (such as displays related to pricing and promotion) in order to keep these items up to date.

[0045] Utilizing these traditional methods of retailing product cases, refrigerators and freezers in grocery stores can be difficult areas for brands to effectively advertise and promote "on-site." Brands can often rely on other advertising media (e.g., print, billboards, online and TV ads, etc.) to promote products that are retailed in refrigerator / freezer and / or refrigerator shelves in retail and convenience stores.

[0046] Furthermore, using these traditional methods mentioned above, advertising efforts may not be as effective and impactful. Due to the limitations of traditional methods, market penetration is very small.

[0047] 1 illustrates a computing platform 100 supporting one or more retail product containers according to an embodiment. The computing platform 100, including a controller 101 and retail product containers 102 and 103, may generate a selected advertisement at one of the retail product containers based on the stored products at the retail product container.

[0048] Retail product cases 102 and 103 may include refrigerators, freezers, vending machines, etc., and typically store different types of products (eg, milk, frozen meals, beer, ice cream, etc.) in corresponding interior volumes.

[0049] refer to Figure 1A In one embodiment of the system disclosed herein, the edge processor (controller) 101 directs the retail product cabinet 102 or 103 to display the selected advertisement based on one or more criteria, such as customers near the retail product cabinet, products stored at the retail product cabinet, and generalized information not specific to the retail product cabinet (e.g., weather conditions). At the same time, other embodiments of the system disclosed herein operate / function without knowing the individual shopper / customer, and the system never seeks to identify individuals, that is, it is "identity blind" because it never collects or uses personally identifiable or contactable information. In some embodiments, whether to perform a set of "identity blind" feature selections can be determined based on whether the system is located in a country or state where the consumer expects privacy and / or applicable laws.

[0050] 1, according to some embodiments, the controller 101 can communicate with the user via a wired or wireless communication channel (eg, Wi-Fi, etc.) to communicate with the retail product container and can be located near the retail product container or anywhere in the world via the Internet. Figure 1BAs shown, in some embodiments, the controller 101 may be built into / integrated into an edge computing device 110 in a retail product cabinet. Of course, the computing platform 100 may include one or more routers (e.g., wireless routers) to communicatively connect one or more edge processors 101 corresponding to the retail product cabinets 102, 103 with the cloud computing service 104. In a system 150, the wireless router ( Figure 1A 104; the wireless router can provide a shared wireless network for multiple Internet of Things (IoT) devices at a retail location (including retail product cabinets 102, 103). Figure 1B In another instance of the system 160, the edge computing device 110 may include wireless hardware (e.g., LTE / 5G or other cellular standards) to connect to the cloud computing service 104 without using a shared wireless router and / or modem operated by the retail owner; the edge computing device 110 may also include an edge processor 101 and a data storage area 111.

[0051] Each retail product cabinet 102 or 103 may support a wireless channel through a common communication interface for associated components (e.g., sensors, cameras, etc.) that is not explicitly shown. However, some embodiments may establish separate wireless communication channels with controller 101 for each of the different components supporting the Internet of Things (IoT). In some instances, retail product cabinets 102, 103 may share edge computing devices 110 to reduce the cost and complexity of retail product cabinets 103. In such instances, as Figure 1B As shown, the edge computing device 110 in the first retail product cabinet 102 can be responsible for processing one or more of the computing steps described in this document. Therefore, the second retail product cabinet 103 can distribute the computing load related to the determination of product out-of-stock to the edge computing device 110 in the first retail product cabinet 102.

[0052] According to some embodiments, the controller 101 may interact with additional computing facilities (e.g., cloud computing service 104 or one or more computer servers (not explicitly shown)) to obtain advertising content and interact with customers or employees via wireless devices 106 through wireless services 105. Alternatively, the wireless devices 106 may connect directly to the edge processor 101 through a shared wireless router (not shown in FIG. 1 ) at the retail location. Figure 1B Another architecture of the computing platform 100 is illustrated, where the wireless device 106 can be a tablet computer, a smart phone, a laptop, or a computer that uses a short-range wireless communication channel (e.g., Wi-Fi, etc.) are communicatively connected to an edge computing device 110 residing in or near the retail product cabinets 102, 103.

[0053] According to some embodiments, Figure 1B As shown, edge computing device 110 can communicate with another computing device 115 (e.g., via cloud computing service 104). Computing device 115 can be associated with an expert user and can be used to control and monitor the operation of edge computing device 110. Figure 1C As shown, in some embodiments, the cloud computing service 104 may include a central data storage area 112, which is herein incorporated by reference. Fig. 9 Further description.

[0054] Figure 2 A front view 231 of a door 230 of a retail product case (eg, cases 102 and 103 shown in FIG. 1 ) is shown according to an embodiment.

[0055] Each retail product case 102 and 103 may have a door 230. A display (screen) 216 is mounted on the door 230 along with customer detection hardware 218 such as one or more proximity sensors (heat maps), image sensors, sensors or scanners that detect human form / features, and eye sensors (iris tracking sensors).

[0056] According to some embodiments, the display 216 includes an LCD panel having one or more touch zones 232 that are interacted with by the customer. An access panel 234 may be located near the bottom of the door 230 along with a media player 236 that provides display content to the display 216. A protective panel (not explicitly shown) may be mounted over the display 216.

[0057] Figure 3 A rear view 302 of a door 230 of a retail product cabinet is shown according to an embodiment. A camera or other type of sensor (i.e., inventory counting hardware 304) can be disposed on the rear side 302 of the door 230 and can face the products stored in the retail product cabinet. The controller 101 can use the inventory counting hardware 304 to track the inventory status of the products in the retail product cabinet.

[0058] Figure 4 A retail product cabinet 102 is shown according to an embodiment. Each retail product cabinet 102 includes at least one display 416, such as a display on a door of the retail product cabinet 102.

[0059] Multiple retail product cases (similar to retail product case 102) and associated displays may be arranged side by side along an aisle of a retail store (e.g., a grocery store). Edge processor (controller) 101 (shown in FIG. 1) may be configured to operate display 416 based on, for example, what is detected by customer detector 418 and / or inventory tracker 420.

[0060] According to some embodiments, when the detected object is considered to be a person (natural person) (rather than, for example, a shopping cart), the customer detector 418 can detect a customer. The customer detector 418 can also determine whether the detected natural person is an employee (for example, through the employee's uniform or through an RFID tag on the employee). In such a case, the customer detector 418 can ignore the employee so that the display at the retail product cabinet is not updated.

[0061] According to some embodiments, display 416 comprises a non-transparent display that prohibits a consumer from viewing products stored in the internal storage volume through the display. Display 416 may comprise, for example, a touch screen with which a user may interact. Display 416 may be associated with a display controller.

[0062] Each retail product case 102 may also include a customer detector 418 that can detect any or specific individuals (e.g., customers versus employees), motion (e.g., movement of customers), human form (e.g., human-shaped forms or general faces of customers), heat, etc. (e.g., a customer may be an individual near a retail product case detected by the customer detector 418). According to some embodiments, the customer detector may utilize one or more of a proximity sensor (e.g., via a heat map), an image sensor, a sensor that detects human form / features, a scanner, an eye sensor (e.g., an iris tracking sensor), etc. In addition to the customer detector 418, the retail product case 102 may also include other sensors 423 configured to detect activity within / around / on the retail product case 102. Although for purposes of illustration, the customer detector 418 has been described as an input that triggers / activates additional steps (e.g., in Figure 8802 and 804 in the figure), but the customer detector 418 can be replaced and / or supplemented by one or more other sensors 423, which include but are not limited to the following examples: a sensor configured to detect an open door; a sensor configured to detect the movement of a door from one position to another (e.g., from closed to open, or from open to closed, whether the door is slammed shut, the angular velocity of the door's movement, etc.); a weight sensor on a shelf in the retail product case 102, which is used to detect changes in inventory; an optical sensor (e.g., a camera) configured to detect out-of-stock products in the retail product case 102; and / or other types of sensor / camera operations.

[0063] According to some embodiments, one or more image sensors 423 may be mounted on the door of the retail product cabinet 102. One or more image sensors may have a depth of field of view of 20 feet or more, and may have a field of view range of 170 degrees, with, for example, a detection capability of 150 degrees. For example, the detection capability may detect whether the overall shape or general features of a person / natural person are near the retail product cabinet 102. Computer-readable instructions may be employed in association with the image sensor to monitor shopper interactions, provide relevant advertising content on the display 416 via the media player 417, and track advertising participation in the store. The controller 101 (as shown in FIG. 1 ) may be configured to independently control each image sensor, and to jointly control multiple displays to provide single-screen and / or multi-screen content and interactions. The controller 101 and the display 416 may be configured to allow banner ads 402 and full-screen ads 401 to be displayed on the display 416. The controller 101 may provide advertising content to the display controller for output at the display 416.

[0064] Each retail product cabinet 102 may also include an inventory tracker 420 to, for example, identify, quantify, and / or otherwise track the stored retail products. In some aspects, the inventory tracker 420 may utilize an additional camera and / or sensor 423 that may be disposed within the retail product cabinet 102 and facing the stored products. In some aspects, the inventory tracker 420 may include a processor, a memory, and / or computer-readable instructions for post-processing images and / or other data captured by the camera and / or sensor. In some aspects, the processor and / or computer-readable instructions may be integrated with the controller 101, and the controller 101 may perform post-processing and analysis of the image / data to identify, quantify, and / or otherwise track the stored retail products.

[0065] The controller 101 may be configured to control the display 416 of each retail product case 102 to provide a planogram (e.g., Figure 84). In some aspects, the planogram may relate to retail products physically contained within the interior storage volume of the retail product case 102. The retail products may not necessarily be visible through the display 416. For example, the products stored in the retail product case 102 may not be neatly arranged or may be blocked from view by other products stored therein. However, the planogram may indicate (e.g., indicate to a customer) the retail products stored within the interior storage volume based on inventory information provided by the inventory tracker 420. Thus, the displayed planogram may effectively optimize the content presented to the customer.

[0066] 1 , the controller 101 may be networked with the retail product cabinet 102 via the Internet, Bluetooth, etc., via a wired Ethernet, a wireless LAN, a cellular network, etc. The controller 101 may be configured to control a display 416 of the retail product cabinet 102, and may be configured to receive information from the retail product cabinet 102, including information from the display 416 (e.g., information about touch screen interactions) and information from a customer detector 418 and an inventory tracker 420.

[0067] Each retail product cabinet 102 may also include an interface 422 that may be configured to facilitate, among other things, networking and transfer of information between the controller 101 and the retail product cabinet 102 , and control of the display 416 , customer detector 418 , and inventory tracker 420 .

[0068] According to some embodiments, controller 101 may include a server having one or more processors, memory storage areas, user interfaces, etc., and may be configured to direct content displayed on display 416 and to receive information and data from retail product cases 102. Additionally, controller 101 may be configured to perform analysis based on the received information and data.

[0069] Although not in Figure 4 Although not explicitly shown in FIG. 1 , embodiments of retail product cabinet 102 may include other components necessary to physically store products within its interior volume (eg, a refrigeration unit, etc.).

[0070] Embodiments of the present invention provide an automatic ability to detect inventory of products placed within retail product cases 102 and update a planogram indicating the products on display 416. Inventory tracker 420 may utilize a camera directed to take pictures of the interior shelves of retail product cases 102, combined with software that utilizes intelligent algorithms for image analysis and subsequent analysis. For example, inventory tracker 420 may determine whether a product is in retail product cases 102, the quantity of a product in retail product cases 102, whether a product is in the correct location in retail product cases 102, etc. For example, if any item is not in retail product cases 102, controller 101 is configured to detect and update information on display 416 to direct consumers and store operators.

[0071] Figure 5A and Figure 5B An example placement of one or more cameras 540 is shown in the door 230 of the retail product cabinet 102. The camera 540 may be mounted on the edge of the back side 302 of the door 230 and may be housed on a bracket 542 that is tilted at an angle so that it can take pictures when the door 230 swings open.

[0072] In an example, wide field of view (FOV) cameras (e.g., 180° FOV ("fisheye")) may be utilized because they provide large area coverage. Images generated by wide field of view cameras may be severely distorted at the edges and limit the ability of the software to "recognize" the image during the image processing step. In an example, a miniature camera with a 120° FOV may be mounted on a 45° angled housing along the edge of the handle side of the door 230. Figure 5A As shown, the camera 540 can be housed within the empty bezel space 543 between the display panel assembly 544 and the mechanical frame 546 of the door 230 .

[0073] Any number of cameras 540 may be provided on the inside of each door of each retail product cabinet 102. Figure 5B As shown, three cameras 540 can be equally spaced along the height of door 230 to allow full height and width coverage of the shelf space (i.e., within retail product cabinet 14). When door 230 is opened at any angle greater than about 25°, the three cameras can capture the entire shelf of the retail product cabinet. In some other examples, the type of camera used and / or the specifications of the camera can result in an angle greater than or less than 25°.

[0074] The cameras 540 can be triggered and capture images during door operation (i.e., when the door 230 is opened). The cameras 540 can be triggered in response to one or more events. For example, the cameras 540 can be triggered based on sensing motion. At least one of the cameras 540 can be used to sense motion. Additionally or alternatively, one or more accelerometers 548 (see FIG. 5A ) on the door 230 can be used to sense motion. Figure 5B ) can be used to sense the movement of the door. For example, one or more accelerometers 548 can sense that the door is closing and trigger the camera 540. The camera 540 can be triggered based on the swing state and position of the door 230. For example, the camera 540 can be triggered when the door 230 is opened.

[0075] Camera 540 may be triggered when door 230 is closed. This may enable inventory tracker 420 to generate an accurate assessment of product inventory in at least some example situations. For example, if retail product case 102 has a single bottle of a particular soft drink, display 416 may display a planogram that includes the soft drink. If a consumer opens door 230 and removes retail product case 102 and closes door 230, inventory tracker 420 may update a record associated with the soft drink and may then update display 416 to indicate (e.g., on a planogram) that the soft drink is now out of stock at retail product case 102. In some embodiments, capturing images when door 230 is closed may enable inventory tracker 420 to accurately update product inventory after a consumer visits retail product case 102.

[0076] Multiple images may be captured during door operation. For example, a single camera 540 may capture multiple images. For example, when three cameras 540 are used, a first photo may be captured by the first camera, a second photo may be captured by the second camera, and a third photo may be captured by the third camera. Alternatively, each of the three cameras 540 may capture multiple images. This may enable images to be captured from different positions / angles relative to the interior contents of the retail product case.

[0077] In addition, in other examples, a motion camera can be used. The motion camera can capture multiple video frames from the start position of the door rotation to a predefined end position. The end position can be a set predefined position, a predefined range, or a predefined relative rotation angle with the start position.

[0078] The controller (e.g., inventory tracker 420 and / or controller 101) can adjust the operating parameters of camera 540 based on various factors. The controller can adjust the exposure, focus position, sensor gain / ISO speed, aperture size, etc. of camera 540. In one example, the controller can change the frequency of image capture based on the rotation speed of the door. For example, if the controller receives a measurement result (e.g., one or more accelerometers 548 or other motion sensors) that the door is swinging at a higher rotation speed, the controller can increase the number of images captured. In low light conditions, the controller can adjust the operating parameters of one or more cameras to adapt to the low light. In other examples, the controller can detect moisture or condensation on the camera lens and modify the operating parameters of the affected camera.

[0079] The controller may perform post-processing on the captured images. The controller may execute computer executable instructions to digitally combine / stitch multiple images into a single composite image. For example, the controller may combine multiple photos from a first camera with one or more photos taken by a second camera and a third camera into a single composite image. In some embodiments, the composite image may undergo post-processing to reduce the image size. In one instance, artificial intelligence using a neural network such as a convolutional neural network may be used to identify the boundary edges of the relevant portions of the contents of the retail product cabinet. The area of ​​the image outside the boundary edge may be discarded by the controller. The area outside the boundary edge may correspond to a location outside the internal storage volume of the retail product cabinet 102. Smaller image sizes may reduce memory requirements at the controller and may also reduce the bandwidth required to transmit the image from the controller to one or more remote servers in the cloud.

[0080] In some embodiments, the captured image may undergo post-processing to reduce blur. A blur mitigation module in the controller may identify repeated portions of the captured image and select appropriate portions from these repeated portions where blur (and / or other undesirable image features) are reduced. The controller may generate a composite image by digitally combining / stitching the selected portions identified by the blur mitigation module.

[0081] like Figure 5A As shown, camera 540 can be positioned so that rotation of camera 540 can cause camera 540 to protrude beyond the edge of display panel assembly 544. At least one benefit of such positioning is that the camera's field of view is not obstructed. In some embodiments, such considerations may result in the location of camera 540 being approximately 28". At the same time, in other embodiments, camera 540 may be located at other distances from the hinge axis.

[0082] In one illustrative embodiment, the camera to shelf distance may be 4" and the retail products on the shelf may be approximately centered relative to the door frame. The camera may have a diagonal FOV of approximately 120° and may be oriented so that the view is wider in the direction of the small dimension of the camera. The camera may be positioned near the handle side of the door, tilted 45° toward the hinge side, without having its view blocked by a display screen within the door. In some instances, the camera to shelf distance may be set to a value other than 4". In other instances of door cabinet assemblies having varying thicknesses or dimensions, the camera may be positioned near the handle side of the door, tilted approximately 40° toward the hinge side, without having its view blocked by a display screen within the door.

[0083] In an example, two cameras may be used to capture all contents of the retail product case 102, with the cameras positioned 28" or more from the hinge and rotated approximately 50° toward the hinge edge of the door. In another example, one or more cameras may be set to rotate at an angle of approximately 30° to 90° toward the hinge edge of the door to optimize capture of the contents of the retail product case 102. The rotation angle of the camera may be based on the size of the door, the opening of the door, and / or the shelf configuration. In one example, one camera may be positioned 21" from the top edge of the door 230 and another camera may be positioned 21" from the bottom edge of the door 230, and the two cameras may be 30" from each other. In another example, the cameras may be positioned substantially equidistant vertically along the door to optimize image capture of the contents of the retail product case 102. The various camera positions and / or orientation positions are merely exemplary, and in other examples, other specifications may be used to position and / or orient the cameras.

[0084] Figure 6 An example camera position in the door 230 of the retail product cabinet 102 is illustrated. The camera FOV is shown as a transparent cone. Fig.14 In an example, the top camera C1 and the bottom camera C3 are positioned approximately 16.5" from the top and bottom edges of the door frame. The center camera C2 can be centered on the door edge and can be 20" from each of the other two cameras. In examples where the door 203 is not vertically centered on the products on the shelf within the retail product cabinet 102, the position of one or more of the three cameras can be moved up or down accordingly. The cameras can be angled so that their FOV only slightly lacks the interior surface of the door. Because the hinge is close to the edge of the shelf, this positioning keeps the hinge side of the scene at the edge of the camera image for almost all door angles.

[0085] Figure 7Different views of an illustrative retail product case and door 230 from the front and top are shown when the door 230 is open about 44° or 25°. The camera FOV is represented as a transparent cone in the orthogonal view and as a triangle in the top view. The surface of the shelf is labeled "F". When the door 230 is open 30° or more, the entire scene is contained within the FOV. When the door angle is reduced to 25°, the portion of the scene on the handle side may begin to move outside the camera FOV. Those skilled in the art will understand after reading the entire disclosure herein that the present disclosure is not limited to the specific angles disclosed in the foregoing examples. Instead, the angles and specifications can be adjusted based on the basic teachings of the present disclosure.

[0086] Figure 8 Different types of advertisements 801, 802, 803, 804, 805 are shown that may be presented on the display 416 of the retail product cabinet 102 according to an embodiment. Advertisement 801 shows a full screen advertisement, while advertisements 802 to 805 (corresponding to a banner ad, a hotspot, a tag / label, and a spot deal, respectively) show offers inserted within the planogram of the retail product cabinet 102. For example, on the display screen, a display advertisement may include a label at the bottom of the product image and a label at the top of the product image. The tags and labels may be selected and activated from a predetermined pool of animated icons. For example, a local advertisement for pizza in the display advertisement may be replaced with an animated graphic rendering of a pizza with steam coming out of it. Although for the purposes of the patent description, the display advertisement may be replaced with an animated graphic rendering of a pizza with steam coming out of it. Figure 8 The product displayed on display 416 in FIG. 4 is shown as a generic image without branding, but in actual operation, the displayed graphic image will be brand-specific and include the brand logo, name, colors and other characteristics.

[0087] As will be discussed, in some instances, the inserted offers may be customized based on customers in the vicinity of the retail product case 102 .

[0088] Referring to advertisement 802, customer detector 418 can detect when a customer is viewing (watching) area 851 of an advertisement (eg, advertisement 802). If so, an advertisement for one or more products shown in area 851 can be displayed on screen 416.

[0089] Referring to full screen advertisement 801 , while a single advertisement may occupy the entire area of ​​screen 416 , embodiments may support multiple advertisements that may be displayed on screen 416 simultaneously (in different display areas).

[0090] Fig. 9An input planogram map 902 of the interior storage volume of the retail product cabinet 102 is shown. The planogram map 902 can be stored, for example, in a data store 111 (e.g., associated with the controller 101). The planogram map 902 can be received (e.g., input to a data store) from the cloud service 104, the wireless device 106, and / or a combination thereof. For example, Figure 1B As illustrated, a user at a retail store location may input (e.g., enter, create, update, verify, etc.) a planogram map 902 stored in the data store 111 using a mobile wireless device 106. The user may be an employee at the retail location 100, or may be an authorized user (e.g., a vendor, product supplier, consultant, authorized third party, or other user) who uses the mobile device 106 to verify and update a planogram map 902 previously downloaded from the cloud 104 to the data store 111. In some instances, a central corporate office may have prepared an initial planogram map 902 and distributed it to one or more retail product shelves 102 via the cloud service 104.

[0091] The planogram map 902 may include one or both of: product indicators of products stored within the retail product shelves 102 and corresponding location indicators of locations within the retail product shelves 102 where the products are to be stored. For example, the planogram map 902 may indicate that product P1 may be stored at location L1, product P2 may be stored at location L2, etc. Figure 8 As an example, the digital display 416 on the retail product case 102 may display the digital planogram illustrated in 803. The value of the location L1 stored in the data storage area 111 may indicate that the leftmost aisle / point on the top shelf is designated for thin plastic bottles of a particular type of beverage. Corresponding to the location L1 in the planogram map 902 stored in the data storage area 111, the value of the product P1 may indicate a thin plastic bottle shape of a particular size. Meanwhile, for the location L2 indicating the rightmost aisle on the top shelf, the value of the product P2 may indicate the same thin plastic bottle shape of a particular size, but including color characteristics unique to the bottle. For example, the product P2 in the data storage area 111 may indicate a particular color scheme and a mark of a "Y" with a particular color. Meanwhile, P1 may indicate a mark of an "X" with a particular color.

[0092] Thus, the location may include a particular shelf (e.g., a top shelf, a bottom shelf, a third shelf above the bottom shelf, etc.) and / or a location on a shelf (e.g., the leftmost aisle / point on a shelf, approximately the center of a shelf). In other embodiments, the location may be an x-coordinate and a y-coordinate in a forward-facing plane of the interior storage volume of the retail product cabinet 102 captured by one or more cameras facing inward. For example, the leftmost bottom corner of the composite image generated by the camera may be a (0, 0) coordinate in the xy plane. In some instances, the z-coordinate may also be incorporated into a location corresponding to a distance deeply into the interior storage volume in a direction toward the back of the retail product cabinet 102. The systems 170, 160, 150 may receive a composite image generated by a camera as input and detect shelves in the image using one or more of the method steps described herein. The generation of the composite image may include identifying and removing duplicate shelves, identifying shelf edges, etc. from multiple camera angles. In one instance, the system is trained to identify shelves (e.g., shelf edges) in various situations. Then, in some instances, at a later stage, a separate trained model may identify objects / products on the shelf. In some instances, the detection of shelves in the image can be used in combination with the planogram product image size to identify where the product should be combined with the identification object itself. In other embodiments, the location of the product in the retail product case 102 can be more or less granular in detail; for example, the location can be simply which retail product case 102 should hold the identification of the product, rather than specifically what shelf, shelf position, and / or shelf depth it should be located at. The product indicator can be indexed by the corresponding location indicator. In some instances, the product indicator can be a stock keeping unit (SKU) identifier, or can be another identifier of the product.

[0093] In addition to the input planogram map 902, the data store 111 may also include descriptions 904 of products stored in the retail product shelves 102. The descriptions 904 may be downloaded from the central data store 112, which may include descriptions associated with all products stored by a retailer and / or all / most retailers. For example, when a user at a retail store location inputs a planogram map 902, the edge computing device 110 may download descriptions associated with the products identified in the planogram map 902 from the central data store 112 and store them in the local data store 111. In one example, the file including the descriptions 904 may be manually loaded into the data store 111 by a user using a wired or wireless communication protocol (e.g., a universal serial bus (USB) protocol, a local area network (LAN) protocol, or an Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol).

[0094] In at least some instances, the local data store 111 may include product indicators and descriptions 904 associated with a subset of all products stored by the retailer (not just the products stored in the retail product cabinet 102). The subset of products may include products in the retail product cabinet 102, other products that may be stored near the retail product cabinet 102, and / or other products that may be related to (e.g., in the same category) the products stored in the retail product cabinet 102. For example, if the retail product cabinet 102 includes soft drink bottles, the description 904 may include descriptions associated with the soft drink bottles, and also include descriptions associated with related products such as soft drink cans. In an example, products may be classified into multiple categories (e.g., frozen foods, meat, poultry, dairy products, meat). If the retail product cabinet 102 is used to store a specific category of products, the data store may include descriptions and identities associated with all products in the specific category.

[0095] These descriptions may include one or more of the following: a corresponding shape of the product (e.g., tall bottle, short bottle, tall can, glass bottle, plastic bottle, cardboard box, short can, square box, rectangular box, bag, irregular shape, other shape), a corresponding color of the product packaging (including decorative elements and / or logos in some embodiments) (e.g., red, orange, yellow, green, blue, red background with white stripes, blue background with bold white text, red box with a photo of pizza, and others), a corresponding size of the product (e.g., 6" tall with a diameter of approximately 2" wide; approximately 12" x 12" box with an indeterminate thickness; approximately 10" x 10" box with a 1" thickness; 200 pixel x 200 pixel box, and others), and / or the like. For example, corresponding to location L1 indicating the leftmost aisle on the top shelf, the description of product P1 may indicate a bottle shape having a first size and a red color scheme. For location L2 indicating the rightmost aisle on the top shelf, the description of product P2 may indicate a bottle shape having a second size and a blue color scheme. The description may be received from cloud service 104, wireless device 106, or may be input using interface 422.

[0096] Data storage area 111 may also include advertising content (e.g., illustrations, multimedia content) associated with products stored in retail product cabinet 102 or a subset of products stored by the retailer. For example, if retail product cabinet 102 is for dairy products, data storage area 111 may include advertising content associated with dairy products. Data storage area 111 may also include advertising content associated with other retail product cabinets (e.g., retail product cabinet 103) that may be near retail product cabinet 102, or advertising content associated with products in the same category as the products in the retail product cabinet. For example, if retail product cabinet 102 is for dairy products and a nearby retail product cabinet 103 is for beverages (e.g., soft drinks, juice), data storage area 111 may include advertising content associated with dairy products and beverages.

[0097] The advertising content may be downloaded from the central data store 112, which may include advertising content associated with all products stocked by the retailer. Fig.11 As further described, an advertisement manager associated with controller 111 may request that a group of advertisements (ranging from one advertisement to many (e.g., hundreds) of advertisements) be downloaded to data storage area 111. In other examples, the advertisement content may be manually loaded into the data storage area by a user using a wired or wireless communication protocol (e.g., a USB protocol, a LAN protocol, or an IEEE 802.11 protocol).

[0098] Whenever a new product is introduced, the planogram mapping is modified, and / or a product description is changed, the central data store 112 may be updated. The central data store 112 may push updates to the data store 111. Updates may be periodic (e.g., weekly, monthly), or may be in response to changes in information associated with products stored in the retail product shelves 102 or changes to the planogram mapping 902. At least one advantage of storing local cache memory in the data store 111 is that a temporary loss of connectivity between the retail product shelves 102 and the remote cloud service 104 does not stop the operation of the system 100 at the retail location. The edge processor 101 may operate and subsequently synchronize with the remote cloud service 104 without interrupting operations to customers in the retail location.

[0099] The controller (e.g., edge computing device 110) may post-process the one or more images captured by the camera (as described above) to generate a composite image. Based on the composite image, the planogram map 902, and the description of the product, the controller may determine the inventory status of the product in the retail product shelf 102. Determining the inventory status of the product may include determining whether the product in the input planogram map 102 is present in the retail product shelf 102, the quantity of the product, whether the product is located at the location indicated in the planogram, etc.

[0100] The controller may determine an advertisement to be displayed on display 416 based on the determined inventory status of the product. According to one or more processes described herein, the advertisement may, for example, indicate that the product is out of stock based on a determination that the product is not in retail product case 102. Additionally or alternatively, the controller may generate and transmit a message (e.g., to wireless device 106 or another computing device associated with an employee) indicating that the product is not in retail product case 102. According to one or more processes described herein, the advertisement may, for example, indicate the product (e.g., an illustration associated with the product) based on a determination that the product is in retail product case 102.

[0101] The controller may determine the occupancy of the location based on the post-processed composite image.The controller may determine the occupancy of the location using, for example, an edge detection algorithm.

[0102] The controller may determine, based on post-processing the composite image or using other sensor data, that the location in the retail product case 102 does not include any product. Based on this determination, the controller may determine that the product mapped to the location in the planogram map 902 is not in the retail product case 102. For example, the controller may determine that the leftmost aisle / point on the top shelf (e.g., location L1 designated for product P1) does not include any product. Based on this, the controller may determine that product P1 is not in the retail product case 102. The controller may search for products outside of location L1 to accommodate any offset in the placement of the product in the retail product case 102. If the controller determines that location L1 does not include any product, the controller may generate an advertisement to display on the display 416 indicating that product P1 is out of stock at the retail product case 102. Instead of a tag / label associated with product P1, the advertisement may indicate an illustration showing that product P1 is out of stock. The controller may send the advertisement to a display controller associated with the display 416. The display controller may output the advertisement on the display 416.

[0103] The controller may determine that a location in the retail product shelf 102 includes a stored product (e.g., the location is occupied by an item) based on the post-processed composite image or using other sensor data. The controller may also determine other attributes of the stored product (e.g., shape, color, size, etc.) based on the post-processed composite image. For example, in some instances, post-processing may include determining the dimensions of the product based on the number of pixels occupied by the product in the composite image and the known distance of the shelf from the camera location. For example, a 200 pixel by 200 pixel image of a box in the composite image may correspond to a 10" by 10" box at a predetermined location on the shelf. The post-processing step may scale the image pixel by pixel to account for the additional distance in front of the shelf. Additionally, the controller may compare the determined attributes of the stored product to the description of the product mapped to the location in the planogram map 902. If the determined attributes of the stored product match the description, the controller may determine that the product is in the retail product shelf 102. If the determined attributes of the stored product do not match the description, the controller may determine that the product is not in the retail product shelf 102.

[0104] For example, the controller may detect that the leftmost aisle on the top shelf (e.g., a location L1 designated for product P1) includes a product based on a post-processed composite image or using other sensor data. The controller may also determine other attributes of the product detected at location L1 based on the post-processed composite image. The controller may determine the shape of the detected product, the color of the detected product, and / or the size of the detected product. The controller may determine that the product detected at location L1 is product P1 based on the following: determining that the detected product is in the shape of a bottle; determining that the color of the detected product is red; and / or determining that the detected product has a first size. The controller may, for example, determine individual confidence scores (e.g., probability metrics) associated with whether the detected product is in the shape of a bottle, whether the color of the detected product is red, and / or whether the detected product has a first size. The controller may generate an overall confidence score based on the above confidence scores to determine whether the detected product is product P1. For example, if the overall confidence score is above a threshold (e.g., 60%, 70%, or any other value), the controller may determine whether the detected product is product P1.

[0105] Based on this determination, the controller can generate an advertisement to be displayed on display 416 indicating product P1. The advertisement can indicate an illustration of product P1 (e.g., a label / tag associated with product P1). The controller can send the advertisement to a display controller associated with display 416. The display controller can output the advertisement on display 416.

[0106] In one example, the controller may determine that the product detected at the leftmost aisle on the top shelf is not product P1 based on: determining that the shape of the detected product is not a bottle shape; determining that the color of the detected product is not red; and / or determining that the size of the detected product is not equal to the first size. For example, if any of the individual confidence scores is below a threshold (e.g., 60%, 70%, or any other value), the controller may determine that the detected product is not product P1. For example, if the overall confidence score is below a threshold (e.g., 60%, 70%, or any other value), the controller may determine that the detected product is not product P1.

[0107] If the controller determines that the product detected at location L1 is not product P1 (e.g., it is not red and / or does not have the aforementioned first size), the controller may attempt to identify the detected product. The controller may, for example, scan a description of the product indicated in the input planogram map 902 (e.g., which may be associated with other locations) to identify the detected product. The controller may determine that the detected product is product P2, for example, based on: determining that the detected product is bottle-shaped; determining that the color of the detected product is blue; and / or determining that the detected product has a second size. As described above, the controller may use individual confidence scores and / or an overall confidence score.

[0108] In one example, an artificial intelligence network (on which one or more machine learning algorithms / models are executed) is included in the system disclosed herein. The framework for a machine learning algorithm can involve a combination of one or more components (sometimes three components): (1) representation, (2) evaluation, and (3) optimization components. The representation component refers to a computing unit that performs steps to represent knowledge in different ways, including but not limited to one or more decision trees, rule groups, examples, graphical models, neural networks, support vector machines, model collections, and / or other. The evaluation component refers to a computing unit that performs steps to represent the way in which a hypothesis (e.g., a candidate program) is evaluated, including but not limited to accuracy, prediction and recall, squared error, likelihood, posterior probability, cost, tolerance, entropy kL divergence, and / or other. The optimization component refers to a computing unit that performs steps to generate candidate programs in different ways, including but not limited to combinatorial optimization, convex optimization, constrained optimization, and / or other. In some embodiments, other components and / or subcomponents of the aforementioned components may exist in the system to further enhance and supplement the aforementioned machine learning functions.

[0109] The aforementioned artificial neural network can be configured to perform decision making. As a simplified example for explanation purposes, the artificial neural network can be configured to detect features of a product in an input image. A photo image can be provided as input to the neural network. The layers of nodes of the neural network can each be configured to perform such as Fig. 20 The processing may be further refined by multiple subsequent layers, each layer seeking further more specific tasks, wherein each node performs some form of processing that does not necessarily need to operate in furtherance of the task. The artificial neural network may then predict whether the shape, color, size, and / or other features of the identified product match the product description of the product that should be located at a specific location on a shelf in a specific retail product bin. The prediction may be correct or incorrect, and the neural network may generate a confidence value / score corresponding to whether the prediction is correct or incorrect.

[0110] The feedback system of the artificial neural network can be configured to determine whether the artificial neural network has made a correct decision. The feedback can include an indication of the correct answer and / or an indication of an incorrect answer and / or the degree of correctness (e.g., a percentage). For example, in the product description recognition example provided above, the feedback system can be configured to determine whether the product is correctly identified. The feedback system may already know the correct answer, so that the feedback system can train the artificial neural network by indicating whether it has made a correct decision. The feedback system can include human input, such as an expert user of the calculation and 115 to tell the artificial neural network 700 whether it has made a correct decision. The feedback system can provide feedback to the artificial neural network (e.g., an indication of whether the previous output is correct or incorrect). The feedback system can be coupled to a data storage area in addition or alternatively so that the output is stored. The feedback system may not have a correct answer at all, but is based on feedback for further processing: for example, the feedback system may include a system programmed to identify products on a shelf in an image, so that the feedback allows the artificial neural network to compare its results with the results of the manually programmed / cured system.

[0111] In one instance, an artificial neural network can be dynamically modified to learn and provide better input. Based on, for example, previous inputs and outputs and feedback from a feedback system, the artificial neural network can modify itself. For example, the processing in a node can be changed and / or the connections can be weighted differently. Following the example provided previously, the product identification may be incorrect because the photo provided to the algorithm is colored in a way that all product markings are red. As such, some nodes may detect false positives and may be considered unreliable, and the connection to the node may be significantly less weighted. Additionally or alternatively, the node may be reconfigured to process the image differently. The modification may be a prediction and / or guess by the artificial neural network, so that the artificial neural network can change its nodes and connections to test the hypothesis.

[0112] The artificial neural network need not have a set number of processing nodes or groups of processing nodes, but may increase or decrease its complexity. For example, the artificial neural network may determine that one or more processing nodes are unnecessary or should be repurposed, and based on this, discard or reconfigure the processing nodes. As another example, the artificial neural network may determine that further processing is required for all or part of the input, and based on this, add additional processing nodes and / or groups of processing nodes.

[0113] The feedback provided by the feedback system can be reinforcement only (e.g., providing an indication that the output is correct or incorrect, giving a machine learning algorithm multiple points, etc.), or can be specific (e.g., providing a correct output).

[0114] Artificial neural networks can be supported or replaced by other forms of machine learning. For example, one or more of the nodes of an artificial neural network can implement a decision tree, an association rule set, logic programming, a regression model, a clustering analysis mechanism, a Bayesian network, a propositional formula, a generative model, and / or other algorithms or decision-making forms. Artificial neural networks can implement deep learning.

[0115] In one example, a machine learning algorithm can rely on a unique computing system structure and sometimes make full use of neural networks. Although such a structure is much more complex than a conventional computer system, it is beneficial in implementing machine learning. For example, an artificial neural network can include a large group of nodes, which, like neurons in the brain, can be dynamically configured to implement learning and decision-making. Each of the nodes can be connected to one or more other nodes. These connections can connect the output of a node to the input of another node. The connection can be related to a weighted value. For example, a connection can be weighted to be more important or more significant than another connection, thereby affecting the degree of further processing when the input passes through the artificial neural network. Such a connection can be modified so that the artificial neural network can learn and / or be dynamically reconfigured. Some artificial neural networks can use nonlinear processing, but other forms of nonlinear processing can be used to implement machine learning algorithms according to the features described herein.

[0116] In another example, a trained convolutional neural network (CNN) (an example of a feed-forward network) takes input data (e.g., a picture of a natural person) into a convolutional layer (also called a hidden layer), and applies a series of trained weights or filters to the input data in each of the convolutional layers. The output of the first convolutional layer is an activation map (not shown), which is the input of the second convolutional layer, and the trained weights or filters (not shown) are applied to the second convolutional layer, where the output of the subsequent convolutional layers produces activation maps representing increasingly complex features of the input data to the first layer. After each convolutional layer, a nonlinear layer (not shown) is applied to introduce nonlinearity into the problem, which can include tanh, sigmoid, or ReLU. In some cases, a pooling layer (not shown), also called a downsampling layer, can be applied after the nonlinear layer, which basically takes filters of the same length and stride, applies them to the input, and outputs the maximum number of filters in each sub-region around which the filter is convolved. Other options for pooling are average pooling and L2 norm pooling. The pooling layer reduces the spatial size of the input volume, thereby reducing computational costs and controlling overfitting. The final layer of the network is a fully connected layer that takes the output of the last convolutional layer and outputs an n-dimensional output vector that represents the quantity to be predicted, for example, the probability of the image being classified as 20% car, 75% boat, 5% bus and 0% bicycle, i.e., produces the predicted output (O *), which could be a picture of a boat, for example. The output could be a scalar-valued data point predicted by the network, such as a stock price. The trained weights can be different for each of the convolutional layers, as will be described more fully below. In order to achieve this real-world prediction / detection (e.g., it is a boat), the neural network needs to be trained on known data inputs or training instances that produce a trained CNN. To train the CNN, many different training instances (e.g., many pictures of boats) are input into the model. Those skilled in the art of neural networks will fully appreciate that the above description provides a somewhat simplified view of a CNN to provide some context for this discussion, and will fully appreciate that the application of any CNN alone or in combination with other neural networks will equally apply to and be within the scope of some of the embodiments described herein.

[0117] In one example, the convolutional layers are shown as hidden convolutional layers up to convolutional layer n-1, and the last nth layer is a fully connected layer. It should be understood that the last layer can be more than one fully connected layer. The training instance is input to the convolutional layer, and the nonlinear activation function (not shown) and weights to n are applied serially to the training instance, wherein the output of any hidden layer is input to the next layer, and so on, until the last nth fully connected layer n produces an output. The output or prediction is compared with the training instance (e.g., a picture of a ship), thereby generating a difference between the output or prediction and the training instance. If the difference or loss is less than a preset loss (e.g., the output or prediction predicts that the object is a ship), the CNN is converged and is considered to be trained. If the CNN has not yet converged, the back propagation technique is used to update the weights according to how close the prediction is to the known input. Of course, methods other than back propagation can be used to adjust the weights. In the second training instance (e.g., a different picture of a ship) is input and the process is repeated again with the updated weights, and then the weights are updated again, and so on, until the nth training instance (e.g., the nth picture of the nth ship) has been input. This is repeated over and over again with the same n-training instance until the convolutional neural network (CNN) is trained or converges on the correct output for the known input. Once the CNN is trained, the weights are fixed and used in the trained CNN. Each of the convolutional layers and fully connected layers has different weights. The trained CNN or model is then fed with image data to determine or predict which (e.g., a ship) it is trained to predict / identify, as described herein. Any trained model, CNN, RNN, etc. can be further trained, that is, it can be allowed to modify the weights using additional training instances or using prediction data output by the model, which is then used as a training instance. The machine learning model can be trained "offline", for example, once on a computing platform separate from the platform using / executing the trained model, and then transmitted to the platform. Alternatively, the embodiments described herein can periodically or continuously update the machine learning model based on newly acquired training data. This updated training can occur on a separate computing platform that delivers the updated trained model to a platform that uses / executes the retrained model via a network connection, or the training / retraining / update process can occur on the platform itself when new data is acquired. CNNs can be applied to data in fixed arrays (e.g., pictures, characters, words, etc.) or time series of data. For example, CNNs can be used to model sorted measurement data and other factor data. Some embodiments utilize a feedforward CNN with jump connections and a Gaussian mixture model output to determine a probability distribution of a predicted calculated score (e.g., a calculated score, a weight, an activity level, or some cardiac irregularity).

[0118] Some embodiments may utilize neural networks of other types and configurations. The number of convolutional layers and the number of fully connected layers may be increased or decreased. Typically, the optimal number and ratio of convolutional layers to fully connected layers may be set experimentally by determining which configuration gives the best performance on a given data set. The number of convolutional layers may be reduced to 0, leaving only a fully connected network. The number of convolutional filters and the width of each filter may also be increased or decreased. In some instances, the output of the neural network may be a single scalar value corresponding to an accurate prediction of the primary time series. Alternatively, the output of the neural network may be a logistic regression, where each category corresponds to a specific range or level of primary time series values, which is any number of alternative outputs readily understood by those skilled in the art. At the same time, in some embodiments, the use of Gaussian mixture model outputs is intended to constrain the network to learn well-formed probability distributions and improve generalization of restricted training data. In some embodiments, the use of multiple elements in the Gaussian mixture model is intended to allow the model to learn multimodal probability distributions. Machine learning models that combine or aggregate the results of different neural networks may also be used, where the results may be combined.

[0119] After the controller determines whether the detected product is correct, based on the determination, the controller may generate an advertisement for display on the display 416 indicating product P2 (and / or not indicating product P1). The advertisement may indicate an illustration of product P2 (e.g., a tag / label associated with product P2). The controller may send the advertisement to a display controller associated with the display 416. The display controller may output the advertisement on the display 416 at a location on the display 416 corresponding to the actual location in the internal storage volume. For example, if product P2 is located on the top shelf at the leftmost location in the internal storage volume, the display controller outputs an appropriate illustration graphic at the corresponding pixel location on the display 416. Additionally or alternatively, the controller may generate and transmit a message (e.g., to a wireless device 106 or another computing device associated with the employee) that product P2 is in location L1 corresponding to product P1.

[0120] In at least some instances, the above process for identifying products in retail product cases 102 can be resource efficient because the controller only needs to process planogram maps 902 associated with retail product cases 902. For example, planogram maps 902 can be associated with retail product cases 102 and can include only details associated with the products that retail product cases 902 are configured to store (rather than all products stocked by the retailer). As a result, the process can be completed by an edge processor with relatively low computational capacity and does not need to be sent to a higher edge processor.

[0121] Furthermore, because the planogram map 902 can be indexed by location, the controller can quickly retrieve product descriptions to compare with stored product attributes. For example, the controller can identify the location where the product was detected and simply scan the entry corresponding to the location in the planogram map 902 to identify the product and determine whether the product is stored in the correct location in the retail product shelf.

[0122] The controller may use a machine learning algorithm to identify the stored products in the retail product shelves 102. The controller may use attributes of the stored products (e.g., shape, color, size, etc.) in the machine learning algorithm to identify the stored products. For example, if the controller cannot identify the product based on the planogram map 902 and the description of the product, the controller may use a machine learning algorithm.

[0123] In an example, if the controller cannot identify the product, the controller may send a notification (e.g., to a wireless device 106 associated with an employee or a computing device 115 associated with an expert user at a remote location). The notification may include the identity of the retail product shelf 102, the composite image, and / or the attributes of the unidentified product (e.g., shape, color, size, etc.) for identification by the employee / expert user. The employee / expert user may identify the product and manually enter the identity of the product using the wireless device 106 / computing device 115. The wireless device 106 or computing device 115 may send the identity of the product to the controller. In an example, if the controller cannot identify the product, the controller may send the determined product attributes to a central cloud-based processing system (e.g., cloud service 104) for identification. Based on the identification of the product (e.g., by the employee / expert user or by the cloud-based processing system), the controller may update the planogram map 902 in the data store 111 and / or generate an advertisement including an illustration associated with the product. For example, the controller may add a new entry to the planogram map 902 including information associated with the product (eg, product indicator, location indicator, description, etc.).

[0124] In one instance, when the controller cannot identify the product in the image, the user (e.g., an employee / expert or other person) can manually identify the product and select the identity of the product. In machine learning, tasks can sometimes be broadly classified as unsupervised learning or supervised learning. In unsupervised learning, the machine learning algorithm is left to generate any output (e.g., to be labeled as needed) without feedback (e.g., without human intervention). The machine learning algorithm can teach itself (e.g., observe past outputs), but otherwise operate without (or almost without) feedback from, for example, a human administrator. At the same time, in supervised learning, the machine learning algorithm is provided with feedback on its output. Feedback can be provided in a variety of ways, including via active learning, semi-supervised learning, and / or reinforcement learning. In active learning, the machine learning algorithm is allowed to query answers from the administrator. For example, a machine learning algorithm can make guesses in the form of a human or a general human face detection algorithm, requiring the administrator to identify the photo in the picture, and compare the guesses and the administrator's responses. In semi-supervised learning, a set of example tags and unlabeled data are provided to the machine learning algorithm. For example, a machine learning algorithm may be provided with a data set that includes 700 photos of generic human faces with labels and 10,000 random, unlabeled photos. In reinforcement learning, the machine learning algorithm is rewarded for correct labeling, and is allowed to iteratively observe the conditions until the reward is consistently obtained. For example, for each generic human face correctly identified, the machine learning algorithm may be given a point and / or confidence score (e.g., "75% correct").

[0125] One theoretical basis for supervised learning is inductive learning. In inductive learning, data representation is provided as input sample data (x) and output samples of a function (f(x)). The goal of inductive learning is to learn a good approximation of the function for new data (x), that is, to estimate the output for future new input samples. Inductive learning can be used for various types of functions: (1) classification functions, where the function being learned is discrete; (2) regression functions, where the function being learned is continuous; and (3) probability estimation, where the output of the function is a probability.

[0126] In practice, data scientists tune machine learning systems and their underlying components to perform many steps to refine machine learning systems. This process is sometimes iterative and may require going through a series of steps in sequence: (1) understanding the domain, prior knowledge, and goals; (2) data integration, selection, cleaning, and preprocessing; (3) learning models; (4) interpreting results; and / or (5) incorporating and expanding the discovered knowledge. Given the nearly infinite number of variables that can be optimized in a machine learning system, this may also include consulting with domain experts to refine and make the goals clearer. At the same time, one or more of the data integration, selection, cleaning, and / or preprocessing steps may sometimes be the most time-consuming, as the old adage, “garbage in, garbage out” is also true for machine learning systems.

[0127] In one instance, the AI ​​system can use one of countless machine learning models, including xg enhanced decision trees, autoencoders, perceptrons, decision trees, support vector machines, regression and / or neural networks. The neural network can be any of countless types of neural networks, including feedforward networks, radial basis networks, recursive neural networks, long / short term memories, gated recursive units, autoencoders, variational autoencoders, convolutional networks, residual networks, Kohonen networks and / or other types. The middle layer of the neural network can be a convolutional layer, a pooling layer, a dense (fully connected) layer and / or other types. The input layer can pass the input to the middle layer. In one instance, the layer in the neural network can use an activation function, such as a sigmoid function, a Tanh function, a ReLu function and / or other functions. In addition, the neural network can include a loss function. When training a neural network, the output of the output layer can be used as a prediction. The output data in the machine learning system can be represented as a multidimensional array, which is an extension of a two-dimensional table (such as a matrix) to data with higher dimensions. In addition, the neural network can be compared with the target value of the training example to determine the error. The error can be used to update the weights in each layer of the neural network. The neural network can use gradient descent to update the weights. Alternatively, the neural network can use an optimizer to update the weights in each layer. For example, the optimizer can use various techniques or combinations of techniques to update the weights in each layer. Where appropriate, the neural network can include mechanisms to prevent overfitting - regularization (such as L1 or L2), dropout, and / or other techniques. The neural network can also increase the amount of training data used to prevent overfitting.

[0128] Once the data for machine learning is created, an optimization process can be used to transform the machine learning model. In one example, the cloud service 104 can be used to incorporate images captured at the retailer product cabinet 102, and the image data can be used to train and test new models, such as the trained convolutional neural network (CNN) and other models described herein. In some examples, such as Figure 1C As illustrated, the model can be run on-site at the retail location on the edge computing device, however, the computationally intensive training / retraining can be offloaded to and performed in the cloud computing service 104. For example, to provide a flexible solution, a portion of the captured image can be uploaded to the cloud so that the model can be trained / updated there. Once the model is updated and / or confirmed, the cloud can distribute the new model version to one or more edge computing devices 110 at one or more retail locations for execution. Of course, as the processing power and speed of the edge computing processor 101 increases and the constraints of the local data storage area 111 improve, the use of the cloud computing service 104 for off-site processing can also be reduced. In one example, if the edge computing device 110 properly executes the model at the edge controller 101 and uses the local data storage area 111, a temporary loss of connectivity between the computing platform at the retail location 100 and the cloud computing service 104 for a period of time may not matter. Then, upon reconnection, the cloud computing service 104 can synchronize with the retail location 100 and exchange data and other information.

[0129] Using the trained and updated model, an optimization process can be used to transform the machine learning model. The optimization process can include (1) training data to predict outcomes, (2) defining a loss function that serves as an accurate measure for evaluating the performance of the machine learning model, (3) minimizing the loss function, such as by a gradient descent algorithm or other algorithms, and / or (4) optimizing a sampling method, such as using a stochastic gradient descent (SGD) method, where instead of feeding the entire data set to the machine learning algorithm for computation at each step, subsets of the data are sequentially sampled.

[0130] For example, the controller may not be able to identify the product detected at location L1. The controller may determine that the detected product is a box shape having a third size and is brown. The controller may send a notification of the composite image and the determined attributes of the unidentified product to a wireless device 106 associated with the employee. The employee may identify the product as product P3. The wireless device may send an indication of product P3 to the controller, and the controller may then update the planogram map 902 to indicate product P3 at location L1. The controller may also include a description of product P3 at location L1 to indicate that product P3 is a box shape having a third size and is brown. The controller may generate an advertisement to be displayed on the display 416 to indicate product P3.

[0131] In an example, the controller can determine descriptions of the products based on the training process. For example, an employee can stock the retail product shelves 102 based on the planogram map 902 and initiate a training process at the controller. The controller can determine associated descriptions (e.g., shape, color, size, etc.) of the products stored in the retail product shelves 102 based on the composite image and the planogram map 902. For example, the controller can scan location L1 and determine a description associated with product P1 stored at location L1.

[0132] Determining the inventory status of the product may include determining the quantity of the product (e.g., the number of product packages) in the retail product case 102. In an example, the product packages in the retail product case may be stacked on top of each other, and the controller may use an image processing algorithm on the composite image and further use a description of the product (e.g., shape, size, color) to determine the quantity.

[0133] In an example, a first package of products may obscure other packages of products placed behind the package. Removing the first package may reveal the next package. The controller may compare multiple composite images to determine slight changes in package placement / orientation to determine whether the first package was removed. For example, if the controller determines that a first composite image shows a package oriented in a first manner, and a next composite image shows a package oriented in a different manner, the controller may determine that the package was removed.

[0134] In an example, each package can be associated with a unique SKU identifier, and the controller can compare the plurality of composite images to determine the SKU identifier associated with the product. If the controller determines that a first composite image shows a first package with a first SKU identifier and a next composite image shows a second package with a different SKU identifier, the controller can determine that the first package is removed.

[0135] The controller may count the number of times packages are removed. Based on the initial number of packages stored in the retail product case 102 and the count, the controller may determine the number of products remaining in the retail product case. The initial number of packages may be indicated in the planogram map 902 or in the description of the product. The initial number may be entered by an employee stocking the retail product case 102.

[0136] The controller may generate and transmit a message (e.g., to the wireless device 106 or another computing device) indicating the quantity of the product. The controller may generate and transmit a message indicating that the product is low in stock based on determining the quantity of the product and further determining that the quantity is below a (e.g., predetermined) threshold quantity. The threshold quantity for each product may be included in a description associated with the product.

[0137] Determining the inventory status of the product may include determining the shelf time of the packages in the retail product cabinet 102. In an instance where each package of the product is associated with a unique SKU identifier, the controller may use the composite image and the time at which the composite image was captured to determine the shelf time associated with the package corresponding to the specific SKU identifier. For example, if the first composite image (at 12:00 pm on Tuesday) shows a package with a first SKU, and the second composite image (at 9 pm) shows the same package with the first SKU, the controller may determine that the shelf time of the package is 9 hours. In an example, the controller may determine that the shelf time is equal to the time period during which the product count does not decrease. For example, if the shelf time of the product has exceeded a threshold, the controller may generate and transmit a notification message (e.g., to the wireless device 106 or another computing device). Referring to the above example, if the threshold is defined as 14 days, the controller may transmit a notification message indicating that the shelf time of the package has exceeded the threshold. The threshold time for each product may be included in the description associated with the product.

[0138] In an example, the controller can use the composite image to determine an expiration date (e.g., printed on a product's label, or using a SKU). For example, if a product has exceeded its expiration date, the controller can send, generate, and transmit a notification message (e.g., to wireless device 106 or another computing device).

[0139] The controller may be integrated with the retail product cabinet 102 or may be located nearby (e.g., in a retail store housing the retail product cabinet). This may enable the controller to operate without requiring any external connectivity (e.g., utilizing a cloud-based service) to perform the various processes described herein.

[0140] Fig.10 Different advertisements 1004-1020 are shown that may be displayed on display 416 based on the determined inventory status of the products in retail product cabinet 102. The controller may determine that product 1024 is out of stock at retail product cabinet 102. As shown in advertisement 1008, the controller may overlay an out-of-stock notification (e.g., a standardized out-of-stock image) on a tag / label associated with product 1024.

[0141] Alternatively, as shown in advertisement 1012, the controller may overlay a tag / label associated with a different product over a tag / label associated with product 1024. The different product and product 1024 may be associated with the same brand, or may be associated with different brands.

[0142] To avoid displaying a mark / label associated with a different brand, the brand corresponding to product 1024 can protect the mark / label associated with product 1024 by setting restriction rules on computing platform 100 so that retail product cabinet 102 allows or does not allow the display of graphics of a specific product on a specific retail product cabinet. Based on these rules, the advertising graphics can be configured to display a mark / label associated with another product of the brand. For example, as shown in advertisement 1016, the controller can cover the mark / label of other products advertised by the brand. The brand can negotiate to reach an agreement to protect all marks / labels in advertisement 1008 in a similar manner.

[0143] Alternatively, the controller may overlay a tag / label associated with a promotion, as shown in advertisement 1020. For example, a retailer housing retail product shelves 102 may overlay a tag / label associated with a promotion that is associated with another product offered by the retailer.

[0144] Fig.11 An advertising computer system 1100 supporting one or more retail product cabinets according to an embodiment is shown. Referring to Figure 1, the computer system 1100 includes an edge processor (controller) 101, retail product cabinets 102-103, and an interface with a cloud service 104. For example, according to some embodiments, components 1102-1105 and 1107 can be implemented at the edge processor 101, and components 1106 and 1108-1109 can be implemented at the retail product cabinets 102-103.

[0145] refer to Fig.11 , the advertisement manager 1107 interacts with the advertisement storage space 1104 to download the advertisement content from the cloud service 1101. The advertisement manager 1107 can request the selected advertisement content (e.g., based on the product type and corresponding to multiple advertisements) from the cloud service 1101 via the advertisement storage space 1104, and can direct one of the multiple advertisements to be displayed on the screen (display) 1109 through the screen player 1108. As previously described, the screen player 1108 can include the media player 417 and can be implemented at the retail product cabinet 102, 103 and / or the controller 101.

[0146] Advertisement manager 1107 may request to download a set of advertisements (from one advertisement to many (e.g., hundreds) of advertisements) for retail product cabinet 102. The request may be based on the type of products stored at retail product cabinet 102. For example, the request by ad manager 1107 may be based on the number of advertisements downloaded from inventory tracker 420 (e.g., Figure 4 Inventory information received by the user (as shown), or a mapping from a container identifier to a specified stored product.

[0147] Advertisement manager 1107 may be configured to monitor the door based on information from door sensor / camera 1106 (e.g., located at a Figure 2 230 as shown), sensor / camera information of the third party data module 11011 and / or the phone application support module 1103 to select one of the downloaded advertisements.

[0148] As will be discussed in further detail, the advertisement evaluator 1102 may determine the effectiveness of the advertisement presented on the screen 1109 based on information from the door sensor / camera 1106 and the phone application support module 1103. For example, the door image sensor 1106 may indicate that the customer is viewing a particular area of ​​the displayed advertisement and / or the customer may indicate effectiveness through the mobile application. Fig.16 The tiers (shown in blocks 1606 to 1610 in the example above) may be determined in a variety of ways. For example, the tiers may be mapped to different values ​​of the duration that the customer views the product in the displayed advertisement, or may be obtained directly from the customer's input via the mobile application.

[0149] Although cloud services may be used to provide additional computer resources required by the advertising computer system 1100, the computer resources may be provided by one or more computer servers, or they may be provided by the cloud service 1101. For example, the advertising manager 1107, the advertising evaluator 1102, the phone application support 1103, and / or the third-party data 11011 may all reside in the cloud 1110. In addition, the aforementioned systems may all be referred to as the cloud service 1101, either together or individually. For example, a seller may provide advertising content through the advertising storage space 1104, so that the advertising manager 1107 can download the advertising content from the cloud 1110. The advertising manager 1107 may request the selected advertising content (e.g., based on the product type and corresponding to multiple advertisements) from the cloud 1110 and display it on the screen (display) 1109 through the screen player 1108. Although Fig.11 The components in are shown as being separate, but the invention is not limited thereto; and one or more components may be merged and / or branched.

[0150] Fig.12 An edge computing device (controller) 1200 is shown that can support one or more retail product cabinets (eg, retail product cabinets 102 and 103) according to an embodiment. Controller 1200 can correspond to controller 101.

[0151] The controller 1200 may include a processing device 1201, in which the processes discussed herein may be implemented (e.g., Figures 13 to 191900). Processing device 1201 can control the overall operation of controller 1200 and its associated components. Processing device 1201 can access and execute computer-readable instructions from memory device 1204, which can employ various computer-readable media. For example, computer-readable media can be any available media that can be accessed by processing device 1201, and can include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media can include a combination of computer storage media and communication media.

[0152] In addition, processing device 1201 may download advertising content from a cloud service via cloud interface 1203, store the downloaded content at memory device 1205, access the downloaded content (e.g., the selected advertisement) from memory device 1205, and provide the advertising content to display 416 via screen interface 1206. As previously described, the advertising content may be displayed on a display device 416, such as a display device 416, or a display device 416. Figure 4 Media player 417 is shown presenting to display 412 .

[0153] As previously described, the controller 1200 may obtain advertising content from the cloud service 1101 via the cloud computing interface 1203 (via Fig.11 The controller 1200 may also include an advertisement storage space 1104 (shown) and third-party data (via a third-party data module 1105). In addition, the controller 1200 may interact with customers via wireless communications (e.g., via a phone application support module 1103), and may provide advertisement ratings (e.g., via advertisement ratings 1102) via a cloud computing interface 1203.

[0154] Computer storage media may include volatile and nonvolatile media, as well as removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computing device.

[0155] Communication media may include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. A modulated data signal may be a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct wired connection, and wireless media such as acoustic media, RF media, infrared media and other wireless media.

[0156] Processing device 1201 may also obtain image sensor data from customer detector 418 and inventory tracker 420 through interface 1202. Processing device 1201 may use this data to determine the location of customers relative to retail product case 102 and the current inventory of retail product case 102 (e.g., via customer detector 418).

[0157] The controller 1200 (executing computer readable instructions from the memory 1204) may be configured to control the display 416 (eg, Figure 4 102) to display current pricing information about products physically contained in the interior storage volume of the retail product cabinet 102, where pricing can be determined by the controller 1200. In some examples, the current pricing information and other product details can be received from an on-premises point-of-sale pricing system at the retail location. This point-of-sale system can communicate with the controller 1200 at the retail product cabinet 102 by wired or wireless means to update the content on the display device on the retail product cabinet 102. In other examples, the current pricing information and other information can be transmitted to the retail product cabinet 102 at the retail premises via a cloud-based service.

[0158] The controller 1200 may be configured to receive information from a customer detector 418 (eg, Figure 4 1206 ) and may be configured to display a promotion on the display 3112 via the screen interface 1206 based on information from the customer detector 418 .

[0159] The controller 1200 can effectively obtain inventory information for each retail product container 102 (via the inventory tracker 420 and the image recognition software). The controller 1200 can be configured to transmit data to the distributor via the cloud interface 1203 and / or store data about the overall inventory related to the inventory of multiple retail product containers. For example, the system 150 (such as Figure 1A ) can be configured to issue restock notifications, provide sales updates, pricing management, real-time sales data and behavioral consumer analytics.

[0160] The controller 1200 may be configured to direct the display 416 to display a full screen advertisement when the customer detector 418 detects that the customer is not close enough to the retail product case (e.g., Fig.14 401). For example, when a customer is approaching an aisle but at a detection distance greater than a predetermined value, the customer may see a full-screen advertisement on the display 416. As another example, when a customer is not detected within the capabilities of the customer detector 418, the controller 1200 may direct the display 416 to display a full-screen advertisement according to a default process. In one embodiment of a system according to the disclosure herein, the customer detector 418 may use motion to detect whether a customer is present. In other embodiments of a system according to the disclosure herein, the customer detector 418 may operate by detecting a human form. For example, in one embodiment, the customer detector 418 may trigger when a general adult face is detected (rather than simply when motion is detected). Such an embodiment may reduce the number of false positives—for example, a pet, a baby, or non-human movement, such as wind, may not trigger the customer detector 418. In some embodiments, the human form may be anonymized in that only the human form may be detected without any unique identifying features of the natural person. In other embodiments, the customer detector 418 may detect, capture, and / or filter other levels of detail in conjunction with one or more other components of the system disclosed herein (including but limited to the image sensor 423).

[0161] As an example, if motion or a human presence is detected within twenty feet of a retail product case 102 (where a shopper detector 418 is mounted on the door of the retail product case 102), the controller 1200 may be configured to terminate display of the full screen advertisement and transition to displaying a planogram of products located within the retail product case 102. Banner advertisements and pricing tags may also be loaded from the memory 1205 and displayed concurrently with displaying the planogram. The controller 1200 may also be configured to manage sales advertisement objects and layers.

[0162] Controller 1200 may be configured such that as a shopper (e.g., customer) approaches retail product case 102, the advertising content on display 416 may subsequently change. For example, display 416 may change to include indicia (e.g., such as Figure 8 Promotional indicia and labels may be activated via controller 1200 and may be selected from a predetermined set of static animated icons or may be customized by the retailer and / or brand based on business and marketing needs. In one embodiment according to aspects of the systems disclosed herein, controller 1200 may be configured to use one or more consumer review sources (e.g., using Fig.11The third-party data 1105 shown in the figure can be combined with a consumer rating score (e.g., 1 to 5 stars, trends, or favorites). At the same time, other embodiments of the system disclosed herein operate / function without knowing the individual shopper / customer, and the system never seeks to identify individuals, that is, it is "identity blind" because it never collects or uses personally identifiable or contactable information. In some embodiments, whether to implement a set of "identity blind" features can be determined based on whether the system is located in a country or state where the consumer expects privacy and / or is governed by applicable laws.

[0163] The controller 1200 can be configured so that when a shopper stands or stays in front of a given retail product cabinet 102, the content (e.g., graphic rendering) output at the display 416 can change again. In some embodiments, the controller 1200 can use information from the customer detector 418 to effectively learn more about the specific customer, such as gender, age, mood, etc. (Alternatively or in combination, as will be discussed, when the customer chooses to join in response to the displayed invitation, the controller 1200 can obtain customer information through a mobile application.) In such an embodiment, the controller 1200 can be configured to process the detected customer information to determine a specific or customized advertisement to be presented to the specific customer on the display 416. By tracking shopper data while tracking the advertising content presented on the displays of all retail product cabinets within the shopper's viewing range, retailers have new influence opportunities at the moment of purchase decision, thereby optimizing marketing expenses and generating new revenue streams relative to traditional methods. At the same time, other embodiments of the system disclosed herein are designed to have privacy and security features integrated into the technology platform and operating model. And, such an embodiment operates / works without knowing the individual shopper / customer, and the system never seeks to identify an individual, i.e., it never collects or uses personally identifiable or contactable information. In such an embodiment, an "identity-blind" retail consumer security platform is provided. In some embodiments, it can be determined whether to perform a set of "identity-blind" feature selections based on whether the system is located in a country or state where the consumer expects privacy and / or applicable laws. The controller 1200 can be configured so that when a shopper stands in front of a display 416 (which can be detected by an image sensor installed on the door of this particular retail product cabinet 102), relevant conditional advertisements are displayed on the display 416 based on real-time data related to news events, weather, sports events, etc.

[0164] As described above, controller 1200 may be configured to track the inventory of retail product case 102 via inventory tracker 420, which may include one or more cameras and / or sensors on the inside of retail product case 102. When controller 1200 has determined that a given product is out of stock, controller 1200 may modify the content displayed on display 416. For example, an out of stock illustration may be displayed and / or an advertisement may be swapped.

[0165] As will be discussed, processes 1300 to 1900 (respectively Figures 13 to 19 ) are some of the processes that can be performed according to aspects of one or more embodiments. Fig.12 , the processing device 1201 can execute computer-readable instructions from the memory device 1204 to perform processes 1300 to 1900.

[0166] Fig.13 A flow chart 1300 is shown for configuring the computing platform 100 (shown in FIG. 1 ) of the advertising-enabled computer system 1100 , according to an embodiment.

[0167] At block 1301, controller 1200 discovers retail product containers (e.g., containers 102 and 103) to be supported by controller 1200. (A retail product container is supported by a controller when the controller controls advertisements displayed at the retail product containers.) Embodiments may utilize a known list of retail product containers and associated devices including sensors and cameras, or may use a discovery process (e.g., for discovering IoT devices including sensors and cameras and associating them to corresponding retail product containers).

[0168] When a supported retail product cabinet is found, at block 1302, the controller 1200 determines the types of products / manufacturers stored in a given retail product cabinet (e.g., retail product cabinet 103). For example, a retail product cabinet may report a scanned universal product code of the products it stores. Alternatively or in combination, the controller 1200 may access a mapping between each supported retail product cabinet and the corresponding stored product type. At block 1303, the controller 1200 may download relevant advertisements (which may be referred to as an advertisement group) related to the stored products of the given retail product cabinet.

[0169] If there are additional supported retail product containers, as determined at block 1304, blocks 1302 through 1303 may be repeated for the next supported retail product container.

[0170] After the content related to the advertisements has been downloaded, controller 1200 may direct each retail product shelf to display one of the downloaded advertisements based on various criteria (eg, as determined by processes 1400 , 1500 , 1600 , and 1700 ).

[0171] Fig.14 A flow chart 1400 for generating advertisements on a screen of a retail product cabinet according to an embodiment is shown. Initially, a full screen advertisement is displayed on a display of a retail product cabinet at block 1401. For example, as previously described, the controller 1200 may direct the retail product cabinet 102 to display appropriate advertisements based on the products stored at the retail product cabinet 102.

[0172] When the customer detector 418 (such as Figure 4 When it is determined at block 1402 that the customer is within the predetermined range, the controller 1200 may direct the retail product cabinet 102 to display a planogram reflecting the products stored at the retail product cabinet 102 at block 803.

[0173] When the customer detector 418 determines at block 1404 that the customer is closer to the retail product case 102 (within a predetermined distance, e.g., within 12 to 15 feet), the controller 1200 may generate an opt-in invitation within the displayed planogram at block 1405. For example, the invitation may include a quick response (QR) code in the center of the display. If the customer wishes to opt-in, the customer may scan the QR code via the customer's smartphone to open a link or download a mobile application associated with the QR code. In other instances, instead of or in addition to the QR code, an image other than a QR code (e.g., an image with a machine-detectable watermark), a Bluetooth (or other short-range wireless protocol) beacon, or other mechanism may be provided to indicate the invitation to the customer. In some instances, multiple invitations may be transmitted simultaneously to the user's device to provide the customer with the option of selecting whichever mechanism is most preferred. In other examples, the indicated invitations may be transmitted serially such that a first method (e.g., a wireless beacon) may be used when the customer is within a certain distance range from the retail product cabinet 102, and a second method (e.g., a QR code) may be used when the customer is closer to the cabinet 102. Although an example of a predetermined distance of 12 to 15 feet is described in one of the foregoing examples, the present disclosure is not limited thereto, and for example, the predetermined distance may be within a range greater than 15 feet or within a range less than 12 feet depending on the mechanism used to indicate the invitation to the customer.

[0174] When the customer chooses to enter, as determined in box 1406, controller 1200 can generate personalized advertisements on the display of retail product cabinet 102 based on the customer information obtained by mobile application. (For example, when the customer accepts the invitation by scanning the QR code presented and accepting the use agreement of the mobile application, the customer chooses to join.) According to the retail consumer security platform, until the consumer chooses to enter, the system can remain "identity blind" because it never collects or uses personal identifiable or contactable information. The retail consumer security platform can be designed to have privacy and security features integrated into the technology platform and the operating model. Such an embodiment operates / works without knowing the individual shopper / customer, and the system never seeks to identify an individual. In some embodiments, it can be determined whether to perform a set of "identity blind" feature selections according to whether the system is located in a country or state governed by consumer expectation privacy and / or applicable law.

[0175] At block 1409, the controller 1200 may generate special offers via the customer's smartphone for one or more products stored at the retail product shelf 102. The process 1400 continues in this mode at block 1410 while the customer detector 418 continues to detect the presence of a customer. Otherwise, the process 1400 returns to block 1401 where a full screen advertisement is displayed.

[0176] When the customer has not opted in, as determined at block 1406 , an advertisement may be inserted into the displayed planogram based on generalized information (not specific to the customer) at block 1407 . Figure 8 Examples of advertisements 802 to 805 are shown in FIG. Fig.10 Examples of advertisements 1008 and 1020 are shown in FIG. When a customer is detected, process 1400 continues in this mode at block 1408. Otherwise, process 1400 returns to block 1401.

[0177] When process 1400 returns to block 1401 from block 1408 or 1410 , process 1400 repeats blocks 1403 to 1410 when a subsequent customer is detected by customer detector 418 at block 1402 .

[0178] Fig.15 A flowchart 1500 is shown for displaying an advertisement selected from a rule set according to an embodiment. The computing platform 100 determines the advertisement to present on the display of the cabinet based on the rule set, which may cover weather conditions, dates, and / or characteristics of customers near the cabinet. For example, the advertisement presented may be selected because the customer stood by the beverage cabinet for 20 seconds, it was 98 degrees outside, and the date was July 3 (i.e., just before the Fourth of July).

[0179] refer to Fig.15, the rule group 1505 selects the appropriate advertisement to be displayed from the advertisement group 1504 based on different factors. In one embodiment according to various aspects of the present disclosure, these factors may include customer characteristics 1501, customer viewing information 1502, and / or generalized information 1503. For example, a customer may express through a mobile application (e.g., after the customer opts in) that he / she does not like broccoli and likes ice cream. In addition, the customer detector 418 may indicate that the customer is viewing a specific area of ​​the displayed planogram where an ice cream product appears. In such a scenario, the computer platform 100 may launch an advertisement for the ice cream product. However, other embodiments of the system disclosed herein operate / function without knowing the individual shopper / customer, so that the rule group 1505 is "identity blind". Moreover, those embodiments of the system never seek to identify an individual, that is, it never collects or uses personally identifiable or contactable information. In some embodiments, whether to perform the selection of a set of "identity blind" features may be determined based on whether the system is located in a country or state where consumers expect privacy and / or applicable laws.

[0180] When the computing platform 100 has identified appropriate advertisements at block 1506 , the identified advertisements may be displayed at corresponding retail product cases at block 1507 .

[0181] Fig.16 A flow chart 1600 is shown for determining the effectiveness level of a displayed advertisement according to an embodiment. The computing platform 100 evaluates the effectiveness of an advertisement presented on a display of a retail product case. For example, the computing platform 100 may detect whether a customer is within a determined distance of the retail product case and / or whether one or more customers are viewing the presented advertisement.

[0182] refer to Fig.16 At box 1601, the computing platform 100 can obtain information (e.g., the detected distance of the customer from the cabinet, the area of ​​the display area that the customer is viewing, and / or the location where the customer is viewing).

[0183] At block 1602, the computer platform 100 determines whether the customer is within a predetermined distance (in other words, considered close enough to the retail product case). If so, at block 1604, the customer detector 418 (e.g., via an iris scanning sensor) can track the customer's eye movements and provide the eye movement information to the controller 101 to determine an effectiveness level (e.g., a success metric) based on the eye movement information.

[0184] At blocks 1605, 1606, and 1609, computing platform 100 may evaluate the information to determine a level of effectiveness at blocks 1607, 1608, 1610, and 1611. For example, a customer may gaze at display area 851 of advertisement 802 (e.g., Figure 8 The effectiveness level can be measured in a variety of ways. For example, it can be measured by the amount of time the customer views the displayed product or by the customer's direct input through the mobile application (if the customer has opted in).

[0185] However, when the computing platform 100 cannot determine the validity level, the process 1600 terminates at block 1603 .

[0186] Fig.17 A flowchart 1700 for generating product-level advertisements according to an embodiment is shown. The computing platform 100 can generate product-level advertisements on a display of a retail product case. For example, a customer detector can detect that a customer is viewing a particular product displayed in a planogram at a retail product case.

[0187] refer to Fig.17 At block 1701 , customer detector 418 may determine the display area that the detected customer is viewing. For example, the customer may be viewing lower area 851 of advertisement 802 .

[0188] At block 1702, computing platform 100 may identify one or more products stored at the detected display area. Computing platform 100 may identify the one or more products in a variety of ways. For example, computing platform 100 may access a mapping of products to different areas of a displayed planogram. Alternatively or in combination, computing platform 100 may obtain inventory information about inventory products at a retail product shelf from inventory tracker 420.

[0189] At block 1703, the computing platform 100 may display an appropriate advertisement regarding the identified product on a display of the case.

[0190] Fig.18A flowchart 1800 is shown of displaying advertisements for products stored at a second retail product container at a first retail product container according to an embodiment. Advertisements displayed at a first retail product container (e.g., retail product container 102) can promote products stored at a second retail product container (e.g., retail product container 103). In one embodiment of the system disclosed herein, the computing platform 100 can predict that a customer will shop at a second retail product container after shopping at a first retail product container based on customer information. At the same time, other embodiments of the system disclosed herein operate / function without knowing the individual shopper / customer, and the system never seeks to identify an individual, i.e., it is "identity blind" because it never collects or uses personally identifiable or contactable information. In some embodiments, it can be determined whether to perform a set of "identity blind" feature selections based on whether the system is located in a country or state where the consumer expects privacy and / or applicable laws.

[0191] refer to Fig.18 , at box 1801, in one embodiment of the system disclosed herein, the computing platform 100 can obtain customer information about customers near the first retail product cabinet - for example, the computing platform 100 can obtain customer information about the customer's likes and dislikes through a mobile application when the customer chooses to opt in. Continuing with the previous example, the customer may hate broccoli but love ice cream. At the same time, other embodiments of the system disclosed herein are designed to have privacy and security features integrated into the technology platform and operating model. Such an embodiment operates / functions without knowing the individual shopper / customer, and the system never seeks to identify individuals or their likes / dislikes, that is, it never collects or uses personally identifiable or contactable information. In such an embodiment, an "identity-blind" retail consumer security platform is provided. In some embodiments, whether to perform a set of "identity-blind" feature selections can be determined based on whether the system is located in a country or state where the consumer expects privacy and / or is governed by applicable law.

[0192] At block 1802, computing platform 100 may determine (e.g., via detection by a customer detector) whether a customer is at a first product cabinet. If the customer is at the first product cabinet, computing platform 100 may determine at block 1803 whether the customer typically goes to a second retail product cabinet that stores, for example, ice cream. Continuing with this example, at block 1804, computing platform 100 may generate an advertisement at the first retail product cabinet for an identified product (e.g., ice cream) stored at the second retail product cabinet.

[0193] Fig.19A flowchart 1900 for determining and displaying advertisements in a display associated with a retail product case is shown according to an embodiment. At block 1902, a controller (e.g., an edge computing device 101 including a processor and circuitry to control a retail product case) may receive (e.g., based on user input or from a cloud computing service) a planogram map associated with a retail product case (e.g., retail product case 102) and a description of products associated with the retail product case. The controller may store the planogram map and the description in a data storage area 111. For example, the planogram map may indicate that a top shelf of the retail product case corresponds to beverages that are bottle-shaped, red, and have a first size. At block 1904, the computing platform 100 may determine a composite image based on one or more images captured by one or more cameras in the retail product case.

[0194] At step 1906, the controller may determine whether the product is located at a particular location (e.g., a top shelf) of the retail product case. The controller may use the composite image to make this determination. If the controller determines that there is no product in the top shelf, the controller may determine (at step 1918) that the beverage is out of stock at the retail product case. At step 1920, the controller may determine a first advertisement for display based on determining that the beverage is out of stock. The first advertisement may include, for example, an illustration indicating that the beverage is out of stock. The controller may send the first advertisement to a display controller for display on a display device.

[0195] Continuing with this example, if the controller determines that the product is stored in the top shelf, the controller may determine the attributes of the stored product based on the composite image. The controller may compare the determined attributes with the description of the beverage. At step 1908, the controller may determine whether the stored product is in the shape of a bottle. At step 1910, the controller may determine whether the stored product is red. At step 1912, the controller may determine whether the stored product has a first size. The controller may perform steps 1908 to 1912 in a different order, or may omit one or more of steps 1908 to 1912. In an example, the composite image may be blurred, and the controller may perform one or more post-processing steps to reduce the effect of blur on the above comparison.

[0196] At each of steps 1908 to 1912, the controller may determine a corresponding confidence score (e.g., probability, a value between 1 and 10, where a higher value indicates a higher confidence) associated with whether the attribute of the stored product corresponds to the description of the beverage. For example, the controller may determine a confidence score associated with whether the stored product is in the shape of a bottle, a confidence score associated with whether the stored product is red, and / or a confidence score associated with whether the stored product has a first size. Based on the determined confidence scores, the controller may determine an overall single confidence score corresponding to the probability that the attribute of the stored product matches the description of the beverage. If the overall confidence score is above a threshold (e.g., 60%, 70%, or other appropriate value), the controller may determine (e.g., at step 1914) that the stored product is a beverage and that the beverage is in stock at a retail product shelf. At step 1916, the controller may determine a second advertisement for display based on determining that the beverage is in stock. The second advertisement may include, for example, an illustration associated with the beverage. The controller may send the second advertisement to the display controller for display on the display device.

[0197] Continuing with this example, if the controller determines based on steps 1908 to 1912 that the shape, color, and / or size of the stored product at the top shelf does not match the shape, color, and size of the beverage, the controller may attempt to identify the stored product. For example, the controller may scan a description of the product (e.g., which is stored in data storage area 111) to identify the stored product (e.g., Fig. 20 ). At step 1922, the controller may determine that the stored product is a second beverage. At step 1924, the controller may determine a third advertisement for display based on determining that the stored product is the second beverage. The third advertisement may include, for example, an illustration associated with the second beverage. The controller may send the second advertisement to the display controller for display on the display device.

[0198] Fig. 20 Flowchart 2000 for determining and displaying advertisements in a display associated with a retail product shelf according to an embodiment is shown. For example, if the controller determines based on steps 1908 to 1912 that the shape, color, and / or size of the stored products at the top shelf do not match the shape, color, and size of the beverage, the controller may perform Fig. 20 At step 2002, the controller may scan the descriptions of other products in the data storage area 111 to determine the shapes, colors, and sizes associated with the other products. Continuing with the above example, the data storage area 111 may include a description of a second beverage that is bottle-shaped, blue, and has a second size.

[0199] The controller may determine attributes of the stored product based on the composite image. The controller may compare the determined attributes to the description of the second beverage. At step 2004, the controller may determine whether the stored product is in the shape of a bottle. At step 2006, the controller may determine whether the stored product is blue. At step 2008, the controller may determine whether the stored product has a second size. The controller may perform steps 2004 to 2008 in a different order, or may omit one or more of steps 2004 to 2008. The controller may determine an overall confidence score corresponding to a probability that the attributes of the stored product match the description of the second beverage (e.g., as described above with reference to FIG. 20). Fig.19 ). If the overall confidence score is above a threshold (e.g., 60%, 70%, or any other value), the controller may determine that the stored product is a second beverage and that the second beverage is in stock at the retail product shelf. At step 2010, the controller may determine an advertisement for display based on determining that the second beverage is in stock. The advertisement may include, for example, an illustration associated with the second beverage. The controller may send the advertisement to a display controller for display on a display device.

[0200] Continuing with this example, if the controller determines based on steps 2004-2008 that the shape, color, and / or size of the stored products at the top shelf do not match the shape, color, and size of any product indicated in the planogram map, the controller may send a notification (e.g., to the wireless device 106 or another computing device accessible to the retailer inventory manager) at step 1916. The notification may indicate that there are unidentified products in the retail product case.

[0201] A number of illustrative embodiments are listed below according to one or more aspects disclosed herein. Although many of the embodiments listed below are described as being dependent on other embodiments, the dependence is not limited thereto. For example, embodiment #3 (below) is explicitly described as being in combination with the features of embodiment #2 (below), however, the present disclosure is not limited thereto. For example, embodiment #3 may rely on any one or more of the aforementioned embodiments (i.e., embodiment #1, embodiment #2, and / or other appropriate embodiments). In addition, the present disclosure contemplates that any one or more of embodiments #2 to #14 may be combined with embodiment #1 alone or in combination with other embodiments. Similarly, any one of embodiments #1, 15, 18, and / or 32 may be combined with one or more of the features listed in embodiments #2 to 14, 16 to 17, 19 to 31, and / or 33 to 34. In addition, the present disclosure contemplates that any one or more of the features in embodiments #1, 15, 18, and / or 32 may be combined. In addition, the present disclosure contemplates that any one or more of the features in embodiments #1 to 34 may be combined.

[0202] Embodiment #1. A smart movable enclosure system configured for installation in a retail environment requiring a cabinet adapted to be maintained at a temperature cooler than an external cabinet ambient temperature, the system comprising:

[0203] an interior storage volume of a retail product case that stores perishable items for purchase in the retail environment;

[0204] a removable door that prohibits viewing of said interior storage volume;

[0205] a display device affixed to the movable door, wherein the display device is configured to display a planogram of at least the interior storage volume, wherein the planogram includes an out-of-stock indicator;

[0206] an optical sensor mounted on the movable door to capture one or more images of the interior storage volume of the retail product case;

[0207] a controller communicatively coupled to the display device;

[0208] a data storage area including the input planogram mapping of the interior storage volume and descriptions of a plurality of products, wherein the descriptions include a shape, a color, and a size of each of the plurality of products;

[0209] one or more processors located proximate to the retail product case, and

[0210] a memory storing computer readable instructions that, when executed by the one or more processors, cause the one or more processors to:

[0211] post-processing the one or more images captured by the optical sensor into a composite image;

[0212] determining an inventory status of a product among the plurality of products in the retail product shelf based on the composite image, the input planogram map, and the descriptions of the plurality of products;

[0213] determining an advertisement to be displayed on the display device based on the determined inventory status, wherein when the inventory status of the product is out of stock, the advertisement includes the out of stock indicator; and

[0214] The advertisement is sent to the controller for display on the display device.

[0215] Embodiment #2. The intelligent movable enclosure system of embodiment #1, wherein the input planogram mapping comprises:

[0216] an indication of the product in question; and

[0217] An indication of the designated location of the product.

[0218] Embodiment #3. The intelligent movable enclosure system of Embodiment #2, wherein determining the inventory status of the product comprises determining, based on the composite image, that a product is not detected at the designated location.

[0219] Embodiment #4. The smart movable closure system of Embodiment #3, wherein the optical sensor is configured to capture multiple images during operation of the movable door, wherein the operation includes opening the movable door and closing the movable door.

[0220] Embodiment #5. The intelligent movable enclosure system of Embodiment #3, wherein the advertisement includes illustrations associated with different products.

[0221] Embodiment #6. The smart movable closure system of Embodiment #3, wherein the inventory tracker, when executed by the one or more processors, causes the one or more processors to transmit a message to a computing device indicating that the product is not in the retail product case.

[0222] Embodiment #7. The intelligent movable enclosure system of Embodiment #2, wherein determining the inventory status of the product comprises identifying the stored product in the retail product case based on the composite image, and further determining at least one selected from the following:

[0223] the location of the stored product;

[0224] the shape of the stored product;

[0225] the color of the stored product;

[0226] The sizes of the stored products; and combinations thereof.

[0227] Embodiment #8. The intelligent movable enclosure system of Embodiment #7, wherein determining the inventory status of the product comprises determining the presence of the product in the retail product case based on at least one selected from the group consisting of:

[0228] determining that the location of the stored product corresponds to the designated location;

[0229] determining that the shape of the stored product corresponds to a shape of the product in the description;

[0230] determining that the color of the stored product corresponds to a color of the product in the description; and

[0231] Determining that the size of the stored product corresponds to the size of the product in the description; and combinations thereof.

[0232] Embodiment #9. The intelligent movable enclosure system of Embodiment #2, wherein determining the inventory status of the product comprises determining that another product is present at the designated location in the retail product case based on at least one of:

[0233] determining that a shape of the other product does not correspond to a shape of the product in the description;

[0234] determining that a color of the other product does not correspond to a color of the product in the description;

[0235] Determining that a size of the other product does not correspond to a size of the product in the description; and combinations thereof.

[0236] Embodiment #10. The intelligent movable closure system of Embodiment #7, wherein the advertisement includes an illustration associated with the product.

[0237] Embodiment #11. The intelligent movable enclosure system of Embodiment #1, wherein determining the inventory status of the product comprises determining the quantity of the product in the retail product case.

[0238] Embodiment #12. The intelligent movable enclosure system of Embodiment #1, wherein the advertisement comprises a planogram comprising illustrations associated with the plurality of products.

[0239] Embodiment #13. The intelligent movable enclosure system of Embodiment #1, further comprising receiving the input planogram mapping from a computing device.

[0240] Embodiment #14. The intelligent movable closure system of Embodiment #1, wherein the data storage area is located in the retail environment, and wherein the one or more images are captured based on the operation of the movable door.

[0241] Embodiment #15. A smart movable enclosure system configured for installation in a retail environment requiring a cabinet adapted to be maintained at a temperature cooler than an external cabinet ambient temperature, the system comprising:

[0242] a customer's wireless communication device having the application installed thereon;

[0243] A first retail product container, the first retail product container comprising:

[0244] Internal storage capacity;

[0245] a removable door that prohibits viewing of said interior storage volume;

[0246] a first display affixed to the movable door, wherein the first display is configured to display a planogram of at least the interior storage volume; and

[0247] a customer detector configured to detect the customer; and

[0248] A controller, the controller comprising:

[0249] one or more processors; and

[0250] a memory storing controller instructions that, when executed by the one or more processors, cause the controller to:

[0251] detecting, by the customer detector, a human form of the customer within a predetermined distance of the first retail product case; and

[0252] In response to the detecting, interacting with the customer via the application executing on the wireless communication device of the customer.

[0253] Embodiment #16. The intelligent movable enclosure system of Embodiment #15, wherein the human form of the customer comprises a human face, and wherein the controller instructions, when executed by the one or more processors, further cause the controller to:

[0254] responsive to said detecting, generating a text message within a first graphical rendering already output on said first display of said first retail product case;

[0255] receiving a notification when the customer responds to the text message;

[0256] in response to said receiving, interacting with said customer via said application executing on said wireless communication device of said customer;

[0257] When the customer responds, receiving customer information via the application; and

[0258] A second graphical rendering is customized on the first display of the first retail product case based on the customer information.

[0259] Embodiment #17. The intelligent movable enclosure system of Embodiment #15, wherein the controller instructions, when executed by the one or more processors, further cause the controller to:

[0260] Predicting a second retail product shelf that the customer will browse based on the customer information; and

[0261] A third graphical rendering is displayed on the first display of the first retail product case for products stored at the second retail product case.

[0262] Embodiment #18. A computer platform with a cooling system, comprising:

[0263] A first retail product container, the first retail product container comprising:

[0264] Internal storage capacity;

[0265] a removable door blocking view of said interior storage volume;

[0266] a first display affixed to the movable door, wherein the first display is configured to display a planogram of at least the interior storage volume; and

[0267] Customer detectors; and

[0268] A controller, the controller comprising:

[0269] one or more processors; and

[0270] a memory storing controller instructions that, when executed by the one or more processors, cause the controller to:

[0271] establishing communication with the first retail product container via a communication channel;

[0272] Instructing the first retail product container to display a full-screen advertisement;

[0273] receiving an indication from the first retail product case that a customer is detected by the customer detector within a first predetermined distance from the first retail product case; and

[0274] The first retail product container is directed to terminate the full screen advertisement and display a planogram of products located in the interior storage volume upon receiving the instruction.

[0275] Embodiment #19. The computer platform of Embodiment #18, wherein the first display comprises a non-transparent display.

[0276] Embodiment #20. A computer platform according to embodiment #18,

[0277] Wherein the first retail product container is configured as:

[0278] receiving, via the customer detector, first sensor information regarding the customer proximate the first retail product case, wherein the first sensor information includes eye movement information regarding eye movements of the customer; and

[0279] sending the first sensor information about the customer to the controller;

[0280] and wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0281] determining customer information about the customer from the received first sensor information about the customer;

[0282] selecting an advertisement suitable for the customer based on the customer information;

[0283] directing the first retail product shelf to display a selected advertisement on the first display based on the customer information; and

[0284] A success metric for the selected advertisement is determined based on the eye movement information.

[0285] Embodiment #21. The computer platform of Embodiment #18, wherein the first retail product cabinet further comprises a sensor configured to detect movement of the movable door from the first position to the second position, and wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0286] The first retail product bin is directed to terminate the display of the planogram on the first display.

[0287] Embodiment #22. A computer platform according to embodiment #20,

[0288] Wherein the first retail product container is configured as:

[0289] determining, via the customer detector, a number of customers who viewed the selected advertisement; and

[0290] sending information to the controller regarding the number of the customers who viewed the selected advertisement; and

[0291] Wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0292] receiving said information from said first retail product case regarding a number of said customers who viewed said selected advertisement; and

[0293] An effectiveness level of the selected advertisement is determined based on the number of customers.

[0294] Embodiment #23. A computer platform according to embodiment #20,

[0295] Wherein the first retail product container is configured as:

[0296] receiving, via the customer detector, second sensor information regarding an area of ​​the planogram proximate the first retail product case that the customer is viewing; and

[0297] sending the first sensor information about the customer to the controller; and

[0298] Wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0299] receiving the second sensor information about the area of ​​the planogram from the first retail product case; and

[0300] identifying a product in the planogram that the customer is viewing based on the area of ​​the planogram; and

[0301] wherein selecting the advertisement appropriate for the customer is further based on a product identified in the planogram that the customer is viewing; and

[0302] Wherein directing the first retail product container to display the selected advertisement on the first display is further based on the identified product.

[0303] Embodiment #24. A computer platform according to embodiment #20,

[0304] Wherein the first retail product container is configured as:

[0305] receiving, via the customer detector, real-time data indicating that the customer is in front of the first display; and

[0306] providing the real-time data to the controller;

[0307] and wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0308] receiving the real-time data from the first retail product bin;

[0309] determining conditional advertising for the customer based on the real-time data; and

[0310] The first retail product shelf is directed to display the conditional advertisement on the first display based on the real-time data.

[0311] Embodiment #25. The computer platform of Embodiment #24, wherein determining the conditional advertisement is further based on a rule set accessed by the controller, and wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0312] A duration of time the customer was within a second predetermined distance and weather conditions proximate the first retail product case are determined from the real-time data.

[0313] Embodiment #26. The computer platform of Embodiment #20 further comprising a second retail product cabinet,

[0314] Wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0315] determining a first product type stored in the first retail product container and a second product type stored in the second retail product container, wherein the first product type is different from the second product type;

[0316] downloading a first set of files associated with the first product type and a second set of files associated with the second product type, wherein the first set of files includes a first plurality of files, wherein the second set of files includes a second plurality of files, and wherein each file in the first set of files or the second set of files corresponds to an associated advertisement;

[0317] launching a first advertisement from the first set of files on the first display at the first retail product case; and

[0318] A second advertisement is launched from the second set of files on a second display at the second retail product case, wherein the second advertisement is different from the first advertisement.

[0319] Embodiment #27. The computer platform of Embodiment #26, wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0320] determining the first product type based on the first identification information of the first retail product container; and

[0321] The second product type is determined based on the second identification information of the second retail product bin.

[0322] Embodiment #28. A computer platform according to Embodiment #26, wherein determining the first product type is further based on first inventory information obtained from the first retail product container, and wherein determining the second product type is further based on second inventory information obtained from the second retail product container.

[0323] Embodiment #29. The computer platform of Embodiment #20 further comprising a second retail product cabinet,

[0324] Wherein the second retail product container is further configured to:

[0325] receiving, via a customer detector of the second retail product case, an indication that the customer is within a third predetermined distance; and

[0326] sending the indication that the customer is within the third predetermined distance to the controller;

[0327] and wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0328] receiving, from the second retail product cabinet, the indication that the customer is within the third predetermined distance;

[0329] generating an opt-in invitation indication on the first display of the first retail product case;

[0330] directing the first retail product container to display the invitation to opt-in on the first display of the first retail product container; and

[0331] receiving, by the customer scanning the invitation indication via a communication device, a notification from the first retail product case that the customer has opted in;

[0332] and wherein the first retail product cabinet is further configured to:

[0333] displaying the invitation to opt-in on the first display of the first retail product cabinet;

[0334] receiving, by the customer scanning the invitation indication via the communication device, an indication that the customer has opted in;

[0335] sending, by the customer scanning the invitation indication via the communication device, a notification to the controller that the customer has opted in; and

[0336] Interacting with the customer via an application executing on the communication device.

[0337] Embodiment #30. The computer platform of Embodiment #29, wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0338] when the customer opts in, as indicated by the notification, customizing the advertisement displayed on the first display of the first retail product case for the customer based on the customer information obtained through the application;

[0339] The customized displayed advertisement includes customized pricing for the customer.

[0340] Embodiment #31. The computer platform of Embodiment #29, wherein the controller instructions, when executed by the one or more processors, cause the controller to:

[0341] predicting a third retail product shelf that the customer will browse;

[0342] customizing displayed advertisements for products stored at the third retail product case; and

[0343] When the customer does not opt-in in response to the invitation indication, continuing to generate a planogram based on the generalized information.

[0344] Embodiment #32. A method for generating advertising content on a display screen, the display screen being affixed to a removable door, the removable door being attached to a retail product case, the method comprising:

[0345] displaying a first advertisement on the display screen of the first retail product case;

[0346] detecting a customer's human form within a predetermined distance of the first retail product cabinet; and

[0347] In response to the detecting, directing the first retail product case to terminate the first advertisement and display a planogram of products located in the interior storage volume of the first retail product case on the display screen,

[0348] Wherein the products located in the interior storage volume of the first retail product container are blocked from view by the movable door.

[0349] Embodiment #33. The method of embodiment #32, wherein the human form of the customer includes a human face, the method further comprising:

[0350] responsive to said detecting, generating an opt-in invitation indication on said display of said first retail product case;

[0351] When the customer chooses to enter, receiving a notification through the communication interface of the first retail product cabinet;

[0352] In response to the receiving, interacting with the customer via an application executing on the customer's communication device; and

[0353] A second advertisement is customized on the display of the first retail product case based on preferences corresponding to the customer.

[0354] Embodiment #34. The method according to embodiment #33 further includes:

[0355] predicting that the customer will browse a second retail product shelf; and

[0356] A third advertisement is displayed on the display of the first retail product case for products stored in the second retail product case.

[0357] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon is disclosed. One or more aspects of the present disclosure may be embodied in computer-usable data or computer-executable instructions, such as in one or more program modules stored on a computer-readable medium, which are executed by one or more computers or other devices to perform the operations described herein. Typically, program modules include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types when executed by one or more processors in a computer or other data processing device. Computer-executable instructions may be stored as computer-readable instructions on computer-readable media such as hard disks, optical disks, removable storage media, solid-state memory, RAM, etc. In various embodiments, the functions of program modules may be combined or distributed as needed. In addition, the functions may be embodied in firmware or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Specific data structures may be used to more effectively implement one or more aspects of the present disclosure, and it is expected that such data structures are within the scope of computer-executable instructions and computer-usable data described herein.

[0358] Various aspects described herein may be embodied as one or more computer-readable media storing computer executable instructions, methods, devices, or computer-readable instructions. Therefore, these aspects may take the form of a complete hardware embodiment, a complete software embodiment, a complete firmware embodiment, or an embodiment combining software, hardware, and firmware aspects in any combination. In addition, various signals representing data or events described herein may be transmitted between a source and a destination in the form of light waves or electromagnetic waves passing through a signal conducting medium such as a metal wire, an optical fiber, and / or a wireless transmission medium (e.g., air and / or space). Typically, one or more computer-readable media may be and / or include one or more non-transitory computer-readable media.

[0359] As described herein, various methods and actions may operate across one or more computing servers and one or more networks. Functionality may be distributed in any manner, or may be located in a single computing device (e.g., a server, a client computer, etc.). For example, in an alternative embodiment, one or more of the computing systems discussed above may be combined into a single computing system, and the various functions of each computing system may be performed by the single computing system. In such an arrangement, any and / or all of the above communications between computing systems may correspond to data accessed, moved, modified, updated, and / or otherwise used by a single computing system.

[0360] Various aspects of the present disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, and variations within the scope and spirit of the appended claims will occur to those of ordinary skill in the art from a review of the present disclosure. For example, one or more of the steps depicted in the illustrative figures may be performed in an order other than the order listed, and one or more of the depicted steps may be optional according to various aspects of the present invention.

Claims

1. A system for inventory management, marketing and advertising on a removable non-transparent display on a surface of a retail product case, comprising: A server, the server comprising: one or more server processors; and a server memory storing computer executable instructions that, when executed by the one or more server processors, cause the server to: identifying inventory of one or more retail products physically contained in a first retail product container of a plurality of retail product containers via image recognition using inventory counting hardware attached to the first retail product container, wherein the inventory counting hardware includes at least one camera and the inventory counting hardware is configured to: triggering the at least one camera from time to time based on motion sensed by the at least one camera; and receiving a trigger from one or more accelerometers attached to the first retail product case based on a swing state and position of a door of the first retail product case; directing the first retail product container to display a full screen advertisement on at least one non-transparent display on a surface of the first retail product container, wherein the at least one non-transparent display blocks viewing of the one or more retail products physically contained in the first retail product container from outside of the first retail product container; determining, via customer detection hardware at the first retail product case, that a customer is within a predetermined distance from the first retail product case; After the determining, generating a planogram of the one or more retail products physically contained in the first retail product case based on the inventory identified by the inventory counting hardware; and The first retail product case is directed to terminate the full screen advertisement and display the planogram on the at least one non-transparent display.

2. The system according to claim 1, further comprising: Inventory management systems, and Wherein, the computer executable instructions, when executed by the one or more server processors, further cause the server to: determining that a particular product is out of stock based on the inventory of the one or more retail products physically contained in the first retail product container; sending a signal to a content management system to update the planogram with an indication that the particular product is out of stock; directing the first retail product case to display an updated planogram on the non-transparent display of the first retail product case having an indication that the particular product is out of stock; and Based on determining that the particular product is out of stock, the inventory management system is notified of a need to replenish the particular product in the first retail product bin.

3. The system of claim 1, wherein the at least one non-transparent display on the surface of the first retail product case is movable from an open position to a closed position, and wherein, When in the closed position, the one or more retail products physically contained in the first retail product case are blocked from viewing from outside the first retail product case.

4. The system of claim 1 , wherein the computer executable instructions, when executed by the one or more server processors, cause the server to: receiving, via customer detection hardware at the first retail product case, first sensor data corresponding to a customer; and The first retail product case is directed to display a promotion for a retail product of the one or more retail products for a customer on the at least one non-transparent display on a surface of the first retail product case.

5. The system of claim 4, wherein the first sensor data comprises eye movement information of the customer, wherein: The computer executable instructions, when executed by the one or more server processors, further cause the server to: A success metric for the promotion is determined based on the eye movement information.

6. The system of claim 4, wherein the promotion comprises a planogram including artwork associated with the one or more retail products physically contained in the first retail product case.

7. The system of claim 1, wherein the computer executable instructions, when executed by the one or more server processors, further cause the server to: determining a shortage of at least one of the one or more retail products based on an inventory of the one or more retail products physically contained in the first of the plurality of retail product containers; and Based on the shortage, a replenishment order or direct store delivery (DSD) reminder is generated for at least one of the one or more retail products.

8. The system of claim 1, wherein the at least one camera is housed within an empty frame space between the non-transparent display on the first retail product case and a mechanical frame of the door.

9. The system of claim 1, wherein each of the plurality of retail product containers is communicatively linked to the server via a network, and wherein each retail product container comprises: an interior storage volume for storing the one or more retail products; at least one non-transparent display on a surface of each retail product case, said at least one non-transparent display prohibiting viewing of said interior storage volume of said retail product case; Customer testing hardware; and Inventory counting hardware, wherein the inventory counting hardware includes one or more cameras and is configured to collectively measure the planogram and learn to transfer the planogram.

10. The system of claim 1, wherein the computer executable instructions, when executed by the one or more server processors, further cause the server to: Prior to instructing the first retail product case to terminate the full screen advertisement, a determination is made that a movable non-transparent display on a surface of the retail product case moves from a first position to a second position.

11. A method for generating advertising content on a display screen, the display screen being affixed to a removable door, the removable door being attached to a retail product case including an interior storage volume, the method comprising: identifying inventory of one or more retail products physically contained in a first retail product container of a plurality of retail product containers via image recognition using inventory counting hardware attached to the first retail product container, wherein the inventory counting hardware includes at least one camera and the inventory counting hardware is configured to: triggering the at least one camera from time to time based on motion sensed by the at least one camera; receiving a trigger from one or more accelerometers attached to the first retail product case based on a swing state and position of a door of the first retail product case; directing the first retail product container to display a full screen advertisement on at least one non-transparent display on a surface of the first retail product container, wherein the at least one non-transparent display blocks viewing of the one or more retail products physically contained in the first retail product container from outside of the first retail product container; determining, via customer detection hardware at the first retail product case, that a customer is within a predetermined distance from the first retail product case; After said determining, generating a planogram of said one or more retail products physically contained in said first retail product case based on the inventory identified by said inventory counting hardware; as well as The first retail product case is directed to terminate the full screen advertisement and display the planogram on the at least one non-transparent display.

12. The method according to claim 11, further comprising: determining that a particular product is out of stock based on the inventory of the one or more retail products physically contained in the first retail product container; sending a signal to a content management system to update the planogram with an indication that the particular product is out of stock; directing the first retail product case to display an updated planogram on the non-transparent display of the first retail product case having an indication that the particular product is out of stock; and Based on determining that the particular product is out of stock, an inventory management system is notified of a need to replenish the particular product in the first retail product bin.

13. The method of claim 11, wherein the at least one non-transparent display on the surface of the first retail product case is movable from an open position to a closed position, and wherein, When in the closed position, the one or more retail products physically contained in the first retail product case are blocked from viewing from outside the first retail product case.

14. The method according to claim 11, further comprising: receiving, via the customer detection hardware at the first retail product case, first sensor data corresponding to a customer; as well as The first retail product case is directed to display a promotion for a retail product of the one or more retail products for a customer on the at least one non-transparent display on a surface of the first retail product case.

15. The method of claim 14, wherein the first sensor data comprises eye movement information of the customer, the method further comprising: A success metric for the promotion is determined based on the eye movement information.

16. The method of claim 14, wherein the promotion comprises a planogram including artwork associated with the one or more retail products physically contained in the first retail product case.

17. The method according to claim 11, further comprising: determining a shortage of at least one of the one or more retail products based on an inventory of the one or more retail products physically contained in the first retail product case in the plurality of retail product cases; as well as Based on the shortage, a replenishment order or direct store delivery (DSD) reminder is generated for at least one of the one or more retail products.

18. The method of claim 11, wherein the at least one camera is housed within an empty frame space between the non-transparent display on the first retail product case and a mechanical frame of the door.

19. The method of claim 11, wherein each retail product case comprises: an interior storage volume for storing the one or more retail products; at least one non-transparent display on a surface of each retail product case, said at least one non-transparent display prohibiting viewing of said interior storage volume of said retail product case; Customer testing hardware; and Inventory counting hardware, wherein the inventory counting hardware includes one or more cameras and is configured to collectively measure the planogram and learn to transfer the planogram.

20. The method of claim 11, wherein directing the first retail product container to terminate the full screen advertisement comprises: Prior to instructing the first retail product case to terminate the full screen advertisement, it is determined that the non-transparent display on the surface of the retail product case moves from a first position to a second position.

Images