Shopping cart with on-board computing system that collects contextual data and displays commodity-related information

Abstract

A shopping cart system detects initiation of a customer's shopping session within a physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors. The system tracks contextual information associated with the shopping cart received by the sensor during the shopping session, the contextual information describing one or more locations of the shopping cart within the store, a status of the shopping cart, and a set of items within the shopping cart. In response to identifying an opportunity for presenting content to the customer based on the contextual information, the system identifies a set of content items associated with one or more items within the store based on the contextual information. The system generates a user interface including a set of content items, and sends the user interface to a display area associated with the customer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. utility patent application serial number 17 / 824,696, filed on May 25, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] The present disclosure relates generally to displaying information related to merchandise in a physical store, and more particularly, to displaying information related to merchandise in a physical retail store based on contextual information associated with a shopping cart system.

[0004] Physical retail stores often offer a wide variety of merchandise to their customers because different customers may be interested in different merchandise for their own specific reasons (e.g., personal preference, quality, price, material, sustainability, etc.). For example, a physical grocery store customer who is on a limited budget and is a vegetarian may be interested in the store's brand of meat-free products, while another customer who enjoys grilling and often splurges on food is unlikely to be interested in the same products. To encourage customers to purchase new items that they may not be familiar with and to purchase more of items that they are already familiar with, physical retail stores may provide incentives for customers to purchase different items (e.g., by periodically offering promotions or discounts on those items). For example, a physical retail store may offer promotions or discounts on different items each week and may rotate samples or promotional displays in the store that showcase different featured items.

[0005] However, despite the incentives provided by physical retail stores, customers who might have purchased certain goods from the store may still not make the purchase for a variety of reasons. In some cases, customers may give up purchasing because of a lack of information about the goods. For example, if a physical grocery store has an exotic fruit, customers who would like the fruit but know nothing about it except for the basic information provided by the grocery store (such as its name and price) are unlikely to buy the fruit without additional information about the fruit (e.g., how it tastes, how to prepare it, nutritional content, etc.). In addition, in the example described above, even if most customers who encounter the fruit have client devices (e.g., smart phones) that can be used to query the fruit, customers may find it too cumbersome to do so, especially if they originally did not intend to buy the fruit. Customers may also not buy goods they would have bought because they do not know the existence of certain goods. For example, if a customer who likes cookies and is on a diet does not know that their favorite cookies now have a sugar-free version, then as long as they do not know the existence of the sugar-free version, they cannot buy this new version of the cookies. In addition, customers may also not buy certain goods because they forget to buy them during a shopping session. For example, without a reminder, a customer buying pasta might forget to buy the pasta sauce that goes with it. Summary of the invention

[0006] Physical retail stores often offer a wide variety of merchandise to customers because different customers may be interested in different merchandise for their own specific reasons. In order to encourage customers to purchase new merchandise that they may not be familiar with, and to purchase more merchandise that they are already familiar with, physical retail stores may offer incentives (e.g., promotions or discounts) for customers to purchase different merchandise. However, despite these incentives, customers who might otherwise purchase certain merchandise from physical retail stores may still fail to purchase them for a variety of reasons (e.g., if information about the merchandise is not readily available, if they are unaware of the merchandise's existence, or if they simply forget to purchase the merchandise).

[0007] According to one or more aspects of the present disclosure, in order to encourage customers to buy new products that they may not be familiar with and to buy more products that they are already familiar with, a physical retail store displays information related to in-store products based on context information associated with a shopping cart system. More specifically, a shopping cart system detects the initiation of a shopping session of a customer in a physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a sensor set. The shopping cart system tracks context information associated with the shopping cart received by the sensor during the shopping session. The context information tracked by the shopping cart system describes one or more locations of the shopping cart in the physical retail store, the state of the shopping cart, and the set of products in the shopping cart, wherein the state of the shopping cart indicates whether the shopping cart is moving or stationary. The shopping cart system identifies opportunities for presenting content to customers based in part on the context information. The shopping cart system then identifies a set of content items associated with one or more products in the physical retail store based on the context information, generates a user interface including the content items, and sends the user interface to a display area associated with the customer. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of a system environment in which a shopping cart system and an online system (such as an online concierge system) operate according to one or more embodiments.

[0009] Figure 2 is an environmental illustration of a shopping cart system and an online concierge system according to one or more embodiments.

[0010] Figure 3 is a schematic diagram of a shopping cart system and an online concierge system according to one or more embodiments.

[0011] Figure 4 is a flow chart of a method for displaying information related to items in a physical retail store based on contextual information associated with a shopping cart system according to one or more embodiments.

[0012] Figure 5 is an illustration of a floor plan associated with a physical retail store according to one or more embodiments.

[0013] Figure 6 is an illustration of content items related to one or more items within a physical retail store according to one or more embodiments.

[0014] The drawings present embodiments of the present disclosure for illustrative purposes only. Alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles or interests of the present disclosure described herein. DETAILED DESCRIPTION

[0015] System Structure

[0016] Figure 1 is a block diagram of a system environment 100, wherein a shopping cart system 150 and an online system (such as the one described below in conjunction with Figure 2 and Figure 3 The online concierge system 102) further described operates in this environment. Figure 1 The system environment 100 shown includes one or more client devices 110, a network 120, one or more third-party systems 130, one or more physical retail stores 104, and an online concierge system 102. In alternative configurations, the system environment 100 may include different and / or additional components. Additionally, in other embodiments, the online concierge system 102 may be replaced by an online system configured to retrieve content for display to a user and transmit the content to one or more client devices 110 for display.

[0017] The client device 110 is one or more computing devices capable of receiving user input and transmitting and / or receiving data via the network 120. In one embodiment, the client device 110 is a computer system, such as a desktop computer or a laptop computer. Alternatively, the client device 110 may also be a device with computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smart phone, or other suitable device. The client device 110 is configured to communicate via the network 120. In one embodiment, the client device 110 executes an application that allows a user of the device to interact with the online concierge system 102. For example, the client device 110 executes a customer mobile application 206 or a shopper mobile application 212 (described below in conjunction with Figure 2 102) to enable interaction between the client device 110 and the online concierge system 102. As an additional example, the client device 110 executes a browser application to enable interaction between the client device 110 and the online concierge system 102 via the network 120. In another embodiment, the client device 110 executes a browser application running on a native operating system of the client device 110 (such as or Android TM ) to interact with the online concierge system 102 through an application programming interface (API) on the .

[0018] The client device 110 includes one or more processors 112 configured to control the operation of the client device 110 by performing various functions. In various embodiments, the client device 110 includes a memory 114, which includes a non-transitory storage medium on which instructions are encoded. The memory 114 has instructions encoded thereon, which, when executed by the processor 112, cause the processor 112 to perform functions to execute the customer mobile application 206 or the shopper mobile application 212 to provide the following in conjunction with Figure 2 Further description of the functionality.

[0019] The client device 110 is configured to communicate via a network 120, which may include any combination of a local area network and / or a wide area network, and use wired and / or wireless communication systems. In one embodiment, the network 120 uses standard communication technologies and / or protocols. For example, the network 120 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, 5G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of network protocols for communicating via the network 120 include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged via the network 120 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or part of the communication links of the network 120 may be encrypted using any suitable technology or techniques.

[0020] One or more third-party systems 130 may be coupled to the network 120 for communicating with the online concierge system 102, the client device(s) 110, or the physical retail store(s) 104. In one embodiment, the third-party system 130 is an application provider that communicates information describing an application for execution by the client device 110, or communicates data to the client device 110 for use by an application executing on the client device 110. In other embodiments, the third-party system 130 provides content or other information for presentation via the client device 110. For example, the third-party system 130 stores one or more web pages and transmits these web pages to the client device 110 or the online concierge system 102. The third-party system 130 may also communicate information to the online concierge system 102, such as advertisements, content, or related information about the application provided by the third-party system 130.

[0021] The online concierge system 102 includes one or more processors 142 configured to control the operation of the online concierge system 102 by performing various functions. In various embodiments, the online concierge system 102 includes a memory 144, which includes a non-transitory storage medium on which instructions are encoded. The memory 144 may have encoded thereon, in conjunction with the following Figure 3 The instructions corresponding to the modules further described, when these instructions are executed by the processor 142, cause the processor 142 to execute the following combined Figure 2 as well as Figures 4 to 6 For example, the memory 144 has instructions encoded thereon that, when executed by the processor 142, cause the processor 142 to display information related to items in the physical retail store 104 based on contextual information associated with the shopping cart system 150. Additionally, the online concierge system 102 includes a communication interface that is configured to connect the online concierge system 102 to one or more networks (such as the network 120) or otherwise communicate with a device (such as the client device 110) connected to the network(s).

[0022] One or more physical retail stores 104 may be coupled to the network 120 to communicate various types of information with the online concierge system 102, the client device(s) 110, or the third-party system(s) 130. In some embodiments, a physical retail store 104 (e.g., a physical grocery store) communicates a floor plan associated with the physical retail store 104 that indicates the location of each item (e.g., each retail product) within the physical retail store 104 and the layout of the physical retail store 104. For example, the floor plan associated with the physical retail store 104 may indicate the location of the items on shelves, display cases, aisles, or any other suitable organizational elements within the physical retail store 104 for organizing the items. In this example, the floor plan may also include the layout of the physical retail store 104, which describes the location of these organizational elements relative to each other and relative to architectural elements of the physical retail store 104 (e.g., walls, doors, stairs, elevators, etc.). In some embodiments, the physical retail store 104 may also communicate updates to the floor plan of the physical retail store 104 to the online concierge system 102, the client device(s) 110, and / or the third-party system(s) 130 (e.g., periodically, when the location of items within the physical retail store 104 or the layout of the physical retail store 104 is updated, etc.). The floor plan associated with the physical retail store 104 may be used to identify one or more items within a threshold distance of a shopping cart 156 within the physical retail store 104 and / or to detect events within the physical retail store 104, as further described below.

[0023] The physical retail store 104 may include one or more shopping cart systems 150 that are capable of collecting information and transmitting and / or receiving data via the network 120. Each shopping cart system 150 within the physical retail store 104 may include one or more processors 152 that are configured to control the operation of the shopping cart system 150 by performing various functions. In various embodiments, each shopping cart system 150 also includes a memory 154 that includes a non-transitory storage medium on which instructions are encoded. The memory 154 may have a computer program that is encoded thereon and that is used in conjunction with the following instructions. Figure 3 The instructions corresponding to some modules further described, when these instructions are executed by the processor 152, cause the processor 152 to execute the following combined Figure 2 as well as Figures 4 to 6 Further description of the functionality.

[0024] Each shopping cart system 150 within a physical retail store 104 may also include a shopping cart 156 that customers of the physical retail store 104 or users of the online concierge system 102 may use to carry the items they have collected during shopping. In some embodiments, the shopping cart 156 is a wheeled cart made of plastic, metal, or any other suitable material or combination of materials. In various embodiments, the shopping cart system 150 may also alternatively include a shopping basket, an electric walker, a handcart, a multi-purpose cart, or any other object that can be used to carry items during shopping.

[0025] Each shopping cart system 150 within the physical retail store 104 may also include various sensors 158 for collecting contextual information related to the shopping carts 156 included in the shopping cart system 150. The contextual information related to the shopping carts 156 may describe one or more locations of the shopping carts 156 within the physical retail store 104, the state of the shopping carts 156, and / or the collection of items within the shopping carts 156. The state of the shopping carts 156 may indicate whether the shopping carts 156 are moving or stationary. In some embodiments, the state of the shopping carts 156 may also or alternatively indicate whether the speed of movement of the shopping carts 156 is less than a threshold speed, whether the shopping carts 156 have moved a distance less than a threshold within a specified time, whether the shopping carts 156 have changed direction, the speed of the shopping carts 156, the direction of the shopping carts 156, or any other suitable information describing the state of the shopping carts 156. In some embodiments, the sensors 158 may be mounted on the shopping carts 156 (e.g., along the perimeter of the shopping carts 156). In embodiments where the shopping cart system 150 includes objects other than the shopping cart 156 (e.g., a shopping basket, a powered walker, etc.), the sensors 158 may collect contextual information related to the object and may be mounted on the object (e.g., along the perimeter of the shopping basket).

[0026] In some embodiments, one or more sensors 158 included in the shopping cart system 150 may be capable of identifying physical objects within the physical retail store 104 (e.g., via machine vision, object recognition sensors 158, etc.). For example, the sensors 158 mounted on the shopping cart 156 may include one or more cameras facing the interior of the shopping cart 156 (e.g., video cameras or digital cameras capable of capturing still images) that are capable of identifying items (e.g., retail products) added to the shopping cart 156. In the above example, the sensors 158 mounted on the shopping cart 156 may also include one or more cameras facing the exterior of the shopping cart 156 that are capable of identifying items on shelves, display cases, aisles, or any other suitable organizational elements within the physical retail store 104 for organizing items. Additionally, in the above example, the cameras facing the exterior of the shopping cart 156 may also be capable of identifying organizational elements and architectural elements (e.g., walls, doors, stairs, elevators, etc.) of the physical retail store 104. In some embodiments, the sensors 158 have various functions to help identify items, organizational elements, and / or architectural elements within the physical retail store 104. For example, if the sensors 158 of the shopping cart system 150 include one or more cameras, these cameras may have features such as facial recognition, text recognition, infrared detection, night vision, motion activation, etc.

[0027] One or more sensors 158 included in the shopping cart system 150 may have additional functionality for collecting contextual information associated with a shopping cart 156 (or other object) included in the shopping cart system 150. In some embodiments, the sensors 158 may be capable of collecting the following contextual information associated with the shopping cart 156 (or other object): location information (e.g., GPS coordinates), motion, proximity / distance (e.g., distance relative to merchandise, organizational elements for organizing merchandise, and / or architectural elements of the physical retail store 104), various wavelengths (e.g., visible light, infrared, etc.), color, sound, speed, weight, vibration, etc. For example, a GPS sensor 158 and a proximity sensor 158 (e.g., a laser or ultrasonic proximity sensor 158) mounted on the shopping cart 156 may collect information describing the location of the shopping cart 156 with respect to merchandise, aisles, service counters, etc. within the physical retail store 104. In this example, information describing the location of the shopping cart 156 may include GPS coordinates associated with the shopping cart 156, as well as the distance (e.g., in feet or meters), direction (e.g., north, south, east, or west), and altitude (e.g., in feet or meters) of each item, aisle, service counter, etc. relative to one or more proximity sensors 158 mounted on the shopping cart 156.

[0028] Within the physical retail store 104, the contextual information collected by the one or more sensors 158 of each shopping cart system 150 may be associated with various types of data. Examples of data associated with the information include the name of the physical retail store 104, the geographic location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, etc. For example, if the physical retail store 104 is a grocery store that belongs to a chain of grocery stores, then the contextual information collected by the sensors 158 included in the shopping cart system 150 of the physical retail store 104 may be associated with data including the name identifying the chain and the geographic location (e.g., city and state) identifying the particular physical retail store 104. In the example described above, the contextual information may also be associated with a timestamp indicating the time of collection.

[0029] In some embodiments, the shopping cart system 150 may include a display area 159 (e.g., a screen mounted to a shopping cart 156 included in the shopping cart system 150). For example, the display area 159 of the shopping cart system 150 may correspond to a touch screen display mounted on the shopping cart 156 of the shopping cart system 150. The display area 159 may be a liquid crystal display (LCD), an in-plane switching liquid crystal display (IPS-LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or any other suitable display type. The display area 159 of the shopping cart system 150 may be used to display a user interface to customers of the physical retail store 104, as further described below.

[0030] In some embodiments, customers of the physical retail store 104 can interact with the shopping cart system 150. Customers can interact with the shopping cart system 150 to initiate a shopping session and / or to perform user authentication, as further described below. In embodiments where the shopping cart system 150 includes a display area 159, the customer can interact with the shopping cart system 150 through the display area 159. For example, if the display area 159 of the shopping cart system 150 is a touch screen display, the customer can initiate a shopping session by touching a button displayed in the display area 159, which would submit a request to the shopping cart system 150 to initiate a shopping session. The display area 159 may include one or more built-in accessories, such as a speaker, a microphone, etc. Although Figure 1Not shown, in some embodiments, the shopping cart system 150 may also include one or more accessories (e.g., speakers, microphones, buttons, dials, knobs, barcode scanners, cameras, etc.) that are coupled to the shopping cart 156 included in the shopping cart system 150, and customers of the physical retail store 104 may use these accessories to interact with the shopping cart system 150. For example, if the display area 159 installed on the shopping cart 156 of the shopping cart system 150 is not a touch screen display, the customer may initiate a shopping session by pressing a button located next to the display area 159, and pressing the button submits a request to initiate a shopping session to the shopping cart system 150. In this example, if the customer presses the button, a confirmation of the request may then be communicated to the customer (e.g., visually displayed via the display area 159 and audibly played via a speaker installed on the shopping cart 156).

[0031] One or more of the client device 110, the third party system 130, the online concierge system 102, or the shopping cart system 150 may be a specially configured computing device for performing specific functions, as described below in conjunction with Figures 2 to 6 as further described, and may include specific computing components, such as processors, memory, communication interfaces, etc.

[0032] System Overview

[0033] Figure 2 An environment 200 of a shopping cart system 150 and an online platform (such as an online concierge system 102) is illustrated according to one or more embodiments. The same reference numbers are used in the figures to identify the same elements. A letter following a reference number (e.g., "104a") indicates that the text specifically refers to the element with that particular reference number. A reference number in the text that is not followed by a letter (e.g., "104") refers to any or all elements with that reference number in the figures. For example, "104" in the text may refer to "104a," "104b," and / or "104c" in the figures.

[0034] Environment 200 includes an online concierge system 102. Online concierge system 102 is configured to receive orders from one or more customers 204 (only one is shown for simplicity). The order specifies a list of items (multiple objects or products) to be delivered to customer 204, the location where the items are to be delivered, and the time period within which the items should be delivered. In some embodiments, the order also specifies one or more retailers from which the items should be purchased. Customer 204 can place an order using a customer mobile application (CMA) 206 configured to communicate with online concierge system 102.

[0035] The online concierge system 102 is configured to send the order received from the customer 204 to one or more shoppers 208. The shopper 208 can be a person (e.g., a contractor, an employee, etc.), an entity, or an autonomous device (e.g., a robot) capable of fulfilling the order received by the online concierge system 102. The shopper 208 travels between the physical retail store 104 and the delivery location (e.g., the customer's home or office), and can do so by car, truck, bicycle, scooter, walking, or any other mode of transportation. In some embodiments, the delivery can be partially or fully automated, for example, using an autonomous vehicle. The environment 200 also includes three physical retail stores 104a, 104b, and 104c (although only three are shown for simplicity, the environment 200 may include hundreds of physical retail stores 104). The physical retail store 104 can be a physical retailer, such as a grocery store, a discount store, a department store, etc., or a non-public physical retail store 104 that stores merchandise that can be collected and delivered to the customer 204. Each shopper 208 fulfills an order received from the online concierge system 102, delivers the order to the customer 204, or performs both order fulfillment and delivery at one or more physical retail stores 104. In one embodiment, the shopper 208 uses a shopper mobile application 212 that is configured to interact with the online concierge system 102.

[0036] One or more physical retail stores 104 within environment 200 may also include one or more shopping cart systems 150 (only one is shown for simplicity). Shopping cart systems 150 may be configured to communicate various types of information with online concierge system 102 and / or customer mobile application (CMA) 206. Likewise, online concierge system 102 and / or CMA 206 may also be configured to communicate various types of information with shopping cart system 150. In various embodiments, shopping cart system 150 may send a user interface to a display area of ​​a client device 110 associated with a customer 204 (e.g., in the form of a push notification via CMA 206). In some embodiments, online concierge system 102 may send to shopping cart system 150 a user interface to be presented to a customer 204 in physical retail store 104.

[0037] In some embodiments, the shopping cart system 150 may communicate information to the online concierge system 102 describing performance metrics associated with content items presented to customers 204 of the physical retail store 104. The performance metrics associated with the content items (e.g., conversion rates) may indicate whether one or more customers 204 to whom the content items were presented subsequently performed an action associated with the content items. For example, if the content items are associated with a particular item within the physical retail store 104, the performance metrics associated with the content items may indicate the rate at which customers 204 to whom the content items were presented subsequently visited the aisle where the item was located, placed the item in their shopping carts 156, and / or purchased the item.

[0038] The shopping cart system 150 within the physical retail store 104 may also communicate contextual information collected by one or more sensors 158 included in the shopping cart system 150 to the online concierge system 102 and / or the customer management system 206. For example, assume that a customer 204 interacts with the shopping cart system 150 to log into an account associated with the customer 204 maintained in the online concierge system 102. In this example, the shopping cart system 150 may then communicate the route of the customer 204 within the physical retail store 104 based on information describing the various locations of the shopping cart 156 included in the shopping cart system 150 within the physical retail store 104 during the shopping session. In the above example, the shopping cart system 150 may also communicate location information where the shopping cart 156 stops within the physical retail store 104, as well as information describing items that the customer 204 added to the shopping cart 156 and ultimately purchased during the shopping session.

[0039] Figure 3 is a schematic diagram of a shopping cart system 150 and an online concierge system 102 according to one or more embodiments. In various embodiments, the shopping cart system 150 and / or the online concierge system 102 may include Figure 3 Furthermore, in some embodiments, the shopping cart system 150 and / or the online concierge system 102 may include modules that are comparable to those described above. Figure 3 There are fewer modules describing the combination.

[0040] The online concierge system 102 includes an inventory management engine 302 that interacts with the inventory system of each physical retail store 104. In one embodiment, the inventory management engine 302 requests and receives inventory information maintained by the physical retail store 104. The inventory management engine 302 may also receive inventory information from one or more shopping cart systems 150 included in the physical retail store 104, where the inventory information is collected by one or more sensors 158 included in the shopping cart system(s) 150. The inventory of each physical retail store 104 is unique and may change over time. The inventory management engine 302 monitors inventory changes at each participating physical retail store 104. The inventory management engine 302 is also configured to store inventory records in an inventory database 304. The inventory database 304 may store information in separate records: one record for each participating physical retail store 104, or may consolidate or merge inventory information into a unified record. The inventory information includes the attributes of the merchandise, which include qualitative and quantitative information about the merchandise, such as size, color, weight, SKU (stock keeping unit), serial number, etc. In one embodiment, the inventory database 304 may also store purchase rules associated with each merchandise, if any. For example, age-restricted merchandise, such as alcohol and tobacco, may be marked accordingly in the inventory database 304. The inventory database 304 may also store other useful inventory information for predicting merchandise availability. For example, for each merchandise-store combination (i.e., a particular merchandise at a particular physical retail store 104), the inventory database 304 may store the last time the merchandise was found, the last time the merchandise was not found (e.g., if the shopper 208 looked for the merchandise but did not find it), the rate at which the merchandise was found, and the popularity of the merchandise.

[0041] For each commodity, the inventory database 304 identifies one or more attributes of the commodity and the value corresponding to each attribute. For example, the inventory database 304 includes an entry for each commodity provided by the physical retail store 104, wherein the entry for the commodity includes a commodity identifier that can uniquely identify the commodity. The entry includes different fields, each field corresponding to an attribute of the commodity. The field of the entry includes the value of the attribute corresponding to the field, which enables the inventory database 304 to maintain different attribute values ​​for different commodities.

[0042] In various embodiments, the inventory management engine 302 maintains an assortment of merchandise for purchase provided by one or more physical retail stores 104. For example, the inventory management engine 302 receives a catalog of merchandise from the physical retail store 104 that identifies the merchandise for purchase provided by the physical retail store 104. Based on the catalog of merchandise, the inventory management engine 302 determines an assortment of merchandise provided by the physical retail store 104, wherein different levels of the assortment provide different levels of specificity regarding the merchandise included in the level. In various embodiments, the assortment identifies a category and associates one or more specific merchandise with the category. For example, the category may be identified as "milk," and the assortment associates identifiers of different milk merchandise (e.g., milk provided by different brands, milk with one or more different attributes, etc.) with the category. Thus, the assortment maintains an association between the category and specific merchandise matching the category provided by the physical retail store 104. In some embodiments, different levels of the assortment identify merchandise with different levels of specificity based on any suitable attribute or combination of attributes of the merchandise. For example, different levels of the taxonomy specify different combinations of attributes of the merchandise, and thus, merchandise in lower levels of the hierarchical taxonomy have more attributes, corresponding to higher specificity in the category, and merchandise in higher levels of the hierarchical taxonomy have fewer attributes, corresponding to lower specificity in the category. In various embodiments, higher levels of the taxonomy include fewer details about the merchandise, and thus include a greater number of merchandise in the higher levels (e.g., higher levels include a greater number of merchandise that satisfy broader categories). Similarly, lower levels of the taxonomy include more details about the merchandise, and thus include a smaller number of merchandise in the lower levels (e.g., lower levels include a smaller number of merchandise that satisfy more specific categories). In various embodiments, the taxonomy may be received from a physical retail store 104. In other embodiments, the inventory management engine 302 applies a trained taxonomy model to a catalog of merchandise received from the physical retail store 104 to include different merchandise in the levels of the taxonomy, and thus, applying the trained taxonomy model associates specific merchandise with categories corresponding to the levels in the taxonomy.

[0043] The inventory information provided by the inventory management engine 302 can supplement the training data set 320. The inventory information provided by the inventory management engine 302 does not necessarily include information about the delivery order fulfillment results associated with the items, while the data in the training data set 320 is structured to include the delivery order fulfillment results (e.g., whether the items in the order are collected).

[0044] In some embodiments, the online concierge system 102 includes an order fulfillment engine 306 that is configured to compose and display an order interface to each customer 204 (e.g., via the customer mobile application 206). The order fulfillment engine 306 is also configured to access the inventory database 304 to determine which items are available at which physical retail store 104. The order fulfillment engine 306 can supplement the item availability information in the inventory database 304 with the item availability information predicted by the machine-learned item availability model 316. The order fulfillment engine 306 determines the selling price for each item ordered by each customer 204. The price set by the order fulfillment engine 306 can be the same as the in-store price set by the retailer (i.e., the price that the customer 204 and shopper 208 need to pay at the physical retail store 104), or it can be different. The order fulfillment engine 306 is also responsible for facilitating the transaction associated with each order. In one embodiment, when a customer 204 places an order, the order fulfillment engine 306 charges the payment instrument associated with the customer. The order fulfillment engine 306 can transmit the payment information to an external payment gateway or payment processor. The order fulfillment engine 306 stores payment and transaction information associated with each order in a transaction record database 308 .

[0045] In various embodiments, the order fulfillment engine 306 generates and transmits a search interface to the client device 110 of the customer 204, which is displayed via the customer mobile application 206. The order fulfillment engine 306 receives a query from the customer 204 that includes one or more terms and retrieves items that meet the query criteria, such as items that have descriptive information that at least partially matches the query. In various embodiments, the order fulfillment engine 306 utilizes embeddings for the items to retrieve the items based on the received query. For example, the order fulfillment engine 306 generates an embedding for the query and determines a similarity measure between the embedding of the query and the embeddings of various items included in the inventory database 304.

[0046] In some embodiments, the order fulfillment engine 306 also shares order details with the physical retail store 104. For example, after an order is successfully fulfilled, the order fulfillment engine 306 can send a summary of the order to the corresponding physical retail store 104. The details of the order can include the items purchased, the total value of the items, and in some cases, the identity of the shopper 208 and customer 204 associated with the order. In one embodiment, the order fulfillment engine 306 pushes the transaction and / or order details to the retailer system in an asynchronous manner. This can be achieved through the use of webhooks, which enable programmatic or system-driven information transmission between web applications. In another embodiment, the retailer system can be configured to periodically poll the order fulfillment engine 306, which provides details of all orders that have been processed since the last request.

[0047] The order fulfillment engine 306 can interact with a shopper management engine 310, which is responsible for managing communications with and utilization of shoppers 208. In one embodiment, the shopper management engine 310 receives new orders from the order fulfillment engine 306. Based on one or more parameters, such as the probability of item availability determined by the machine-learned item availability model 316, the order content, the inventory of the physical retail store 104, and the delivery location of the closest physical retail store 104, the shopper management engine 310 determines the appropriate physical retail store 104 to fulfill the order. The shopper management engine 310 then determines one or more appropriate shoppers 208 to fulfill the order based on one or more parameters (e.g., the shopper's proximity to the appropriate physical retail store 104 (and / or customer 204), the shopper's familiarity with the particular physical retail store 104, etc.). Additionally, the shopper management engine 310 accesses a shopper database 312 that stores information describing each shopper 208, including the shopper's name, gender, rating, previous shopping history, etc.

[0048] In the process of fulfilling an order, the order fulfillment engine 306 and / or the shopper management engine 310 can access a customer database 314 storing information describing each customer 204. In some embodiments, the information about each customer 204 stored in the customer database 314 is included in a user profile of the customer 204. This information can include each customer's name, geographic location (e.g., home or business address), age, gender, shopping preferences, favorite items, disliked items, stored payment instruments, profile pictures, etc.

[0049] The user profile of the customer 204 stored in the customer database 314 may also include historical information associated with the customer 204. In some embodiments, the historical information associated with the customer 204 may describe one or more shopping trips of the customer 204 to the physical retail store 104. In this embodiment, the historical information may be collected by one or more sensors 158 included in the shopping cart system 150 within the physical retail store 104. For example, the historical information associated with the customer 204 describing its shopping trips to the physical retail store 104 may include information identifying the physical retail store 104, the time period and amount of time of the shopping session at the physical retail store 104, the route that the customer 204 walked within the physical retail store 104, the total time that the customer 204 stayed at a specific location within the physical retail store 104, information describing each location where the customer 204 stayed, etc. In this example, the historical information associated with the customer 204 may also include information identifying one or more content items associated with merchandise in the physical retail store 104 that were sent to the customer 204 while the customer 204 was in the physical retail store 104, the time each content item was sent, and information identifying one or more merchandise associated with each content item. In addition, in the example described above, the historical information associated with the customer 204 may also describe the interactions between the customer 204 and merchandise in the physical retail store 104 (e.g., picking up merchandise, placing merchandise in the shopping cart 156, purchasing merchandise, etc.) and the time associated with each interaction.

[0050] In some embodiments, the historical information associated with customer 204 included in the user profile of customer 204 may also describe actions performed by customer 204 within online concierge system 102. For example, the historical information associated with customer 204 may describe orders placed by customer 204 using online concierge system 102 (e.g., descriptions of items included in each order, the time each order was placed, information identifying the physical retail store 104 associated with each order, etc.). In this example, the historical information associated with customer 204 may also indicate whether customer 204 viewed information associated with the items (e.g., product information, ingredient lists, etc.). Additionally, in the example described above, the historical information associated with customer 204 may also describe one or more content items sent by online concierge system 102 to customer 204 for display (e.g., information identifying each content item, the time each content item was sent, information indicating whether customer 204 interacted with each content item, and information identifying one or more items associated with each content item).

[0051] In some embodiments, the user profile information associated with the customer 204 may be stored in multiple databases. For example, in addition to the customer database 314, historical information associated with the customer 204 may also be stored in the transaction record database 308 and / or the training data set 320. In such an embodiment, the information included in the user profile information of the customer 204 may be identified based on information associated with the customer 204 (e.g., the user name or other identifier of the customer 204). In addition, in some embodiments, the user profile information associated with the customer 204 may also be stored in one or more databases of the shopping cart system 150 (e.g., the event database 324, the context database 330, and / or the attribution database 338), which will be further described below.

[0052] In various embodiments, the order fulfillment engine 306 determines whether to delay presentation of the received order to the shopper 208 for fulfillment by the time interval. In the event that it is determined that presentation of the received order is to be delayed by the time interval, the order fulfillment engine 306 evaluates subsequent orders received during the time interval for inclusion in one or more batches that also include the received order. After the time interval, the order fulfillment engine 306 presents the order to one or more shoppers 208 via the shopper mobile application 212; if the order fulfillment engine 306 generates one or more batches that include the received order and one or more subsequent orders received during the time interval, the batch(es) are also presented to the one or more shoppers 208 via the shopper mobile application 212.

[0053] Shopping cart system components

[0054] The shopping cart system 150 includes an event detection engine 322 that detects various types of events associated with a customer 204 within a physical retail store 104. Examples of events that the event detection engine 322 may detect include a customer 204 of a physical retail store 104 initiating a shopping session within the physical retail store 104, parking a shopping cart 156 within a threshold distance of an item within the physical retail store 104 for at least a threshold time, moving a shopping cart 156 within a threshold distance of an item within the physical retail store 104, picking up an item from a particular location within the physical retail store 104 (e.g., a promotional display), adding an item to a shopping cart 156, purchasing an item from the physical retail store 104, ending a shopping session, or any other type of event associated with a customer 204 within the physical retail store 104. For example, the event detection engine 322 may detect a shopping session initiated by a customer 204 of the physical retail store 104 upon receiving a reading from one or more sensors 158 (e.g., motion or speed sensors 158) of the shopping cart system 150, or upon receiving one or more interactions by the customer 204 with the display area 159 and / or one or more accessories (e.g., buttons, knobs, dials, microphones, barcode scanners, cameras, etc.) in the shopping cart system 150 corresponding to a request to initiate a shopping session. Similarly, the event detection engine 322 may detect the end of a shopping session upon not receiving a reading from one or more sensors 158 (e.g., motion or speed sensors 158) included in the shopping cart system 150 for at least a threshold time, or upon receiving one or more interactions by the customer 204 with the display area 159 and / or one or more accessories in the shopping cart system 150 corresponding to a request to end the shopping session. In instances where an event detected by the event detection engine 322 corresponds to a shopping cart 156 being parked within a threshold distance of an item for at least a threshold time within a physical retail store 104, the threshold time and / or distance may be determined by a machine learning model (e.g., a model trained based on anonymous aggregated data describing customer behavior associated with the shopping cart 156 when deciding whether to purchase an item).

[0055] In some embodiments, a shopping session may be initiated after customer 204 authenticates. In this embodiment, information provided by customer 204 may be used by event detection engine 322 and / or online concierge system 102 to authenticate customer 204. Examples of such information include a username and password combination, biometric information (e.g., fingerprint), or any other suitable information that may be used to authenticate customer 204. For example, information from customer 204 corresponding to login credentials for online concierge system 102 may be received by event detection engine 322 via touch screen display area 159 included in shopping cart system 150. In this example, event detection engine 322 may communicate the login credentials to online concierge system 102, and if the credentials match the credentials of customer 204 stored in customer database 314 of online concierge system 102, the system may authenticate customer 204. Alternatively, in the example described above, the event detection engine 322 may access the customer database 314 of the online concierge system 102 and authenticate the customer 204 if the credentials match the credentials of the customer 204 of the online concierge system 102 stored in the customer database 314. As another example, the online concierge system 102 may generate a quick response (QR) code that uniquely identifies the customer 204 in the online concierge system 102 and display it on the client device 110 associated with the customer 204 (e.g., via the CMA 206). In this example, if a camera included in the shopping cart system 150 scans the QR code and the event detection engine 322 communicates the QR code to the online concierge system 102, which then authenticates the customer 204, the customer 204 may be authenticated. In embodiments where the shopping session is initiated after the customer 204 is authenticated, the shopping session may be associated with the user profile of the customer 204 after the customer 204 is authenticated. For example, once customer 204 is authenticated, information describing the shopping session may be stored in event database 324 (described below) and included in user profile information associated with customer 204 .

[0056] The event detection engine 322 can detect events associated with the customer 204 at the physical retail store 104 based on the context information of the shopping cart 156 tracked by the context tracking engine 328 as described below, and / or the floor plan associated with the physical retail store 104 stored in the floor plan database 326 as described below. For example, assume that a particular item is placed in two different locations within the physical retail store 104, one of which is its normal location within an aisle, and the other is a promotional display area at the end of another aisle. In this example, when the item is added to the shopping cart 156 included in the shopping cart system 150, the event detection engine 322 can detect a corresponding event, and the event detection engine 322 can also detect the location where the item was added to the shopping cart 156 (e.g., based on the floor plan associated with the physical retail store 104). As an additional example, the event detection engine 322 can detect the end of a shopping session based on the context information monitored by the context tracking engine 328. The context information indicates that the shopping cart 156 included in the shopping cart system 150 is within a predetermined distance of a shopping cart return area, or has successfully passed through a checkout lane located within the physical retail store 104 .

[0057] The event detection engine 322 can store information describing one or more events associated with one or more customers 204 of one or more physical retail stores 104. In some embodiments, the information can be stored in an event database 324 of the shopping cart system 150. In various embodiments, the information can also or alternatively be communicated to the online concierge system 102 and stored therein (e.g., stored in the transaction record database 308 and / or the training data set 320). The event detection engine 322 can store the information describing the event in association with information identifying the customer 204 associated with the event (e.g., based on login credentials provided by the customer 204), information identifying the physical retail store 104 at which the event was detected, the time at which the event was detected, the location within the physical retail store 104 at which the event was detected, or any other information associated with the event. For example, assume that the event detection engine 322 detects that a customer 204 adds an item to a shopping cart 156 included in the shopping cart system 150 while shopping at the physical retail store 104. In this example, information describing the event, such as the type of event (i.e., the item was added to the shopping cart 156) and information describing the item (e.g., the type, size, brand, etc. of the item) can be stored (e.g., in the event database 324, the transaction record database 308, and / or the training data set 320). In the example described above, the information describing the event can be stored in association with information identifying the physical retail store 104 (e.g., its name and address), the time when the item was added to the shopping cart 156, the aisle in the physical retail store 104 where the customer 204 added the item to the shopping cart 156, and the username associated with the customer 204 in the online concierge system 102.

[0058] In some embodiments, the shopping cart system 150 also includes a floor plan database 326. The floor plan database 326 stores one or more floor plans, each of which is associated with a physical retail store 104 and describes the placement of multiple items within the physical retail store 104. The floor plan associated with the physical retail store 104 may indicate the placement of items on shelves, display cabinets, aisles, or any other suitable organizational elements within the physical retail store 104 for organizing items. The floor plan may also include the layout of the physical retail store 104, which describes the location of the organizational elements relative to each other and relative to the architectural elements of the physical retail store 104 (e.g., walls, doors, stairs, elevators, etc.). In some embodiments, the floor plans of the physical retail stores 104 stored in the floor plan database 326 can be updated (e.g., periodically, or when the placement of items within the physical retail store 104 or the layout of the physical retail store 104 is updated, etc.). In some embodiments, the floor plan database 326 may also or alternatively be included in the online concierge system 102. In this embodiment, upon receiving information describing the update from the physical retail store 104 , the online concierge system 102 may update the floor plan of the physical retail store 104 .

[0059] The shopping cart system 150 also includes a context tracking engine 328. The context tracking engine 328 tracks context information associated with a shopping cart 156 (or a similar object used to carry collected items during a shopping session) included in the shopping cart system 150, which is received by one or more sensors 158 included in the shopping cart system 150. As described above, the one or more sensors 158 included in the shopping cart system 150 are capable of identifying physical objects (e.g., merchandise, organizational elements, and / or architectural elements) within the physical retail store 104. Also as described above, the context information associated with the shopping cart 156 may describe one or more locations of the shopping cart 156 within the physical retail store 104 and / or the collection of items within the shopping cart 156. For example, the context information associated with the shopping cart 156 describing the location of the shopping cart 156 within the physical retail store 104 may include GPS coordinates collected by a GPS sensor 158 coupled to the shopping cart 156. In this example, the contextual information collected by one or more proximity sensors 158 coupled to the shopping cart 156 may also include the distance (e.g., in feet or meters), direction (e.g., north, south, east, or west), and height (e.g., in feet or meters) between one or more items, aisles, service counters, etc. within the physical retail store 104 and the shopping cart 156. Continuing with this example, the contextual information collected by the sensor 158 (e.g., a camera) mounted on the shopping cart 156 and facing the interior of the shopping cart 156 may also include information identifying one or more items (e.g., retail items) within the shopping cart 156.

[0060] In some embodiments, the context tracking engine 328 may track context information associated with the shopping cart 156 within the physical retail store 104 by comparing information collected by one or more sensors 158 included in the shopping cart system 150 to a floor plan associated with the physical retail store 104. For example, if the one or more sensors 158 included in the shopping cart system 150 correspond to one or more cameras, the context tracking engine 328 may compare video or image data collected by the cameras to the floor plan associated with the physical retail store 104 and determine the location of the shopping cart 156 based on the comparison. In this example, if the items identified in the video or image data and their placement within the physical retail store 104 have at least a threshold degree of similarity to a portion of the floor plan, the context tracking engine 328 may determine that the location of the shopping cart 156 within the physical retail store 104 corresponds to the portion of the floor plan.

[0061] The contextual information of the shopping cart 156 tracked by the context tracking engine 328 may also describe the state of the shopping cart 156. As described above, the state of the shopping cart 156 may indicate whether the shopping cart 156 is moving or stationary. In some embodiments, the state of the shopping cart 156 may also indicate whether the speed of movement of the shopping cart 156 is less than a threshold speed, whether the distance moved by the shopping cart 156 in a specified time is less than a threshold distance, whether the shopping cart 156 has reversed direction, the speed of the shopping cart 156, the direction of the shopping cart 156, or any other suitable information that may describe the state of the shopping cart 156.

[0062] The context information tracked by the context tracking engine 328 can be associated with a variety of types of data. Examples of data that can be associated with the context information include: the name of the physical retail store 104 at which the context information was collected, the geographic location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, information describing the shopping session during which the context information was collected (e.g., the time of day), information identifying the customer 204 associated with the shopping session (e.g., based on information provided to the customer 204 for authentication), etc. For example, if the physical retail store 104 is a grocery store that belongs to a chain of grocery stores, then the context information tracked by the context tracking engine 328 can be associated with data including the name identifying the grocery store chain and the geographic location (e.g., city and state) identifying the particular physical retail store 104. In the example described above, the context information can also be associated with a timestamp that indicates the time when the sensor 158 included in the shopping cart system 150 collected the context information.

[0063] In some embodiments, context tracking engine 328 may store the context information it tracks and / or data associated with the context information in context database 330. In embodiments where the context information stored in context database 330 includes information identifying customer 204 associated with the context information, such information may be included in a user profile associated with customer 204. For example, if customer 204 provides information that uniquely identifies the customer 204 when initiating a shopping session, and such information is used to verify the identity of customer 204, then information identifying customer 204 (e.g., a user name) may be stored in context database 330 in association with the context information tracked by context tracking engine 328 during the shopping session. In this example, the context information may be retrieved later in association with other user profile information associated with customer 204.

[0064] In various embodiments, the context tracking engine 328 may also communicate the context information it tracks and / or data associated with the context information to the online concierge system 102. In this embodiment, the online concierge system 102 may use this information for various purposes. In some embodiments, the online concierge system 102 may use this information to customize the experience of the customer 204 in the online concierge system 102. For example, if a customer 204 of a physical retail store 104 is also a customer 204 of the online concierge system 102, the online concierge system 102 may customize the online version of the physical retail store based on the route that the customer 204 typically takes in the physical retail store 104.

[0065] The shopping cart system 150 also includes a user interface engine 332. The user interface engine 332 identifies opportunities for presenting content to a customer 204 of the physical retail store 104 based in part on context information associated with a shopping cart 156 included in the shopping cart system 150. In various embodiments, the user interface engine 332 may receive context information associated with the shopping cart 156 from the context tracking engine 328, while in other embodiments, the user interface engine 332 may retrieve context information from the context database 330. In some embodiments, the user interface engine 332 may identify opportunities for presenting context to the customer 204 upon detecting that the shopping cart 156 has been stationary for at least a threshold amount of time. For example, the user interface engine 332 may receive context information from the context tracking engine 328 that describes a state of a shopping cart 156 used during a shopping session associated with the customer 204 of the physical retail store 104. In this example, if the context information indicates that the shopping cart 156 has been stationary for at least the threshold amount of time, the user interface engine 332 may identify opportunities for presenting context to the customer 204. In some embodiments, the user interface engine 332 may also or alternatively identify an opportunity to present context to the customer 204 upon detecting the presence of one or more particular items in the shopping cart 156. For example, if the context information indicates that an item has been added to the shopping cart 156, the user interface engine 332 may identify an opportunity to present content associated with the item to the customer 204.

[0066] The user interface engine 332 also identifies a set of content items associated with one or more items within the physical retail store 104 to include in a user interface for display to be sent to a customer 204 of the physical retail store 104. In some embodiments, the user interface engine 332 may use a machine learning model to identify the set of content items. In addition, the user interface engine 332 may identify the set of content items based in part on context information associated with a shopping cart 156 included in the shopping cart system 150. In embodiments where the user interface engine 332 identifies an opportunity to present context to the customer 204 when it detects that the shopping cart 156 has been stationary for at least a threshold amount of time, the set of content items identified by the user interface engine 332 may be associated with one or more items within a threshold distance of the shopping cart 156. For example, the user interface engine 332 may access context information stored in the context database 330 that describes the location of the shopping cart 156 within the physical retail store 104 when the shopping cart 156 has been stationary for at least a threshold amount of time. In this example, the user interface engine 332 can then access the floor plan database 326 to retrieve a floor plan associated with the physical retail store 104 and compare the location of the shopping cart 156 to the floor plan. Continuing with this example, the user interface engine 332 can then identify one or more items within the physical retail store 104 that are within a threshold distance of the location of the shopping cart 156, and access the content database 334 (described below) to identify one or more content items associated with the identified (multiple) items. In embodiments where the user interface engine 332 identifies an opportunity to present context to the customer 204 when it detects the presence of one or more specific items in the shopping cart 156, the set of content items identified by the user interface engine 332 can be associated with these items. For example, upon detecting that an item has been added to the shopping cart 156 (e.g., detected by the event detection engine 322), the user interface engine 332 can access context information stored in the context database 330 that describes one or more items in the shopping cart 156. In this example, if the context information indicates that the item in the shopping cart 156 is pasta, the user interface engine 332 can identify a set of content items associated with pasta.

[0067] In some embodiments, the user interface engine 332 may also identify a set of content items associated with one or more items within the physical retail store 104 based in part on user profile information associated with the customer 204 in the physical retail store 104, so as to include the content items in a user interface to be sent to the customer 204 for display. In this embodiment, the user interface engine 332 may identify a set of content items based on the customer's geographic location, age, gender, shopping preferences, favorite items, dislikes, historical information associated with the customer 204 (e.g., information in the physical retail store 104 or the online concierge system 102), or any other information included in the customer's user profile information. For example, assume that the user profile of the customer 204 indicates that the customer does not like nuts because of an allergy to nuts. In this example, if one or more sensors 158 of the shopping cart system 150 identify that an item is added to the shopping cart 156 that the customer 204 is using and that the item contains an ingredient corresponding to nuts, the user interface engine 332 may identify a content item that includes a warning about the ingredient of the item. As an additional example, assume that historical information associated with customer 204 indicates that customer 204 frequently purchases a particular item during each shopping session or on a consistent basis, such that customer 204 is likely to purchase the item during the current shopping session. In this example, if contextual information associated with a shopping cart 156 included in a shopping cart system 150 being used by customer 204 indicates that shopping cart 156 is in a checkout line and the item is not in shopping cart 156, user interface engine 332 may identify a content item that reminds customer 204 of the item. In embodiments where user interface engine 332 identifies content items based on user profile information associated with customer 204, user interface engine 332 may access user profile information stored in online concierge system 102 (e.g., in customer database 314, transaction record database 308, and / or training data set 320), and / or online concierge system 102 may communicate user profile information to user interface engine 332.

[0068] In various embodiments, the user interface engine 332 may also identify a collection of content items associated with one or more items within the physical retail store 104 based in part on the time of the shopping session to include the content items in a user interface for display sent to the customer 204 of the physical retail store 104. In this embodiment, the user interface engine 332 may identify content items to include in the collection based on seasonal demand for items associated with the content items, the day of the week, the time of day, etc. For example, during an early morning shopping session, the user interface engine 332 may identify one or more content items associated with breakfast items (e.g., pastries, coffee, tea, etc.). As an additional example, one month before Thanksgiving, the user interface engine 332 may identify content items associated with items associated with Thanksgiving (e.g., Thanksgiving recipes, promotions on ham or gravy, etc.).

[0069] In some embodiments, the user interface engine 332 may also identify a set of content items associated with one or more merchandise within the physical retail store 104 based in part on information associated with the set of content items and / or information about one or more merchandise associated with the content items to include the content items in a user interface for display that is to be sent to a customer 204 of the physical retail store 104. In various embodiments, each content item may be associated with a value that the user interface engine 332 may use to identify a set of content items to include in the user interface, such that content items associated with higher values ​​are more likely to be identified by the user interface engine 332 than content items associated with lower values. For example, if the content items correspond to advertisements, content items associated with higher bid amounts are more likely to be identified by the user interface engine 332 for inclusion in the user interface than content items associated with lower bid amounts. As an additional example, content items associated with higher merchandise prices and / or higher stock levels are more likely to be identified by the user interface engine 332 for inclusion in the user interface than content items associated with lower merchandise prices and / or lower stock levels.

[0070] In some embodiments, the user interface engine 332 may also identify a set of content items associated with one or more items within the physical retail store 104 based in part on information received from the online concierge system 102 for inclusion in a user interface for display that is to be sent to customers 204 of the physical retail store 104. For example, assume that the online concierge system 102 presents various content items (e.g., articles, recipes, etc.) to its customers 204 via the customer mobile application 206. In this example, the online concierge system 102 may communicate information to the user interface engine 332 that identifies certain content items that are popular among the customers 204 of the online concierge system 102. In the examples described above, the user interface engine 332 may identify these popular content items for inclusion in a user interface for display that is to be sent to customers 204 of the physical retail store 104 (e.g., if contextual information associated with the shopping cart 156 used by the customer 204 indicates that the shopping cart 156 is within a threshold distance of one or more items associated with the popular content items)

[0071] In some embodiments, the user interface engine 332 may not identify content items associated with merchandise within the physical retail store 104 for inclusion in a user interface to be sent to the customer 204 for display based on the type associated with the content items and the contextual information associated with the shopping cart 156. In various embodiments, the user interface engine 332 may not identify a content item if it is unlikely to encourage the customer 204 to perform an action associated with the merchandise (e.g., purchase the merchandise). For example, assume that the content item corresponds to an advertisement for spaghetti sauce, and the contextual information associated with the customer 204 using the shopping cart 156 within the physical retail store 104 indicates that the merchandise in the shopping cart 156 includes spaghetti sauce. In this example, since the customer 204 is likely to have intended to purchase the spaghetti sauce if he or she has already placed the spaghetti sauce in the shopping cart 156, the content item may not be included in the set of content items identified by the user interface engine 332 for inclusion in the user interface to be sent to the customer 204 for display. However, in the example described above, if the content item is a coupon for spaghetti sauce, the content item may be included in the set of content items identified by the user interface engine 332 because the customer 204 is more likely to purchase more spaghetti sauce with the coupon.

[0072] Once the user interface engine 332 identifies a set of content items associated with one or more items within the physical retail store 104 to be included in a user interface for display to be sent to a customer 204 of the physical retail store 104, the user interface engine 332 also generates a user interface that includes the content items. For example, if the user interface engine 332 identifies multiple content items, the user interface engine 332 can generate a user interface that organizes the content items in a grid format or a horizontally or vertically scrollable carousel format. In an embodiment where the user interface engine 332 identifies multiple content items to be included in a user interface for display to be sent to a customer 204, the content items can be arranged based on the likelihood that the customer 204 will subsequently perform an action associated with each content item. For example, if the user interface includes multiple content items, the content items associated with the action that the customer 204 is most likely to perform will be arranged in the most prominent position of the user interface, while the content items associated with the action that the customer 204 is least likely to perform will be arranged in the least prominent position of the user interface. In this embodiment, the user interface engine 332 can determine the likelihood of being associated with each content item based on information included in the user profile of the customer 204. For example, user interface engine 332 may determine the likelihood that customer 204 will perform an action associated with a content item based on information included in a user profile for customer 204 maintained in attribution database 338 (described below). In this example, the information included in the user profile may describe whether events associated with similar actions performed by customer 204 were attributed to the inclusion of similar content items in a user interface previously sent to a display area associated with customer 204.

[0073] Once the user interface engine 332 generates the user interface, the user interface engine sends the user interface to a display area associated with the customer 204. In some embodiments, the user interface engine 332 sends the user interface to a display area included in a client device 110 associated with the customer 204. In various embodiments, the user interface engine 332 also or alternatively sends the user interface to a display area 159 included in the shopping cart system 150.

[0074] The shopping cart system 150 also includes a content database 334. The content database 334 stores content items associated with items within one or more physical retail stores 104. The content database 334 can store various types of content items, such as product information associated with the item (e.g., source / manufacturing information, material / ingredient warnings), promotions for the item, coupons for the item, sets of instructions for using the item, suggested uses for the item, advertisements for the item, reminders for the item, videos, images, or social media posts featuring the item, etc. For example, if the item is a retail item sold in a grocery store, the content items corresponding to the product information associated with the item may include nutritional information associated with the item, information indicating whether the item is an organic item, origin information, etc. As an additional example, the content items associated with the item in the grocery store may correspond to recipes that include the item as an ingredient. As another example, the content items associated with the item may correspond to reminders to purchase the item if the customer 204 frequently purchases the item during a shopping session, or if the item is often paired with another item in the shopping cart 156 that the customer 204 is using. In some embodiments, one or more content items stored in content database 334 may be associated with multiple items within physical retail store 104. For example, a content item corresponding to a recipe may be associated with multiple items, each corresponding to an ingredient used in the recipe.

[0075] The shopping cart system 150 also includes an attribution engine 336. The attribution engine 336 is responsible for determining whether an event detected by the event detection engine 322 should be attributed to a content item presented in a user interface to a customer 204 of the physical retail store 104. The attribution engine 336 can make a determination based on two times: a first time is the time when the user interface is sent to a display area associated with the customer 204, and a second time is the time when the event is detected by the event detection engine 322. In some embodiments, once the user interface is sent to a display area associated with the customer 204 during a shopping session, the attribution engine 336 can determine that any subsequent events detected by the event detection engine 322 during the shopping session can be attributed to the inclusion of the content item associated with the event in the user interface. For example, assume that the user interface engine 332 sends a user interface to a display area associated with the customer 204 that includes a content item corresponding to a promotion. In this example, the attribution engine 336 can determine that subsequent additions of an item to the shopping cart 156 by the customer 204 and purchases of the item by the customer 204 detected by the event detection engine 322 should be attributed to the inclusion of the content item in the user interface sent to the customer 204 for presentation.

[0076] In some embodiments, once the attribution engine 336 determines whether to attribute an event detected by the event detection engine 322 to the inclusion of a content item in a user interface for display sent to a customer 204 of a physical retail store 104, the attribution engine 336 may store information describing the determination in an attribution database 338. The information stored in the attribution database 338 may include information describing a content item included in a user interface sent to a display area associated with a customer 204 of a physical retail store 104, information describing a time when the user interface was sent to the display area, information describing the display area (e.g., whether the display area is included in a client device 110 or a shopping cart system 150), information identifying the customer 204, information describing whether to attribute the event to the inclusion of a content item in the user interface (e.g., a time when the event was detected, information identifying a product associated with the event, etc.), or any other suitable information describing the determination made by the attribution engine 336. In embodiments where the information stored in the attribution database 338 includes information identifying a customer 204, the information may be included in user profile information associated with the customer 204.

[0077] In some embodiments, the shopping cart system 150 also includes a performance engine 340. The performance engine 340 calculates a performance metric associated with the content item based in part on the determination of the attribution engine 336 (i.e., whether the event is attributed to the inclusion of the content item in the user interface sent to the display area associated with the customer 204 of the physical retail store 104). The performance metric can correspond to a conversion rate, a click-through rate (CTR), or any other suitable metric describing the performance of the content item. For example, the performance metric associated with the content item can correspond to a conversion rate, wherein the conversion rate describes the ratio of the customer 204 of the physical retail store 104 who sees the content item during his shopping and then adds the goods associated with the content item to his shopping cart 156. As an additional example, the performance metric associated with the content item can correspond to a click-through rate, wherein the click-through rate describes the ratio of the customer 204 who sees the content item to click on the content item to view the recipe associated with the goods. The performance engine 340 can be configured to store information describing the performance metric in a performance database 342. Furthermore, in some embodiments, the performance engine 340 and / or performance database 342 may also or alternatively be included in the online concierge system 102. In embodiments where the performance engine 340 is included in the online concierge system 102, the attribution engine 336's determination as to whether to attribute an event to the inclusion of the content item in a user interface sent to a display area associated with a customer 204 of a physical retail store 104 may be communicated to the online concierge system 102 and used by the performance engine 340 to calculate a performance metric associated with the content item. In this embodiment, the performance engine 340 of the online concierge system 102 may access the attribution engine 336 of the shopping cart system 150 to retrieve the determination, or the attribution engine 336 of the shopping cart system 150 may communicate the determination to the performance engine 340 of the online concierge system 102.

[0078] Displaying information related to items in a physical retail store based on contextual information associated with a shopping cart system interest

[0079] Figure 4 is a flow chart of a method for displaying information related to items in a physical retail store 104 based on contextual information associated with a shopping cart system 150 according to one or more embodiments. In various embodiments, the method may include: Figure 4 In addition, in some embodiments, the steps of the method may be performed in the same manner as described above. Figure 4 Combine the different orders described. Figure 4 In the described method, one or more steps may be performed by the online concierge system 102 in various embodiments, while in other embodiments, the steps of the method are performed by any online system capable of retrieving items.

[0080] The shopping cart system 150 detects 405 (e.g., via the event detection engine 322) that a customer 204 within the physical retail store 104 has initiated a shopping session. The initiation of the shopping session can be detected 405 by one or more components of the shopping cart system 150. In some embodiments, the initiation of the shopping session is detected 405 by one or more sensors 158 included in the shopping cart system 150. For example, upon receiving a reading from one or more sensors 158 (e.g., a motion or speed sensor 158) of the shopping cart system 150, the shopping cart system 150 can detect 405 the initiation of the shopping session. In various embodiments, the initiation of the shopping session can also or alternatively be detected 405 by one or more interactions with a display area 159 and / or one or more accessories (e.g., buttons, dials, knobs, microphones, barcode scanners, cameras, etc.) in the shopping cart system 150. For example, when the customer 204 requests to initiate a shopping session via a touch screen display area 159 included in the shopping cart system 150, the shopping cart system 150 can detect 405 the initiation of the shopping session.

[0081] In some embodiments, a shopping session may be initiated after verifying the identity of customer 204. In this embodiment, information provided by customer 204 may be used by shopping cart system 150 and / or online concierge system 102 to verify the identity of the customer. Example information includes a username and password combination, biometric information (e.g., fingerprint), or any other information that may be used to verify the identity of the customer. For example, login credential information provided by customer 204 to online concierge system 102 may be received by shopping cart system 150 via touch screen display area 159 in shopping cart system 150. In this example, shopping cart system 150 may communicate the login credential to online concierge system 102, and if the credential matches the credential of customer 204 in online concierge system 102 (e.g., in customer database 314), online concierge system 102 may verify the identity of the customer. Alternatively, in the example described above, shopping cart system 150 may access information associated with customer 204 of online concierge system 102 stored in online concierge system 102 (e.g., information in customer database 314), and if the credential matches, the identity of the customer may be verified. As an additional example, the online concierge system 102 may generate a quick response (QR) code that uniquely identifies the customer 204 in the online concierge system 102 and displays it on a client device 110 associated with the customer 204 (e.g., via the CMA 206). In this example, if a camera included in the shopping cart system 150 scans the QR code and the shopping cart system 150 communicates the QR code to the online concierge system 102, the online concierge system 102 then authenticates the customer 204. In embodiments where the shopping session is initiated after the customer 204 authenticates, once the customer 204 is authenticated, the shopping session may be linked to the user profile of the customer 204. For example, once the customer 204 is authenticated, information describing the shopping session may be stored by the shopping cart system 150 (e.g., in the event database 324) and included in the user profile information associated with the customer 204.

[0082] The shopping cart system 150 tracks (e.g., using the context tracking engine 328) context information associated with a shopping cart 156 (or a similar object used to carry collected items during a shopping session) included in the shopping cart system 150, which is received by one or more sensors 158 included in the shopping cart system 150. As described above, the one or more sensors 158 included in the shopping cart system 150 are capable of identifying physical objects (e.g., merchandise, organizational elements, and / or architectural elements) within the physical retail store 104. Also as described above, the context information associated with the shopping cart 156 may describe one or more locations of the shopping cart 156 within the physical retail store 104 and / or the collection of items within the shopping cart 156. For example, the context information associated with the shopping cart 156 describing the location of the shopping cart 156 within the physical retail store 104 may include GPS coordinates collected by a GPS sensor 158 coupled to the shopping cart 156. In this example, the contextual information collected by one or more proximity sensors 158 coupled to the shopping cart 156 may also include the distance (e.g., in feet or meters), direction (e.g., north, south, east, west), and height (e.g., in feet or meters) between one or more items, aisles, service counters, etc. within the physical retail store 104 and the shopping cart 156. Continuing with this example, the contextual information collected by the sensor 158 (e.g., a camera) mounted on the shopping cart 156 and facing the interior of the shopping cart 156 may also include information identifying one or more items (e.g., retail items) within the shopping cart 156.

[0083] In some embodiments, the shopping cart system 150 may track 410 contextual information associated with the shopping cart 156 by comparing information collected by one or more sensors 158 included in the shopping cart system 150 to a floor plan associated with the physical retail store 104. Figure 5As shown, the figure illustrates a floor plan 500 associated with a physical retail store 104. According to one or more embodiments, the floor plan 500 may indicate the placement of merchandise 505 in shelves, display cases, aisles, or any other suitable organizational elements within the physical retail store 104 for organizing merchandise 505. The floor plan 500 may also include a layout of the physical retail store 104 that describes the locations of the organizational elements relative to each other and relative to architectural elements (e.g., walls, doors, stairs, elevators, etc.) of the physical retail store 104. To illustrate how the example shopping cart system 150 tracks 410 contextual information associated with the shopping cart 156 by comparing information collected by sensors 158 included in the shopping cart system 150 with the floor plan 500 associated with the physical retail store 104, assume that one or more sensors 158 correspond to one or more cameras. In this example, the shopping cart system 150 may compare video or image data collected by the cameras with the floor plan 500 associated with the physical retail store 104, and determine the location of the shopping cart 156 based on the comparison. In the example above, if the items 505 identified in the video or image data and their placement within the physical retail store 104 have at least a threshold degree of similarity to a portion of the floor plan 500 , the shopping cart system 150 can determine the location of the shopping cart 156 within the physical retail store 104 that corresponds to the portion of the floor plan 500 .

[0084] The contextual information associated with the shopping cart 156 tracked 410 by the shopping cart system 150 may also describe the state of the shopping cart 156. As described above, the state of the shopping cart 156 may indicate whether the shopping cart is moving or stationary. In some embodiments, the state of the shopping cart 156 may also indicate whether the shopping cart is moving at a speed below a threshold, whether the distance moved within a specified time is less than a threshold, whether the direction has changed, the speed of the shopping cart, the orientation of the shopping cart, or any other suitable information describing the state of the shopping cart 156.

[0085] The context information associated with the shopping cart 156 tracked 410 by the shopping cart system 150 may be associated with a variety of types of data. Examples of data associated with the context information include: the name of the physical retail store 104 at which the context information was collected, the geographic location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, information describing the shopping session period (e.g., time of day) when the context information was collected, information identifying the customer 204 associated with the shopping session (e.g., based on information provided to the customer 204 for authentication), etc. For example, if the physical retail store 104 is a grocery store that belongs to a chain of grocery stores, then the context information associated with the shopping cart 156 tracked 410 by the shopping cart system 150 may be associated with data including the name identifying the grocery store chain and data identifying the geographic location of the particular physical retail store 104 (e.g., city and state). In the example described above, the context information may also be associated with a timestamp that indicates the time when the sensor 158 included in the shopping cart system 150 collected the context information.

[0086] although Figure 4 4. Although not illustrated, in some embodiments, the shopping cart system 150 may store the context information that it tracks 410 and / or data associated with the context information (e.g., in the context database 330). In embodiments where the context information stored in the shopping cart system 150 includes information identifying the customer 204 associated with the context information, the information may be included in a user profile associated with the customer 204. For example, if the customer 204 provides information that uniquely identifies the customer 204 when a shopping session is initiated, and the information is used to verify the identity of the customer 204, the information identifying the customer 204 (e.g., a user name) may be stored in the shopping cart system 150 in association with the context information that the shopping cart system 150 tracks 410 during the shopping session. In this example, the context information may then be retrieved in association with other user profile information associated with the customer 204.

[0087] In various embodiments, the shopping cart system 150 may also communicate the contextual information and / or data associated with the contextual information that it tracks 410 to the online concierge system 102. In this embodiment, the online concierge system 102 may use this information for various purposes. In some embodiments, the online concierge system 102 may use this information to customize the experience of the customer 204 in the online concierge system 102. For example, if a customer 204 of a physical retail store 104 is also a customer 204 of the online concierge system 102, the online concierge system 102 may customize the online version of the physical retail store 104 based on the route that the customer 204 typically takes in the physical retail store 104.

[0088] Back to reference Figure 4 , the shopping cart system 150 subsequently identifies 415 (e.g., via the user interface engine 332) an opportunity to present content to the customer 204 based in part on the contextual information associated with the shopping cart 156. In some embodiments, the shopping cart system 150 may identify 415 an opportunity to present content to the customer 204 upon detecting that the shopping cart 156 has been stationary for at least a threshold amount of time. For example, the shopping cart system 150 may receive contextual information describing a state of the shopping cart 156. In this example, if the contextual information indicates that the shopping cart 156 has been stationary for at least a threshold amount of time, the shopping cart system 150 may identify 415 an opportunity to present content to the customer 204. In some embodiments, the shopping cart system 150 may also or alternatively identify 415 an opportunity to present content to the customer 204 upon detecting that one or more particular items 505 are within the shopping cart 156. For example, if the contextual information indicates that an item 505 has been added to the shopping cart 156, the shopping cart system 150 may identify 415 an opportunity to present content associated with the item 505 to the customer 204.

[0089] The shopping cart system 150 then identifies step 420 (e.g., using the user interface engine 332) a set of content items associated with the one or more items 505 within the physical retail store 104 to include in a user interface for display to be sent to the customer 204. In some embodiments, the shopping cart system 150 can use a machine learning model to identify 420 the set of content items. In addition, the shopping cart system 150 can also identify 420 the set of content items based in part on contextual information associated with the shopping cart 156.

[0090] The collection of content items may be of various types maintained by the shopping cart system 150 (e.g., in the content database 334), such as product information associated with the item 505 (e.g., source / manufacturing information, material / ingredient warnings), promotions for the item 505, coupons for the item 505, sets of instructions for using the item 505, suggested uses for the item 505, advertisements for the item 505, reminders about the item 505, videos, images, or social media posts featuring the item 505, etc. For example, if the item 505 is a retail item sold in a grocery store, then the content items corresponding to the product information associated with the item 505 may include nutritional information associated with the item 505, information indicating whether the item 505 is an organic product, its origin, etc. As an additional example, the content items associated with the item 505 in the grocery store may correspond to recipes that include the item 505 as an ingredient. As another example, a content item associated with an item 505 may correspond to a reminder to purchase the item 505 if the customer 204 frequently purchases the item 505 during shopping sessions or if the item 505 is often paired with another item 505 in the shopping cart 156 .

[0091] In some embodiments, one or more content items may be associated with multiple items 505 within a physical retail store 104. Figure 6 As shown, according to one or more embodiments, various content items 605 can be associated with one or more commodities 505. For example, content items 605a, 605b, 605d, and 605e are associated only with commodity 505a, while content items 605c and 605f are associated with both commodities 505a and 505b, and content item 605g is associated with commodities 505a, 505b, and 505c.

[0092] Back to reference Figure 4, in embodiments where the shopping cart system 150 identifies 415 an opportunity to present content to the customer 204 when it detects that the shopping cart 156 has been stationary for at least a threshold amount of time, the set of content items 605 identified 420 by the shopping cart system 150 may be associated with one or more items 505 within a threshold distance of the shopping cart 156. For example, the shopping cart system 150 may access context information stored in the shopping cart system 150 (e.g., in the context database 330) that describes the location of the shopping cart 156 within the physical retail store 104 when the shopping cart 156 has been stationary for at least the threshold amount of time. In this example, the shopping cart system 150 may then retrieve a floor plan 500 associated with the physical retail store 104 (e.g., from the floor plan database 326) and compare the location of the shopping cart 156 to the floor plan 500. Continuing with this example, the shopping cart system 150 may then identify one or more items 505 within the physical retail store 104 that are within a threshold distance of the location of the shopping cart 156 and identify 420 one or more content items 605 associated with the identified item(s) 505 (e.g., by accessing a content database 334).

[0093] In embodiments where the shopping cart system 150 identifies 415 an opportunity to present content to the customer 204 when it detects the presence of one or more particular items 505 in the shopping cart 156, the set of content items 605 identified 420 by the shopping cart system 150 may be associated with the item(s) 505. For example, upon detecting that an item 505 has been added to the shopping cart 156, the shopping cart system 150 may access contextual information (e.g., stored in the context database 330) describing the one or more items 505 in the shopping cart 156. In this example, if the contextual information indicates that the item 505 in the shopping cart 156 is pasta, the shopping cart system 150 may identify 420 one or more content items 605 associated with pasta.

[0094] In some embodiments, the shopping cart system 150 may also identify 420 a set of content items 605 to include in a user interface to be sent to the customer 204 for display based in part on user profile information associated with the customer 204. In this embodiment, the shopping cart system 150 may identify 420 a set of content items 605 based on the customer's geographic location, age, gender, shopping preferences, favorite items 505, dislikes, historical information associated with the customer 204 (e.g., information in a physical retail store 104 or an online concierge system 102), or any other information included in the customer's user profile. For example, assume that the user profile of the customer 204 indicates that the customer does not like nuts due to a nut allergy. In this example, if one or more sensors 158 of the shopping cart system 150 identify that an item 505 is added to the shopping cart 156 and that the item 505 includes an ingredient corresponding to nuts, the shopping cart system 150 may identify 420 a content item 605 that includes an ingredient warning about the item 505. As an additional example, assume that historical information associated with customer 204 indicates that customer 204 frequently purchases a particular item 505 during each shopping session or on a consistent basis, such that customer 204 is likely to purchase the item 505 during the current shopping session. In this example, if contextual information associated with shopping cart 156 indicates that shopping cart 156 is in a checkout line and the item 505 is not in shopping cart 156, shopping cart system 150 may identify 420 a content item 605 that reminds customer 204 about the item 505. In embodiments where shopping cart system 150 identifies (step 420) content item 605 based on user profile information associated with customer 204, shopping cart system 150 may access user profile information stored in online concierge system 102 (e.g., in customer database 314, transaction records database 308, and / or training data set 320), and / or online concierge system 102 may communicate the user profile information to shopping cart system 150.

[0095] In various embodiments, the shopping cart system 150 may also identify 420 a collection of content items 605 to be included in a user interface for display sent to the customer 204 based in part on the time of the shopping session. In this embodiment, the shopping cart system 150 may identify 420 content items 605 to include in the collection based on seasonal demand for the items 505 associated with the content items 605, the day of the week, the time of day, etc. For example, if the shopping session is in the early morning, the shopping cart system 150 may identify 420 one or more content items 605 associated with breakfast items 505 (e.g., pastries, coffee, tea, etc.). As an additional example, if the shopping session is one month before Thanksgiving, the shopping cart system 150 may identify (step 420) content items 605 associated with items 505 related to Thanksgiving (e.g., Thanksgiving recipes, promotions for ham or gravy, etc.).

[0096] In some embodiments, the shopping cart system 150 may also identify 420 a set of content items 605 to include in a user interface sent to the customer 204 for display based in part on information associated with the set of content items 605 and / or information associated with one or more items 505 associated with the set of content items 605. In various embodiments, each content item 605 may be associated with a value that the shopping cart system 150 may use to identify 420 the set of content items 605 to include in the user interface, such that content items 605 associated with higher values ​​are more likely to be identified 420 by the shopping cart system 150 than content items 605 associated with lower values. For example, if the content items 605 correspond to advertisements, content items 605 associated with higher bid amounts may be more likely to be identified 420 by the shopping cart system 150 for inclusion in the user interface than content items 605 associated with lower bid amounts. As an additional example, content items 605 associated with higher priced items 505 and / or items 505 with higher stock quantities may be more likely to be identified 420 by the shopping cart system 150 for inclusion in the user interface than content items 605 associated with lower priced items 505 and / or items 505 with lower stock quantities.

[0097] In some embodiments, the shopping cart system 150 may also identify 420 a set of content items 605 to be included in a user interface for display sent to a customer 204 based in part on information received from the online concierge system 102. For example, assume that the online concierge system 102 presents various content items 605 (e.g., articles, recipes, etc.) to its customers 204 via the CMA 206. In this example, the online concierge system 102 may communicate information to the shopping cart system 150 that identifies certain content items 605 that are popular among the customers 204 of the online concierge system 102. In the above example, the popular content items 605 may be included among those content items identified 420 by the shopping cart system 150 for inclusion in a user interface for display sent to the customer 204 (e.g., if contextual information associated with the shopping cart 156 indicates that the shopping cart 156 is within a threshold distance of one or more items 505 associated with the popular content items 605).

[0098] In some embodiments, shopping cart system 150 may not identify 420 content item 605 for inclusion in a user interface for display sent to customer 204 based on the type associated with content item 605 and contextual information associated with shopping cart 156. In various embodiments, shopping cart system 150 may not identify 420 content item 605 if content item 605 is unlikely to encourage customer 204 to perform an action associated with item 505 (e.g., purchase item 505). For example, assume that content item 605 corresponds to an advertisement for spaghetti sauce, and contextual information associated with shopping cart 156 indicates that spaghetti sauce is one of items 505 within shopping cart 156. In this example, since customer 204 is likely to have intended to purchase spaghetti sauce if they have already placed spaghetti sauce in shopping cart 156, content item 605 may not be included in the set of content items 605 identified 420 by shopping cart system 150 for inclusion in a user interface for display sent to customer 204. However, in the example above, if content item 605 is a coupon for spaghetti sauce, the content item may be included in the set of content items 605 identified 420 by shopping cart system 150 because customer 204 is more likely to purchase more spaghetti sauce with the coupon.

[0099] The shopping cart system 150 then generates 425 (e.g., using the user interface engine 332) a user interface that includes the identified set of content items 605. For example, if the shopping cart system 150 identifies 420 multiple content items 605, the shopping cart system 150 may generate 425 a user interface that organizes the content items 605 in a grid format or a horizontally or vertically scrollable carousel format. In embodiments where the shopping cart system 150 identifies 420 multiple content items 605 to include in a user interface for display to be sent to the customer 204, the content items 605 may be arranged based on the likelihood that the customer 204 will subsequently perform an action associated with each content item 605. For example, if the user interface includes multiple content items 605, the content items 605 associated with the actions that the customer 204 is most likely to perform may be arranged in the most prominent position in the user interface, while the content items 605 associated with the actions that the customer 204 is least likely to perform may be arranged in the least prominent position. In this embodiment, the shopping cart system 150 may determine the likelihood associated with each content item 605 based on information included in the user profile of the customer 204. For example, shopping cart system 150 may determine the likelihood that customer 204 performs an action associated with content item 605 based on information included in a user profile of customer 204 maintained in shopping cart system 150 (e.g., in attribution database 338). In this example, the information included in the user profile may describe whether events associated with similar actions previously performed by customer 204 were attributed to the inclusion of similar content item 605 in a user interface previously sent to a display area associated with customer 204. In some embodiments, a server in communication with shopping cart system 150 may generate 425 the user interface. For example, a server associated with online concierge system 102 may generate 425 the user interface and communicate it to shopping cart system 150.

[0100] Once the user interface is generated 425, the shopping cart system 150 sends 430 (e.g., using the user interface engine 332) the user interface to a display area associated with the customer 204. In some embodiments, the shopping cart system 150 sends 430 the user interface to a display area included in a client device 110 associated with the customer 204. For example, the shopping cart system 150 may send 430 the user interface to a display area included in a client device 110 associated with the customer 204 (e.g., via a push notification sent by the CMA 206). In various embodiments, the shopping cart system 150 also or alternatively sends 430 the user interface to a display area 159 included in the shopping cart system 150.

[0101] In some embodiments, the shopping cart system 150 may then detect 435 (e.g., using the event detection engine 322) an event associated with one or more items 505 associated with the set of content items 605 included in the user interface sent 430 to the display area associated with the customer 204. Examples of events that the shopping cart system 150 may detect 435 include: parking the shopping cart 156 within a threshold distance of an item 505 within the physical retail store 104 for at least a threshold amount of time; moving the shopping cart 156 within the physical retail store 104 to within a threshold distance of an item 505; picking up an item 505 from a particular location within the physical retail store 104 (e.g., a promotional display); adding an item 505 to the shopping cart 156; purchasing an item 505 from the physical retail store 104; or any other type of event that may be associated with a customer 204 within the physical retail store 104. In embodiments where the event detected 435 by the shopping cart system 150 corresponds to a shopping cart 156 being parked within a threshold distance of an item 505 within a physical retail store 104 for at least a threshold amount of time, the threshold amount of time and / or distance may be determined by a machine learning model (e.g., a model trained based on anonymous aggregated data describing customer behavior associated with the shopping cart 156 when deciding whether to purchase the item 505).

[0102] The shopping cart system 150 may detect 435 an event of one or more items 505 associated with a set of content items 605 included in a user interface sent to a display area associated with a customer 204 based on the contextual information associated with the shopping cart 156 tracked 410 by the shopping cart system 150 and / or a floor plan 500 associated with the physical retail store 104. For example, assume that a particular item 505 associated with a content item 605 included in a user interface sent to a display area associated with a customer 204 is placed in two different locations within the physical retail store 104, one of which is its normal location within an aisle and the other is a promotional display area at the end of another aisle. In this example, the shopping cart system 150 may detect 435 an event corresponding to adding the item 505 to the shopping cart 156, wherein the shopping cart system 150 may also detect the location at which 435 the item 505 was added to the shopping cart 156 (e.g., based on the floor plan 500 associated with the physical retail store 104). In the above example, the shopping cart system 150 may subsequently also detect 435 an event corresponding to the purchase of the item 505 based on the contextual information associated with the shopping cart 156 and the floor plan 500 associated with the physical retail store 104, which indicates that the shopping cart 150 left the checkout lane within the physical retail store 104 with the item 505.

[0103] although Figure 44. In some embodiments, the shopping cart system 150 may store information describing the event (e.g., in the event database 324). In various embodiments, the information may also or alternatively be transmitted to and stored in the online concierge system 102 (e.g., in the transaction record database 308 and / or the training data set 320). The shopping cart system 150 may store the information describing the event and information associated with the event that identifies the customer 204 (e.g., based on login credentials provided by the customer 204), identifies the physical retail store 104 at which the event 435 was detected, the time at which the event 435 was detected, the location within the physical retail store 104 at which the event 435 was detected, or any other information associated with the event. For example, assume that the shopping cart system 150 detects 435 that the customer 204 adds the item 505 to the shopping cart 156. In this example, information describing the event (such as the event type (i.e., adding the item 505 to the shopping cart 156)) and information describing the item 505 (e.g., the type of item 505, the size of the item 505, the brand of the item 505, etc.) can be stored (e.g., in the event database 324, the transaction record database 308, and / or the training data set 320). In the example described above, the information describing the event can be stored in association with information identifying the physical retail store 104 (e.g., its name and address), the time when the item 505 was added to the shopping cart 156, the aisle within the physical retail store 104 where the customer 204 added the item 505 to the shopping cart 156, and the user name associated with the customer 204 in the online concierge system 102.

[0104] Shopping cart system 150 may then determine 440 (e.g., using attribution engine 336) whether to attribute event 435 detected by shopping cart system 150 to inclusion of content item 605 in a user interface sent 430 to customer 204 for display. Shopping cart system 150 may make this determination based on a first time the user interface is sent 430 to a display area associated with customer 204 and a second time the shopping cart system 150 detects 435 the event. In some embodiments, once the user interface is sent 430 to a display area associated with customer 204 during a shopping session, shopping cart system 150 may determine 440 that any event 435 subsequently detected by shopping cart system 150 during the shopping session may be attributed to inclusion of content item 605 associated with the event in the user interface. For example, assume shopping cart system 150 sends 430 a user interface to a display area associated with customer 204, where the user interface includes content item 605 corresponding to a coupon for product 505. In this example, shopping cart system 150 may determine 440 that shopping cart system 150's detection of customer 204 adding item 505 to shopping cart 156 and customer 204's subsequent purchase of item 505 should be attributed to including content item 605 in the user interface sent 430 to customer 204 for display.

[0105] although Figure 4 4, once the shopping cart system 150 determines 440 whether to attribute the detected event 435 to the inclusion of the content item 605 in the user interface sent 430 to the customer 204 for display, the shopping cart system 150 may store information describing the determination (e.g., in the attribution database 338). The stored information may include information describing the content item 605, information describing the time when the user interface was sent 430 to the display area associated with the customer 204, information describing the display area (e.g., whether the display area was included in the client device 110 or the shopping cart system 150), information identifying the customer 204, information describing whether to attribute the event to the inclusion of the content item 605 in the user interface (e.g., the time when the event was detected 435, information identifying the item 505 associated with the event, etc.), or any other suitable information describing the determination made by the shopping cart system 150. In embodiments where the information describing the determination identifies the customer 204, the information may be included in the user profile information associated with the customer 204.

[0106] In some embodiments, the shopping cart system 150 also calculates 445 (e.g., using the performance engine 340) a performance metric associated with the content item 605 included in the user interface sent 430 to the customer 204 for display. The shopping cart system 150 may calculate 445 the performance metric based in part on its determination whether to attribute the event 435 detected by the shopping cart system 150 to the inclusion of the content item 605 in the user interface sent 430 to the display area associated with the customer 204. The performance metric may correspond to a conversion rate, a click-through rate (CTR), or any other suitable metric describing the performance of the content item 605. For example, the performance metric associated with the content item 605 may correspond to a conversion rate, wherein the conversion rate describes the rate at which customers 204 of the physical retail store 104 subsequently add items 505 associated with the content item 605 to their shopping carts 156 after being presented with the content item 605 during their shopping session. As an additional example, the performance metric associated with content item 605 may correspond to a click-through rate, wherein the click-through rate describes the rate at which customers 204 click on content item 605 to view a recipe associated with product 505 after content item 605 is presented to them. Figure 4 342 , in some embodiments, shopping cart system 150 may be configured to store information describing the performance metric (e.g., in performance database 342). Additionally, in some embodiments, a determination by shopping cart system 150 whether to attribute an event to including content item 605 in a user interface sent 430 to a display area associated with customer 204 may be communicated to online Concierge system 102 and used by online Concierge system 102 to calculate 445 a performance metric associated with content item 605. In this embodiment, online Concierge system 102 may access shopping cart system 150 to retrieve the determination, or shopping cart system 150 may communicate the determination to online Concierge system 102.

[0107] The shopping cart system 150 may then communicate 450 the performance metric associated with the content item 605 to an entity associated with the physical retail store 104, the third-party system 130, and / or the online concierge system 102. For example, assume that the performance metric associated with the content item 605 corresponds to a conversion rate associated with the content item 605, and the content item 605 corresponds to a coupon provided by the physical retail store 104 for the item 505. In this example, the performance metric may indicate the rate at which customers 204 subsequently add the item 505 to their shopping cart 156 after being presented with the content item 605, and the performance metric may be communicated 450 to the entity associated with the physical retail store 104 as an indication of the effectiveness of the coupon in encouraging customers 204 to add the item 505 to their shopping cart 156. As an additional example, if the performance metric associated with the content item 605 corresponds to a click-through rate, and the content item 605 corresponds to an advertisement, the performance metric may be communicated 450 to the third-party system 130 associated with the advertisement as an indication of the effectiveness of the advertisement. As yet another example, assume that the performance metric associated with content item 605 corresponds to a conversion rate, and content item 605 corresponds to product information associated with item 505. In this example, the performance metric may be communicated 450 to online concierge system 102, which may increase or decrease the prominence of the product information presented in online concierge system 102 to its customers 204 based on the performance metric.

[0108] In various embodiments, information associated with one or more items 505 included in a user interface sent 430 for display to a customer 204 may be updated 455, the one or more items 505 associated with the content item 605, based on the performance metric associated with the content item 605. In some embodiments, the updated information may be maintained in the shopping cart system 150, while in other embodiments, the updated information may be maintained in the online concierge system 102. For example, if the performance metric associated with the content item 605 calculated 445 by the shopping cart system 150 corresponds to a low conversion rate for the item 505 associated with the content item 605, and the content item 605 corresponds to a recipe that includes the item 505 as an ingredient, then the content item 605 may be updated 455 with a different recipe in the shopping cart system 150. As an additional example, assume that the performance metric associated with the content item 605 calculated 445 by the shopping cart system 150 corresponds to a high conversion rate for the item 505 associated with the content item 605, and the content item 605 corresponds to nutritional information for the item 505. In this example, if the performance metrics are communicated 450 to the online concierge system 102 , the online concierge system 102 may update 455 the product information for the item 505 to highlight the nutritional information.

[0109] although Figure 4Not shown in the figure, in some embodiments, upon detecting (e.g., using the event detection engine 322) that a shopping session has ended, the same shopping cart system 150 may subsequently detect 405 (e.g., via the event detection engine 322) that an additional customer 204 has initiated an additional shopping session within the physical retail store 104. Some or all of the additional steps described above (e.g., steps 410 to 455) may then be performed for the additional customer 204.

[0110] Additional Notes

[0111] The above description of the embodiments of the present invention is presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise form disclosed.Many modifications and variations are possible in light of the disclosure described above.

[0112] Parts of this description describe embodiments of the present invention in the form of algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are commonly used in the field of data processing to effectively convey the essence of their work to others. Although these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs or equivalent circuits, microcodes, or the like. In addition, sometimes for convenience, without loss of generality, these operations are arranged to be referred to as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0113] Any step, operation or process described herein can be performed alone or in combination with other devices using one or more hardware or software modules. In one embodiment, the software module is implemented by a computer program product including a computer readable medium (the medium contains a computer program code executable by a computer processor) to perform any or all steps, operations or processes described herein.

[0114] Embodiments of the present invention may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose and / or it may include a computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a tangible computer-readable storage medium, which includes any tangible medium suitable for storing electronic instructions and coupled to a computer system bus. In addition, any computing system referred to in the specification may include a single processor, or may be an architecture that uses multiple processors to increase computing power.

[0115] Embodiments of the present invention may also relate to a computer data signal embodied in a carrier wave, wherein the computer data signal includes any embodiment of the computer program product or other data combination described herein. The computer data signal is a product presented in a tangible medium or carrier wave, and is modulated or otherwise encoded in a carrier wave, which is tangible and transmitted according to any suitable transmission method.

[0116] Finally, the language used in the specification is selected primarily for readability and teaching purposes, and may not be selected to define or limit the subject matter of the invention. Therefore, the scope of the invention is not intended to be limited by this detailed description, but rather by any claims based on the application filed herewith. Therefore, the disclosure of the embodiments of the invention is intended to be illustrative and not limiting, and the scope of the invention is set forth by the following claims.

Claims

1. A method comprising: detecting, at a shopping cart system, initiation of a shopping session within the physical retail store by a customer of the physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors; During the shopping session, tracking contextual information associated with the shopping cart received by the set of sensors, the contextual information describing one or more locations of the shopping cart within the physical retail store, a state of the shopping cart, and a set of items within the shopping cart; identifying an opportunity for presenting content to the customer based at least in part on the contextual information associated with the shopping cart; responsive to identifying the opportunity for presenting the content to the customer, identifying a set of content items associated with one or more items within the physical retail store based at least in part on the contextual information associated with the shopping cart; generating a user interface including the set of content items; as well as The user interface is sent to a display area associated with the customer, wherein sending the user interface to the display area associated with the customer causes the user interface including the set of content items to be displayed to the customer.

2. The method of claim 1 , wherein identifying the opportunity for presenting the content to the customer comprises one or more of: detecting that the shopping cart has been stationary for at least a threshold amount of time, and detecting a particular item in the set of items within the shopping cart.

3. The method of claim 2, wherein based at least in part on the contextual information associated with the shopping cart, identifying the set of content items associated with the one or more items within the physical retail store comprises: when the shopping cart has been stationary for at least the threshold amount of time, comparing the location of the shopping cart at the one or more locations within the physical retail store to a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; identifying the one or more items within the physical retail store based at least in part on the comparison, wherein the one or more items are within a threshold distance of the location of the one or more locations of the shopping cart; as well as The set of content items is identified based at least in part on the identified one or more items.

4. A method according to claim 1, wherein the collection of content items includes one or more of the following: product information associated with an item among the one or more items, a promotion for the item among the one or more items, a coupon for the item among the one or more items, a set of instructions for using the item among the one or more items, a suggested use for the item among the one or more items, and an advertisement for the item among the one or more items.

5. The method according to claim 1, further comprising: detecting an event associated with the one or more items based at least in part on one or more of: the contextual information associated with the shopping cart and a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; determining whether to attribute the event to the set of content items included in the user interface that is sent to the display area associated with the customer, the determination being based at least in part on a first time when the user interface is sent to the display area associated with the customer and a second time when the event is detected; calculating a performance metric associated with content items in the set of content items based at least in part on determining whether the event is attributed to the set of content items included in the user interface that is sent to the display area associated with the customer; as well as The performance metric associated with the content item is communicated to one or more of: an entity associated with the physical retail store, a third party system associated with the content item, and an online concierge system associated with the physical retail store. 6 . The method of claim 5 , wherein the event is selected from the group consisting of: adding the one or more items to the shopping cart and purchasing the one or more items.

7. The method according to claim 5, further comprising: Information maintained in the online concierge system is updated based at least in part on the performance metric, wherein the updated information is associated with the one or more items.

8. A computer program product comprising a non-transitory computer-readable storage medium having encoded thereon instructions that, when executed by a processor, cause the processor to: detecting, at a shopping cart system, initiation of a shopping session within the physical retail store by a customer of the physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors; During the shopping session, tracking contextual information associated with the shopping cart received by the set of sensors, the contextual information describing one or more locations of the shopping cart within the physical retail store, a state of the shopping cart, and a set of items within the shopping cart; identifying an opportunity for presenting content to the customer based at least in part on the contextual information associated with the shopping cart; responsive to identifying the opportunity for presenting the content to the customer, identifying a set of content items associated with one or more items within the physical retail store based at least in part on the contextual information associated with the shopping cart; generating a user interface including the set of content items; as well as The user interface is sent to a display area associated with the customer.

9. The computer program product of claim 8, wherein identifying the opportunity for presenting the content to the customer comprises one or more of: detecting that the shopping cart has been stationary for at least a threshold amount of time, and detecting a particular item in the set of items within the shopping cart.

10. The computer program product of claim 9, wherein based at least in part on the contextual information associated with the shopping cart, identifying the set of content items associated with the one or more items within the physical retail store comprises: when the shopping cart has been stationary for at least the threshold amount of time, comparing the location of the shopping cart at the one or more locations within the physical retail store to a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; identifying the one or more items within the physical retail store based at least in part on the comparison, wherein the one or more items are within a threshold distance of the location of the one or more locations of the shopping cart; as well as The set of content items is identified based at least in part on the identified one or more items.

11. A computer program product according to claim 8, wherein the collection of content items includes one or more of the following: product information associated with an item among the one or more items, a promotion for the item among the one or more items, a coupon for the item among the one or more items, an instruction set for using the item among the one or more items, a suggested use for the item among the one or more items, and an advertisement for the item among the one or more items.

12. The computer program product of claim 8, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to: detecting an event associated with the one or more items based at least in part on one or more of: the contextual information associated with the shopping cart and a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; determining whether to attribute the event to the set of content items included in the user interface that is sent to the display area associated with the customer, the determination being based at least in part on a first time when the user interface is sent to the display area associated with the customer and a second time when the event is detected; calculating a performance metric associated with content items in the set of content items based at least in part on determining whether the event is attributed to the set of content items included in the user interface that is sent to the display area associated with the customer; as well as The performance metric associated with the content item is communicated to one or more of: an entity associated with the physical retail store, a third party system associated with the content item, and an online concierge system associated with the physical retail store.

13. The computer program product of claim 12, wherein the event is selected from the group consisting of: adding the one or more items to the shopping cart and purchasing the one or more items.

14. The computer program product of claim 12, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to: Information maintained in the online concierge system is updated based at least in part on the performance metric, wherein the updated information is associated with the one or more items.

15. A shopping cart system, comprising: processor; A non-transitory computer-readable storage medium storing instructions, which, when executed by the processor, cause the shopping cart system to perform actions, the actions comprising: detecting, at the shopping cart system, initiation of a shopping session within the physical retail store by a customer of the physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors; During the shopping session, tracking contextual information associated with the shopping cart received by the set of sensors, the contextual information describing one or more locations of the shopping cart within the physical retail store, a state of the shopping cart, and a set of items within the shopping cart; identifying an opportunity for presenting content to the customer based at least in part on the contextual information associated with the shopping cart; responsive to identifying the opportunity for presenting the content to the customer, identifying a set of content items associated with one or more items within the physical retail store based at least in part on the contextual information associated with the shopping cart; generating a user interface including the set of content items; and The user interface is sent to a display area associated with the customer.

16. The shopping cart system of claim 15, wherein identifying the opportunity for presenting the content to the customer comprises one or more of: detecting that the shopping cart has been stationary for at least a threshold amount of time, and detecting a particular item in the set of items within the shopping cart.

17. The shopping cart system of claim 16, wherein based at least in part on the contextual information associated with the shopping cart, identifying the set of content items associated with the one or more items within the physical retail store comprises: when the shopping cart has been stationary for at least the threshold amount of time, comparing the location of the shopping cart at the one or more locations within the physical retail store to a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; identifying the one or more items within the physical retail store based at least in part on the comparison, wherein the one or more items are within a threshold distance of the location of the one or more locations of the shopping cart; as well as The set of content items is identified based at least in part on the identified one or more items.

18. A shopping cart system according to claim 15, wherein the collection of content items includes one or more of the following: product information associated with items in the one or more items, promotions for the items in the one or more items, coupons for the items in the one or more items, a set of instructions for using the items in the one or more items, suggested uses for the items in the one or more items, and advertisements for the items in the one or more items.

19. The shopping cart system of claim 15, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the shopping cart system to perform actions comprising: detecting an event associated with the one or more items based at least in part on one or more of: the contextual information associated with the shopping cart and a floor plan associated with the physical retail store, wherein the floor plan indicates placement of each of a plurality of items within the physical retail store; determining whether to attribute the event to the set of content items included in the user interface that is sent to the display area associated with the customer, the determination being based at least in part on a first time when the user interface is sent to the display area associated with the customer and a second time when the event is detected; calculating a performance metric associated with content items in the set of content items based at least in part on determining whether the event is attributed to the set of content items included in the user interface that is sent to the display area associated with the customer; as well as The performance metric associated with the content item is communicated to one or more of: an entity associated with the physical retail store, a third party system associated with the content item, and an online concierge system associated with the physical retail store.

20. The shopping cart system of claim 19, wherein the event is selected from the group consisting of: adding the one or more items to the shopping cart and purchasing the one or more items.

Images