Shopping cart equipped with an on-board computing system that collects context data and displays information about items

Abstract

A shopping cart system detects the start of a shopping session by a customer, and the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors. During the shopping session, context information related to the shopping cart received by the sensors is tracked, and the context information describes one or more positions of the shopping cart within the retail store, the state of the shopping cart, and a set of items within the shopping cart. In response to identifying an opportunity to present content to the customer based on the context information, the system identifies a set of content items related to one or more items within the retail store based on the context information. The system generates a user interface including the set of content items and transmits the user interface to a display area relevant to the customer.

Description

Technical Field

[0001] The present disclosure generally relates to displaying information about items in a physical store, and more particularly to displaying information about items in a physical retail store based on context information related to a shopping cart system.

Background Art

[0002] This application claims the benefit of U.S. Utility Patent Application No. 17 / 824,696, filed May 25, 2022, the entire disclosure of which is incorporated herein by reference.

[0003] Since various customers can be interested in various items for reasons specific to each customer (e.g., personal preferences, quality, price, materials, sustainability, etc.), physical retail stores generally offer a wide range of items to their customers. For example, a customer of a physical grocery store who is on a limited budget and is a vegetarian may be interested in the store's own-brand products that do not contain meat, while another customer who enjoys barbecuing and often spends a large amount of money on food is less likely to be interested in the same products. To encourage customers to purchase new items that they may not be familiar with and to further purchase items they are already familiar with, physical retail stores can offer customers incentives to purchase various items (e.g., by periodically offering sales promotions or discounts on items). For example, every week, a physical retail store can offer special sales or discounts on various items and can change the samples or sales promotion displays featuring various items across the store.

[0004] However, despite the incentives offered by a physical retail store, customers who would otherwise be expected to purchase a particular item from that store may, for various reasons, not do so. In some cases, customers may refrain from purchasing an item due to lack of information about the item. For example, when a physical grocery store stocks foreign fruits, customers who enjoy the fruits but know nothing about them beyond the basic information provided by the grocery store, such as the item name and price, are less likely to purchase the fruits without additional information about them (e.g., what they taste like, how to prepare them, nutritional component information, etc.). Further, in the above example, even if most of the customers who encounter the fruits have access to a client device (e.g., a smartphone) that they could use to research the fruits, they may consider it too much trouble to do so, especially if they did not originally intend to purchase the fruits. Customers may also fail to purchase a particular item that they would otherwise have bought because they are unaware of its existence. For example, a customer who enjoys cookies and is on a diet may not purchase the sugar-free version of their favorite cookies if they are not aware that the new version is currently available. Additionally, customers may fail to purchase a particular item because they forget to do so during their shopping session. For example, a customer who purchases pasta may simply forget to buy a pasta sauce to go with the pasta if there is no reminder. SUMMARY OF THE INVENTION

[0005] Because various customers may be interested in various items for reasons specific to each customer, physical retail stores generally offer a wide range of items to those customers. To encourage those customers to purchase new items they may not be familiar with and to further purchase items they are already familiar with, physical retail stores may offer customers incentives (e.g., sales promotions or discounts) to purchase various items. However, despite these incentives, customers who would otherwise be expected to purchase specific items from a physical retail store may not do so for various reasons (e.g., when information about the item is not readily available, when they are not aware of the existence of the item, or when they simply forget to purchase those items).

[0006] According to one or more aspects of the present disclosure, information about items in a physical retail store is presented to a customer based on context information associated with a shopping cart system in order to encourage the customer to purchase new items they may not be familiar with and to further purchase items they are already familiar with. More specifically, a shopping cart system including a shopping cart, a processor, a memory, and a set of sensors detects the start of a shopping session by a customer in a physical retail store. The shopping cart system tracks context information associated with the shopping cart received by the sensors during the shopping session. The context information tracked by the shopping cart system describes one or more positions of the shopping cart in the physical retail store, the state of the shopping cart, and the set of items in the shopping cart, and the state of the shopping cart indicates whether the shopping cart is moving or stationary. The shopping cart system identifies an opportunity to present content to the customer based in part on the context information. The shopping cart system then identifies a set of content items associated with one or more items in the physical retail store based on the context information, generates a user interface including the set of content items, and transmits the user interface to a display area associated with the customer.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0008] The figures depict embodiments of the present disclosure for illustrative purposes only. Alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the present disclosure or the required benefits described herein.

Best Mode for Carrying Out the Invention

[0009] System Architecture FIG. 1 is a block diagram of a system environment 100 in which online systems such as a shopping cart system 150 and an online concierge system 102 operate, which will be further described below with respect to FIGS. 2 and 3. The system environment 100 shown in FIG. 1 includes one or more client devices 110, a network 120, one or more third-party systems 130, one or more physical retail stores 104, and an online concierge system 102. In alternative configurations, different and / or additional components may be included within the system environment 100. Additionally, in other embodiments, the online concierge system 102 may be replaced by an online system configured to retrieve content for display to a user and transmit the content to one or more client devices 110 for display.

[0010] Client device 110 is one or more computing devices capable of receiving user input and transmitting and / or receiving data via network 120. In one embodiment, client device 110 is a computer system such as a desktop or laptop computer. Alternatively, client device 110 can be a device having computer capabilities, such as a personal digital assistant (PDA), mobile phone, smartphone, or another suitable device. Client device 110 is configured to communicate via network 120. In one embodiment, client device 110 runs an application to enable a user of client device 110 to interact with the online concierge system 102. For example, client device 110 runs a customer mobile application 206 or a shopper mobile application 212, as further described below with respect to FIG. 2, to enable interaction between client device 110 and the online concierge system 102. As an additional example, client device 110 runs a browser application to enable interaction between client device 110 and the online concierge system 102 via network 120. In another embodiment, client device 110 interacts with the online concierge system 102 through an application programming interface (API) running on the native operating system of client device 110, such as IOS® or ANDROID™.

[0011] Client device 110 includes one or more processors 112 configured to control the operation of client device 110 by performing various functions. In various embodiments, client device 110 includes a memory 114 that includes a non-transitory storage medium on which instructions are encoded. When executed by processor 112, memory 114 may encode thereon instructions that cause processor 112 to execute customer mobile application 206 or shopping agent mobile application 212 to provide functions further described below with respect to FIG. 2.

[0012] Client device 110 is configured to communicate via a network 120 that may include any combination of local area and / or wide area networks, using both wired and / or wireless communication systems. In one embodiment, network 120 uses standard communication techniques and / or protocols. For example, network 120 includes communication links that use technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, 5G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), and the like. Examples of networking protocols used to communicate via network 120 include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged via network 120 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 120 may be encrypted using any suitable one or more techniques.

[0013] One or more third-party systems 130 may be coupled to the network 120 to communicate with the online concierge system 102, the client device 110, or the physical retail store 104. In one embodiment, the third-party system 130 is an application provider that communicates information describing an application for execution by the client device 110 or communicates data for use by an application running on the client device 110 to the client device 110. In other embodiments, the third-party system 130 provides content or other information for presentation via the client device 110. For example, the third-party system 130 stores one or more web pages and transmits the web pages to the client device 110 or the online concierge system 102. The third-party system 130 may also communicate information such as advertisements, content, or information about an application provided by the third-party system 130 to the online concierge system 102.

[0014] The online concierge system 102 includes one or more processors 142 configured to control the operation of the online concierge system 102 by performing various functions. In various embodiments, the online concierge system 102 includes a memory 144 that includes a non-transitory storage medium having instructions encoded thereon. The memory 144 may have instructions encoded thereon that, when executed by the processor 142, cause the processor 142 to perform functions further described below with respect to FIGS. 2 and 4-6 and that correspond to modules further described below with respect to FIG. 3. For example, the memory 144 may have instructions encoded thereon that, when executed by the processor 142, cause the processor 142 to display information about items in the physical store 104 based on context information related to the shopping cart system 150. Additionally, the online concierge system 102 includes a communication interface configured to connect the online concierge system 102 to one or more networks, such as network 120, or, in some cases, communicate with devices connected to the network (e.g., client device 110).

[0015] One or more physical retail stores 104 may be coupled to a network 120 to communicate various types of information to an online concierge system 102, a client device 110, or a third-party system 130. In some embodiments, a physical retail store 104 (e.g., a physical grocery store) communicates a shelf map associated with the physical retail store 104, where the shelf map indicates the placement of each item (e.g., each retail product) within the physical retail store 104, as well as the layout of the physical retail store 104. For example, a shelf map associated with a physical retail store 104 may indicate the placement of items on shelves, within display cases, within aisles, or within any other suitable organizing element used to organize items within the physical retail store 104. In this example, the shelf map may include the layout of the physical retail store 104 that describes the positions of these organizing elements relative to each other or relative to building elements (e.g., walls, doors, stairs, elevators, etc.) of the physical retail store 104. In some embodiments, the physical retail store 104 may also communicate updates to the shelf map of the physical retail store 104 to the online concierge system 102, the client device 110, and / or the third-party system 130 (e.g., periodically, as soon as the placement of items within the physical retail store 104 or the layout of the physical retail store 104 is updated, etc.). The shelf map associated with the physical retail store 104 may be used to identify one or more items within a threshold distance of a shopping cart 156 within the physical retail store 104 and / or to detect events within the physical retail store 104, as further described below.

[0016] The physical retail store 104 may include one or more shopping cart systems 150 capable of collecting information and transmitting and / or receiving data via the network 120. Each shopping cart system 150 within the physical retail store 104 may include one or more processors 152 configured to control the operation of the shopping cart system 150 by performing various functions. In various embodiments, each shopping cart system 150 may also include a memory 154 including a non-transitory storage medium having instructions encoded thereon. When executed by the processor 152, the memory 154 may encode thereon instructions corresponding to some of the modules further described below with respect to FIG. 3, causing the processor 152 to perform some of the functions further described below with respect to FIGS. 2 and 4-6.

[0017] Each shopping cart system 150 within the physical retail store 104 may also include a shopping cart 156 that can be used by a customer of the physical retail store 104 or a shopper for the online concierge system 102 to carry the items they collect during a shopping session. In some embodiments, the shopping cart 156 is a wheeled cart made of plastic, metal, or any other suitable material or combination of materials. In various embodiments, the shopping cart system 150 may alternatively include a shopping basket, a mobility scooter, a handcart, a utility cart, or any other object that can be used to carry items collected during a shopping session.

[0018] Each shopping cart system 150 within the physical retail store 104 may also include various sensors 158 for collecting context information related to a shopping cart 156 included in the shopping cart system 150. Context information related to the shopping cart 156 may describe one or more positions of the shopping cart 156 within the physical retail store 104, the state of the shopping cart 156, and / or a set of items within the shopping cart 156. The state of the shopping cart 156 may indicate whether the shopping cart 156 is moving or stationary. In some embodiments, the state of the shopping cart 156 may also or alternatively indicate whether the shopping cart 156 is moving at less than a threshold speed, whether the shopping cart 156 has moved less than a threshold distance within a specified amount of time, whether the shopping cart 156 has a reverse direction, the speed of the shopping cart 156, the orientation of the shopping cart 156, or any other suitable information describing the state of the shopping cart 156. In some embodiments, the sensors 158 may be mounted on the shopping cart 156 (e.g., along the perimeter of the shopping cart 156). In embodiments where the shopping cart system 150 includes objects other than the shopping cart 156 (e.g., a shopping basket, a mobility scooter, etc.), the sensors 158 can collect context information related to the object and may be mounted on the object (e.g., along the perimeter of the shopping basket).

[0019] In some embodiments, one or more sensors 158 included in the shopping cart system 150 may be capable of identifying physical objects within the physical retail store 104 (e.g., via machine vision, object recognition sensors 158, etc.). For example, the sensors 158 mounted on the shopping cart 156 may include one or more cameras (e.g., a video camera or a digital camera that captures still images) facing the interior of the shopping cart 156 that are capable of identifying items (e.g., retail products) added to the shopping cart 156. In the above example, the sensors 158 mounted on the shopping cart 156 may also include one or more cameras facing the outside of the shopping cart 156 that are capable of identifying items on the shelves, within display cases, within aisles, or within any other suitable organizing element used to organize items within the physical retail store 104. Additionally, in the above example, the cameras facing the outside of the shopping cart 156 may also be capable of identifying the organizing elements and building elements (e.g., walls, doors, stairs, elevators, etc.) of the physical retail store 104. In some embodiments, the sensors 158 have various capabilities to assist in identifying items, organizing elements, and / or building elements within the physical retail store 104. For example, if the sensors 158 of the shopping cart system 150 include one or more cameras, the cameras may have features such as face detection, text detection, infrared detection, night vision, motion activation, etc.

[0020] One or more sensors 158 included in the shopping cart system 150 may have an additional ability to collect context information related to a shopping cart 156 (or other object) included in the shopping cart system 150. In some embodiments, the sensor 158 may be capable of collecting the following context information related to the shopping cart 156 (or other object): location information (e.g., GPS coordinates), movement, proximity / distance (e.g., from items, organizing elements used to organize items, and / or building elements of the actual retail store 104), various wavelengths (e.g., visible light, infrared, etc.), color, sound, speed, weight, vibration, etc. For example, a GPS sensor 158 and a proximity sensor 158 (e.g., a laser or ultrasonic proximity sensor 158) attached to the shopping cart 156 may collect information describing the position of the shopping cart 156 relative to items, aisles, service counters, etc. within the actual retail store 104. In this example, the information describing the position of the shopping cart 156 may include the GPS coordinates related to the shopping cart 156, as well as the distance (e.g., in feet or meters), direction (e.g., north, south, east, or west), and altitude (e.g., in feet or meters) of each item, aisle, service counter, etc. relative to one or more proximity sensors 158 attached to the shopping cart 156.

[0021] The context information collected by one or more sensors 158 of each shopping cart system 150 within the physical retail store 104 can be associated with various types of data. Examples of data related to the information include the name of the physical retail store 104, the geographical location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, and the like. For example, if the physical retail store 104 is a grocery store belonging to a grocery store chain, the context information collected by the sensors 158 included in the shopping cart system 150 within the physical retail store 104 can be associated with data including the name identifying the chain and the geographical location (e.g., city and state) identifying the specific physical retail store 104. In the above example, the context information can also be associated with a timestamp indicating the time at which it was collected.

[0022] In some embodiments, the shopping cart system 150 can include a display area 159 (e.g., a screen attached to the shopping cart 156 included in the shopping cart system 150). For example, the display area 159 of the shopping cart system 150 can correspond to a touch screen display attached to the shopping cart 156 of the shopping cart system 150. The display area 159 can be a liquid crystal display (LCD), an in-plane switching liquid crystal display (IPS-LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or any other suitable display type. The display area 159 of the shopping cart system 150 can be used to display a user interface to a customer of the physical retail store 104, as further described below.

[0023] In some embodiments, a customer of the physical retail store 104 may interact with the shopping cart system 150. The customer may interact with the shopping cart system 150 to start a shopping session and / or authenticate the user, as further described below. In embodiments where the shopping cart system 150 includes a display area 159, the customer may interact with the shopping cart system 150 via the display area 159. For example, if the display area 159 of the shopping cart system 150 is a touch screen display, the customer can start a shopping session by touching a button displayed within the display area 159, in which case touching the button sends a request to start a shopping session to the shopping cart system 150. The display area 159 may include one or more built-in accessories, such as speakers, microphones, etc. Although not shown in FIG. 1, in some embodiments, the shopping cart system 150 may also include one or more accessories (e.g., speakers, microphones, buttons, dials, knobs, barcode scanners, cameras, etc.) coupled to a shopping cart 156 included in the shopping cart system 150 that a customer of the physical retail store 104 can use to interact with the shopping cart system 150. For example, if the display area 159 mounted on the shopping cart 156 of the shopping cart system 150 is not a touch screen display, the customer can start a shopping session by pressing a button located beside the display area 159, in which case pressing the button sends a request to start a shopping session to the shopping cart system 150. In this example, if the customer presses the button, confirmation of the request can then be communicated to the customer (e.g., visually via the display area 159 and aurally via a speaker mounted on the shopping cart 156).

[0024] One or more of the client device 110, third-party system 130, online concierge system 102, or shopping cart system 150 may be a dedicated computing device configured to perform specific functions as further described below with respect to FIGS. 2-6, and may include specific computing components such as a processor, memory, communication interface, etc.

[0025] System Overview FIG. 2 illustrates an environment 200 of an online platform such as a shopping cart system 150 and an online concierge system 102 according to one or more embodiments. These figures use like reference numerals to identify like elements. Characters following a reference numeral, such as “104a”, indicate that the text specifically refers to the element having that particular reference numeral. A reference numeral in the text that is not followed by a character, such as “104”, refers to any or all of the elements in the drawing having that reference numeral. For example, “104” in the text may refer to reference numerals “104a”, “104b”, and / or “104c” in the drawing.

[0026] The environment 200 includes an online concierge system 102. The online concierge system 102 is configured to receive orders from one or more customers 204 (only one is shown for simplicity). The order specifies a list of items (items or products) to be delivered to the customer 204, the location where the items are to be delivered, and the time window within which the items are to be delivered. In some embodiments, the order specifies one or more retail stores from which the items are to be purchased. The customer 204 may use a customer mobile application (CMA) 206 configured to communicate with the online concierge system 102 to place an order.

[0027] The online concierge system 102 is configured to send orders received from a customer 204 to one or more shopper agents 208. The shopper agents 208 can be a person (e.g., a contractor, an employee, etc.), an entity, or an autonomous device (e.g., a robot) enabled to fulfill the orders received by the online concierge system 102. The shopper agents 208 can travel between a physical retail store 104 and a delivery location (e.g., the customer's home or office) by car, truck, bicycle, scooter, on foot, or via any other mode of transportation. In some embodiments, the delivery can be partially or fully automated, for example, using a self-driving vehicle. The environment 200 also includes three physical retail stores 104a, 104b, and 104c (only three are shown for simplicity, but the environment 200 can include hundreds of physical retail stores 104). The physical retail stores 104 can be physical retail stores such as grocery stores, discount stores, department stores, or non-public physical retail stores 104 that store items that can be collected and delivered to the customer 204. Each shopper agent 208 fulfills the orders received from the online concierge system 102 at one or more physical retail stores 104, delivers the orders to the customer 204, or performs both fulfillment and delivery. In one embodiment, the shopper agent 208 utilizes a shopper agent mobile application 212 configured to interact with the online concierge system 102.

[0028] One or more physical retail stores 104 within the environment 200 may also include one or more shopping cart systems 150 (only one is shown for simplicity). The shopping cart system 150 may be configured to communicate various types of information to the online concierge system 102 and / or the CMA 206. Similarly, the online concierge system 102 and / or the CMA 206 may be configured to communicate various types of information to the shopping cart system 150. In various embodiments, the shopping cart system 150 may send a user interface to the display area of the client device 110 associated with the customer 204 (e.g., via a push-type notification through the CMA 206). In some embodiments, the online concierge system 102 may send a user interface to the shopping cart system 150 for display to the customer 204 of the physical retail store 104.

[0029] In some embodiments, the shopping cart system 150 may communicate to the online concierge system 102 information that describes performance criteria associated with content items presented to the customer 204 of the physical retail store 104. Performance criteria associated with a content item (e.g., a conversion rate) may indicate whether one or more customers 204 to whom the content item was presented subsequently performed an activity associated with the content item. For example, if the content item is associated with a particular item within the physical retail store 104, the performance criteria associated with the content item may indicate the percentage of customers 204 to whom the content item was presented who subsequently visited the aisle where the item is located, placed the item in their shopping cart 156, and / or purchased the item.

[0030] The shopping cart system 150 within the physical retail store 104 may also communicate context information collected by one or more sensors 158 included in the shopping cart system 150 to the online concierge system 102 and / or the CMA 206. For example, assume that a customer 204 interacts with the shopping cart system 150 to log in to an account associated with the customer 204 maintained in the online concierge system 102. In this example, the shopping cart system 150 may then communicate the path of the customer 204 within the physical retail store 104 based on information describing multiple positions of the shopping cart 156 included in the shopping cart system 150 within the physical retail store 104 during the shopping session. In the above example, the shopping cart system 150 may also communicate information describing the location within the physical retail store 104 where the shopping cart 156 stopped, and the items that the customer 204 added to the shopping cart 156 and then purchased during the shopping session.

[0031] FIG. 3 is a diagram of a shopping cart system 150 and an online concierge system 102 according to one or more embodiments. In various embodiments, the shopping cart system 150 and / or the online concierge system 102 may include different or additional modules than those described with respect to FIG. 3. Further, in some embodiments, the shopping cart system 150 and / or the online concierge system 102 may include fewer modules than those described with respect to FIG. 3.

[0032] The online concierge system 102 includes an inventory management engine 302 that interacts with the inventory systems associated with each physical retail store 104. In one embodiment, the inventory management engine 302 requests and receives inventory information maintained by the physical retail store 104. The inventory management engine 302 can also receive inventory information from one or more shopping cart systems 150 included within the physical retail store 104, in which case the inventory information is collected by one or more sensors 158 included within the shopping cart system 150. The inventory of each physical retail store 104 is unique and can change over time. The inventory management engine 302 monitors changes to the inventory for each participating physical retail store 104. The inventory management engine 302 is also configured to store inventory records in an inventory database 304. The inventory database 304 can store the information in separate records, one for each participating physical retail store 104, or can integrate or combine the inventory information into a unified record. The inventory information includes the attributes of the items, including both quantitative and qualitative information about the items, such as size, color, weight, SKU, serial number, etc. In one embodiment, the inventory database 304 also stores any purchase rules associated with each item, if they exist. For example, age-restricted items such as alcohol and tobacco are flagged accordingly within the inventory database 304. Additional inventory information useful for predicting the availability of items can be stored within the inventory database 304. For example, for each item-store combination (a particular item at a particular physical retail store 104), the inventory database 304 can store the time the item was last found, the time the item was last not found (e.g., the shopper 208 was looking for the item but could not find it), the percentage of times the item was found, and the popularity of the item.

[0033] For each item, the inventory database 304 identifies one or more attributes of the item and corresponding values for each attribute of the item. For example, the inventory database 304 includes an entry for each item provided by the physical retail store 104, in which case the entry for the item includes an item identifier that uniquely identifies the item. The entry includes different fields, each field corresponding to an attribute of the item. The fields of the entry include values for the attributes corresponding to the fields, enabling the inventory database 304 to maintain values of different attributes for different items.

[0034] In various embodiments, the inventory management engine 302 maintains a classification of items offered for purchase by one or more physical retail stores 104. For example, the inventory management engine 302 receives from the physical retail store 104 an item catalog that identifies items offered for purchase by the physical retail store 104. From the item catalog, the inventory management engine 302 determines a classification of the items offered by the physical retail store 104, where different levels of classification provide different levels of specificity regarding the items included in the level. In various embodiments, the classification identifies categories and associates one or more specific items with the categories. For example, a category identifies "milk" and the classification associates identifiers of different milk items (e.g., milk offered by different brands, milk having one or more different attributes, etc.) with the category. In this way, the classification maintains the relationship between the category and the specific items offered by the physical retail store 104 that match the category. In some embodiments, different levels of classification identify items with different levels of specificity based on any suitable attribute or combination of attributes of the items. For example, different levels of classification specify different combinations of attributes of the items, and thus, items at lower levels of the hierarchical classification have a greater number of attributes corresponding to higher specificity within the category, and items at higher levels of the hierarchical classification have a fewer number of attributes corresponding to lower specificity within the category. In various embodiments, higher-level classifications include less detail regarding the items and thus a greater number of items are included at the higher level (e.g., the higher level includes a greater number of items that satisfy a broader category). Similarly, lower-level classifications include more detail regarding the items and thus a fewer number of items are included at the lower level (e.g., the lower level includes a fewer number of items that satisfy a more specific category). The classification may be received from the physical retail store 104 in various embodiments.In other embodiments, the inventory management engine 302 applies the trained classification model to the item catalog received from the physical retail store 104 to include different items at the levels of the classification method, and thus, the application of the trained classification model associates a particular item with a category corresponding to a level within the classification method.

[0035] The inventory information provided by the inventory management engine 302 may supplement the training dataset 320. The inventory information provided by the inventory management engine 302 may not necessarily include information regarding the results of fulfilling the shipping instructions related to the item, although the data within the training dataset 320 is constructed to include the results of fulfilling the shipping instructions (e.g., whether the items within an order were collected or not collected).

[0036] In some embodiments, the online concierge system 102 includes an order fulfillment engine 306 configured to synthesize an order interface for display to each customer 204 (e.g., via the customer mobile application 206). The order fulfillment engine 306 is also configured to access the inventory database 304 to determine which items are available at which physical retail stores 104. The order fulfillment engine 306 may supplement the item availability information from the inventory database 304 with item availability information predicted by the machine-learned item availability model 316. The order fulfillment engine 306 determines the selling price for each item ordered by the customer 204. The price set by the order fulfillment engine 306 may or may not match the in-store price determined by the retail store (the price that the customer 204 and the shopper 208 will pay at the physical retail store 104). The order fulfillment engine 306 also facilitates the transactions associated with each order. In one embodiment, the order fulfillment engine 306 charges the payment means associated with the customer 204 when the customer 204 places an order. The order fulfillment engine 306 may transmit payment information to an external payment gateway or payment processor. The order fulfillment engine 306 stores the payment and transaction information associated with each order in the transaction record database 308.

[0037] In various embodiments, the order fulfillment engine 306 generates a search interface and transmits it to the customer's client device 110 for display via the customer mobile application 206. The order fulfillment engine 306 receives a query from the customer 204 that includes one or more conditions and retrieves items that satisfy the query, such as items having descriptive information that matches at least a portion of the query. In various embodiments, the order fulfillment engine 306 utilizes item embeddings for items to retrieve items based on the received query. For example, the order fulfillment engine 306 generates an embedding for the query and determines a measure of similarity between the embedding for the query and the item embeddings for various items included within the inventory database 304.

[0038] In some embodiments, the order fulfillment engine 306 also shares order details with the physical retail store 104. For example, after successfully fulfilling an order, the order fulfillment engine 306 may transmit an order summary to the appropriate physical retail store 104. The order details may indicate the purchased items, the total value of the items, and, optionally, the identification of the shopper 208 and the customer 204 associated with the order. In one embodiment, the order fulfillment engine 306 pushes transactions and / or order details to the retail store system asynchronously. This may be accomplished via the use of webhooks that enable a programmatic or system-driven transmission between web applications of information. In another embodiment, the retail store system may be configured to periodically poll the order fulfillment engine 306, which provides details of all orders processed since the last request.

[0039] The order fulfillment engine 306 can interact with a shopper management engine 310 that manages communication with and utilization of the shopper 208. In one embodiment, the shopper management engine 310 receives a new order from the order fulfillment engine 306. The shopper management engine 310 identifies an appropriate physical retailer 104 for fulfilling the order based on one or more parameters such as the probability of item availability determined by the machine-learned item availability model 316, the content of the order, the inventory of the physical retailer 104, and the proximity of the physical retailer 104 to the delivery location. The shopper management engine 310 then identifies one or more appropriate shoppers 208 for fulfilling the order based on one or more parameters such as the proximity of the shopper to the appropriate physical retailer 104 (and / or the customer 204), his / her level of familiarity with that particular physical retailer 104, etc. Additionally, the shopper management engine 310 accesses a shopper database 312 that stores information describing him / her, such as the name, gender, rating, previous shopping history, etc. of each shopper 208.

[0040] As part of fulfilling the order, the order fulfillment engine 306 and / or the shopper management engine 310 can access a customer database 314 that stores information describing each customer 204. In some embodiments, the information stored within the customer database 314 that describes each customer 204 is included within the user profile of the customer 204. This information can include the name, geographical location (e.g., home or business address), age, gender, shopping preferences, favorite items, dislikes, stored payment methods, profile photo, etc. of each customer.

[0041] The user profile of customer 204 stored within customer database 314 may also include historical information related to customer 204. In some embodiments, the historical information related to customer 204 may describe one or more shopping trips of customer 204 to physical retail store 104. In such embodiments, the historical information may be collected by one or more sensors 158 included within shopping cart system 150 within physical retail store 104. For example, the historical information related to customer 204 that describes a shopping trip to physical retail store 104 may include information identifying physical retail store 104, the time and duration of a shopping session at physical retail store 104, the path within physical retail store 104 traversed by customer 204, the amount of time customer 204 stopped at a particular location within physical retail store 104, information describing each location at which customer 204 stopped, etc. In this example, the historical information related to customer 204 may also include information identifying one or more content items related to items within physical retail store 104 that were sent for display to customer 204 while at physical retail store 104, the time at which each content item was sent, information identifying one or more items related to each content item. Additionally, in the above example, the historical information related to customer 204 may describe interactions of customer 204 with items at physical retail store 104 (e.g., picking up an item, placing an item within shopping cart 156, purchasing an item, etc.) and the time associated with each interaction.

[0042] In some embodiments, the historical information related to customer 204 included within the user profile of customer 204 may also describe activities performed by customer 204 within the online concierge system 102. For example, the historical information related to customer 204 may describe orders placed by customer 204 using the online concierge system 102 (e.g., a description of the items included within each order, the time at which each order was placed, information identifying the physical retail store 104 at which each order was placed, etc.). In this example, the historical information related to customer 204 may also indicate whether customer 204 viewed information related to an item (e.g., product information, ingredient list, etc.). Additionally, in the above example, the historical information related to customer 204 may also describe one or more content items transmitted by the online concierge system 102 for display to customer 204 (e.g., information identifying each content item, the time at which each content item was sent, information indicating whether customer 204 interacted with each content item, information identifying one or more items related to each content item).

[0043] In some embodiments, the user profile information related to customer 204 may be stored in multiple databases. For example, in addition to the customer database 314, the historical information related to customer 204 may be stored in the transaction record database 308 and / or the training dataset 320. In such embodiments, the information included within the user profile of customer 204 may be identified based on information identifying customer 204 related to the information (e.g., the username or other identifier of customer 204). Further, in some embodiments, the user profile information related to customer 204 may also be stored within one or more databases within the shopping cart system 150 (e.g., within the event database 324, within the context database 330, and / or within the attribute database 338), as further described below.

[0044] In various embodiments, the order fulfillment engine 306 determines whether to delay the display of an order received for fulfillment to a shopper 208 by a time interval. In response to a determination that the display of the received order should be delayed by a time interval, the order fulfillment engine 306 evaluates subsequent orders received during that time interval for inclusion in one or more batches that also include the received order. After the time interval, the order fulfillment engine 306 displays the order to one or more shoppers 208 via the shopper mobile application 212. If the order fulfillment engine 306 generates one or more batches that include the received order and one or more subsequent orders received during the time interval, the batches are displayed to one or more shoppers 208 via the shopper mobile application 212.

[0045] Shopping Cart System Components The shopping cart system 150 includes an event detection engine 322 that detects various types of events related to a customer 204 within a physical retail store 104. Examples of events that can be detected by the event detection engine 322 include the start of a shopping session within the physical retail store 104 by a customer 204 of the physical retail store 104, stopping the shopping cart 156 within a threshold distance of an item within the physical retail store 104 for at least a threshold amount of time, moving the shopping cart 156 within a threshold distance of an item within the physical retail store 104, picking up an item from a specific location (e.g., a sales promotion display area) within the physical retail store 104, adding an item to the shopping cart 156, purchasing an item from the physical retail store 104, completing a shopping session, or any other type of event that can be associated with the customer 204 within the physical retail store 104. For example, the event detection engine 322 can detect the start of a shopping session by a customer 204 of the physical retail store 204 as soon as it receives an indication from one or more sensors 158 (e.g., a motion or speed sensor 158) of the shopping cart system 150, or as soon as it receives one or more interactions by the customer 204 with a display area 159 and / or one or more accessories (e.g., buttons, dials, knobs, microphones, barcode scanners, cameras, etc.) included within the shopping cart system 150 that correspond to a request to start a shopping session. Similarly, the event detection engine 322 can detect the completion of a shopping session as soon as it receives no indication from one or more sensors 158 (e.g., a motion or speed sensor 158) included within the shopping cart system 150 for at least a threshold amount of time, or as soon as it receives one or more interactions by the customer 204 with a display area 159 and / or one or more accessories included within the shopping cart system 150 that correspond to a request to complete a shopping session.In an embodiment where an event detected by the event detection engine 322 corresponds to stopping the shopping cart 156 within a threshold distance of an item within the physical retail store 104 for at least a threshold amount of time, the threshold amount of time and / or distance can be determined by a machine-learned model (e.g., a model trained based on anonymized aggregated data that describes the behavior of customers associated with the shopping cart 156 when determining whether to purchase an item).

[0046] In some embodiments, a shopping session may be initiated upon authentication of customer 204. In such embodiments, information provided by customer 204 may be used by event detection engine 322 and / or online concierge system 102 to authenticate customer 204. Examples of such information include a combination of username and password, biometric information (e.g., fingerprint), or any other suitable information that may be used to authenticate customer 204. For example, information from customer 204 corresponding to login credentials for online concierge system 102 may be received by event detection engine 322 via touch screen display area 159 included in shopping cart system 150. In this example, event detection engine 322 may communicate the login credentials to online concierge system 102, which can authenticate customer 204 if the credentials match those of customer 204 in online concierge system 102 stored in customer database 314. Alternatively, in the above example, event detection engine 322 may access customer database 314 of online concierge system 102 and authenticate customer 204 if the credentials match those of customer 204 in online concierge system 102 stored in customer database 314. As an additional example, a quick response (QR) code that uniquely identifies customer 204 within online concierge system 102 may be generated by online concierge system 102 and displayed on client device 110 associated with customer 204 (e.g., via CMA 206). In this example, customer 204 may be authenticated if a camera included in shopping cart system 150 scans the QR code and event detection engine 322 then communicates the QR code to online concierge system 102, which authenticates customer 204. In embodiments where the shopping session is initiated upon authentication of customer 204, the shopping session may be linked to the user profile of customer 204 upon authentication of customer 204.For example, when customer 204 is authenticated, information describing the shopping session can be stored in an event database 324 (described below) and can be included in the user profile information associated with customer 204.

[0047] Event detection engine 322 can detect events associated with customer 204 in physical retail store 104 based on context information associated with shopping cart 156 tracked by context tracking engine 328 (described below) and / or based on a shelf map associated with physical retail store 104 stored in shelf map database 326 (also described below). For example, assume that a particular item is located at two different locations within physical retail store 104, one location being its normal location in an aisle and the other location being a sales promotion display area at the end of a different aisle. In this example, event detection engine 322 can detect an event corresponding to adding an item to shopping cart 156 included in shopping cart system 150, in which case event detection engine 322 can also detect the location at which the item was added to shopping cart 156 (e.g., based on the shelf map associated with physical retail store 104). As an additional example, event detection engine 322 can detect the completion of a shopping session based on context information tracked by context tracking engine 328 indicating that shopping cart 156 included in shopping cart system 150 is within threshold processing of a shopping cart return area or that shopping cart 156 has passed through a checkout queue within physical retail store 104.

[0048] The event detection engine 322 may store information describing one or more events associated with one or more customers 204 of the physical retail store 104. In some embodiments, this information may be stored within the event database 324 of the shopping cart system 150. In various embodiments, this information may also or alternatively be communicated to the online concierge system 102 and stored therein (e.g., within the transaction record database 308 and / or the training data set 320). The event detection engine 322 may store information describing the event, along with information identifying the customer 204 associated with the event (e.g., based on login credential information provided by the customer 204), information identifying the physical retail store 104 where the event was detected, the time at which the event was detected, the location within the physical retail store 104 where the event was detected, or any other information associated with the event. For example, assume that the event detection engine 322 detects that a customer 204 added an item to a shopping cart 156 included in the shopping cart system 150 during shopping at the physical retail store 104. In this example, information describing the event, such as the type of the event (i.e., adding an item to the shopping cart 156) and information describing the item (e.g., the type of the item, the size of the item, the brand of the item, etc.), may be stored (e.g., within the event database 324, the transaction record database 308, and / or the training data set 320). In the above example, the information describing the event may be stored together with information identifying the physical retail store 104 (e.g., its name and address), the time at which the item was added to the shopping cart 156, the aisle within the physical retail store 104 where the customer 204 added the item to the shopping cart 156, and the username associated with the customer 204 within the online concierge system 102.

[0049] In some embodiments, the shopping cart system 150 further includes a shelf map database 326. The shelf map database 326 stores one or more shelf maps, each of which is associated with a physical retail store 104 and describes the arrangement of each of a plurality of items within the physical retail store 104. The shelf map associated with the physical retail store 104 may indicate the arrangement of items on a shelf, within a display area, within an aisle, or within any other suitable organizing element used to organize items within the physical retail store 104. The shelf map may include the layout of the physical retail store 104 that describes the positions of the organizing elements relative to each other and relative to the building elements (e.g., walls, doors, stairs, elevators, etc.) of the physical retail store 104. In some embodiments, the shelf map of the physical retail store 104 stored within the shelf map database 326 may be updated (e.g., periodically, as soon as the arrangement of items within the physical retail store 104 or the layout of the physical retail store 104 is updated). In some embodiments, the shelf map database 326 may also or alternatively be included within the online concierge system 102. In such embodiments, the shelf map of the physical retail store 104 may be updated by the online concierge system 102 as soon as information from the physical retail store 104 that describes the update is received.

[0050] The shopping cart system 150 further includes a context tracking engine 328. The context tracking engine 328 tracks context information related to a shopping cart 156 (or a similar object used to carry items collected during a shopping session) included within the shopping cart system 150, received by one or more sensors 158 included within the shopping cart system 150. As described above, one or more sensors 158 included within the shopping cart system 150 may be capable of identifying physical objects (e.g., items, organizing elements, and / or building elements) within the physical retail store 104. Also as described above, the context information related to the shopping cart 156 may describe one or more positions of the shopping cart 156 within the physical retail store 104 and / or a set of items within the shopping cart 156. For example, the context information related to the shopping cart 156 that describes the position of the shopping cart 156 within the physical retail store 104 may include GPS coordinates collected by a GPS sensor 158 coupled to the shopping cart 156. In this example, the context information collected by one or more proximity sensors 158 coupled to the shopping cart 156 may also include the distance (e.g., in feet or meters), direction (e.g., north, south, east, or west), and altitude (e.g., in feet or meters) of one or more items, aisles, service counters, etc. within the physical retail store 104 relative to the shopping cart 156. Continuing with this example, the context information collected by a sensor 158 (e.g., a camera) mounted on the shopping cart 156 and facing the interior of the shopping cart 156 may also include information identifying one or more items (e.g., retail items) within the shopping cart 156.

[0051] In some embodiments, the context tracking engine 328 can track context information related to the shopping cart 156 within the physical retail store 104 by comparing information collected by one or more sensors 158 included in the shopping cart system 150 with a shelf map associated with the physical retail store 104. For example, if one or more sensors 158 included in the shopping cart system 150 correspond to one or more cameras, the context tracking engine 328 can compare video or image data collected by the cameras with a shelf map associated with the physical retail store 104 and, based on the comparison, determine the position of the shopping cart 156. In this example, if the items identified in the video or image data and their placement within the physical retail store 104 have at least a threshold measure of similarity to a portion of the shelf map, the context tracking engine 328 can determine that the shopping cart 156 is at a position within the physical retail store 104 corresponding to that portion of the shelf map.

[0052] The context information related to the shopping cart 156 tracked by the context tracking engine 328 can also describe the state of the shopping cart 156. As described above, the state of the shopping cart 156 can indicate whether the shopping cart 156 is moving or stationary. In some embodiments, the state of the shopping cart 156 can also indicate whether the shopping cart 156 is moving at a speed less than a threshold speed, whether the shopping cart 156 has moved a distance less than a threshold distance within a specified amount of time, whether the shopping cart 156 has a reverse direction, the speed of the shopping cart 156, the orientation of the shopping cart 156, or any other suitable information that describes the state of the shopping cart 156.

[0053] The context information tracked by the context tracking engine 328 can be associated with various types of data. Examples of data that can be associated with the context information include the name of the physical retail store 104 where the context information was collected, the geographical location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, information (e.g., time) describing the shopping session during which the context information was collected, information identifying the customer 204 associated with the shopping session (e.g., based on the information provided to authenticate the customer 204), and the like. For example, if the physical retail store 104 is a grocery store belonging to a grocery store chain, the context information tracked by the context tracking engine 328 can be associated with data including the name identifying the chain and the geographical location (e.g., city and state) identifying the specific physical retail store 104. In the above example, the context information can also be associated with a timestamp indicating the time at which it was collected by the sensor 158 included in the shopping cart system 150.

[0054] In some embodiments, the context tracking engine 328 can store the context information it tracks and / or the data associated with the context information in the context database 330. In embodiments where the context information stored in the context database 330 includes information identifying the customer 204 associated with the context information, this information can be included in the user profile associated with the customer 204. For example, if information uniquely identifying the customer 204 is provided as soon as the customer 204 starts a shopping session and the information is used to authenticate the customer 204, the information identifying the customer 204 (e.g., username) can be stored in the context database 330 together with the context information tracked by the context tracking engine 328 during the shopping session. In this example, the context information can then be retrieved together with other user profile information associated with the customer 204.

[0055] In various embodiments, the context tracking engine 328 may also communicate context information it tracks and / or data related to the context information to the online concierge system 102. In such embodiments, the online concierge system 102 may use this information for various purposes. In some embodiments, the online concierge system 102 may use this information to customize the experience of a customer 204 of the online concierge system 102. For example, if a customer 204 of the physical retail store 104 is also a customer 204 of the online concierge system 102, the online concierge system 102 may customize the online version of the physical retail store 104 based on the paths typically traversed by the customer 204 through the physical retail store 104.

[0056] The shopping cart system 150 further includes a user interface engine 332. The user interface engine 332 identifies an opportunity to present content to a customer 204 of the physical store 104, based in part on context information related to the shopping cart 156 included in the shopping cart system 150. In various embodiments, the user interface engine 332 may receive context information related to the shopping cart 156 from the context tracking engine 328, although in other embodiments, the user interface engine 332 may retrieve the context information from the context database 330. In some embodiments, the user interface engine 332 may identify an opportunity to present content to the customer 204 as soon as it detects that the shopping cart 156 has been stationary for at least a threshold amount of time. For example, the user interface engine 332 may receive from the context tracking engine 328 context information that describes a state in which the shopping cart 156 is being used during a shopping session related to a customer 204 of the physical store 104. In this example, if the context information indicates that the shopping cart 156 has been stationary for at least a threshold amount of time, the user interface engine 332 may identify an opportunity to present content to the customer 204. In some embodiments, the user interface engine 332 may also or alternatively identify an opportunity to present content to the customer 204 as soon as it detects one or more specific items within the shopping cart 156. For example, if the context information indicates that an item has been added to the shopping cart 156, the user interface engine 332 may identify an opportunity to present content related to the item to the customer 204.

[0057] The user interface engine 332 also identifies a set of content items associated with one or more items within the physical retail store 104 for inclusion in the user interface that is to be sent for display to the customer 204 of the physical retail store 104. In some embodiments, the user interface engine 332 may identify the set of content items using a machine-learned model. Further, the user interface engine 332 may identify the set of content items based in part on context information associated with the shopping cart 156 included in the shopping cart system 150. In embodiments where the user interface engine 332 identifies an opportunity to present content to the customer 204 as soon as it detects that the shopping cart 156 has been in a stationary state for at least a threshold amount of time, the set of content items identified by the user interface engine 332 may be associated with one or more items within a threshold distance of the shopping cart 156. For example, the user interface engine 332 may access context information stored in the context database 330 that describes the position of the shopping cart 156 within the physical retail store 104 while the shopping cart 156 has been in a stationary state for at least a threshold amount of time. In this example, the user interface engine 332 may then access the shelf map database 326 to retrieve the shelf map associated with the physical retail store 104 and compare the position of the shopping cart 156 to the shelf map. Continuing with this example, the user interface engine 332 may then identify one or more items within the physical retail store 104 within a threshold distance of the position of the shopping cart 156 and access the content database 334 (described below) to identify one or more content items associated with the identified items. In embodiments where the user interface engine 332 identifies an opportunity to present content to the customer 204 as soon as it detects one or more specific items within the shopping cart 156, the set of content items identified by the user interface engine 332 may be associated with the items.For example, as soon as it is detected (e.g., by the event detection engine 322) that an item has been added to the shopping cart 156, the user interface engine 332 can access the context information stored in the context database 330 that describes one or more items within the shopping cart 156. In this example, if the context information indicates that the item within the shopping cart 156 is spaghetti, the user interface engine 332 can identify a set of content items related to spaghetti.

[0058] In some embodiments, the user interface engine 332 may also identify a set of content items related to one or more items within the physical retail store 104 for inclusion in a user interface that is to be sent for display to the customer 204 of the physical retail store 104, based in part on user profile information related to the customer 204. In such embodiments, the user interface engine 332 may identify the set of content items based on the customer's geographical location, age, gender, shopping preferences, favorite items, dislikes, historical information related to the customer 204 (e.g., within the physical retail store 104 or the online concierge system 102), or any other information included within the customer's user profile. For example, assume that the user profile of the customer 204 indicates that the customer 204 is allergic to nuts and thus dislikes them. In this example, one or more sensors 158 of the shopping cart system 150 identify items added to the shopping cart 156 being used by the customer 204, and if an item includes a component corresponding to nuts, a content item including a note regarding the content of the item may be identified by the user interface engine 332. As an additional example, assume that historical information related to the customer 204 indicates that the customer 204 typically purchases certain items according to a consistent cadence, such as during each shopping session or is likely to purchase an item during the current shopping session. In this example, context information related to the shopping cart 156 included in the shopping cart system 150 being used by the customer 204, if the shopping cart 156 is in the checkout queue and an item is not within the shopping cart 156, a content item that reminds the customer 204 about the item may be identified by the user interface engine 332.In an embodiment where the user interface engine 332 identifies content items based on user profile information associated with customer 204, the user interface engine 332 can access user profile information stored within the online concierge system 102 (e.g., within the customer database 314, the transaction record database 308, and / or the training data set 320), and / or the online concierge system 102 can communicate the user profile information to the user interface engine 332.

[0059] In various embodiments, the user interface engine 332 can also identify a set of content items related to one or more items within the physical retail store 104 for inclusion in a user interface that is to be sent for display to customer 204 of the physical retail store 104, based in part on the time of the shopping session. In such embodiments, the user interface engine 332 can identify the content items included in the set based on seasonal demand for the items related to the content items, day of the week, time of day, etc. For example, during an early morning shopping session, the user interface engine 332 can identify one or more content items related to breakfast items (e.g., pastries, coffee, tea, etc.). As an additional example, during the month prior to Thanksgiving, the user interface engine 332 can identify content items related to items for Thanksgiving (e.g., Thanksgiving recipes, sales promotions for ham or gravy, etc.).

[0060] In some embodiments, the user interface engine 332 may also identify a set of content items related to one or more items within the physical retail store 104 for inclusion in a user interface that will be transmitted for display to a customer 204 of the physical retail store 104, based at least in part on information related to a set of content items and / or information related to one or more items related to the content items. In various embodiments, each content item may be associated with a value that the user interface engine 332 may use to identify a set of content items for inclusion in the user interface such that content items related to higher values are more likely to be identified by the user interface engine 332 for inclusion in the user interface than content items related to lower values. For example, if a content item corresponds to an advertisement, a content item related to a higher bid amount may be more likely to be identified by the user interface engine 332 for inclusion in the user interface than a content item related to a lower bid amount. As an additional example, content items related to higher priced items and / or items with higher inventory may be more likely to be identified by the user interface engine 332 for inclusion in the user interface than content items related to lower priced items and / or items with lower inventory.

[0061] In some embodiments, the user interface engine 332 may also identify a set of content items related to one or more items within the physical retail store 104 for inclusion in a user interface that will be transmitted for display to the customers 204 of the physical retail store 104, based in part on information received from the online concierge system 102. For example, assume that the online concierge system 102 presents various content items (e.g., articles, recipes, etc.) to its customers 204 via the CMA 206. In this example, the online concierge system 102 may communicate to the user interface engine 332 information identifying certain content items that are popular among the customers 204 of the online concierge system 102. In the above example, the popular content items may be identified by the user interface engine 332 for inclusion in a user interface that will be transmitted for display to the customers 204 of the physical retail store 104 (e.g., if context information related to the shopping cart 156 used by the customer 204 indicates that the shopping cart 156 is within a threshold distance of one or more items related to the popular content items).

[0062] In some embodiments, the user interface engine 332 may not identify content items related to items in the physical store 104 for inclusion in the user interface to be sent for display to the customer 204, based on the type related to the content item and the context information related to the shopping cart 156. In various embodiments, if the content item is unlikely to incentivize the customer 204 to perform an activity related to the item (e.g., to purchase the item), the user interface engine 332 may not identify the content item. For example, assume that the content item corresponds to an advertisement for spaghetti sauce and the context information related to the shopping cart 156 used by the customer 204 in the physical store 104 indicates that the spaghetti sauce is included among the items in the shopping cart 156. In this example, if the spaghetti sauce is in the customer 204's shopping cart 156, they are likely to have already purchased it, so the content item may not need to be included in the set of content items identified by the user interface engine 332 for inclusion in the user interface to be sent for display to the customer 204. However, in the above example, if the content item is a coupon for spaghetti sauce, the content item may be included in the set of content items identified by the user interface engine 332 because the customer 204 is more likely to use the coupon to purchase more spaghetti sauce.

[0063] When the user interface engine 332 identifies a set of content items related to one or more items within the physical retail store 104 for inclusion in a user interface that is to be sent for display to a customer 204 of the physical retail store 104, the user interface engine 332 also generates a user interface that includes the set of content items. For example, when the user interface engine 332 identifies multiple content items, the user interface engine 332 may generate a user interface with the content items arranged in a grid format or in a carousel format that is scrollable horizontally or vertically. In embodiments where the user interface engine 332 identifies multiple content items for inclusion in a user interface that is to be sent for display to the customer 204, the content items may be arranged based on the likelihood that the customer 204 will subsequently perform an activity related to each content item. For example, if the user interface includes multiple content items, the content item related to the activity that the customer 204 is most likely to perform is placed within the most prominent position of the user interface, and the content item related to the activity that the customer 204 is least likely to perform is placed within the least prominent position. In such embodiments, the user interface engine 332 may determine the likelihood associated with each content item based on information included within the user profile of the customer 204. For example, the user interface engine 332 may determine the likelihood that the customer 204 will perform an activity related to a content item based on information (described below) included within the user profile of the customer 204 maintained within the attribute database 338. In this example, the information included within the user profile may describe whether events related to similar activities performed by the customer 204 were due to similar content items being included in a user interface previously sent to a display area related to the customer 204.

[0064] When the user interface engine 332 generates a user interface, the user interface engine 332 transmits the user interface to a display area associated with customer 204. In some embodiments, the user interface engine 332 transmits the user interface to a display area included in the client device 110 associated with customer 204. In various embodiments, the user interface engine 332 also or alternatively transmits the user interface to a display area 159 included in the shopping cart system 150.

[0065] The shopping cart system 150 further includes a content database 334. The content database 334 stores content items related to items within one or more physical retail stores 104. The content database 334 can store various types of content items, such as product information related to the item (e.g., supplier / manufacturing information, cautions regarding materials / components), sales promotions for the item, coupons for the item, a set of instructions for using the item, suggested usage for the item, advertisements related to the item, reminders related to the item, videos, images, or social media posts featuring the item. For example, if the item is a retail item sold in a grocery store, the content item corresponding to the product information related to the item can include nutritional component information related to the item, information indicating whether the item is organic, where it is made, etc. As an additional example, the content item related to an item in a grocery store can correspond to a recipe in which the item is included as an ingredient. As yet another example, the content item related to an item can correspond to a reminder to purchase the item if customer 204 frequently purchases the item during a shopping session, or if the item is commonly paired with another item within the shopping cart 156 that is being used by customer 204. In some embodiments, one or more content items stored within the content database 334 can be associated with multiple items within the physical retail store 104. For example, a content item corresponding to a recipe can be associated with multiple items each corresponding to an ingredient for that recipe.

[0066] The shopping cart system 150 further includes an attribute engine 336. The attribute engine 336 determines whether to cause content items to be included in the user interface transmitted for display to the customer 204 of the physical store 104 for the events detected by the event detection engine 322. The attribute engine 336 may make a determination based on a first time at which the user interface is transmitted to a display area associated with the customer 204 and a second time at which the event is detected by the event detection engine 322. In some embodiments, when the user interface is transmitted to a display area associated with the customer 204 during a shopping session, the attribute engine 336 may determine that any subsequent events detected by the event detection engine 322 during the shopping session may cause content items related to the event to be included in the user interface. For example, assume that the user interface engine 332 transmits a user interface including a content item corresponding to a coupon for an item to a display area associated with the customer 204. In this example, the attribute engine 336 may determine that the addition of an item to the shopping cart 156 by the customer 204 and the subsequent purchase of the item by the customer 204 detected later by the event detection engine 322 should cause content items to be included in the user interface transmitted for display to the customer 204.

[0067] In some embodiments, when the attribute engine 336 determines whether to cause content items to be included in a user interface sent to display to a customer 204 of the physical retail store 104 as a result of an event detected by the event detection engine 322, the attribute engine 336 may store information describing the determination in the attribute database 338. The information stored in the attribute database 338 includes information describing content items included in a user interface sent to a display area associated with a customer 204 of the physical retail store 104, information describing the time at which the user interface was sent to the display area, information describing the display area (e.g., whether the display area is included in the client device 110 or in the shopping cart system 150), information identifying the customer 204, information describing an event that did or did not result in the content items being included in the user interface (e.g., the time at which the event was detected, information identifying items associated with the event, etc.), or any other suitable information describing a determination made by the attribute engine 336. In embodiments where the information stored in the attribute database 338 includes information identifying the customer 204, the information may be included among user profile information associated with the customer 204.

[0068] In some embodiments, the shopping cart system 150 further includes a performance engine 340. The performance engine 340 calculates performance criteria associated with content items, based in part on a determination of whether to cause the attribute engine 336 to include the content items in the user interface sent to the display area associated with the customer 204 of the physical retail store 104 for an event. The performance criteria may correspond to a conversion rate, a click-through rate (CTR), or any other suitable criteria that describe the performance of the content items. For example, the performance criteria associated with a content item may correspond to a conversion rate that describes the percentage of customers 204 of the physical retail store 104 to whom the content item was presented during a shopping session and who subsequently added an item associated with the content item to their shopping cart 156. As an additional example, the performance criteria associated with a content item may correspond to a click-through rate that describes the percentage of customers 204 to whom the content item was presented who clicked on the content item to view a recipe associated with the item. The performance engine 340 may be configured to store information describing the performance criteria in a performance database 342. Further, in some embodiments, the performance engine 340 and / or the performance database 342 may also or alternatively be included within the online concierge system 102. In embodiments where the performance engine 340 is included within the online concierge system 102, the determination of whether to cause the attribute engine 336 to include the content items in the user interface sent to the display area associated with the customer 204 for an event may be communicated to the online concierge system 102 and used by the performance engine 340 to calculate the performance criteria associated with the content items. In such embodiments, the performance engine 340 of the online concierge system 102 may access the attribute engine 336 of the shopping cart system 150 to retrieve the determination, or the attribute engine 336 of the shopping cart system 150 may communicate the determination to the performance engine 340 of the online concierge system 102.

[0069] Display information about items in a physical retail store based on context information related to a shopping cart system FIG. 4 is a flowchart of a method for displaying information about items in a physical retail store 104 based on context information related to a shopping cart system 150, according to one or more embodiments. In various embodiments, the method includes steps different from or additional to those described with respect to FIG. 4. Further, in some embodiments, the steps of the method may be performed in an order different from the order described with respect to FIG. 4. One or more steps of the method described with respect to FIG. 4 may be performed by an online concierge system 102 in various embodiments, but in other embodiments, the steps of the method are performed by any online system capable of retrieving items.

[0070] The shopping cart system 150 detects (405), e.g., via the event detection engine 322, the start of a shopping session within the physical retail store 104 by a customer 204 of the physical retail store 104. The start of the shopping session can be detected (405) by one or more components of the shopping cart system 150. In some embodiments, the start of the shopping session is detected (405) by one or more sensors 158 included in the shopping cart system 150. For example, the shopping cart system 150 can detect the start of the shopping session as soon as it receives readings from one or more sensors 158 (e.g., motion or speed sensors 158) of the shopping cart system 150 (405). In various embodiments, the start of the shopping session is also or alternatively detected (405) as soon as the shopping cart system 150 receives one or more interactions with the display area 159 and / or one or more accessories (e.g., buttons, dials, knobs, microphones, barcode scanners, cameras, etc.) included in the shopping cart system 150. For example, the shopping cart system 150 can detect the start of the shopping session by the customer 204 as soon as it receives a request to start a shopping session via the touch screen display area 159 included in the shopping cart system 150 (405).

[0071] In some embodiments, the shopping session may be initiated upon authentication of customer 204. In such embodiments, the information provided by customer 204 may be used by the shopping cart system 150 and / or the online concierge system 102 to authenticate customer 204. Examples of such information include a combination of username and password, biometric information (e.g., fingerprint), or any other suitable information that may be used to authenticate customer 204. For example, information from customer 204 corresponding to login qualification information for the online concierge system 102 may be received by the shopping cart system 150 via the touch screen display area 159 included in the shopping cart system 150. In this example, the shopping cart system 150 may communicate the qualification information to the online concierge system 102 that can authenticate customer 204 if the login qualification information matches those of the customers in the online concierge system 102 (e.g., within the customer database 314). Alternatively, in the above example, the shopping cart system 150 may access the information maintained within the online concierge system 102 (e.g., within the customer database 314) related to customer 204 of the online concierge system 102 and authenticate customer 204 if the qualification information matches those of customer 204 of the online concierge system 102. As an additional example, a quick response (QR) code that uniquely identifies customer 204 within the online concierge system 102 may be generated by the online concierge system 102 and displayed on the client device 110 associated with customer 204 (e.g., via the CMA 206). In this example, customer 204 may be authenticated if the camera included in the shopping cart system 150 scans the QR code and the shopping cart system 150 then communicates the QR code to the online concierge system 102 that authenticates customer 204. In embodiments where the shopping session is initiated upon authentication of customer 204, the shopping session may be linked to the user profile of customer 204 upon authentication of customer 204.For example, when customer 204 is authenticated, information describing the shopping session can be stored by shopping cart system 150 (e.g., within event database 324) and can be included in user profile information associated with customer 204.

[0072] Shopping cart system 150 tracks (410), using, for example, context tracking engine 328, context information related to shopping cart 156 (or a similar object used to carry items collected during a shopping session) included in shopping cart system 150 that is received by one or more sensors 158 included in shopping cart system 150. As described above, one or more sensors 158 included in shopping cart system 150 may be capable of identifying physical objects (e.g., items, organizing elements, and / or building elements) within physical retail store 104. Also as described above, context information related to shopping cart 156 may describe one or more positions of shopping cart 156 within physical retail store 104 and / or a set of items within shopping cart 156. For example, context information related to shopping cart 156 that describes the position of shopping cart 156 within physical retail store 104 may include GPS coordinates collected by a GPS sensor 158 coupled to shopping cart 156. In this example, context information collected by one or more proximity sensors 158 coupled to shopping cart 156 may also include distances (e.g., in feet or meters), directions (e.g., north, south, east, or west), and altitudes (e.g., in feet or meters) of one or more items, aisles, service counters, etc. within physical retail store 104 relative to shopping cart 156. Continuing with this example, context information collected by a sensor 158 (e.g., a camera) mounted on shopping cart 156 that faces the interior of shopping cart 156 may include information identifying one or more items (e.g., retail items) within shopping cart 156.

[0073] In some embodiments, the shopping cart system 150 may track context information related to the shopping cart 156 by comparing information collected by one or more sensors 158 included in the shopping cart system 150 with a shelf map related to the physical retail store 104 (410). As shown in FIG. 5, which illustrates a shelf map 500 related to the physical retail store 104, according to one or more embodiments, the shelf map 500 may show the arrangement of items 505 on shelves, within display cases, within aisles, or within any other suitable organizing element used to organize items 505 within the physical retail store 104. The shelf map 500 may also include a layout of the physical retail store 104 that describes the positions of the organizing elements relative to each other and relative to the building elements of the physical retail store 104 (e.g., walls, doors, stairs, elevators, etc.). To illustrate an example of how the shopping cart system 150 may track context information related to the shopping cart 156 by comparing information collected by sensors 158 included in the shopping cart system 150 with the shelf map 500 related to the physical retail store 104 (410), assume that one or more of the sensors 158 correspond to one or more cameras. In this example, the shopping cart system 150 may compare video or image data collected by the cameras with the shelf map 500 related to the physical retail store 104 and, based on the comparison, determine the position of the shopping cart 156. In the above example, if the items 505 identified in the video or image data and their arrangement within the physical retail store 104 have at least a threshold measure of similarity to a portion of the shelf map 500, the shopping cart system 150 may determine that the shopping cart 156 is at a position within the physical retail store 104 corresponding to that portion of the shelf map 500.

[0074] The context information associated with the shopping cart 156 tracked by the shopping cart system 150 (410) may describe the state of the shopping cart 156. As described above, the state of the shopping cart 156 may indicate whether the shopping cart 156 is moving or in a stationary state. In some embodiments, the state of the shopping cart 156 may also indicate whether the shopping cart 156 is moving at a speed less than a threshold speed, whether the shopping cart 156 has moved less than a threshold distance within a specified amount of time, whether the shopping cart 156 has a reverse direction, the speed of the shopping cart 156, the orientation of the shopping cart 156, or any other suitable information describing the state of the shopping cart 156.

[0075] The context information associated with the shopping cart 156 tracked by the shopping cart system 150 (410) may be associated with various types of data. Examples of data that may be associated with the context information include the name of the physical retail store 104 where the context information was collected, the geographical location associated with the physical retail store 104, the time at which the information was collected at the physical retail store 104, information (e.g., time) describing the shopping session during which the context information was collected, information identifying the customer 204 associated with the shopping session (e.g., based on information provided to authenticate the customer 204), and the like. For example, if the physical retail store 104 is a grocery store belonging to a grocery store chain, the context information associated with the shopping cart 156 tracked by the shopping cart system 150 (410) may be associated with data including the name identifying the chain and the geographical location (e.g., city and state) identifying the specific physical retail store 104. In the above example, the context information may also be associated with a timestamp indicating the time at which it was collected by the sensor 158 included in the shopping cart system 150.

[0076] Although not illustrated in FIG. 4, in some embodiments, the shopping cart system 150 may store the context information it tracks (410) and / or data related to the context information (e.g., within the context database 330). In embodiments where the context information stored within the shopping cart system 150 includes information that identifies the customer 204 related to the context information, this information may be included within a user profile related to the customer 204. For example, when the customer 204 starts a shopping session and provides information that uniquely identifies the customer 204, and the information is used to authenticate the customer 204, the information that identifies the customer 204 (e.g., a username) may be stored within the shopping cart system 150 together with the context information (410) tracked by the shopping cart system 150 during the shopping session. In this example, the context information may then be retrieved together with other user profile information related to the customer 204.

[0077] In various embodiments, the shopping cart system 150 may also communicate the context information 410 it tracks and / or data related to the context information to the online concierge system 102. In such embodiments, the online concierge system 102 may use this information for various purposes. In some embodiments, the online concierge system 102 may use this information to customize the experience of the customer 204 within the online concierge system 102. For example, if the customer 204 of the physical retail store 104 is also the customer 204 of the online concierge system 102, the online concierge system 102 may customize the online version of the physical retail store 104 based on the paths typically traversed by the customer 204 through the physical retail store 104.

[0078] Referring back to FIG. 4, the shopping cart system 150 then identifies (at 415) an opportunity to present content to the customer 204 (e.g., via the user interface engine 332), based in part on the context information associated with the shopping cart 156. In some embodiments, the shopping cart system 150 may identify an opportunity to present content to the customer 204 as soon as it detects that the shopping cart 156 has been stationary for at least a threshold amount of time (415). For example, the shopping cart system 150 may receive context information that describes the state of the shopping cart 156. In this example, if the context information indicates that the shopping cart 156 has been stationary for at least a threshold amount of time, the shopping cart system 150 may identify an opportunity to present content to the customer 204 (415). In some embodiments, the shopping cart system 150 may also or alternatively identify an opportunity to present content to the customer 204 as soon as it detects one or more specific items 505 within the shopping cart 156 (415). For example, if the context information indicates that an item 505 has been added to the shopping cart 156, the shopping cart system 150 may identify an opportunity to present content related to the item 505 to the customer 204 (415).

[0079] The shopping cart system 150 then identifies (at 420) a set of content items related to one or more items 505 within the physical store 104 for inclusion in a user interface that will be sent for display to the customer 204 (e.g., using the user interface engine 332). In some embodiments, the shopping cart system 150 may identify the set of content items using a machine-learned model (420). Additionally, the shopping cart system 150 may identify the set of content items based in part on the context information associated with the shopping cart 156 (420).

[0080] A set of content items can be of various types maintained by the shopping cart system 150 (e.g., within the content database 334), such as product information related to item 505 (e.g., supplier / manufacturer information, cautions regarding materials / components), sales promotion for item 505, coupons for item 505, a set of usage instructions for item 505, suggested usage methods for item 505, advertisements related to item 505, reminders related to item 505, videos, images, or social media posts featuring item 505. For example, if item 505 is a retail item sold in a grocery store, the content item corresponding to the product information related to item 505 can include nutritional component information related to item 505, information indicating whether item 505 is organic, where it is made, etc. As an additional example, the content item related to item 505 in a grocery store can correspond to a recipe in which item 505 is included as an ingredient. As yet another example, the content item related to item 505 can correspond to a reminder to purchase item 505 if customer 204 frequently purchases item 505 during a shopping session, or if item 505 is normally paired with another item 505 in the shopping cart 156.

[0081] In some embodiments, one or more content items can be associated with multiple items 505 within the physical retail store 104. As shown in FIG. 6, which illustrates content items 605 related to one or more items 505 within the physical retail store 104, according to one or more embodiments, various content items 605 can be associated with one or more items 505. For example, content items 605a, 605b, 605d, and 605e are associated only with item 505a, while content items 605c and 605f are associated with both items 505a and 505b, and content item 605g is associated with items 505a, 505b, and 505c.

[0082] Referring back to FIG. 4, in an embodiment where the shopping cart system 150 identifies an opportunity to present content to the customer 204 as soon as it detects that the shopping cart 156 has been stationary for at least a threshold amount of time (415), the set of content items 605 identified (420) by the shopping cart system 150 can be associated with one or more items 505 within a threshold distance of the shopping cart 156. For example, the shopping cart system 150 can access context information stored (e.g., within the context database 330) within the shopping cart system 150 that describes the location of the shopping cart 156 within the physical retail store 104 while the shopping cart 156 is stationary for at least a threshold amount of time. In this example, the shopping cart system 150 can then retrieve the shelf map 500 associated with the physical retail store 104 (e.g., from the shelf map database 326) and compare the location of the shopping cart 156 to the shelf map 500. Continuing with this example, the shopping cart system 150 can then identify one or more items 505 within the physical retail store 104 within a threshold distance of the location of the shopping cart 156 and identify one or more content items 605 associated with the identified items 505 (420) (e.g., by accessing the content database 334).

[0083] In an embodiment where shopping cart system 150 identifies an opportunity to present content to customer 204 as soon as one or more specific items 505 within shopping cart 156 are detected (415), a set of content items 605 identified (420) by shopping cart system 150 may be associated with item 505. For example, shopping cart system 150 may access context information (e.g., stored within context database 330) that describes one or more items 505 within shopping cart 156 as soon as it detects that item 505 has been added to shopping cart 156. In this example, if the context information indicates that the item 505 within shopping cart 156 is spaghetti, shopping cart system 150 may identify (420) one or more content items 605 related to spaghetti.

[0084] In some embodiments, the shopping cart system 150 may also identify (420) a set of content items 605 for inclusion in a user interface to be sent for display to the customer 204, based in part on user profile information associated with the customer 204. In such embodiments, the shopping cart system 150 may identify (420) the set of content items 605 based on the customer's geographical location, age, gender, shopping preferences, favorite items 505, dislikes, (e.g., historical information associated with the customer 204 within the physical retail store 104 or the online concierge system 102), or any other information included within the customer's user profile. For example, assume that the user profile of the customer 204 indicates that the customer 204 has an allergy to nuts and thus dislikes them. In this example, one or more sensors 158 of the shopping cart system 150 identify the item 505 added to the shopping cart 156, and if the item 505 includes a component corresponding to nuts, a content item 605 including a note regarding the content of the item 505 may be identified (420) by the shopping cart system 150. As an additional example, assume that historical information associated with the customer 204 indicates that the customer 204 typically purchases a particular item 505 according to a consistent cadence, such as during each shopping session or is likely to purchase the item 505 during the current shopping session. In this example, if the context information associated with the shopping cart 156 indicates that the shopping cart 156 is in the checkout queue and the item 505 is not within the shopping cart 156, a content item 605 that reminds the customer 204 about the item 505 may be identified (420) by the shopping cart system 150.In an embodiment where the shopping cart system 150 identifies content item 605 based on user profile information associated with customer 204 (step 420), the shopping cart system 150 can access user profile information stored within the online concierge system 102 (e.g., within customer database 314, transaction record database 308, and / or training data set 320), and / or the online concierge system 102 can communicate the user profile information to the shopping cart system 150.

[0085] In various embodiments, the shopping cart system 150 may also identify (420) a set of content items 605 for inclusion in a user interface to be sent for display to customer 204, based in part on the time of the shopping session. In such embodiments, the shopping cart system 150 may identify (420) the content items 605 included in the set based on seasonal demand for items 505 associated with the content item 605, day of the week, time of day, etc. For example, if the shopping session is early in the morning, the shopping cart system 150 may identify (420) one or more content items 605 (e.g., pastries, coffee, tea, etc.) associated with breakfast items 505. As an additional example, if the shopping session occurs in the month prior to Thanksgiving, the shopping cart system 150 may identify (step 420) content items 605 associated with items 505 related to Thanksgiving (e.g., Thanksgiving recipes, sales promotions for hams or gravy, etc.).

[0086] In some embodiments, the shopping cart system 150 may also identify (420) a set of content items 605 for inclusion in a user interface that is to be transmitted for display to the customer 204, based in part on information related to the set of content items 605 and / or information related to one or more items 505 related to the set of content items 605. In various embodiments, each content item 605 may be associated with a value that the shopping cart system 150 may use to identify (420) the set of content items 605 for inclusion in the user interface such that content items 605 associated with higher values are more likely to be identified (420) by the shopping cart system 150 for inclusion in the user interface than content items 605 associated with lower values. For example, if the content item 605 corresponds to an advertisement, the content item 605 associated with a higher bid amount may be more likely to be identified (420) by the shopping cart system 150 for inclusion in the user interface than the content item 605 associated with a lower bid amount. As an additional example, content items 605 related to higher priced items 505 and / or items 505 having higher inventory may be more likely to be identified (420) by the shopping cart system 150 for inclusion in the user interface than content items 605 related to lower priced items 505 and / or items 505 having lower inventory.

[0087] In some embodiments, the shopping cart system 150 may also identify (420) a set of content items 605 for inclusion in a user interface that is to be sent for display to the customer 204, based in part on information received from the online concierge system 102. For example, assume that the online concierge system 102 presents various content items 605 (e.g., articles, recipes, etc.) to its customer 204 via the CMA 206. In this example, the online concierge system 102 may communicate to the shopping cart system 150 information identifying certain content items 605 that are popular among the customers 204 of the online concierge system 102. In the above example, the popular content items 605 may be included among those identified (420) by the shopping cart system 150 for inclusion in the user interface that is to be sent for display to the customer 204 (e.g., if context information related to the shopping cart 156 indicates that the shopping cart 156 is within a threshold distance of one or more items 505 related to the popular content items 605).

[0088] In some embodiments, the shopping cart system 150 may not identify (420) content items 605 for inclusion in a user interface to be sent for display to the customer 204, based on the type associated with the content item 605 and the context information associated with the shopping cart 156. In various embodiments, the shopping cart system 150 may not identify (420) content items 605 if the content item 605 is unlikely to incentivize the customer 204 to perform an activity associated with the item 505 (e.g., to purchase the item 505). For example, assume that the content item 605 corresponds to an advertisement for spaghetti sauce and the context information associated with the shopping cart 156 indicates that the spaghetti sauce is included among the items 505 in the shopping cart 156. In this example, if the spaghetti sauce is in the customer 204's shopping cart 156, they are likely to have already purchased it, so the content item 605 may not be included in the set of content items 605 identified (420) by the shopping cart system 150 for inclusion in a user interface to be sent for display to the customer 204. However, in the above example, if the content item 605 is a coupon for spaghetti sauce, the customer 204 may be more likely to use the coupon to purchase more spaghetti sauce, so the content item 605 may be included in the set of content items 605 identified (420) by the shopping cart system 150.

[0089] The shopping cart system 150 then generates (425) a user interface that includes the identified set of content items 605 (e.g., using the user interface engine 332). For example, if the shopping cart system 150 identifies (420) multiple content items 605, the shopping cart system 150 may generate a user interface with the content items 605 arranged in a grid format or in a carousel format that is scrollable horizontally or vertically (425). In embodiments where the shopping cart system 150 identifies (420) multiple content items 605 for inclusion in the user interface that is to be sent for display to the customer 204, the content items 605 may be arranged based on the likelihood that the customer 204 will subsequently perform an activity associated with each content item 605. For example, if the user interface includes multiple content items 605, the content items 605 associated with the activities that the customer 204 is most likely to perform are arranged within the most prominent positions of the user interface, and the content items 605 associated with the activities that the customer 204 is least likely to perform are arranged within the least prominent positions. In such embodiments, the shopping cart system 150 may determine the likelihood associated with each content item 605 based on information included within the user profile of the customer 204. For example, the shopping cart system 150 may determine the likelihood that the customer 204 will perform an activity associated with the content item 605 based on information included within the user profile of the customer 204 maintained within the shopping cart system 150 (e.g., within the attribute database 338). In this example, the information included within the user profile may describe whether events associated with similar activities performed by the customer 204 were due to similar content items 605 being included in a user interface previously sent to a display area associated with the customer 204. In some embodiments, a server that communicates with the shopping cart system 150 may generate the user interface (425).For example, a server associated with the online concierge system 102 may generate (425) a user interface and communicate the user interface to the shopping cart system 150.

[0090] When the user interface is generated (425), the shopping cart system 150 transmits (430) the user interface to a display area associated with the customer 204 (e.g., using the user interface engine 332). In some embodiments, the shopping cart system 150 transmits (430) the user interface to a display area included within the client device 110 associated with the customer 204. For example, the shopping cart system 150 may transmit (430) the user interface to a display area included within the client device 110 associated with the customer 204 (e.g., via push-type notification through the CMA 206). In various embodiments, the shopping cart system 150 also or alternatively transmits (430) the user interface to a display area 159 included within the shopping cart system 150.

[0091] In some embodiments, the shopping cart system 150 may then detect (435), using, for example, the event detection engine 322, an event associated with one or more items 505 related to a set of content items 605 included in a user interface transmitted (430) to a display area associated with the customer 204. Examples of events that may be detected (435) by the shopping cart system 150 include stopping the shopping cart 156 within a threshold distance of an item 505 within the physical store 104 for at least a threshold amount of time, moving the shopping cart 156 within a threshold distance of an item 505 within the physical store 104, picking up an item 505 from a particular location (e.g., a sales promotion display area) within the physical store 104, adding an item 505 to the shopping cart 156, purchasing an item 505 from the physical store 104, or any other type of event that may be associated with the customer 204 within the physical store 104. In embodiments where the event detected (435) by the shopping cart system 150 corresponds to stopping the shopping cart 156 within a threshold distance of an item 505 within the physical store 104 for at least a threshold amount of time, the threshold amount of time and / or distance may be determined by a machine-learned model (e.g., a model trained based on anonymized aggregated data that describes the behavior of customers associated with the shopping cart 156 when determining whether to purchase an item 505).

[0092] Shopping cart system 150 can detect (435) an event related to one or more items 505, related to a set of content items 605 included in a user interface transmitted (430) to a display area related to customer 204, based on context information related to shopping cart 156 tracked by shopping cart system 150 and / or a shelf map 500 related to physical retail store 104. For example, assume that a particular item 505 related to content item 605 included in a user interface transmitted (430) to a display area related to customer 204 is placed at two different locations within physical retail store 104, where one location is its normal location in an aisle and the other location is a sales promotion display area at the end of a different aisle. In this example, shopping cart system 150 can detect (435) an event corresponding to adding item 505 to shopping cart 156, in which case shopping cart system 150 can also detect (435) the location where item 505 was added to shopping cart 156 (e.g., based on shelf map 500 related to physical retail store 104). In the above example, shopping cart system 150 can also subsequently detect (435) an event corresponding to purchasing item 505, based on context information related to shopping cart 156 and shelf map 500 related to physical retail store 104, indicating that shopping cart 156 has left the checkout queue within physical retail store 104 with item 505.

[0093] Although not illustrated in FIG. 4, in some embodiments, the shopping cart system 150 may store information describing an event (e.g., within the event database 324). In various embodiments, this information may also or alternatively be communicated to the online concierge system 102 and stored therein (e.g., within the transaction record database 308 and / or the training data set 320). The shopping cart system 150 may store information describing the event along with information identifying the customer 204 associated with the event (e.g., based on login credentials provided by the customer 204), information identifying the physical retail store 104 in which the event was detected (435), the time at which the event was detected (435), the location within the physical retail store 104 where the event was detected (435), or any other information associated with the event. For example, assume that the shopping cart system 150 detects (435) that the customer 204 added the item 505 to the shopping cart 156. In this example, information describing the event, such as the type of the event (i.e., adding the item 505 to the shopping cart 156) and information describing the item 505 (e.g., the type of the item 505, the size of the item 505, the brand of the item 505, etc.), may be stored (e.g., within the event database 324, the transaction record database 308, and / or the training data set 320). In the above example, the information describing the event may be stored along with information identifying the physical retail store 104 (e.g., its name and address), the time at which the item 505 was added to the shopping cart 156, the aisle within the physical retail store 104 where the customer 204 added the item 505 to the shopping cart 156, and the username associated with the customer 204 within the online concierge system 102.

[0094] Shopping cart system 150 may determine (e.g., using attribute engine 336) whether to cause content item 605 to be included in a user interface (430) sent for display to customer 204 for an event (435) detected by shopping cart system 150. Shopping cart system 150 may make the determination based on a first time (430) at which the user interface is sent to a display area associated with customer 204 and a second time (435) at which the event is detected by shopping cart system 150. In some embodiments, when the user interface is sent (430) to a display area associated with customer 204 during a shopping session, shopping cart system 150 may determine (440) that any subsequent event (435) detected by shopping cart system 150 during the shopping session may cause content item 605 associated with the event to be included in the user interface. For example, assume that shopping cart system 150 sends (430) to a display area associated with customer 204 a user interface that includes content item 605 corresponding to a coupon for item 505. In this example, shopping cart system 150 may determine (440) that the addition of item 505 to shopping cart 156 by customer 204 and the subsequent purchase of item 505 by customer 204 detected (435) thereafter by shopping cart system 150 should cause content item 605 to be included in the user interface (430) sent for display to customer 204.

[0095] Although not illustrated in FIG. 4, in some embodiments, the shopping cart system 150 determines (440) whether to cause the user interface transmitted (430) to the customer 204 to include the content item 605 in order to display the event (435) detected by the shopping cart system 150, and the shopping cart system 150 may store information describing the determination (e.g., within the attribute database 338). The stored information includes information describing the content item 605, information describing the time at which the user interface was transmitted (430) to the display area associated with the customer 204, information describing the display area (e.g., whether the display area is included within the client device 110 or within the shopping cart system 150), information identifying the customer 204, information describing the event that did or did not result from including the content item 605 in the user interface (e.g., the time at which the event was detected (435), information identifying the item 505 associated with the event, etc.), or any other suitable information describing the determination made by the shopping cart system 150. In embodiments where the information describing the determination identifies the customer 204, the information may be included within the user profile information associated with the customer 204.

[0096] In some embodiments, the shopping cart system 150 also calculates (445) performance criteria associated with content item 605 included in the user interface sent (430) for display to customer 204 (e.g., using performance engine 340). The shopping cart system 150 can calculate the performance criteria (445) based in part on a determination by the shopping cart system 150 as to whether to cause an event detected (435) by the shopping cart system 150 to include content item 605 in the user interface sent (430) to a display area associated with customer 204. The performance criteria can correspond to a conversion rate, a click-through rate (CTR), or any other suitable criteria that describes the performance of content item 605. For example, the performance criteria associated with content item 605 can correspond to a conversion rate that describes the percentage of customers 204 in physical store 104 to whom content item 605 was presented during a shopping session who subsequently added an item 505 associated with content item 605 to their shopping cart 156. As an additional example, the performance criteria associated with content item 605 can correspond to a click-through rate that describes the percentage of customers 204 to whom content item 605 was presented who clicked on content item 605 to view a recipe associated with item 505. Although not illustrated in FIG. 4, in some embodiments, the shopping cart system 150 can be configured to store information that describes the performance criteria (e.g., within performance database 342). Further, in some embodiments, the determination by the shopping cart system 150 as to whether to cause an event detected (435) by the shopping cart system 150 to include content item 605 in the user interface sent (430) to a display area associated with customer 204 can be communicated to online concierge system 102 and used by online concierge system 102 to calculate (445) the performance criteria associated with content item 605.In such an embodiment, the online concierge system 102 may access the shopping cart system 150 to retrieve a decision, or the shopping cart system 150 may communicate the decision to the online concierge system 102.

[0097] The shopping cart system 150 may then communicate performance criteria related to the content item 605 to an entity related to the physical store 104, a third - party system 130, and / or the online concierge system 102 (450). For example, assume that the performance criteria related to the content item 605 corresponds to a conversion rate related to the content item 605, and the content item 605 corresponds to a coupon provided by the physical store 104 for item 505. In this example, the performance criteria can indicate the percentage of customers 204 presented with the content item 605 who then added item 505 to their shopping cart 156, and can be communicated to an entity related to the physical store 104 (450) as an indication of the effectiveness of the coupon in motivating customer 204 to add item 505 to their shopping cart 156. As an additional example, assume that the performance criteria related to the content item 605 corresponds to a click - through rate and the content item 605 corresponds to an advertisement. In this case, the performance criteria can be communicated to the third - party system 130 related to the advertisement (450) as an indication of the effectiveness of the advertisement. As yet another example, assume that the performance criteria related to the content item 605 corresponds to a conversion rate and the content item 605 corresponds to product information related to item 505. In this example, the performance criteria can be communicated to the online concierge system 102 (450), which can increase or decrease the prominence of the product information displayed to that customer 204 within the online concierge system 102 based on the performance criteria.

[0098] In various embodiments, information related to one or more items 505 associated with content item 605 included in the (430) user interface sent for display to customer 204 may be updated (455) based on performance criteria associated with content item 605. In some embodiments, the updated information may be maintained within shopping cart system 150, and in other embodiments, the updated information may be maintained within online concierge system 102. For example, if the performance criteria associated with content item 605 calculated (445) by shopping cart system 150 corresponds to a low conversion rate for item 505 associated with content item 605, and content item 605 corresponds to a recipe in which item 505 is included as an ingredient, content item 605 may be updated (455) within shopping cart system 150 with a different recipe. As an additional example, assume that the performance criteria associated with content item 605 calculated (445) by shopping cart system 150 corresponds to a high conversion rate for item 505 associated with content item 605, and content item 605 corresponds to nutritional component information related to item 505. In this example, if the performance criteria is communicated (450) to online concierge system 102, online concierge system 102 may update (455) the product information related to item 505 to highlight the nutritional component information.

[0099] Although not shown in FIG. 4, in some embodiments, after detecting the completion of a shopping session (e.g., using event detection engine 322), the same shopping cart system 150 may then detect (405) the start of an additional shopping session within physical store 104 by an additional customer 204 of physical store 104 (e.g., via event detection engine 322). Some or all of the additional steps described above (e.g., steps 410 - 455) may then be performed with respect to the additional customer 204.

[0100] Additional Considerations The foregoing description of the embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above disclosure.

[0101] Some portions of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used in data processing to effectively convey the substance of these operations to others. These operations are functionally, computationally, or logically described, but it is understood that they may be implemented by a computer program or equivalent electrical circuit, microcode, etc. Further, it has also proven convenient at times, without loss of generality, to refer to these arrangements of operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0102] Any of the steps, operations, or processes described herein may be executed or implemented alone or in combination with one or more hardware or software modules. In one embodiment, a software module is implemented with a computer program product that includes a computer-readable medium containing computer program code that may be executed by a computer processor to execute any or all of the described steps, operations, or processes.

[0103] Embodiments of the present invention may also relate to an apparatus for performing the operations of this specification. This apparatus may be specially constructed for the required purposes and / or it may comprise a computing device selectively activated or reconfigured by a computer program stored within a computer. Such a computer program may be stored in a tangible computer-readable storage medium including any type of tangible medium suitable for storing electronic instructions and coupled to a computer system bus. Further, any of the computing systems referred to herein may include a single processor or may be an architecture that employs a plurality of processor designs for increased computing capability.

[0104] Embodiments of the present invention may also relate to a computer data signal embodied in a carrier wave, where the computer data signal includes any embodiment of a computer program product or other data combination described herein. The computer data signal is a product presented in a tangible medium or carrier wave, modulated or otherwise encoded within a tangible carrier wave, and transmitted according to any suitable transmission method.

[0105] Finally, the language used herein has been principally selected for readability and instructional purposes and may not have been selected to delineate or limit the subject matter of the present invention. Accordingly, the scope of the present invention is intended to be limited not by this detailed description but rather by any claims issued on an application based thereon. Thus, the present disclosure of embodiments of the present invention is intended to be illustrative, not limiting, of the scope of the present invention as set forth in the following claims.

Claims

1. In a shopping cart system, detecting the start of a shopping session in the physical retail store by a customer of the physical retail store, the shopping cart system including a shopping cart, a processor, a memory, and a set of sensors, and tracking, during the shopping session, context information related to the shopping cart received by the set of sensors, the context information describing one or more positions of the shopping cart in the physical retail store, the state of the shopping cart, and a set of items in the shopping cart, and identifying an opportunity to present content to the customer, at least in part based on the context information related to the shopping cart, and in response to identifying the opportunity to present the content to the customer, identifying a set of content items related to one or more items in the physical retail store, at least in part based on the context information related to the shopping cart, and generating a user interface including the set of content items, and transmitting the user interface to a display area associated with the customer, transmitting the user interface to the display area associated with the customer including causing the set of content items to be displayed to the customer in the user interface, and A method comprising.

2. Identifying the opportunity to present the content to the customer comprises detecting that the shopping cart has been in a stationary state for at least a threshold amount of time, and detecting a particular item of the set of items in the shopping cart, including one or more of The method according to claim 1.

3. Identifying the set of content items related to the one or more items in the physical retail store, at least in part based on the context information related to the shopping cart, comprises While the shopping cart is in a stationary state for at least a threshold amount of time, comparing a position among one or more positions of the shopping cart within the physical retail store with a shelf layout diagram associated with the physical retail store, wherein the shelf layout diagram shows the arrangement of each of a plurality of items within the physical retail store, Identifying the one or more items within the physical retail store based at least in part on the comparing, wherein the one or more items are within a threshold distance of the position among the one or more positions of the shopping cart, Identifying the set of content items based at least in part on the identified one or more items, The method according to claim 2, comprising.

4. The set of content items, Product information related to an item among the one or more items, promotion of the item among the one or more items, a coupon for the item among the one or more items, a set of instructions for using the item among the one or more items, a usage method suggested for the item among the one or more items, and an advertisement for the item among the one or more items, including one or more of, The method according to claim 1.

5. Detecting an event related to the one or more items based at least in part on one or more of the context information related to the shopping cart and a shelf layout diagram related to the physical retail store, wherein the shelf layout diagram shows the arrangement of each of a plurality of items within the physical retail store, Determining whether to cause the event to the set of content items included in the user interface transmitted to the display area related to the customer, wherein the determining is based at least in part on a first time when the user interface is transmitted to the display area related to the customer and a second time when the event is detected, Calculating performance criteria associated with content items in the set of content items included in the user interface sent to the display area associated with the customer, at least in part based on determining whether to cause the event to occur to the set of content items; Communicating the performance criteria associated with the content items to one or more of the entity associated with the physical retail store, the third-party system associated with the content items, and the online concierge system associated with the physical retail store; The method according to claim 1, further comprising.

6. The event is selected from the group consisting of adding the one or more items to the shopping cart and purchasing the one or more items. The method according to claim 5.

7. Updating information maintained in the online concierge system, at least in part based on the performance criteria, wherein the updated information is associated with the one or more items. The method according to claim 5, further comprising.

8. When executed by a processor, the processor is caused to In a shopping cart system, detecting the start of a shopping session in a physical retail store by a customer of the physical retail store, the shopping cart system including a shopping cart, a processor, a memory, and a set of sensors; Tracking context information related to the shopping cart received by the set of sensors during the shopping session, the context information describing one or more positions of the shopping cart in the physical retail store, the state of the shopping cart, and a set of items in the shopping cart; Identifying an opportunity to present content to the customer, at least in part based on the context information related to the shopping cart; In response to identifying the opportunity to present the content to the customer, identifying a set of content items related to one or more items in the physical retail store, at least in part based on the context information related to the shopping cart; Generating a user interface including the set of content items; Transmitting the user interface to a display area associated with the customer; A computer program product comprising a non-transitory computer-readable storage medium having encoded instructions for causing the above.

9. Identifying the opportunity to present the content to the customer comprises: Detecting that the shopping cart has been stationary for at least a threshold amount of time, and Detecting a particular item of the set of items in the shopping cart, Including one or more of: The computer program product according to claim 8.

10. Identifying the set of content items related to the one or more items in the physical retail store, at least partially based on the context information related to the shopping cart, comprises: Comparing, while the shopping cart has been stationary for at least a threshold amount of time, a position among one or more positions of the shopping cart in the physical retail store with a shelf layout associated with the physical retail store, the shelf layout indicating the arrangement of each of a plurality of items in the physical retail store; Identifying the one or more items in the physical retail store, at least partially based on the comparing, the one or more items being within a threshold distance of the position among the one or more positions of the shopping cart; Identifying the set of content items, at least partially based on the one or more identified items; The computer program product according to claim 9, including the above.

11. The set of content items includes: Product information related to an item among the one or more items, promotion of the item among the one or more items, a coupon for the item among the one or more items, a set of instructions for using the item among the one or more items, a usage method suggested for the item among the one or more items, and an advertisement for the item among the one or more items, Including one or more of: The computer program product according to claim 8.

12. When the computer-readable storage medium is executed by the processor, it causes the processor to detect an event related to the one or more items, at least in part based on one or more of the context information related to the shopping cart and a shelf layout diagram related to the physical retail store, the shelf layout diagram showing the placement of each of a plurality of items within the physical retail store; determine whether to cause the event in a set of content items included in the user interface sent to the display area related to the customer, the determining being at least in part based on a first time when the user interface was sent to the display area related to the customer and a second time when the event was detected; calculate performance criteria related to a content item in the set of content items, at least in part based on determining whether to cause the event in the set of content items included in the user interface sent to the display area related to the customer; communicate the performance criteria related to the content item to one or more of an entity related to the physical retail store, a third-party system related to the content item, and an online concierge system related to the physical retail store; further comprising encoded instructions to cause the above to be performed The computer program product according to claim 8.

13. The event is selected from the group consisting of adding the one or more items to the shopping cart and purchasing the one or more items. The computer program product according to claim 12.

14. When the computer-readable storage medium is executed by the processor, it causes the processor to update information maintained in the online concierge system, at least in part based on the performance criteria, the updated information being associated with the one or more items; further comprising encoded instructions to cause the above to be performed The computer program product according to claim 12.

15. A shopping cart system, A processor, and when executed by the processor, cause the shopping cart system to In a shopping cart system, detecting the start of a shopping session in the physical retail store by a customer of the physical retail store, wherein the shopping cart system includes a shopping cart, a processor, a memory, and a set of sensors, During the shopping session, tracking context information related to the shopping cart received by the set of sensors, wherein the context information describes one or more positions of the shopping cart in the physical retail store, the state of the shopping cart, and a set of items in the shopping cart, Identifying an opportunity to present content to the customer based at least in part on the context information related to the shopping cart, In response to identifying the opportunity to present the content to the customer, identifying a set of content items related to one or more items in the physical retail store based at least in part on the context information related to the shopping cart, Generating a user interface including the set of content items, Transmitting the user interface to a display area associated with the customer, A non-transitory computer-readable storage medium storing instructions for causing an action including A shopping cart system comprising

16. Identifying the opportunity to present the content to the customer includes Detecting that the shopping cart has been stationary for at least a threshold amount of time, and Detecting a particular item of the set of items in the shopping cart, including one or more of The shopping cart system according to claim 15.

17. Identifying the set of content items related to the one or more items in the physical retail store based at least in part on the context information related to the shopping cart is While the shopping cart is in a stationary state for at least a threshold amount of time, comparing a position among one or more positions of the shopping cart in the physical retail store with a shelf layout diagram related to the physical retail store, wherein the shelf layout diagram shows the arrangement of each of a plurality of items in the physical retail store, Identifying the one or more items in the physical retail store based at least in part on the comparing, wherein the one or more items are within a threshold distance of the position among the one or more positions of the shopping cart, Identifying the set of content items based at least in part on the one or more identified items, The shopping cart system according to claim 16, comprising.

18. The set of content items, Product information related to the item among the one or more items, promotion of the item among the one or more items, a coupon for the item among the one or more items, a set of instructions for using the item among the one or more items, a usage method suggested for the item among the one or more items, and an advertisement for the item among the one or more items, including one or more of, The shopping cart system according to claim 15.

19. When executed by the processor, the computer-readable storage medium causes the shopping cart system to, Detecting an event related to the one or more items based at least in part on one or more of the context information related to the shopping cart and a shelf layout diagram related to the physical retail store, wherein the shelf layout diagram shows the arrangement of each of a plurality of items in the physical retail store, Determining whether to cause the event to occur in the set of content items included in the user interface transmitted to the display area related to the customer, wherein the determining is based at least in part on a first time when the user interface is transmitted to the display area related to the customer and a second time when the event is detected, Calculating performance criteria associated with content items of the set of content items, at least in part based on determining whether to cause the event to the set of content items included in the user interface sent to the display area associated with the customer; Communicating the performance criteria associated with the content item to one or more of the entity associated with the physical retail store, a third-party system associated with the content item, and an online concierge system associated with the physical retail store; Further having encoded instructions to cause an action to be performed, The shopping cart system according to claim 15.

20. The event is selected from the group consisting of adding the one or more items to the shopping cart and purchasing the one or more items. The shopping cart system according to claim 19.

Images