Early intervention and correction in online conversations using large-scale language machine learning models

Abstract

The online system, in conjunction with a model-providing system or interface system, performs inference tasks and continuously monitors conversations between users and shoppers to determine whether a message sent by a sender can be responded to automatically rather than prompting a manual response from the recipient. The online system automatically responds to the message without manual involvement from the recipient. In one or more embodiments, the online system can infer whether a question is likely to be intervened and / or suggest one or more available answers that the sender can consider as feedback without a manual response from the recipient.

Description

Technical Field

[0001] The present invention relates to early intervention and correction in online conversations using large language machine learning models.

Background Art

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 458,942, filed Apr. 13, 2023, which is hereby incorporated by reference in its entirety.

[0003] An online system is an online platform that connects users and retailers. For example, a user can place an order to purchase items such as groceries from a participating retailer via the online system, and shopping is conducted by individual shoppers. In some instances, the online system generates a communication interface that enables a user to communicate with a picker who is processing the user's order. Often, a user or picker may make an inquiry or request about an order to the other party, but there are cases where the other party has to manually respond to fulfill the order for that inquiry, or cases where the other party cannot respond because they do not have access to appropriate information. This can lead to undesirable delays in fulfilling a user's order.

Summary of the Invention

[0004] According to one or more aspects of the present disclosure, an online system, in conjunction with a model-providing system or interface system, performs inference tasks to continuously monitor a conversation between a user and a shopper to determine, for example, whether a message sent by a sender can be responded to, rather than prompting a manual response from the recipient. The online system automatically responds to messages without the recipient's involvement. In one or more embodiments, the online system can infer whether a question can be intervened and / or suggest one or more available answers that the sender can consider as feedback without a manual response from the recipient. The online system can sequentially provide one or more messages from the conversation to the model-providing system or interface system, receive responses to each question indicating whether the question can be intervened, and automatically generate responses. If the message can be intervened, the online system can also suggest one or more answers to the question that are specific actions or suggestions that the message sender can consider as feedback. [Brief explanation of the drawing]

[0005] [Figure 1A] Figure 1A shows an exemplary system environment for an online system according to one or more embodiments. [Figure 1B] Figure 1B shows an exemplary system environment for an online system according to one or more embodiments. [Figure 2] Figure 2 shows an exemplary system architecture for an online system according to one or more embodiments. [Figure 3] Figure 3 shows an exemplary conversation between a shopper and a user in one or more embodiments. [Figure 4] Figure 4 shows an exemplary conversation between a shopper and a user in one or more embodiments. [Figure 5]Figure 5 is a flowchart illustrating a method for predicting whether an automated response can be generated for a message, according to one or more embodiments. [Modes for carrying out the invention]

[0006] Figure 1A shows an exemplary system environment for an online system 140 in one or more embodiments. The system environment shown in Figure 1A includes a customer client device 100, a picker client device 110, a retailer computing system 120, a network 130, and an online system 140. Alternative embodiments may include more, fewer, or different components than those shown in Figure 1A, and the functions of each component may be divided among components different from those described below. Furthermore, each component may perform its respective function in response to a human request or automatically without human intervention.

[0007] As used herein, customers, pickers, and retailers may generally be referred to as “users” of the online system 140. Furthermore, although one customer client device 100, a picker client device 110, and a retailer computing system 120 are shown in Figure 1, any number of customers, pickers, and retailers may interact with the online system 140. Thus, there may be two or more customer client devices 100, picker client devices 110, or retailer computing systems 120.

[0008] The customer client device 100 is a client device that allows the customer to interact with the picker client device 110, the retailer computing system 120, or the online system 140. The customer client device 100 may be a personal computing device or mobile computing device such as a smartphone, tablet, laptop computer, or desktop computer. In some embodiments, the customer client device 100 runs a client application that uses an application programming interface (API) and communicates with the online system 140.

[0009] A customer places an order with the online system 140 using a customer client device 100. The order specifies a set of items to be delivered to the customer. As used herein, “item” means a commodity or product that can be offered to the customer via the online system 140. The order may include an item identifier (e.g., an inventory management unit or price lookup code) for the items to be delivered to the user, and may include the quantity of items to be delivered. Furthermore, the order may further include the delivery location where the ordered items will be delivered and the period over which the items should be delivered. In some embodiments, the order also specifies one or more retailers from which the ordered items should be collected.

[0010] The customer client device 100 presents the customer with an ordering interface. The ordering interface is a user interface that the customer can use to place an order with the online system 140. The ordering interface may be part of a client application running on the customer client device 100. The ordering interface allows the customer to search for available items via the online system 140 and to select items to add to their “shopping list.” As used herein, the “shopping list” is a provisional set of items that the user has selected for an order but has not yet confirmed for the order. The ordering interface allows the customer to update their shopping list, for example, by changing the quantity of items, adding or removing items, or adding item instructions that specify how the items should be collected.

[0011] The customer client device 100 may receive additional content from the online system 140 and present it to the customer. For example, the customer client device 100 may receive coupons, recipes, or item suggestions. The customer client device 100 may present the received additional content to the customer when the customer places an order using the customer client device 100 (for example, as part of the ordering interface).

[0012] Furthermore, the customer client device 100 includes a communication interface that allows the customer to communicate with the picker processing the customer's order. This communication interface allows the user to enter text-based messages and send them to the picker client device 110 via the network 130. The picker client device 110 receives messages from the customer client device 100 and presents the messages to the picker. The picker client device 110 also includes a communication interface that allows the picker to communicate with the customer. The picker client device 110 sends messages provided by the picker to the customer client device 100 via the network 130. In some embodiments, messages sent between the customer client device 100 and the picker client device 110 are transmitted via the online system 140. In addition to text messaging, the communication interfaces of the customer client device 100 and the picker client device 110 may allow the customer and the picker to communicate via voice or video communication, such as telephone, voice over IP call, or video call.

[0013] The picker client device 110 is a client device that allows a picker to interact with a customer client device 100, a retailer computing system 120, or an online system 140. The picker client device 110 can be a personal computing device or a mobile computing device such as a smartphone, tablet, laptop computer, or desktop computer. In some embodiments, the picker client device 110 runs a client application using an application programming interface (API) and communicates with the online system 140.

[0014] The picker client device 110 receives orders from the online system 140 for the picker to process. The picker processes the orders by collecting the items listed in the orders from the retailer. The picker client device 110 presents the items included in the customer's order to the picker in the collection interface. The collection interface is a user interface that provides the picker with information about the items to be collected for the customer's order and the quantities of each item. In some embodiments, the collection interface provides multiple orders from multiple customers for the picker to process simultaneously from the same retail store. The collection interface further presents any instructions that the customer may have included in relation to collecting the items in the order. Furthermore, the collection interface may present the location of each item in the retail store and may specify the order in which the items should be collected to improve the efficiency of the picker's collection of items. In some embodiments, the picker client device 110 transmits the items collected by the picker in real time to the online system 140 or the customer client device 100 as the picker collects the items.

[0015] A picker can use the picker client device 110 to track the items they have collected and ensure that they have collected all items for an order. The picker client device 110 may include a barcode scanner capable of determining an item identifier encoded in a barcode attached to an item. The picker client device 110 compares this item identifier to an item in the order the picker is processing, and if the item identifier corresponds to an item in the order, the picker client device 110 identifies that item as collected. In some embodiments, instead of using a barcode scanner, or in addition to it, the picker client device 110 captures one or more images of an item and determines the item identifier of the item based on the images. The picker client device 110 may determine the item identifier directly or by transmitting the images to an online system 140. Furthermore, the picker client device 110 determines the weight of items that are priced by weight. The picker client device 110 may prompt the picker to manually enter the weight of the item, or it may communicate with the retailer's weighing system to receive the weight of the item.

[0016] When a picker has collected all the items in an order, the picker client device 110 instructs the picker on where to deliver the items in the customer's order. For example, the picker client device 110 displays the delivery location from the order to the picker. The picker client device 110 also provides navigation instructions for the picker to travel from the retail store to the delivery location. If the picker is processing multiple orders, the picker client device 110 identifies which items should be delivered to which delivery location. The picker client device 110 may provide navigation instructions from the retail store to each of the delivery locations. The picker client device 110 may receive one or more delivery locations from the online system 140 and provide the picker with these delivery locations so that the picker can deliver the corresponding one or more orders to those locations. The picker client device 110 may also provide navigation instructions to the picker from the retail store where the picker collected the items to one or more delivery locations.

[0017] In some embodiments, the picker client device 110 tracks the picker's location as the picker delivers the order to the delivery location. The picker client device 110 collects location data and transmits it to an online system 140. The online system 140 may transmit the location data to a customer client device 100 for display to the customer so that the customer can track when their order will be delivered. Furthermore, the online system 140 may generate updated navigation instructions for the picker based on the picker's location. For example, if the picker takes an incorrect turn while moving to the delivery location, the online system 140 determines the picker's updated location based on the location data from the picker client device 110 and generates updated navigation instructions for the picker based on the updated location.

[0018] In one or more embodiments, a picker is a single person who collects items for an order from a retail store and delivers the order to a delivery location. Alternatively, multiple people may act as pickers for an order. For example, multiple people may pick up items at a retail store for a single order. Similarly, the person who delivers the order to its delivery location may be different from the person who received the items from the retail store. In these embodiments, each person may have a picker client device 110 that can be used to interact with an online system 140.

[0019] Furthermore, while the description herein may primarily refer to the picker as a human, in some embodiments some or all of the steps performed by the picker may be automated. For example, a semi-autonomous or fully autonomous robot may collect items in a retail store for an order, and an autonomous vehicle may deliver the order from the retail store to the customer.

[0020] The retailer computing system 120 is a computing system operated by a retailer interacting with the online system 140. As used herein, “retailer” refers to an entity that operates a “retail store,” meaning a store, warehouse, or other building from which pickers can collect items. The retailer computing system 120 may store and provide item data to the online system 140 and periodically update the online system 140 with updated item data. For example, the retailer computing system 120 may provide item data indicating which items are available in the retail store and the quantities of those items. Furthermore, the retailer computing system 120 may transmit updated item data to the online system 140 when an item is no longer available in the retail store. Furthermore, the retailer computing system 120 may provide the online system 140 with updated item prices, sales, or availability. Furthermore, the retailer computing system 120 may receive payment information from the online system 140 for orders processed by the online system 140. Alternatively, the retailer computing system 120 may provide payment to the online system 140 for a portion of the overall cost of the user's order (for example, as a commission).

[0021] The customer client device 100, the picker client device 110, the retailer computing system 120, and the online system 140 can communicate with each other via the network 130. The network 130 is a collection of computing devices that communicate via wired or wireless connections. The network 130 may include one or more local area networks (LANs) or one or more wide area networks (WANs). Network 130 is a comprehensive term that can refer to any or all of the standard layers used to describe a physical or virtual network, such as the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, and application layer, as referred to herein. The network 130 may include a physical medium for communicating data from one computing device to another, such as an MPLS line, fiber optic cable, cellular connection (e.g., 3G, 4G, or 5G spectrum), or satellite. The network 130 may also use networking protocols such as TCP / IP, HTTP, SSH, SMS, or FTP to transmit data between computing devices. In some embodiments, the network 130 may include Bluetooth® or Near Field Communication (NFC) technology or protocol for local communication between computing devices. The network 130 may transmit encrypted or unencrypted data.

[0022] The online system 140 is an online system that allows customer users to order items provided by a picker from a retailer. The online system 140 receives an order from the customer client device 100 via the network 130. The online system 140 selects a picker to process the customer's order and sends the order to the picker client device 110 associated with the picker. The picker collects the ordered items from the retail store and delivers the ordered items to the customer. The online system 140 bills the customer for the cost of the order and provides a portion of the payment from the customer to the picker and the retailer.

[0023] As an example, the online system 140 may enable a customer to order groceries from a grocery retail store. In the customer's order, the groceries to be delivered from the grocery store and the quantity of each grocery can be specified. The customer's client device 100 sends the customer's order to the online system 140, and the online system 140 selects a picker to move to the grocery store retail store to collect the groceries ordered by the customer. When the picker user collects the groceries ordered by the customer user, the picker delivers the groceries to the location sent to the picker client device 110 by the online system 140. The online system 140 will be described in more detail below with reference to FIG. 2.

[0024] The model-providing system 150 receives requests from the online system 140 and performs one or more inference tasks using a machine learning model. The inference tasks include, but are not limited to, natural language processing (NLP) tasks, speech processing tasks, image processing tasks, and video processing tasks. In one or more embodiments, the machine learning model deployed by the model-providing system 150 is a model configured to perform one or more NLP tasks. NLP tasks include, but are not limited to, text generation, query processing, machine translation, and chatbot applications. In one or more embodiments, the language model is configured as a transformer neural network architecture. Specifically, the transformer model is coupled to receive tokenized sequential data into a sequence of input tokens and generates a sequence of output tokens depending on the inference task being performed.

[0025] The model providing system 150 receives a request including input data (e.g., text data, audio data, image data, or video data), and encodes the input data into a set of input tokens. In one or more embodiments, the machine learning model is a generative multimodal transformer architecture combined to receive various data modalities (e.g., images, text, videos) and generate tokens corresponding to the various data modalities (e.g., images, text, videos). The model providing system 150 applies the machine learning model to generate a set of output tokens. Each token in the set of input tokens or the set of output tokens may correspond to a text unit. For example, a token may correspond to a word, punctuation mark, space, phrase, paragraph, etc. For an exemplary query processing task, the language model may receive a sequence of input tokens representing a query and generate a sequence of output tokens representing a response to the query. For a translation task, the transformer model may receive a sequence of input tokens representing a paragraph in German and generate a sequence of output tokens representing a translation of the paragraph or text in English. For a text generation task, the transformer model may receive a prompt and continue a conversation or expand a given prompt with human-like text.

[0026] If the machine learning model is a language model, the sequence of input or output tokens is arranged as a tensor having one or more dimensions, e.g., 1-dimensional, 2-dimensional, or 3-dimensional. For example, one dimension of the tensor may represent the number of tokens (e.g., sentence length), one dimension of the tensor may represent the number of samples in a batch of input data being processed together, and one dimension of the tensor may represent space in the embedding space. However, in other embodiments, it is understood that the input or output data can be configured as any number of appropriate dimensions, depending on whether the data is in the form of image data, video data, audio data, etc. For example, for 3-dimensional image data, the input data may be a series of pixel values ​​arranged along a first and second dimension, and further arranged along a third dimension corresponding to the RGB channels of the pixels.

[0027] In one or more embodiments, the language model is a Large-Scale Language Model (LLM) trained on a large corpus of training data to produce outputs for NLP tasks. LLMs can be trained on vast amounts of text data, often containing billions of words or text units. Large amounts of training data from diverse data sources allow LLMs to produce outputs for many inference tasks. An LLM in a deep neural network (e.g., a transformer architecture) may have a considerable number of parameters, e.g., at least 1 billion, at least 15 billion, at least 135 billion, at least 175 billion, at least 500 billion, at least 1 trillion, or at least 1.5 trillion.

[0028] Because LLMs have a considerable parameter size and require a large amount of computational power to infer or train, LLMs can be deployed on infrastructure consisting of supercomputers that provide enhanced computing power (e.g., graphics processing units (GPUs) for training or deploying deep neural network models). In one example, an LLM could be trained and hosted on a cloud infrastructure service. An LLM can be trained by an online system 140 or by entities / systems different from the online system 140. An LLM can be trained on large amounts of data from various data sources, such as websites, articles, and web posts. From this vast amount of data combined with the computing power of the LLM, it can perform various inference tasks and synthesize and formulate output responses based on information extracted from the training data.

[0029] In one or more embodiments, when a machine learning model including an LLM is a transformer-based architecture, the transformer has a generative pre-training (GPT) architecture including a set of decoders, each performing one or more operations for inputting data into each decoder. The decoders may include attention operations that generate keys, queries, and values ​​from input data to the decoders and produce attention outputs. In another embodiment, the transformer architecture may have an encoder-decoder architecture, which may include a set of encoders coupled to a set of decoders. The encoders or decoders may include one or more attention operations.

[0030] While LLMs with transformer-based architectures are described as the primary embodiment, it is understood that in other embodiments, the language model may be configured as any other suitable architecture, including but not limited to, long-time short-term memory (LSTM) networks, Markov networks, BARTs, generative adversarial networks (GANs), and diffusion models (e.g., diffusion LMs). The LLM is configured to receive prompts and generate responses to those prompts. Prompts may contain additional contextual information useful for responding to task requests and queries. The LLM infers responses to queries from knowledge trained on the contextual information contained in the prompts and / or from the contextual information contained in the prompts.

[0031] In one or more embodiments, the inference task for the model-providing system 150 may be based primarily on the inference and summarization of knowledge specific to the online system 140, rather than on general knowledge encoded in the weights of the machine learning model of the model-providing system 150. Thus, one type of inference task may, in conjunction with the machine learning model of the model-providing system 150, perform various types of queries on large amounts of data in an external corpus. For example, the inference task may perform question answering, text summarization, text generation, etc., based on the information contained in the external corpus.

[0032] Therefore, in one or more embodiments, the online system 140 is connected to the interface system 160. The interface system 160 receives an external corpus of data from the online system 140 and constructs a structured index on the data using another machine learning language model or heuristic. Based on the external data, the interface system 160 receives one or more task requests from the online system 140. The interface system 160 constructs one or more prompts for input to the model providing system 150. The prompts may include the user's task request and context obtained from the structured index of the external data. In one example, the context in the prompt includes a portion of the structured index as contextual information for the query. The interface system 160 obtains one or more responses to the query from the model providing system 150 and synthesizes the responses. While the online system 140 can generate prompts using the external data as context, often the amount of information in the external data exceeds the prompt size limit configured by the machine learning language model. The interface system 160 can resolve the prompt size limit by generating a structured index of the data, providing a data connector to the external data and a flexible connector to the external corpus.

[0033] In one or more embodiments, the online system 140, in conjunction with the model providing system 150 and / or interface system 160, performs one or more reasoning tasks to continuously monitor the conversation between the user and the shopper to determine whether a message sent by the sender can be responded to by the recipient rather than prompting a manual response. In the main examples referred to throughout this specification, the sender may be a shopper fulfilling an order, the shopper sending a message containing a question to the user of the online system 140 who placed the order, the question seeking feedback from the user.

[0034] For example, a message from a shopper might read, "We're sorry, but the [specific item] you ordered is not available at this store." A message asking, "Is there anything else you need?" is displayed along with an image taken by the shopper at the store where they were looking for the item. The online system 140 constructs a prompt that includes the message, a task request to the LLM regarding whether an automated response can be generated for the message, and other input information, including the current chat history, attached images, geolocation data, and previous conversation data between the shopper and the user. For example, the online system 140 receives a response from the LLM saying, "You are in the wrong section of the store." Figure 3 shows an example of a communication interface, which will be described in more detail below.

[0035] Based on the output from the LLM, the online system 140 identifies a response that includes, for example, actions the shopper can take or alternative options suggested to the user based on previous order data. In one or more embodiments, the online system 140 automatically provides a response to a message without manual involvement from the recipient (for example, without requiring the user who placed the order through the online system 140 to acknowledge and / or respond to the message). The online system 140 sequentially provides one or more messages from the conversation to the model providing system 150 and / or interface system 160, receives a response for each message indicating whether the message can be intervened, and, if possible, automatically responds with a response.

[0036] In one or more embodiments, given a conversation between a shopper and a user, the online system 140 monitors the messages exchanged during the conversation. Regarding the messages, the online system 140 parses them and provides one or more inputs to the model-providing system 150 and / or the interface system 160. Together with the model-providing system 150 or the interface system 160, the online system 140 identifies the portion of the message being questioned and the portion of the message related to the question (e.g., chat messages, attached images, and the shopper's current geographical location within the store). The online system 140 obtains a response indicating whether the question can be automatically intervened. For example, the online system 140 might infer that the shopper's question is incorrect (e.g., the shopper is in the wrong location in the store), and the online system 140 can automatically correct it and intervene with a response containing the appropriate correction.

[0037] Based on the decision, the online system 140 identifies whether the message is one that the online system 140 can automatically respond to with one or more suggestions or corrections. Specifically, if the online system 140 has a high degree of confidence that it can automatically answer the question, the online system 140 may prevent the sender's message from being sent to the end user and instead send a response on its behalf. Returning to the example above, the online system 140 may identify that the question can be answered by finding the correct location of the requested item and may respond to the shopper with instructions to go to a specific section using a store map.

[0038] In this way, the online system 140 identifies messages for which an automated response can be created as automated feedback to the message, and automatically intervenes in the conversation on behalf of the recipient. This allows the online system 140 to eliminate human intervention for messages requiring user feedback, enabling automated and efficient processing of customer orders. Furthermore, recipients (e.g., the user of the order) may not know the response to an inquiry received via the communication interface, even though the inquiry could often be easily answered with the right information. By using the method described herein, the online system 140 can use LLM to extract relevant information for responding to an inquiry received via the communication interface, and as a result, can create a response in real time.

[0039] Furthermore, because the context resides within the online computing system, conversations occurring between users (e.g., pickers and customer users) cannot be monitored by human auditors, for example, due to privacy concerns. Even if they could, auditors might not be able to extract information from large external databases (e.g., item catalogs, retail store maps) when conversations are occurring in real time between users to process orders. Therefore, by configuring the computing system (e.g., machines, servers) to identify opportunities to generate responses intervened in conversations and generating these responses in conjunction with the LLM, the online system 140 can quickly correct information, make suggestions to users, or improve operations and take appropriate actions to enhance user satisfaction.

[0040] Figure 1B shows an exemplary system environment for an online system 140 in one or more embodiments. The system environment shown in Figure 1B includes a customer client device 100, a picker client device 110, a retailer computing system 120, a network 130, and an online system 140. Alternative embodiments may include more, fewer, or different components than those shown in Figure 1B, and the functions of each component may be divided among components different from those described below. Furthermore, each component may perform its respective function in response to a human request or automatically without human intervention.

[0041] The exemplary system environment in Figure 1A shows an environment in which the model-providing system 150 and / or interface system 160 are each managed by entities separate from the entity that manages the online system 140. In one or more embodiments, as shown in the exemplary system environment in Figure 1B, the model-providing system 150 or interface system 160 is managed and deployed by the entity that manages the online system 140.

[0042] Figure 2 shows an exemplary system architecture for an online system 140 in one or more embodiments. The system architecture illustrated in Figure 2 includes a data acquisition module 200, a content presentation module 210, an order management module 220, a question analysis module 225, a predictive response module 227, a machine learning training module 230, and a data store 240. Alternative embodiments may include more, fewer, or different components than those shown in Figure 2, and the functionality of each component may be divided among components different from those described below. Furthermore, each component may perform its respective function in response to a human request or automatically without human intervention.

[0043] The data collection module 200 collects data used by the online system 140 and stores the data in the data store 240. The data collection module 200 may collect data describing the user if the user has previously explicitly consented to the online system 140 collecting such data. Furthermore, the data collection module 200 may encrypt all data, including sensitive or personal data describing the user.

[0044] For example, the data collection module 200 collects customer data, which is information or data describing customer characteristics. Customer data may include the customer's name, address, shopping preferences, favorite items, or saved payment methods. Customer data may also include default settings established by the customer, such as a default retailer / store, payment method, delivery location, or delivery time slot. The data collection module 200 may collect customer data from sensors on the customer client device 100 or based on interactions between the customer and the online system 140.

[0045] The data collection module 200 also collects item data, which is information or data that identifies and describes items available in a retail store. Item data may include item identifiers for available items and the quantity of items associated with each item identifier. Furthermore, item data may also include item attributes such as size, color, weight, stock unit (SKU), or serial number. Item data may further include purchase rules associated with each item, if any exist. For example, age-restricted items such as alcohol and tobacco may be flagged accordingly in the item data. Item data may also include information useful for predicting item availability in a retail store. For example, for each item-retailer combination (a specific item in a particular warehouse), item data may include the time the item was last found, the time the item was last not found (a picker searched for the item but could not find it), the percentage of times the item was found, or the item's popularity. The data collection module 200 may collect item data from a retailer computing system 120, a picker client device 110, or a customer client device 100.

[0046] An item category is a set of items of similar types. Items within an item category may be considered equivalent to one another, or they may be interchangeable within an order. For example, sourdough bread of different brands may be different items, but these items may belong to the "Sourdough Bread" item category. Item categories can be generated by humans, and items may be entered by humans. Item categories can also be generated automatically by the online system 140 (for example, using a clustering algorithm).

[0047] The data collection module 200 also collects picker data, which is information or data describing the characteristics of the picker. For example, a picker's picker data may include the picker's name, location, how often the picker has processed orders from the online system 140, the picker's customer rating, the retailer from which the picker has collected items, or the picker's previous shopping history. Furthermore, the picker data may include preferences expressed by the picker, such as the picker's preferred retailer for collecting items, how far they are willing to travel to deliver items to customers, how many items they are willing to collect at once, the time frame the picker is willing to process orders, or payment information (e.g., bank account) to which the picker will be paid to process orders. The data collection module 200 collects picker data from sensors on the picker client device 110 or from interactions between the picker and the online system 140.

[0048] Furthermore, the data collection module 200 collects order data, which is information or data describing the characteristics of an order. For example, order data may include item data of the items included in the order, the delivery location of the order, the customer associated with the order, the retail store where the customer wants to pick up the ordered items, or the time frame in which the customer wants the order delivered. The order data may further include information describing how the order was processed, such as which picker processed the order, when the order was delivered, or customer feedback on the delivery of the order. In some embodiments, the order data may include user data of the user associated with the order, such as customer data of the customer who placed the order or picker data of the picker who processed the order.

[0049] In one or more embodiments, the data collection module 200 also collects communication data, which is various types of communication between shoppers and users of the online system 140. For example, the data collection module 200 may capture text-based, voice call, and video call-based communications between shoppers and users of the online system 140 as orders are submitted and fulfilled. The data collection module 200 may store the communication information by individual user, individual shopper, geographical region, subset of users with similar attributes, and so on.

[0050] The content presentation module 210 selects content to present to the customer. For example, the content presentation module 210 selects which items to present to the customer while the customer is placing an order. The content presentation module 210 generates and transmits an order interface for the customer to order items. The content presentation module 210 inputs items that the customer may select to add to the order into the order interface. In some embodiments, the content presentation module 210 presents a catalog of all items available to the customer that the customer can browse to select items to order. The content presentation module 210 may also identify the items that the customer is most likely to order and present those items to the customer. For example, the content presentation module 210 may score items and rank them based on those scores. The content presentation module 210 displays items that have scores above some threshold (e.g., the top n items or p-th percentiles of an item).

[0051] The content presentation module 210 may use an item selection model to score items for presentation to the customer. The item selection model is a machine learning model trained to score customer items based on item data of the item and customer data of the customer. For example, the item selection model may be trained to determine the likelihood that a customer will order an item. In some embodiments, the item selection model scores items using item embeddings that describe the item and customer embeddings that describe the customer. These item embeddings and customer embeddings may be generated by separate machine learning models and stored in the data store 240.

[0052] In some embodiments, the content presentation module 210 scores items based on a search query received from a customer client device 100. The search query is free text of a word or set of words that indicates an item of interest to the customer. The content presentation module 210 scores items based on their relevance to the search query. For example, the content presentation module 210 may apply natural language processing (NLP) techniques to the text within the search query to generate a search query representation (e.g., an embedding) that represents the features of the search query. The content presentation module 210 may use the search query representation (e.g., by comparing the search query embedding with an item embedding) to score candidate items for presentation to the customer.

[0053] In some embodiments, the content presentation module 210 scores items based on their predicted availability. The content presentation module 210 may use an availability model to predict the availability of items. The availability model is a machine learning model trained to predict the availability of items in a retail store. For example, the availability model may be trained to predict the likelihood that an item is available in a retail store, or it may predict an estimate of the number of items that are available in a retail store. The content presentation module 210 may weight the item scores based on their predicted availability. Alternatively, the content presentation module 210 may exclude items from presentation to the customer based on whether their predicted availability exceeds a threshold.

[0054] In one or more embodiments, the content presentation module 210 receives one or more recommendations to present to the customer while the customer is operating the ordering interface. The customer's list of ordered items may be called a basket. As illustrated in conjunction with Figures 1A and 1B, the recommendations are generated based on the inferred purpose of the customer's basket and include one or more suggestions to the customer to better fulfill the purpose of the basket.

[0055] In one example, a recommendation is in the form of one or more equivalent baskets, which are modifications to an existing basket that serve the same or similar purpose as the original basket. Equivalent baskets are adjusted with respect to metrics such as cost, health, and whether the basket is sponsored. For example, an equivalent basket might be a healthier option compared to the existing basket, or a cheaper option compared to the existing basket. The content presentation module 210 may present equivalent baskets to the customer via an ordering interface with indicators showing how the equivalent basket is improved or how it differs from the existing basket (e.g., more cost-effective, healthier, or sponsored by a specific organization). The content presentation module 210 may allow the customer to exchange an existing basket for an equivalent basket.

[0056] In one example, if the basket contains a list of edible ingredients, the recommendation may take the form of a list of recipes that can be made with the ingredients, and a list of additional ingredients to make each recipe a reality. The content presentation module 210 may present the customer with each suggested recipe and a list of additional ingredients to make that recipe a reality. The content presentation module 210 may allow the customer to automatically add one or more additional ingredients to their basket.

[0057] An order management module 220 manages customer orders for items. The order management module 220 receives orders from customer client devices 100 and assigns pickers to process the orders based on picker data. For example, the order management module 220 assigns orders to pickers based on the picker's location and the location of the retail store where the ordered items are collected. The order management module 220 may also assign orders to pickers based on the number of items in the order, the vehicle operated by the picker, the delivery location, the picker's preferences regarding the distance to travel to deliver the order, customer ratings of the picker, or how often the picker agrees to process the order.

[0058] In some embodiments, the order management module 220 determines when to assign an order to a picker based on the delivery time frame requested by the customer along with the order. The order management module 220 calculates the estimated time it will take for the picker to collect the items in the order and deliver the ordered items to the delivery location of the order. If the picker is to process the order immediately, the order management module 220 assigns the order to the picker at the point when it is most likely that the picker will deliver the order within the time frame. Therefore, when the order management module 220 receives an order, if the time frame is far in the future, the order management module 220 may delay assigning the order to a picker.

[0059] When the order management module 220 assigns an order to a picker, it sends the order to the picker client device 110 associated with the picker. The order management module 220 may also send navigation instructions from the picker's current location to the retail store associated with the order. If the order includes items to be collected from multiple retail stores, the order management module 220 may identify the retail stores to the picker and specify the order in which the picker should visit them.

[0060] The order management module 220 may track the picker's location via the picker client device 110 to determine when the picker will arrive at the retail store. When the picker arrives at the retail store, the order management module 220 sends the order to the picker client device 110 for display to the picker. When the picker collects items at the retail store using the picker client device 110, the order management module 220 receives item identifiers for the items the picker has collected for the order. In some embodiments, the order management module 220 receives images of the items from the picker client device 110 and applies computer vision techniques to the images to identify the items depicted in the images. The order management module 220 may track the picker's progress as the picker collects items for the order and may send progress updates to the customer client device 100 explaining which items have been collected for the customer's order.

[0061] In some embodiments, the order management module 220 tracks the location of pickers within the retail store. The order management module 220 determines the picker's location within the retail store using sensor data from the picker client device 110 or from sensors located within the retail store. The order management module 220 may send instructions to the picker client device 110 to display a map of the retail store showing where the picker is located. Furthermore, the order management module 220 may instruct the picker client device 110 to display the location of the items the picker is collecting and may further display navigation instructions on how the picker can move from their current location to the location of the next item to collect in the order.

[0062] The order management module 220 determines when the picker has collected all items in an order. For example, the order management module 220 may receive a message from the picker client device 110 indicating that all items for the order have been collected. Alternatively, the order management module 220 may receive item identifiers for the items collected by the picker and determine when all items in the order have been collected. Once the order management module 220 determines that the picker has completed the order, it sends the delivery location of the order to the picker client device 110. The order management module 220 may also send navigation instructions to the picker client device 110 specifying how to get from the retail store to the delivery location, or to the next retail store to collect further items. The order management module 220 tracks the picker's location as they move to the delivery location of the order and updates the picker's location to the customer so that the customer can track the progress of the order. In some embodiments, the order management module 220 calculates the estimated arrival time of the picker at the delivery location and provides the estimated arrival time to the customer.

[0063] In some embodiments, the order management module 220 facilitates communication between the customer client device 100 and the picker client device 110. As described above, the customer may use the customer client device 100 to send a message to the picker client device 110. The order management module 220 receives the message from the customer client device 100 and sends it to the picker client device 110 to present the message to the picker. The picker may use the picker client device 110 to send a message to the customer client device 100 in a similar manner.

[0064] The order management module 220 coordinates customer payments for orders. The order management module 220 receives payments for orders using payment information provided by the customer (e.g., credit card number or bank account). In some embodiments, the order management module 220 stores payment information for use in subsequent orders by the customer. The order management module 220 calculates the total cost of the order and bills the customer for that cost. The order management module 220 may provide a portion of the total cost for processing the order to the picker and another portion of the total cost to the retailer.

[0065] Intervention module 225 acquires messages occurring between customer client device 100 and picker client device 110 and determines whether the message sent by the sender can be processed automatically rather than prompting the recipient for a manual response. Specifically, intervention module 225 continuously monitors and acquires messages between the user and shopper from order management module 220. In one or more embodiments, intervention module 225 determines whether the message sent by the sender can be responded to automatically rather than prompting the recipient for a manual response. Specifically, intervention module 225 determines whether the message, including a question and other contextual information about the question, is similar to a previous instance of a message to which the online system 140 has previously responded and which can suggest one or more answers (e.g., corrections, suggested actions) in response to the message.

[0066] Figure 3 shows an exemplary conversation between a shopper and a user on a communication interface generated on the user's client device 110, according to one embodiment. As shown in Figure 3, at some point in the conversation, the picker sends a message 302 to the user indicating that the ordered item “Ace3850-TSF Heavy Duty Packing Tape” is not in stock and whether the user would like to exchange the item. The intervened response “Hello. You are in the wrong section of the store. Please follow the map to the hardware section of the store and find the Ace3850-TSF Heavy Duty Packing Tape.” 306 instructs the shopper to correct their location based on the intervened message 302.

[0067] In one or more embodiments, the intervention module 225 constructs prompts and task requests to the LLM regarding whether it can intercept a message for a response, instead of manually waiting for a response from the user. The prompt may include one or more inputs related to the message. An exemplary prompt to the LLM of the model-providing system 150 might be, "Given a message named [Message Content] and [Other Inputs Related to the Message], is it possible to provide a response that provides feedback on the message?" The online system 140 might receive a response from the LLM such as, "Based on the information you have provided, the sender of the message is in the wrong section of the store."

[0068] In such an example, the shopper's location within the store can be inferred from the shopper's analyzed messages and other inputs to the online system 140, and it can then be determined that the shopper is not near the requested item. It can also be inferred that the location of the requested item is provided through the online system 140's database, which has a precise data representation of the item listed in the product catalog and the locations where the requested item can be found.

[0069] In one or more embodiments, the intervention module 225 includes a question analysis component and a predictive response component. The question analysis component analyzes the core question being asked and the context of the question, for example, in conjunction with a multimodal LLM. The predictive response component identifies whether the message or request can be automatically responded to by the system. If the predictive response component has a high confidence level, the component prevents the recipient's response from being sent to the sender and instead sends a clear answer on behalf of the recipient.

[0070] In one or more embodiments, the intervention module 225 provides several inputs, such as chat messages analyzed via LLM, attached images analyzed via algorithmic image analysis and object identification, shopper application location data to identify the approximate location of shoppers in the store, storage catalog and inventory data to obtain the expected geographic location of products in the store, past order data within that location in the store, past item location information within that location, past conversation data between shoppers and customers, and past conversations specific to the store (for example, because some stores display products in various locations, often multiple locations within the store).

[0071] In one or more embodiments, the intervention module 225 may also receive time-stamped images and other sensor data from the smart shopping cart. The smart shopping cart includes multiple load sensors and cameras that capture images and sensor data of the environment surrounding the shopping cart. The question analysis module 225 may receive the images and sensor data to determine the item availability of an item. The intervention module 225 receives images of the surrounding aisles, which can provide the LLM with a greater context regarding the expected geographical location of the product in the store, the item's availability, and the approximate location of the shopper in the store. The time-stamped images and sensor data received from the smart shopping cart may also provide the intervention module 225 with greater certainty regarding the item's availability, the expected geographical location of the item in the store, and the approximate location of the shopper in the store.

[0072] Based on the extracted core questions, the intervention module 225 determines one or more answer options that it can provide as answers to the questions. For each option, the intervention module 225 also determines the confidence level that the question is answerable. If the confidence level is high, the intervention module 225 may formulate a response to the message and output the message to the user.

[0073] In the example shown in Figure 3, the intervention module 225 determines with high confidence that the question is a shopper error and can be corrected, for example, by pointing to the correct location of the requested item, which is obtained by searching for the item in the retail store and its location in the database. The intervention module 225 outputs to the shopper that the shopper is in the wrong location in the store, creates a text response to the query, and suggests that the shopper navigate to the hardware section according to the store's map and find the requested item. The user does not need to respond on the communication interface, but in the meantime, can see the communication chain between the picker's original message and the automated message generated using LLM.

[0074] Figure 3 shows one example of how the online system 140 can automatically respond to a message. Another example is when a user, as the sender, inquires with a shopper via message, looking for an item not included in the user's order. The intervention module 225, in conjunction with the model providing system 150 or the interface system 160, parses the message from the user, infers the user's inquiry, and cross-checks the requested item mentioned by the user with items in the catalog of the online system 140. The intervention module 225 determines that it can automatically prompt the user to either select the "best" item or to automatically add the item to the order. The intervention module 225 may output a text response indicating that the requested item has been automatically added to the order, or it may make an API call to the appropriate API to add the requested item to the user's order.

[0075] As another example, when a user, as the sender, queries whether a particular item in an order is the basis or key of the order, intervention module 225 may suggest to the user whether they want to cancel the order if the key item is not found. When the picker processes the order, they notify the application in online system 140 that the item is not found. Intervention module 225 automatically notifies the picker and the user that the order will be canceled and generates an API call to cancel the order.

[0076] As yet another example, a shopper, as the sender, sends a message indicating that they cannot find a particular item, takes an image of the shelf area as evidence, and sends a message to the user indicating that the particular item is unavailable. The intervention module 225, in conjunction with the model providing system 150 or the interface system 160, may infer that the ordered item is actually visible in the image and may determine that it can provide an automated response to the shopper to inform them that the requested item is in the shelf area but has a new packaging label. The intervention module 225 can output a message to the shopper indicating that the requested item is in the shelf area and can examine the requested item in more detail.

[0077] Alternatively, if the requested item is not present in the image, the intervention module 225 may determine that another item that is a good substitute for the requested item is present in the image and can provide an automated response to the shopper informing them of this fact. The intervention module 225 may output an automated response to the shopper indicating that the replacement item in the image is a good substitute for the original item and that the shopper can consider picking up the replacement item instead.

[0078] In the example shown in Figure 4, the intervention module 225 determines with high confidence that the requested item is not present in the image and automatically responds with a substitute for the requested item. In an exemplary chat log 402 between shopper Armstrong A and the user, the shopper submits a photograph of shelf 404 where the requested item, double chocolate chip brownie mix, should be located. The intervention module 225 analyzes the received image according to one or more embodiments and determines that the shopper is in the correct location and that the item is indeed out of stock. The intervention module 225 notifies the user, "This item, double chocolate chip brownie mix, appears to be out of stock." 406. In response to the detection that the requested double chocolate chip brownie mix is ​​out of stock, the intervention module 225 determines and automatically presents a corresponding substitute, milk chocolate chip brownie mix 408, to the user, according to one or more embodiments. The intervention module 225 automatically presents the user with the message, "We're sorry, but this item is out of stock. Would you like milk chocolate chip brownie mix instead?" 408. Fine-tuning of LLM by intervened response In one or more embodiments, the intervention module 225, in conjunction with the machine learning training module 230, may perform fine-tuning of the LLM. To perform fine-tuning, the intervention module 225 retrieves training data from previous instances of conversations between a sender and receiver on a communication interface (e.g., a messaging interface) where one or more intervened responses were generated in conjunction with the LLM.

[0079] In one or more embodiments, the intervention module 225 identifies positive instances of feedback in which positive feedback was received for an intervened response generated based on the LLM. In other embodiments, the intervention module 225 may identify negative instances of feedback in which a user provided negative feedback for an intervened response. The intervention module 225 continues to monitor the conversation after one or more intervened responses have been generated and may classify the response as having received positive feedback if a person (e.g., a receiver) confirms the content of the intervened response (e.g., upvotes), or may classify the response as having received negative feedback if a person (e.g., a receiver) confirms that the content of the intervened response is incorrect (e.g., downvotes).

[0080] As an example, returning to the exemplary conversation in Figure 3, after the intervened response 306 is displayed on the communication interface, the picker might respond, "No, I think I'm in the correct section of the store." The user, as the receiver, might then respond, "Yes, you are in the correct section of the store. I've been to this store before," confirming that the content of the intervened response 306 is incorrect. Therefore, this conversation instance can be classified as negative feedback. In contrast, the user, as the receiver, could alternatively respond, "No, judging from the picture, it looks like you're in the wrong section of the store," confirming that the content of the intervened response 306 is correct. Therefore, this conversation instance can be classified as positive feedback.

[0081] In one or more embodiments, a training data instance may include prompts and positive outputs derived from previous conversations in which positive feedback was received. For example, a prompt may include a picker message indicating that an item in the user's order (e.g., green grapes) is out of stock, an image of a retail store shelf indicating that the item is unavailable, and other types of contextual information such as a map of the retail store. A positive output is a text response confirming that the item is out of stock and a suggestion to replace the item with another item (e.g., red grapes) presented in an image in which positive feedback was received from the user.

[0082] The intervention module 225 obtains such pairs of prompts and positive outputs for the training dataset. The intervention module 225 encodes the data into a set of input tokens, where the tokens are numerical vectors representing words, subwords, phrases, pixels, and latent pixels in a latent space. When the transformer architecture of the machine learning model (e.g., LLM) is an autoregressive architecture, the LLM can be applied to generate one or more output tokens corresponding to positive outputs. The output tokens are decoded to determine the probability that the decoded token corresponds to the corresponding token in the positive output.

[0083] The intervention module 225 determines a loss function across one or more output tokens that represents the difference (e.g., logit difference) between the tokens in the positive output and the output tokens generated by the forward pass of the transformer model. For example, the loss function could be an NLP loss for each token combined across one or more output tokens generated for the positive text. The intervention module 225 takes one or more terms from the loss function and performs backpropagation to update the parameters of the transformer architecture. How to generate an intervened response to a message Figure 5 is a flowchart of the intervention module 225 method in one or more embodiments. The intervention module 225 provides instructions for generating a communication interface for at least one client device 500. The intervention module receives messages from one or more client devices from a conversation sent from sender to receiver 510. The module generates prompts for input to a machine learning language model 520 and provides prompts to a model providing system for execution by the machine learning language model 530. The module receives a response from the machine learning language model 540, parses the response from the model providing system 550, and extracts an automated response to the message. The module determines a suggestion or action to offer to the sender in conjunction with the received response 560, and outputs a response message to the sender containing the extracted response and suggestion or action 570.

[0084] The machine learning training module 230 trains machine learning models used by the online system 140. For example, the machine learning module 230 may train an item selection model, an availability model, or any machine learning model deployed by the model providing system 150. The online system 140 may use the machine learning models to perform the functions described herein. Exemplary machine learning models include regression models, support vector machines, naive Bayes, decision trees, k-nearest neighbors, random forests, boost algorithms, k-means, and hierarchical clustering. Machine learning models may also include neural networks such as perceptrons, multilayer perceptrons, convolutional neural networks, recurrent neural networks, sequence-to-sequence models, generative adversarial networks, or transformers.

[0085] Each machine learning model includes a set of parameters. The set of parameters for a machine learning model is what the model uses to process inputs. For example, the set of parameters for a linear regression model might include the weights applied to each input variable in a linear combination that includes the linear regression model. Similarly, the set of parameters for a neural network might include the weights and biases applied to each neuron in the neural network. The machine learning training module 230 generates a set of parameters for a machine learning model by "training" the machine learning model. Once trained, the machine learning model uses the set of parameters to translate inputs into outputs.

[0086] The machine learning training module 230 trains a machine learning model based on a set of training examples. Each training example contains input data to which the machine learning model applies to generate an output. For example, each training example may include customer data, picker data, item data, or order data. In some cases, the training examples also contain labels that represent the expected output of the machine learning model. In these cases, the machine learning model is trained by comparing its output from the input data of the training example with the labels of the training example.

[0087] The machine learning training module 230 can train a machine learning model by applying an iterative process, thereby training the machine learning model on each of the sets of training examples. To train the machine learning model based on the training examples, the machine learning training module 230 applies the machine learning model to the input data of the training examples to generate an output. The machine learning training module 230 scores the output from the machine learning model using a loss function. The loss function is a function that generates a score on the output of the machine learning model such that the score is high when the machine learning model performs poorly and low when the machine learning model performs well. If the training examples include labels, the loss function is also based on the labels of the training examples. Some exemplary loss functions include the mean squared error function, mean absolute error, hinge loss function, and cross-entropy loss function. Based on the score generated by the loss function, the machine learning training module 230 updates the set of parameters of the machine learning model. For example, the machine learning training module 230 may apply gradient descent to update the set of parameters.

[0088] The data store 240 stores data used by the online system 140. For example, the data store 240 stores customer data, item data, order data, and picker data for use by the online system 140. The data store 240 also stores trained machine learning models trained by the machine learning training module 230. For example, the data store 240 may store a set of parameters for the trained machine learning model in one or more non-temporary computer-readable media. The data store 240 may use computer-readable media to store data and use a database to organize the stored data.

[0089] With respect to machine learning models hosted by the model providing system 150, the machine learning models may already be trained by entities separate from the entity responsible for the online system 140. In another embodiment, when the model providing system 150 is included in the online system 140, the machine learning training module 230 may further train the parameters of the machine learning model based on data specific to the online system 140 stored in the data store 240. As an example, the machine learning training module 230 may take a pre-trained transformer language model and further fine-tune the parameters of the transformer model using training data stored in the data store 240. The machine learning training module 230 may then provide the model to the model providing system 150 for deployment.

[0090] Figure 3 is a flowchart of a method for inferring whether an automated response can be generated to a message, according to several embodiments. Alternative embodiments may include more steps, fewer steps, or different steps than those shown in Figure 3, and the steps may be performed in a different order than those shown in Figure 3. These steps may be performed by an online system (e.g., online system 140). Furthermore, each of these steps may be performed automatically by the online system without human intervention.

[0091] Online system 140 receives messages from one or more client devices from a conversation sent from the sender to the receiver 300. Online system 140 generates prompts 310 for input to a machine learning language model. In one or more embodiments, the prompt specifies at least a message and a request to infer whether an automated response can be generated for the message, providing feedback on the message. Online system 140 provides prompts to a model provider system for execution by the machine learning language model 320. Online system 140 receives responses from the model provider system generated by running the machine learning language model on the prompts 330. Online system 140 parses the responses from the model provider system 340 and extracts an automated response to the message. In response to the automated response, online system 140 determines a suggestion or action to provide to the sender along with the response 350. Online system 140 outputs a response message to the sender containing the extracted response and suggestion or action 360. Additional considerations The foregoing description of embodiments is presented for illustrative purposes only and many modifications and variations are possible, while remaining within the scope of the principles and teachings of the foregoing description. Any step, operation, or process described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In some embodiments, the software module is implemented by a computer program product that includes one or more computer-readable media containing computer program code or instructions, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described. In some embodiments, the computer-readable media includes one or more computer-readable media and, when executed by one or more processors, includes instructions that cause one or more processors to perform steps of the instructions stored in one or more computer-readable media individually or together. Similarly, the processor comprises one or more processors or processing units that perform steps of the instructions stored in the computer-readable media individually or together.

[0092] Embodiments may also relate to products generated by computing processes described herein. Such products may store information obtained from computing processes, which are stored on non-temporary, tangible, computer-readable media, and may include any embodiment of computer program products or other data combinations described herein.

[0093] The description herein may describe processes and systems that use machine learning models in the performance of the functions described. As used herein, “machine learning model” includes one or more machine learning models that perform the functions described. A machine learning model may be stored in one or more computer-readable media containing a set of weights. These weights are parameters used by the machine learning model to transform input data received by the model into output data. The weights may be generated through a training process, thereby training the machine learning model on a set of training examples and the labels associated with those training examples. The training process may include applying the machine learning model to the training examples, comparing the output of the machine learning model to the labels associated with the training examples, and updating the weights associated with the machine learning model through a backpropagation process. The weights may be stored in one or more computer-readable media and used by the system when applying the machine learning model to new data.

[0094] The language used herein has been selected primarily for readability and explanatory purposes, and not to narrow the subject matter of the invention. Therefore, the scope of the patent rights is intended to be limited not by the detailed description of this invention, but by any claims issued in an application based on this specification.

[0095] The terms “to include,” “to have,” “to have,” “to include,” “to have,” and other variations thereof as used herein are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements alone, and may include other elements not expressly described in or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, “or” refers to an inclusive “or” and not an exclusive “or.” For example, the condition “A or B” is satisfied by any of the following: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist). Similarly, the condition “A, B, or C” is satisfied by any combination of A, B, and C being true (or existing). As an unlimited example, the condition “A, B, or C” is satisfied when A and B are true (or exist) and C is false (or does not exist). Similarly, as another non-restrictive example, the condition "A, B, or C" is satisfied if A is true (or exists) and B and C are false (or do not exist).

Claims

1. To provide instructions for generating a communication interface on one or more client devices, The one or more client devices receive messages from a conversation sent from the sender to the receiver, To generate a prompt for input to a machine learning language model, wherein the prompt specifies at least the message and one or more inputs and a request to infer whether a response to the message can be generated. To provide the prompt to the model providing system for execution by the machine learning model, The system receives a response generated by executing the machine learning model on the prompt from the model provisioning system. The process involves analyzing the response from the model-providing system and extracting the response to the message, In response to receiving the aforementioned response, generate a proposal or action to be provided to the sender together with the aforementioned response, Outputting the intervened response, including the extracted response or a description of the proposal or action, to the transmitting side, Methods that include...

2. The method according to claim 1, wherein the intervened response to the message indicates that an error occurred in fulfilling the recipient's order, and the suggestion or action is a suggestion or action by the sender to correct the error.

3. The aforementioned message is from the picker, indicating that the item in the user's order is unavailable at the retail store. The prompt includes the message, an image of part of the retail store, a map of the retail store, and picker location data that identifies the location of the picker within the retail store. The method according to claim 1, wherein the intervened response includes a message indicating that the picker is in the wrong location in the retail store.

4. Deciding on the aforementioned proposal or action is This includes obtaining the location of the item in the aforementioned retail store, The method according to claim 3, wherein the intervened response includes the suggestion to navigate to the location in the retail store in order to find the item.

5. The aforementioned message is from a user requesting the picker to identify items that are not in the user's order. Identifying the aforementioned proposal or action further includes calling an Application Programming Interface (API) call to add the aforementioned item to the user's order, The method according to claim 1, wherein the intervened response indicates that the requested item has been added to the user's order.

6. The aforementioned message is from a user who identifies the key item in the order. The intervened response further includes whether the user wishes to cancel the order if it is determined that the key item cannot be found. The method according to claim 1, further comprising canceling the order and notifying the user and picker in response to receiving a notification that one or more items cannot be found.

7. The aforementioned message is from a picker notifying the user that an item in the order cannot be found and sending an image of where the item should be located in the retail store. Determining the aforementioned proposal or action further includes correctly identifying the item in the image or proposing a substitute for the requested item. The method according to claim 1, wherein the intervened response indicates to the picker that the item is in the image, or suggests to the picker to select the suggested alternative.

8. To identify the confidence level for intervening the message and generating the intervened response, Comparing the aforementioned confidence level with a threshold, In response to the comparison, the intervened response is output to the transmitting side, The method according to claim 1, further comprising:

9. The communication interface is used to obtain one or more subsequent messages that occur after the intervened response, The intervention involves identifying whether the intervened response received positive or negative feedback from the recipient in the one or more subsequent messages. The method according to claim 1, further comprising:

10. The parameters of the machine learning model are fine-tuned based on the prompt, the intervened response, and the feedback obtained regarding the conversation. The method according to claim 9, further comprising:

11. When executed by a computer processor, the computer processor will To provide instructions for generating a communication interface on one or more client devices, The one or more client devices receive messages from a conversation sent from the sender to the receiver, To generate a prompt for input to a machine learning language model, wherein the prompt specifies at least the message and one or more inputs and a request to infer whether a response to the message can be generated. To provide the prompt to the model providing system for execution by the machine learning model, The system receives a response generated by executing the machine learning model on the prompt from the model provisioning system. The process involves analyzing the response from the model-providing system and extracting the response to the message, In response to receiving the aforementioned response, generate a proposal or action to be provided to the sender together with the aforementioned response, Outputting the intervened response, including the extracted response or a description of the proposal or action, to the transmitting side, A non-temporary, computer-readable storage medium that stores instructions for executing steps that include [a specific element].

12. The non-temporary computer-readable storage medium according to claim 11, wherein the intervened response to the message indicates that an error occurred in fulfilling the recipient's order, and the suggestion or action is a suggestion or action by the sender to correct the error.

13. The aforementioned message is from the picker, indicating that the item in the user's order is unavailable at the retail store. The prompt includes the message, an image of part of the retail store, a map of the retail store, and picker location data that identifies the location of the picker within the retail store. The non-temporary computer-readable storage medium according to claim 11, wherein the intervened response includes a message that the picker is in the wrong location in the retail store.

14. To generate the aforementioned proposal or action, This includes obtaining the location of the item in the aforementioned retail store, The non-temporary computer-readable storage medium according to claim 13, wherein the intervened response includes the suggestion for navigating to the location in the retail store to find the item.

15. The aforementioned message is from a user requesting the picker to identify items that are not in the user's order. Identifying the aforementioned proposal or action further includes calling an Application Programming Interface (API) call to add the aforementioned item to the user's order, The non-temporary computer-readable storage medium according to claim 11, wherein the intervened response indicates that the requested item has been added to the user's order.

16. The aforementioned message is from a user who identifies the key item in the order. The intervened response further includes whether the user wishes to cancel the order if it is determined that the key item cannot be found. The method further comprises canceling the order and notifying the user and picker in response to receiving a notification that one or more items cannot be found, according to claim 11, for a non-temporary computer-readable storage medium.

17. The aforementioned message is from a picker notifying the user that an item in the order cannot be found and sending an image of where the item should be located in the retail store. Determining the aforementioned proposal or action further includes correctly identifying the item in the image or proposing a substitute for the requested item. The non-temporary computer-readable storage medium according to claim 11, wherein the intervened response indicates to the picker that the item is in the image or suggests to the picker to select the suggested alternative.

18. To identify the confidence level for intervening the message and generating the intervened response, Comparing the aforementioned confidence level with a threshold, In response to the comparison, the intervened response is output to the transmitting side, A non-temporary computer-readable storage medium according to claim 11, further comprising:

19. The communication interface is used to obtain one or more subsequent messages that occur after the intervened response, The intervention involves identifying whether the intervened response received positive or negative feedback from the recipient in the one or more subsequent messages. A non-temporary computer-readable storage medium according to claim 11, further comprising:

20. The parameters of the machine learning model are fine-tuned based on the prompt, the intervened response, and the feedback obtained regarding the conversation. A non-temporary computer-readable storage medium according to claim 11, further comprising:

Images