Method, program and system for reducing restaurant serving errors

A neural network-based system accurately matches customer images with their orders, significantly reducing serving errors and improving customer satisfaction and operational efficiency in restaurants.

JP7581847B2Active Publication Date: 2024-11-13FUJIFILM BUSINESS INNOVATION CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2020214339
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-16
Filing Date
2020-12-23
Publication Date
2024-11-13
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

Existing restaurant systems fail to accurately deliver food to the correct customer due to similar appearances of customers and the overwhelming workload of waitstaff, leading to significant financial and customer satisfaction losses.

Method used

A system utilizing a convolutional neural network with batch normalization and similarity loss to match customer images with their orders, providing visual or audio guidance to waitstaff for accurate delivery.

Benefits of technology

Reduces serving errors by 90% or more, enhancing customer satisfaction and reducing operational costs and wait times.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581847000012
    Figure 0007581847000012
  • Figure 0007581847000013
    Figure 0007581847000013
  • Figure 0007581847000014
    Figure 0007581847000014
Patent Text Reader

Abstract

To provide a system and method for reducing providing errors of a restaurant.SOLUTION: A method which is executed by a computer receives an order associated with a user to capture information associated with a physical attribute of the user, executes matching operation between the physical attribute of the user about a completed order and information associated with an area of the user including the physical attribute of the user and respective physical attributes of other users to determine an object to be matched most, generates an output for identifying a user associated with the object to be matched most in the information associated with the area of the user, and provides a service provider with the output for delivering the order to the user.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Aspects of the exemplary implementation relate to methods, systems and user experiences relating to reducing delivery errors in restaurants and the like, and more specifically, to matching an image of a user ordering food with an image of an area, and providing a service provider with an assistance system to associate and deliver the completed order to the user. [Background technology]

[0002] In the related art, serving the wrong food to customers is a critical issue for service quality in restaurant businesses. According to the related art approach, waiters / waitresses must play multiple roles (e.g., ordering, serving, cleaning, etc.) and serve multiple tables at the same time, which includes remembering the appearance of customers who place orders, but who switch positions or tables without telling the server, or who look similar to each other. Waiters / waitresses may be overwhelmed with too many diners and serve food that someone else ordered. According to a recent study, such errors cause financial losses of about $9,000 per month for a medium-sized restaurant.

[0003] In addition to the economic loss, there may be other losses: for example, customers will have to wait longer for the waiter / waitress to decide on the correct offering. This delay may significantly increase the risk of poor customer satisfaction and negative reviews (e.g. on social media), further damaging the business's revenue.

[0004] Furthermore, serving errors can cause chaos in the kitchen as the correct dish needs to be re-cooked as soon as possible to remedy the situation, resulting in extra costs that go to waste and increasing the likelihood of service congestion.

[0005] Related art solutions aimed at reducing delivery errors can be broadly categorized into traditional and technology-based approaches.

[0006] Traditional related technology approaches include hiring more restaurant staff and providing better professional training to make staff more experienced. These solutions are very straightforward. However, they are less time-efficient and much more expensive. For example, business owners need to pay extra money to the additionally hired waiters / waitresses, and they may also make the restaurant even more crowded during peak meal times, potentially prolonging the waiting time for customers. Professional training, on the other hand, may be a slow process, taking months to complete and bringing extra burden to waiters / waitresses besides their daily work. More importantly, professional training may be very inefficient because due to low job retention of waiters / waitresses, all new employees need to complete the training before seeing any effect.

[0007] The related technology approach seeks technological support to reduce serving errors. A representative example of the related technology is a restaurant POS (Point of Sale) system. The related technology POS system provides automated services such as menu management, bill splitting, and customer feedback analysis. Such functions focus on improving the customer experience in aspects such as providing convenient food customization options, improving communication between the kitchen and customers, and providing more efficient employee management. However, these related technologies aimed at the restaurant business do not consider the problem of serving food to the wrong customer. Therefore, business owners must still rely only on the traditional approaches of the related technology to reduce such serving errors.

[0008] There is an unmet need in the related art to develop a solution that helps restaurants identify dishes that have been served in error. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] BA, JL, et al., Layer Normalization, July 21, 2016, arXiv:1607.06450, 14 pgs. [Non-Patent Document 2] BERTINETTO, L., et al., Fully-Convolutional Siamese Networks for Object Tracking, European Conference on Computer Vision, 2016, pp.850-865. [Non-Patent Document 3] CHEN, W., et al., Beyond Triplet Loss: A Deep Quadruplet Network for Person Re-Identification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.403-412. [Non-Patent Document 4] GRAVES, A., et al., Speech Recognition with Deep Recurrent Neural Networks, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp.6645-6649. [Non-Patent Document 5] HE, K., et al., Deep Residual Learning for Image Recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.770-778. [Non-Patent Document 6] HERMANS, A., et al., In Defense of the Triplet Loss for Person Re-Identification, November 21, 2017, arXiv:1703.07737, 17 pgs. [Non-Patent Document 7] HOFFER, E., et al., Deep Metric Learning Using Triplet Network, International Workshop on Similarity-Based Pattern Recognition, 2015, pp.84-92. [Non-Patent Document 8] JOFFE, S., et al., Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, March 2, 2015, arXiv:1502.03167, 11 pgs. [Non-Patent Document 9] JIA, J., et al., Frustratingly Easy Person Re-Identification: Generalizing Person Re-lID in Practice, July 22, 2019, arXiv:1905.03422, 14 pgs. [Non-Patent Document 10] KOCH, G., Siamese Neural Networks for One-Shot Image Recognition, Thesis for Masters Degree of Science, Graduate Department of Computer Science, University of Toronto, 2015, 30 pgs. [Non-Patent Document 11] LUO, H., et al., Bag of Tricks and a Strong Baseline for Deep Person Re-Identification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp.4321-4329. [Non-Patent Document 12] REDMON, J., et al., YOLOV3: An Incremental Improvement, April 8, 2018, arXiv:1804.02767, 6 pgs. [Non-Patent Document 13] SCHROFF, F., et al., FaceNet A Unified Embedding for Face Recognition and Clustering, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp.815-823. [Non-Patent Document 14] WEINBERGER, KQ, et al., Distance Metric Learning for Large Margin Nearest Neighbor Classification, Journal of Machine Learning Research, 10, 2009, pp.207-244. [Non-Patent Document 15] ZEGHIDOUR, N., et al., Joint Learning of Speaker and Phonetic Similarities with Siamese Networks, INTERSPEECH, 2016, pp.1295-1299. [Non-Patent Document 16] ZHENG, L., et al., Scalable Person Re-Identification: A Benchmark, Proceedings of the IEEE International Conference on Computer Vision, 2015, pp.1116-1124. Summary of the Invention [Problem to be solved by the invention]

[0010] An object of the present invention is to provide a method, program and system that can reduce serving errors in restaurants. [Means for solving the problem]

[0011] One aspect of an exemplary implementation includes accepting an order associated with a user and obtaining information associated with physical attributes of the user; once the order is completed, performing a matching operation between the physical attributes of the user and information associated with the user's domain, including the physical attributes of the user and the physical attributes of each of other users, to determine a closest match; and generating an output identifying the user associated with the closest match in information associated with the user's domain and providing the output to a service provider to deliver the order to the user.

[0012] An exemplary implementation may also include a non-transitory computer-readable medium having a storage device and a processor that may execute instructions related to reducing serving errors in restaurants and the like, and more specifically, an image of a user ordering food may be matched with an image of the area, and a service provider may be provided with an assistance system to associate and deliver the completed order to the user.

[0013] A first aspect of the present disclosure is a computer-implemented method including a computer receiving an order associated with a user, capturing information associated with physical attributes of the user, performing a matching operation for the completed order between the physical attributes of the user and information associated with the user's domain including the physical attributes of the user and physical attributes of each of other users to determine a best match, generating an output identifying the user associated with the best match in information associated with the user's domain, and providing the output to a service provider for delivery of the order to the user.

[0014] The matching operation includes defining captured information associated with physical attributes of the user as a first input, defining information associated with a region of the user as a second input, applying the first input and the second input to a convolutional neural network to generate predictions of best matches between the physical attributes of the user in the first input and the physical attributes of the user and the other users in the second input, associating the predictions with indicators within the region of the user, and generating outputs as visual and / or audio labels for use by an edge device associated with the service provider.

[0015] The convolutional neural network comprises a feature space that is unified with the space in which batch normalization, similarity loss, and careful fusion are performed.

[0016] Accepting an order associated with the user includes providing a plurality of options to a user in a user interface so that the user selects an option which is transmitted via network communication to a processor that performs the matching operation, and information associated with physical attributes of the user includes capturing an image of the user and location information of the user.

[0017] Providing the output to a service provider for delivery of the order includes at least one of providing an image of the area in which the user is located to identify the user among the other users, and providing one or more audio signals to the service provider that enable the user to physically move the order from its current location to the user's location.

[0018] The users include customers in a restaurant, the orders include selections from a menu of the restaurant, and the service providers include other users, including other customers located in the restaurant, a human or automated delivery machine that receives the orders after completing the orders and providing the output to the customers.

[0019] The physical attributes of the user are captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order.

[0020] A second aspect of the present disclosure is a program that causes a computer to perform the steps of: accepting an order associated with a user, capturing information associated with physical attributes of the user, for a completed order, performing a matching operation between the physical attributes of the user and information associated with the user's domain including the physical attributes of the user and physical attributes of each of other users to determine a best match, generating an output that identifies the user associated with the best match in the information associated with the user's domain, and providing the output to a service provider to deliver the order to the user.

[0021] The matching operation includes defining captured information associated with physical attributes of the user as a first input, defining information associated with a region of the user as a second input, applying the first input and the second input to a convolutional neural network to generate predictions of best matches between the physical attributes of the user in the first input and the physical attributes of the user and the other users in the second input, associating the predictions with indicators within the region of the user, and generating outputs as visual and / or audio labels for use by an edge device associated with the service provider.

[0022] The convolutional neural network comprises a feature space that is unified with the space in which batch normalization, similarity loss, and careful fusion are performed.

[0023] Accepting an order associated with the user includes providing a plurality of options to a user in a user interface so that the user selects an option which is transmitted via network communication to a processor that performs the matching operation, and information associated with physical attributes of the user includes capturing an image of the user and location information of the user.

[0024] Providing the output to a service provider for delivery of the order includes at least one of providing an image of the area in which the user is located to identify the user among the other users, and providing one or more audio signals to the service provider that enable the user to physically move the order from its current location to the user's location.

[0025] The users include customers in a restaurant, the orders include selections from a menu of the restaurant, and the service providers include other users, including other customers located in the restaurant, a human or automated delivery machine that receives the orders after completing the orders and providing the output to the customers.

[0026] The physical attributes of the user are captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order.

[0027] A third aspect of the present disclosure is a system comprising an input device for accepting an order associated with a user, a sensor for capturing information associated with physical attributes of the user, a processor for performing a matching operation between the physical attributes of the user and information associated with the user's realm including the physical attributes of the user and physical attributes of each of other users to determine a best match and generating an output identifying the user associated with the best match in the information associated with the user's realm, and a service provider device for receiving the output and providing the output to a service provider for delivery of the order to the user.

[0028] The matching operation defines captured information associated with physical attributes of the user as a first input and information associated with the user's region as a second input, applies the first input and the second input to a convolutional neural network to generate predictions of best matches between the user's physical attributes in the first input and the user's physical attributes and physical attributes of the other users in the second input, associates the predictions with indicators within the user's region, and generates outputs as visual and / or audio labels for use by an edge device associated with the service provider, the convolutional neural network comprising a feature space unified with the space in which batch normalization, similarity loss, and careful fusion were performed.

[0029] The input device provides the user with a number of options in the user interface so that the user selects an option that is transmitted by network communication to a processor that performs the matching operation, and the sensor includes a camera that captures an image of the user and location information of the user.

[0030] The service provider's device displays an image of the area in which the user is located, identifies the user among the other users, or provides one or more audio signals to the service provider that enable the user to physically move the order from its current location to the user's location.

[0031] The users include customers in a restaurant, the orders include selections from a menu of the restaurant, and the service providers include other users, including other customers located in the restaurant, a human or automated delivery machine that receives the orders after completing the orders and providing the output to the customers.

[0032] The physical attributes of the user are captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order.

[0033] Details of embodiments of the present disclosure, both as to their structure and operation, can be gleaned in part from study of the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief description of the drawings]

[0034] [Figure 1] FIG. 1 is a functional block diagram of an exemplary implementation. [Diagram 2] FIG. 2 is an exemplary user experience. [Diagram 3] FIG. 3 is a functional block diagram of an example implementation related to the order and delivery flow. [Figure 4] FIG. 4 is an example implementation illustrating a schematic of a neural network. [Diagram 5] FIG. 5 is an example implementation illustrating a schematic of a neural network. [Figure 6A] FIG. 6A is a flowchart of an exemplary implementation of a method according to the present disclosure. [Figure 6B] FIG. 6B is a flowchart of an exemplary implementation of the method according to the present disclosure. [Figure 7] FIG. 7 is a functional block diagram of an exemplary implementation of a computing environment according to the present disclosure. [Figure 8] FIG. 8 is a functional block diagram of an exemplary operating environment according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] The following detailed description provides further details of the figures and exemplary implementations of the present application. For elements that overlap between the figures, reference numbers and descriptions are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting.

[0036] An exemplary implementation is directed to a system and method for reducing errors in serving food to the wrong customer. More specifically, this exemplary implementation assists waiters / waitresses in accurately linking dishes to the correct person to deliver them. During the ordering period, visual information of each customer is captured via a camera and dishes are associated with each individual.

[0037] In the serving stage, all people in the camera view are detected, and a person to serve the food is selected by matching the visual information of the detected people with that acquired during ordering. During the process, the waiter / waitress only needs to perform simple operations (e.g., switching between the ordering stage / serving stage). Thus, the server can focus on serving the customer without having to remember the exact individual orders.

[0038] According to an exemplary implementation, the core system has a camera and a customer matching module built based on a deep neural network. The customer matching (MC) module is trained offline to correctly identify whether the person in two images is the same. For deployment, each waiter / waitress is equipped with an edge device with a camera, connected to the cloud that hosts the trained MC module.

[0039] 1 shows a functional overview of an exemplary constraint at 100. More specifically, a customer 101 can place an order. At this time, a dish is associated with the customer at 103. A service provider 107 is then provided with the order associating the dish with the customer. Once the dish is completed, it is matched to the correct customer 105 during the serving process. The dish is then served to the customer 101.

[0040] As shown in FIG. 2, an edge device 200 is provided to perform the functional aspects shown in FIG. 1. The edge device 200 has two modes: an order mode 201 and a serving mode 203. When the device is in the order mode 201, the camera captures an image of the ordering customer and automatically links the food he / she ordered to the captured image. The image is then sent back to the MC module for feature extraction. For example, as shown in 201, a current customer orders a Caesar salad from the menu (e.g., drag and drop), while the camera captures an image of the customer during the ordering process. As disclosed further below, the user can submit the order without the involvement of the service provider, e.g., to maintain social distancing protocols.

[0041] After the device is switched to serving mode at 203, its camera detects all customers displayed on the same screen, and the MC module extracts features of each detected customer. Then, for a particular dish, the features of the image corresponding to that dish captured in the ordering mode are compared with the features of all newly detected customers. The customer whose feature has the smallest distance from the original image is considered as the target. Then, the waiter / waitress is notified to serve the correct person. For example, as shown in 203, when a Caesar salad plate is completed in the kitchen, a matching operation is performed and it is determined that the person whose image matches the customer who ordered the Caesar salad is highlighted in red. Thus, the server does not need to guess the identity of the person who ordered the Caesar salad.

[0042] This is especially useful in large groups where group members may switch seats, switch tables, etc. Because the customer's image is associated with the order at the very beginning, that association is preserved during the serving mode.

[0043] Although the above exemplary implementations show visual service guidance, the exemplary implementations are not limited thereto. For example, but not limited thereto, instead of providing a visual indicator, the guidance may be provided by an audio speaker, vibration, or other modes, alone or in combination with a normal indicator. Thus, the service provider in the restaurant can be assisted in serving the order to the correct customer and does not have to guess whether the customer and dish pairing is correct.

[0044] A functional diagram of an exemplary implementation is shown in Figure 3. The flow is described in order, starting with the ordering stage and then moving to the delivery stage.

[0045] A customer 301 may arrive and be provided with a table and seat of his / her choice. Once the customer is seated, an image of the customer is captured by a handheld camera 305. The handheld camera 305 may optionally be positioned to facilitate capturing an image of the customer 301. For example, but not by way of limitation, the handheld camera 305 may be positioned above the customer in an array along the ceiling, on the table itself, on a point of service (POS) menu, in a corner of the room, or any other location that allows for the capture of an image of the customer 301.

[0046] The image of the customer 301 captured by the mobile camera 305 is provided to a customer matching module 309. The timing of the image capture may occur simultaneously with, immediately before, or immediately after the customer 301 places an order for food. If the customer places an order directly from the POS device without the involvement of a service provider, or if the service provider takes the order, the order is entered through an interface 307. The order is also entered into the customer matching module 309 along with a photograph of the customer. The order is also provided to the service provider 303.

[0047] After some time, the order is prepared and ready to be served. At that point, the waiter is notified that the food is ready to be served. At this point, the customer matching module 309 requests the capture of an image of the room from the mobile camera 305 and executes a matching function. More specifically, the matching function aims to match the first captured image of the customer who submitted the order during the order phase with the best matching image of the customer sitting in the room. The matching result is provided to the interface 307, which the service provider 303 uses to guide him on where to deliver the food. An optional speaker 311 is provided to instruct the service provider 303, who then serves the food to the customer 301.

[0048] Exemplary implementations include, but are not limited to, input / output devices and MC modules. Further details of these features of exemplary implementations are disclosed below.

[0049] For input / output devices, a camera can capture the customer's visual information as input. This camera can be an integrated part of other edge devices such as a POS system terminal. The choice of output device can be flexible. If the waiter / waitress prefers visual feedback, he / she can target the customer using the screen of the POS system terminal, or he / she has a POS system terminal with a speaker / headset and can turn it on for audible instructions when serving the food.

[0050] As shown in element 203 of FIG. 2 and disclosed above, based on various service provider settings, the target customer is highlighted on the POS screen or auditory guidance is provided to the waiter / waitress (e.g., suggestive directional information to specify "in front of you", "to your right / left side", etc.) to associate the order with the correct customer. The input / output devices may be networked, such as connected to the Internet, to send and receive information to and from the MC module.

[0051] With respect to the customer matching aspect of the exemplary implementation, a cloud-based service may be provided that determines and returns the "most similar person" image from all people detected for a query image. For example, but not by way of limitation, a convolutional neural network (CNN) may be used. In one exemplary implementation, the CNN includes a feature extractor that processes all input images into a feature vector.

[0052] FIG. 4 shows a customer matching module 400 according to an exemplary implementation. The image 401 captured during the ordering phase is a query image. When the food is served, the previously captured query image 401 is first extracted into a feature vector together with all detected person images 403. Then, the distance between the feature vector of the query image and the feature vector of the detected image is calculated and ranked in descending order. The detected image with the highest distance score is considered as the target customer to whom the food should be served. This is realized, for example, with a CNN 405. Furthermore, the CNN 405 can be a Siamese network including convolutions 407, 409, a fully connected layer 411, and an output classification or output prediction 413 for matching. A two-stream process is provided to determine the identity loss and similarity loss.

[0053] Further assistance functions may be combined with the customer matching module 400. For example, but not limited to, a detector (e.g., YOLOv3) may be used to detect all people on the camera screen before being sent to perform matching with the query image. Additionally, voice recognition may be used to trigger the matching process if the waiter / waitress's hands are occupied with serving food and unavailable. For example, but not limited to, the waiter / waitress may instruct the customer matching module 400 to perform matching when the order is ready to be served. Optionally, a voice feed may be used to input voice recognition and matching to assist with image matching.

[0054] In the exemplary implementation, we use a Siamese network as the training paradigm, which is a common framework for distance learning. The CNN405 Siamese network aims to minimize the distance between feature vectors belonging to the same person and maximize the distance between feature vectors belonging to different people. It requires a pair of images as input, where a "positive pair" means that both images relate to the same person, and a "negative pair" means that the person in one image is different from the person in the other image.

[0055] The training objective is to minimize the similarity loss at 413, which causes the network to decrease the margin for positive pairs and increase the margin for negative pairs. Optionally, a triplet loss is used as the optimization objective. Triplet Loss JPEG0007581847000001.jpg1017 can be formulated as follows:

[0056]

number

[0057] where f i a are the anchor image features, and f i p , f i n are the features of the positive and negative paired images for the anchors, respectively. N is the number of anchor images selected. α is a margin that controls the minimum measure of whether the features are well separated.

[0058] Optionally, a 413 identity loss can be included in the Siamese network to improve performance. Training a Siamese network can therefore be viewed as a multitask learning problem. Unlike similarity loss, which requires a pair of images, identity loss requires only one image as input and optimizes a single network stream. The goal is to determine the correct identity for the person in the input image. In practice, cross-entropy loss JPEG0007581847000003.jpg1016 is used as the identity loss and can be formulated as follows:

[0059]

number

[0060] where y i is a feature i where M is the ground truth ID of the images, and M is the number of images with ID labels. During the training process, JPEG0007581847000005.jpg1014 and JPEG0007581847000006.jpg916 Both are combined as a joint loss function L and jointly trained as

[0061]

number

[0062] Here, β is a hyperparameter to control the influence of the integrated loss function.

[0063] In addition to training purposes, a new architecture approach is provided to make customer matching more accurate. Identity loss and similarity loss may not be highly compatible with a shared metric space. For example, feature points may be more separable for cosine distance than Euclidean distance with identity loss, as opposed to similarity loss, where Euclidean distance is more likely to converge.

[0064] According to an exemplary implementation, different feature spaces are generated for each loss function during training, and these spaces are combined into one during testing. For example, batch normalization (BN) and layer normalization (LN) can be used to develop separate feature spaces for identity loss and similarity loss, respectively. For any feature f, both normalizations have the form:

[0065]

number

[0066] Here, μ B,L is the batch and layer average, and σ B,L are the batch and layer variances for BN and LN, respectively. γ B,L , η B,L are the weights of the corresponding normalization affine transformation.

[0067] During training, the features f in Eq. 1 and Eq. 2 are B and f L In such a case, f B takes only the gradients from the identity loss, and f L receives the gradients from the similarity loss. To combine them, we use attention weights a B,L is calculated for both normalizations as follows:

[0068]

number

[0069] Here, K is the sum of f and f B,L is a kernel function that generates an attention score based on . The parameters can be trained jointly with the Siamese network.

[0070] After training, the normalized features f B,L We fuse it with the attention score to get the final feature vector: JPEG0007581847000010.jpg1510 can be generated.

[0071]

number

[0072] 5 provides a visualized comparison 500 between a feature space 501 of a Siamese network and a careful fusion space 503-511 according to an exemplary implementation. The feature space 503 is merged by a fusion module 511 with the space where batch normalization 505, similarity loss 507, and careful fusion 509 have been performed. Thus, different inputs provide different signals.

[0073] Related art POS systems for restaurants focus on providing automated solutions to handle problems such as menu management, bill splitting, and customer feedback analysis, which functions are directly targeted at the customer to enhance the customer's dining experience. However, exemplary implementations of the present disclosure can be distinguished from these related art POS system approaches in that they enhance the dining experience by assisting waiters / waitresses to reduce serving errors and be more engaged in serving customers. Thus, exemplary implementations of the present disclosure may be integrated into a POS system.

[0074] An exemplary implementation of the present disclosure optionally provides a person re-identification technique on a Siamese network.Compared with related techniques, the exemplary implementation of the present disclosure focuses on sharing metric spaces for different training purposes.Furthermore, the exemplary implementation of the present disclosure also aims to provide different normalization spaces for corresponding losses and combine them carefully during testing time.

[0075] 6A and 6B are flow charts of an embodiment of a method according to the present disclosure. The method 600 can be implemented by one or more processors in a computing environment (e.g., a computing environment described below). As shown in FIG. 6A, the method 600 can be a combination of multiple sub-processes including accepting an input associated with an ID and an order by a user, such as a food ordered by a POS or an image captured by a camera, at 601. Once the food is prepared, a matching and classification operation is performed at 602. At 603, the service provider is provided with video and / or audio assistance to serve the order to the user. Specific details of the aforementioned operations 601-603 are disclosed above with respect to the description of FIGS. 1-5.

[0076] More specifically, Figure 6B illustrates an example process associated with an example implementation of the present disclosure. Although the operations described herein are ordered according to the drawings, the plantation of the present disclosure is not limited thereto. For example, the order of the operations can be changed, and operations can be combined, separated, or removed without departing from the scope of the present invention.

[0077] At 605, an order is obtained from a user. More specifically, the user may be a customer providing a specific request to order a particular product or service, such as a meal or dish at a restaurant. The order may be submitted through a human interaction, such as a personal interaction with a waiter or waitress. The waiter or waitress enters the order into the system either manually or automatically, and either when receiving the order or when writing it on paper and then transferring the order to an electronic system.

[0078] Alternatively, to provide sufficient social distancing, orders may be entered by a user without human interaction. Examples of order entry performed by a user without human interaction include, but are not limited to, an on-site POS device or counter at the user's table that accepts orders, or a user's mobile device connected to a network that allows orders to be submitted directly from the user's device or other devices known to those skilled in the art.

[0079] At 610, information associated with the user is captured. More specifically, when the user submits an order to a service provider or automatically submits an order, an image of the user may be captured by a sensor, such as a camera. The image may include information, such as the image itself, location coordinate information, the order submitted by the user, and any other information that may be understood to be related to the identity, location, and / or order submitted by the user.

[0080] At 615, the captured information and the order submitted by the user are provided for matching. More specifically, the information may be submitted either by cloud, network, or local storage in a manner usable by the matching module.

[0081] Thus, operations 605, 610, and 615 provide for the receipt of the order and associated user information of 601. Additionally, operations 620-635 refer to matching operations, as described below.

[0082] At 620, the matching module waits for an indication that the order is complete. For example, the user may capture an image and geolocation of themselves before or after taking the order. However, the matching function of the matching module is not performed until the order is finalized. At 620, the matching operation is invoked either by the matching module's automated process that determines that both the order and an image have been provided, and optionally by a service provider or business such as a restaurant that approves or verifies the order. Additionally, a second indication may be provided that the order is ready to be delivered, such as a message from the kitchen, service provider, or other entity that may enable the service provider to indicate to the user that assistance is needed to navigate the order.

[0083] At 625, an image of the user's vicinity is received, including surrounding users, tables, etc. Thus, the camera image is intended to provide the matching module with a view of what the service area looks like to the service provider from where the order to be delivered is received. Thus, multiple users will be present in the received image, including the user who submitted the order.

[0084] At 630, a matching operation is performed between the captured image of the user submitting the order and images of the vicinity of the accepted user. Further details of the matching operation are disclosed and described above with reference to Figures 4 and 5.

[0085] Once a matching operation has been performed, at 635, the best match between the image information of the user displaying the order and the users in the image vicinity of the user is determined.

[0086] Thus, matching operations are performed in operations 620-635. Providing an output to a service provider to assist the service provider in delivering the order, as described below, is described with respect to operations 640-650.

[0087] At 640, the matching module generates an output that is provided to an edge device used by the service provider. For example, and without limitation, the edge device may be a point-of-sale device, a mobile phone, a headset worn by a user, or other edge device as understood by those skilled in the art to be used by a server.

[0088] At 645, the service provider is provided with navigation assistance to the user. For example and without limitation, a video output may be provided identifying the best matching user as determined by the matching module as described above, with that user highlighted to distinguish it from other users. Additionally or alternatively, audio assistance may be provided that provides directional assistance to the service provider on how to navigate to the location of the user who placed the order.

[0089] At 650 the order is delivered to the user.

[0090] 7 is a functional block diagram of one embodiment of a computing environment according to the present disclosure. The computing environment 700 includes an exemplary computer device 705 suitable for use in some exemplary implementations. The computer device 705 in the computing environment 700 may include one or more processing units, cores, or processors 710, memory 715 (e.g., RAM, ROM, and / or the like), internal storage 720 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 725. Any of these may be connected to a communication mechanism or bus 730 for communicating information or may be built into the computer device 705. The environment 700 may support operations related to the system 100, for example.

[0091] According to an exemplary implementation of the present disclosure, processing related to neural activity may be performed on a processor 710 that is a central processing unit (CPU). Alternatively, other processors may be used instead without departing from the concept of the present invention. For example, but not limited to, a graphics processing unit (GPU) and / or a neural processing unit (NPU) may be used in place of or in combination with the CPU to perform the processing of the exemplary implementations described above.

[0092] Computing device 705 may be communicatively connected to input / interface 735 and output device / interface 740. One or both of input / interface 735 and output device / interface 740 may be wired or wireless interfaces and may be removable. Input / interface 735 may include any device, component, sensor, or interface, physical or virtual, that may be used to provide input (e.g., buttons, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, etc.).

[0093] Output devices / interfaces 740 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some exemplary implementations, input / interfaces 735 and output devices / interfaces 740 may be built into computing device 705 or may be physically connected to computing device 705. In other exemplary implementations, other computing devices may function as or provide the functionality of input / interface 735 and / or output device / interface 740 for computing device 705.

[0094] Examples of computing devices 705 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices mounted on vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, server devices, other computers, information kiosks, televisions, radios with and / or connected to one or more processors, etc.).

[0095] Computing device 705 may be communicatively connected (e.g., via I / O interface 725) to external storage 745 and network 750 for communicating with any number of network components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 705 or any connected computing device may function to provide or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or other label service. For example, and without limitation, network 650 may include a blockchain network and / or a cloud.

[0096] I / O interface 725 can include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks in computing environment 600. Network 750 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0097] The computing device 705 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media includes transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media includes magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid state storage), and other non-volatile storage or memory.

[0098] The computing device 705 may be used to execute techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions may be obtained from a transitory medium or may be stored on and obtained from a non-transitory medium. The executable instructions may be generated from one or more of a programming language, a scripting language, and a machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0099] The processor 710 can operate under any operating system (OS) (not shown) in a native or virtual environment. One or more applications may be deployed, including a logic unit 755, an application programming interface (API) unit 760, an input unit 765, an output unit 770, an order and user identification unit 775, a matching unit 780, and a service guidance unit 785, as well as an inter-unit communication mechanism 795 for different units to communicate with each other and with the OS and other applications (not shown).

[0100] The ordering and user identification unit 775 may perform functions related to receiving input, processing the input, and determining a classification of the visitor. The matching unit 780 may perform functions related to detecting the object being delivered. The server guidance unit 785 may perform functions related to receiving input, such as speech, from the visitor and providing the resident with the opportunity to provide output to the visitor.

[0101] For example, the order and user identification unit 775, the matching unit 780, and the server guidance unit 785 may implement one or more processes described above with respect to the above-mentioned structures in addition to the method 300 and the method 400. The described units and elements may be modified in design, function, configuration, or implementation and are not limited to the provided description.

[0102] In some example implementations, once information or execution instructions are obtained by the API unit 760, the information or execution instructions may be communicated to one or more other units (e.g., the logic unit 755, the input unit 765, the order and user identification unit 775, the matching unit 780, and the server guidance unit 785).

[0103] In some examples, logic unit 755 may be configured to control the flow of information between units and, in some example implementations described above, direct the services provided by API unit 760, input unit 765, order and user identification unit 775, matching unit 780, and service provider guidance unit 785. For example, the flow of one or more processes or implementations may be controlled by logic unit 755 alone or in conjunction with API unit 760.

[0104] 8 is a functional block diagram of an example operating environment according to the present disclosure. Environment 800 may be suitable for some example implementations disclosed herein. Environment 800 includes devices 805-845, each communicatively connected to at least one other device, e.g., via network 860 (e.g., by wired and / or wireless connections). Some devices may be communicatively connected to one or more storage devices 830 and 845.

[0105] An example of one or more devices 805-845 may each be the computing device 705 described in Figure 7. Devices 805-845 may include, but are not limited to, a computer device 805 (e.g., a laptop computer device) having a monitor and associated webcam, a mobile device 810 (e.g., a smartphone or tablet), a television 815, a device associated with a vehicle 820, a server computer 825, computer devices 835-840, and storage devices 830 and 845, as described above.

[0106] In some implementations, devices 805-820 can be considered user devices associated with a user that can remotely obtain sensed voice input that is used as input for the above-mentioned example implementations. In example implementations according to the present disclosure, one or more of these user devices 805-820 can be associated with one or more sensors, such as a microphone on the user's phone or a microphone on a POS device, that can sense information required for the example implementations of the present disclosure, as described above.

[0107] This exemplary implementation has various benefits and advantages. For example, but not limited to, by using the approach disclosed above and in FIGS. 1-5, accuracy can be substantially improved, especially in the fusion approach. Accuracy can be achieved to 90% or higher, and fusing normalized features can improve accuracy as well as mAP. As a result, service provider errors in delivering orders can be reduced.

[0108] Exemplary implementations of the present disclosure can also complement existing related art POS approaches. As discussed above, related art POS approaches provide menu management or bill splitting, etc. However, related art POS do not assist the service provider in delivering the order accurately. Additionally, and in contrast to related art approaches of matching, exemplary implementations of the present disclosure provide artificial intelligence for CNNs, including application of normalized feature spaces for training and careful fusion for testing.

[0109] Furthermore, the POS device may be integrated with the user's mobile device in the restaurant, and the user's account is also associated with the order. As a result, the user may be billed individually without making it difficult for multiple people, including the user, to perform the calculations, and without requiring the service provider to manually adjust the bill, while allowing the user to have points, etc., credited to his or her account with the restaurant and share the order with others. Optionally, the user may be provided with ordering suggestions based on the user's account and past ordering information or preferences.

[0110] Moreover, the exemplary implementation can take into account the unique circumstances of a restaurant. When a user moves seats or tables, it may be difficult for the server to identify the location for later order determination. The exemplary implementation of the present disclosure can search the entire area of ​​the restaurant where the user may be located, so that the order can be found and delivered correctly to the user.

[0111] In restaurants where users order at a counter and sit down later (such as with a numbering system), it may be possible for the server to deliver the order without having to look for the table number and without having to ask the user to announce the order, thus allowing the server to improve speed and accuracy.

[0112] This feature is especially useful when the differences in orders between multiple people are subtle (e.g., the same main dish with different spices or toppings) or very important (e.g., allergy-free, gluten-free, etc.). Thus, restaurants can avoid the risk of food contamination and customer illness.

[0113] With respect to online customer reviews, example implementations of the present disclosure may also provide benefits and advantages. For example, but not limited to, for online reviews, reviewers may be verified against actual delivered food to verify customer complaints and feedback and avoid false or misleading reviews. Thus, restaurants may be able to provide better quality control.

[0114] Similarly, since the camera provides continuous input, the camera may also provide some indication of satisfaction after the order has been delivered. For example, but not by way of limitation, the user's reaction may be determined to be positive or negative in a manner that allows the service provider to provide feedback, either in real time or after the fact, and potentially correct any service-related issues.

[0115] With regard to the location of the camera, it is shown to be in the facility (e.g., on a table, ceiling, etc.), but the exemplary implementation of the present disclosure is not limited thereto. For example, but not limited thereto, the camera may be placed on the service provider's wearable device (e.g., glasses, hat, clothing, etc.) so that the service provider's view matches the view processed by the matching module. Optionally, the camera may be integrated with augmented reality, virtual reality, and / or mixed reality. Further optionally, the service provider may wear a badge, carry a POS device, tablet, etc. In addition to the use of sensors such as cameras, audio sensors such as microphones may be used. To some extent, other location tools such as Bluetooth, Wi-Fi, or RF associated with the user's mobile device and their account or ID may assist in determining the location and ID. Optionally, the service provider may have an electronic device, the user may have an electronic device, and all electronic devices are linked by a common communication protocol such as Bluetooth. The user may capture an image and then associate the image with a common device (e.g., using a Bluetooth ID). Further associations may be made to points or business accounts associated with the user's device.

[0116] With respect to retail and restaurant queuing systems and configurations, exemplary implementations of the present disclosure may be provided to increase the accuracy of delivery, for example, but not limited to, in the case of curbside pickup or in-store pickup at retail stores with social distancing between people waiting in queues, to more accurately match the items delivered to users ordering the items. As a result, service providers may be at less risk of unnecessary close contact between extra people that occurs when an order is mistakenly delivered to the wrong person. Similarly, users are less at risk of receiving the wrong item or having unnecessary close interactions with multiple service providers.

[0117] Facial recognition can be used to match objects with users, instead of using tickets or tokens. Such an approach may also be used in drive-thru services with multiple queues, such as fast food, where the service provider needs to distinguish between vehicles in multiple queues, and security camera video can be used to obtain the capture of the initial user image.

[0118] The above exemplary implementations may also be applied to other environments, such as airport baggage claim areas, etc. Thus, as described above, social distancing may be observed at airports by associating a set of luggage with a set of travelers waiting to collect their luggage.

[0119] Although several exemplary implementations have been shown and described, these exemplary implementations are provided to convey the subject matter described herein to those skilled in the art. It should be understood that the subject matter described herein may be embodied in various forms, without being limited to the exemplary implementations described. The subject matter described herein can be implemented using the specifically defined or described items, or using other or different elements or items not described. Those skilled in the art will understand that changes can be made to these exemplary implementations without departing from the subject matter described herein, as defined in the appended claims and their equivalents.

[0120] Aspects of certain non-limiting embodiments of the present disclosure address the features discussed above and / or other features not described above, however, aspects of the non-limiting embodiments need not address the features discussed above, and aspects of the non-limiting embodiments of the present disclosure may not address the features discussed above.

Claims

1. 1. A computer-implemented method comprising: The computer Accepting an order associated with a user and capturing information associated with a physical attribute of the user; performing a matching operation between the physical attributes of the user and information associated with the user's realm, including the physical attributes of the user and the physical attributes of each of the other users, for the completed orders to determine a best match; generating an output identifying the user associated with the best match among information associated with the user's domain, and providing the output to a service provider for delivery of the order to the user; The method includes:

2. The matching operation includes: defining captured information associated with a physical attribute of the user as a first input and information associated with an area of ​​the user as a second input; applying the first input and the second input to a convolutional neural network to generate a prediction of a best match between the physical attributes of the user in the first input and the physical attributes of the user and the other users in the second input; Associating the prediction with an indicator within a region of the user; generating an output as a visual and / or audio label for use on an edge device associated with the service provider; The method of claim 1 , comprising:

3. The convolutional neural network includes: With the feature space unified with the space in which batch normalization, similarity loss, and careful fusion were performed, The method of claim 2.

4. Accepting an order associated with the user providing a plurality of options to a user in a user interface so that the user selects an option which is transmitted by network communication to a processor that performs the matching operation; The information associated with the physical attributes of the user includes capturing an image of the user and location information of the user. The method of claim 1.

5. providing the output to a service provider for delivery of the order; providing an image of the area in which the user is located, identifying the user among the other users; and providing one or more audio signals to the service provider that enable the user to physically move the order from a current location to the user's location. The method of claim 1.

6. The users include customers in a restaurant; the order includes a selection from a menu of the restaurant; the service provider may be a human or an automated delivery machine that receives the order after completing the order and providing the output to the customer among the other users including other customers located in the restaurant; The method of claim 1.

7. The physical attributes of the user include: captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order; The method of claim 1.

8. On the computer, Accepting an order associated with a user and capturing information associated with a physical attribute of the user; performing a matching operation between the physical attributes of the user and information associated with the user's realm, including the physical attributes of the user and the physical attributes of each of the other users, for the completed orders to determine a best match; generating an output identifying the user associated with the best match among information associated with the user's domain, and providing the output to a service provider for delivery of the order to the user; A program that executes a procedure.

9. The matching operation includes: defining captured information associated with a physical attribute of the user as a first input and information associated with an area of ​​the user as a second input; applying the first input and the second input to a convolutional neural network to generate a prediction of a best match between the physical attributes of the user in the first input and the physical attributes of the user and the other users in the second input; Associating the prediction with an indicator within a region of the user; generating an output as a visual and / or audio label for use on an edge device associated with the service provider; The program according to claim 8 , further comprising:

10. The convolutional neural network includes: With the feature space unified with the space in which batch normalization, similarity loss, and careful fusion were performed, The program according to claim 9.

11. Accepting an order associated with the user providing a plurality of options to a user in a user interface so that the user selects an option which is transmitted by network communication to a processor that performs the matching operation; The information associated with the physical attributes of the user includes capturing an image of the user and location information of the user. The program according to claim 8.

12. providing the output to a service provider for delivery of the order; providing an image of the area in which the user is located, identifying the user among the other users; and providing one or more audio signals to the service provider that enable the user to physically move the order from a current location to the user's location. The program according to claim 8.

13. The users include customers in a restaurant; the order includes a selection from a menu of the restaurant; the service provider may be a human or an automated delivery machine that receives the order after completing the order and providing the output to the customer among the other users including other customers located in the restaurant; The program according to claim 8.

14. The physical attributes of the user include: captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order; The program according to claim 8.

15. an input device for accepting orders associated with a user; a sensor for capturing information associated with a physical attribute of the user; a processor that performs a matching operation between the physical attributes of the user and information associated with the user's universe, the matching operation including the physical attributes of the user and the physical attributes of each of the other users, to determine a best match and generate an output that identifies the user associated with the best match in the information associated with the user's universe; a service provider device that receives the output and provides the output to a service provider for delivery of the order to the user; A system comprising:

16. The matching operation includes: defining captured information associated with a physical attribute of the user as a first input and information associated with an area of ​​the user as a second input; applying the first input and the second input to a convolutional neural network to generate a prediction of a best match between the physical attributes of the user in the first input and the physical attributes of the user and the other users in the second input; Associating the prediction with an indicator within a region of the user; generating an output as a visual and / or audio label for use on an edge device associated with the service provider; The convolutional neural network includes: With the feature space unified with the space in which batch normalization, similarity loss, and careful fusion were performed, The system of claim 15.

17. The input device is providing a plurality of options to the user in a user interface such that the user selects an option that is transmitted by network communication to a processor that performs the matching operation; the sensor comprises a camera that captures an image of the user and location information of the user; The system of claim 15.

18. The service provider device includes: displaying an image of the area in which the user is located; providing one or more audio signals to said service provider that identify said user among said other users or that enable said user to physically move said order from a current location to said user's location; The system of claim 15.

19. The users include customers in a restaurant; the order includes a selection from a menu of the restaurant; the service provider may be a human or an automated delivery machine that receives the order after completing the order and providing the output to the customer among the other users including other customers located in the restaurant; The system of claim 15.

20. The physical attributes of the user include: captured by one or more cameras located in the area of ​​the user or on the service provider fulfilling the order; The system of claim 15.

Citation Information

Patent Citations

  • Sales data processor and program

    JP2017215776A

  • Ordered dish provision support system

    JP2019125162A

  • Order processing

    US20050049940A1