Restaurant analysis system, restaurant analysis method, and program

Abstract

To provide a restaurant analysis system that enables provision of appropriate feedback on a restaurant's cuisine or service.SOLUTION: A restaurant analysis system according to the present disclosure includes: image acquisition means for acquiring an image of at least one of a customer using a restaurant table or an object on the table; voice acquisition means for acquiring text data obtained by converting the customer's voice into text; image recognition means for recognizing a body part of the customer or the object from the image; detection means for detecting a menu selection status of the customer based on recognition results of the image recognition means; generation means for generating information indicating the customer's opinion regarding menu selection based on the customer's menu selection status and the text data; and output means for outputting the information indicating the customer's opinion to an employee terminal used by an employee of the restaurant.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a restaurant analysis system and the like. [Background technology]

[0002] Customers at restaurants sometimes communicate their opinions and requests directly to staff. Customers also sometimes communicate their opinions and requests by filling out a questionnaire. Staff will respond to requests that are communicated directly and can be met immediately. Staff will also consider ways to improve the food and service based on customer opinions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-147290 Summary of the Invention [Problem to be solved by the invention]

[0004] It is time-consuming for customers to communicate their requests to staff and answer questionnaires, so restaurants may not receive appropriate feedback on their food or service.

[0005] One of the purposes of the present disclosure is to provide a restaurant analysis system etc. that can provide restaurants with appropriate feedback on their food or services. [Means for solving the problem]

[0006] A restaurant analysis system in one aspect of the present disclosure includes an image acquisition means for acquiring an image of at least one of a customer sitting at a table in a restaurant or an object on the table; a voice acquisition means for acquiring text data that converts the customer's voice into text; an image recognition means for recognizing a part of the customer's body or the object from the image; a detection means for detecting the customer's menu selection status based on the recognition result of the image recognition means; a generation means for generating information indicating the customer's opinion regarding the menu selection based on the customer's menu selection status and the text data; and an output means for outputting the information indicating the customer's opinion to an employee terminal used by an employee of the restaurant.

[0007] A restaurant analysis method in one aspect of the present disclosure acquires an image of at least one of a customer sitting at a table in a restaurant or an object on the table, acquires text data that converts the customer's voice into text, recognizes a part of the customer's body or the object from the image, detects the customer's menu selection status based on the recognition result, generates information indicating the customer's opinion regarding the menu selection based on the customer's menu selection status and the text data, and outputs the information indicating the customer's opinion to an employee terminal used by an employee of the restaurant.

[0008] A program according to one aspect of the present disclosure causes a computer to execute the following processes: acquire an image of at least one of a customer sitting at a table in a restaurant or an object on the table; acquire text data obtained by converting the customer's voice into text; recognize a body part of the customer or the object from the image; detect a menu selection status of the customer based on the recognition result; generate information indicating the customer's opinion regarding the menu selection based on the customer's menu selection status and the text data; and output the information indicating the customer's opinion to an employee terminal used by an employee of the restaurant. The program may be stored in a computer-readable non-transitory recording medium. [Effects of the Invention]

[0009] One example of the effect of the present disclosure is that it can provide restaurants with appropriate feedback on their food or service. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 10 is a diagram illustrating an example of connection between the restaurant analysis system and other devices. [Figure 2] FIG. 1 is a schematic diagram showing an example of a table in a restaurant. [Figure 3] FIG. 1 is a block diagram showing an example of the configuration of a restaurant analysis system. [Figure 4] 10 is a table showing an example of an output. [Figure 5] 10 is a table showing an example of an output. [Figure 6] FIG. 10 is a diagram illustrating an example of a display screen of an employee terminal. [Figure 7] 10 is a flowchart showing an example of the operation of the restaurant analysis system. [Figure 8] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following describes a restaurant analysis system 100 that analyzes the speech of restaurant customers and provides feedback on the food or service. In the following description, we will mainly assume that the restaurant analysis system 100 generates information indicating the customer's opinion on the food or service by analyzing conversations between customers sitting at a table together.

[0012] An example of a connection between a restaurant analysis system 100 and other devices according to the present disclosure will be described using FIG. 1. The restaurant analysis system 100 is connected to other devices via a communication network, either wired or wirelessly. The restaurant analysis system 100 is connected to, for example, a camera 10, a microphone 20, a tabletop terminal 30, and an employee terminal 40. The tabletop terminal 30 may be provided as needed.

[0013] The camera 10 captures images of tables in a restaurant. The camera 10 transmits the captured images to the restaurant analysis system 100. In one example, the camera 10 captures images of objects on the table. In this case, the camera 10 is installed, for example, in a position where it can capture an image of the object from above the table. In another example, the camera 10 captures images of customers sitting at the table. In this case, the camera 10 is installed, for example, in an arbitrary position where it can capture an image of the faces of people around the table. Images captured by such a camera 10 may be used to recognize the line of sight of customers. Therefore, a camera 10 with a resolution sufficient to detect the direction of pupils is used. The camera 10 capturing images of objects on the table and the camera 10 capturing images of customers may be implemented as separate cameras 10, or may be implemented as a single camera 10.

[0014] The microphone 20 collects the voices of customers sitting at tables in the restaurant. In one example, the microphone 20 transmits the collected voice data to the restaurant analysis system 100. In another example, the microphone 20 may be included in a voice recognition device installed at the table. The voice recognition device converts the voice data into text using voice recognition technology and transmits the text data to the restaurant analysis system 100.

[0015] Tabletop terminal 30 is a terminal installed at a table and used by customers. Tabletop terminal 30 displays recommended menu items and a menu list to customers. Tabletop terminal 30 is, for example, a tablet-type terminal, but is not limited to this. Tabletop terminal 30 may also be a self-order terminal included in an order management system (not shown). The order management system manages customer food orders. In this case, tabletop terminal 30 displays a menu list and accepts input of the menu selected by the customer. Tabletop terminal 30 transmits the order details indicating the input menu item to the server of the order management system. The server transmits cooking instructions for the order details to a display terminal for staff. Tabletop terminal 30 may also transmit the order details to restaurant analysis system 100.

[0016] A camera 10, a microphone 20, and a tabletop terminal 30 are installed at each table, for example, as shown in FIG. 2. As shown in FIG. 2, a camera 10 and a microphone 20 may be installed at each table in a restaurant. Multiple cameras 10 and microphones 20 may be installed for one table. Note that one camera 10 may be installed so that it captures the situations of multiple tables. As shown in FIG. 2, the camera 10 and the microphone 20 may be installed separately from the tabletop terminal 30. Alternatively, one tabletop terminal 30 may be equipped with the camera 10 and the microphone 20.

[0017] Employee terminal 40 is a terminal used by an employee of a restaurant. An employee using employee terminal 40 may be a staff member of the restaurant or an office employee who manages the restaurant. The type of employee terminal 40 is not particularly limited, and may be a smartphone, tablet terminal, PC (Personal Computer), or the like. Employee terminal 40 may be part of an order management system. Employee terminal 40 may be a smartphone-type order receiving terminal through which a staff member inputs customer orders. Employee terminal 40 may also be a tablet-type order display terminal that displays cooking instructions. Employee terminal 40 may also be a terminal used for ordering ingredients and for work management, or may be an accounting machine that processes customer payments.

[0018] An example configuration of a restaurant analysis system 100 according to the present disclosure will be described using Fig. 3. The restaurant analysis system 100 includes an image acquisition unit 101, a voice acquisition unit 102, an image recognition unit 103, a detection unit 104, a generation unit 105, and an output unit 106.

[0019] The image acquisition unit 101 acquires an image of at least one of a customer sitting at a table in a restaurant or an object on the table. The image acquisition unit 101 acquires the image by receiving an image from the camera 10, for example. The image acquisition unit 101 may acquire an image of a customer and an image of an object on the table, respectively. The image acquired by the image acquisition unit 101 may be a video.

[0020] The voice acquisition unit 102 acquires text data obtained by converting the voice of a customer using a table into text. The voice acquisition unit 102 acquires voice data collected by, for example, the microphone 20. The voice acquisition unit 102 then acquires the text data by converting the voice into text using a voice recognition technology. The voice acquisition unit 102 may acquire text data obtained by converting the voice data collected by the microphone 20 into text from a voice recognition device (not shown).

[0021] The image recognition unit 103 recognizes a part of the customer's body or an object on the table from the image acquired by the image acquisition unit 101. A part of the customer's body is, for example, the customer's hand or eye. An object on the table is, for example, the customer's hand, a menu list, or food. The image recognition unit 103 recognizes a part of the customer's body from at least one of an image of the customer or an image of an object on the table. The image recognition unit 103 may recognize the customer's body as needed. The image recognition unit 103 may also recognize an object from an image of the object on the table. The image recognition unit 103 may recognize the customer's movements by continuing to recognize video.

[0022] The image recognition unit 103 may use, for example, an object detection algorithm to identify objects on a table in an image. To identify the objects, a machine learning model that learns the relationship between images and correct labels attached to the images may be used to recognize specific objects such as hands or food. The image recognition unit 103 may use segmentation to determine whether each pixel is a part of a customer's body or a specific object. The image recognition unit 103 may use facial recognition technology to detect the customer's face in the image and identify the position of the customer's eyes. The image recognition unit 103 may recognize the position of the pupil or iris using the result of identifying the eye position. The image recognition unit 103 may recognize the direction of the customer's gaze based on the position of the pupil or iris relative to the position of the eyes or face. The image recognition unit 103 may also recognize gaze movement. The image recognition unit 103 may also recognize objects on a table by comparing the image with images pre-stored in a memory unit (not shown) of the restaurant analysis system 100.

[0023] The image recognition unit 103 may recognize the page of the menu list that the customer is viewing and the name of the menu listed on that page from the image acquired by the image acquisition unit 101. Here, the menu refers to the dishes listed on the menu list. The image recognition unit 103 may recognize the page of a paper menu list, or may recognize the menu displayed on the screen of the tabletop terminal 30.

[0024] The image recognition unit 103 may further recognize the type of actual dish on the table from the image acquired by the image acquisition unit 101. Here, the dish may include food and drink, or may include all items pre-stored as a menu. Note that a plurality of dishes may be combined into one menu. The image recognition unit 103 may also recognize ingredients contained in the dish from the image. For example, the image recognition unit 103 may distinguish between a steak on a hot plate and accompanying vegetables.

[0025] The image recognition unit 103 may further recognize tableware such as forks, spoons, chopsticks, plates, etc. on the table. By the image recognition unit 103 recognizing the tableware, the next detection unit 104 can more accurately detect the customer's situation, such as whether the customer has touched the food.

[0026] The above description illustrates a case where the image recognition unit 103 recognizes a customer's face or hands. In such a case, the speech acquisition unit 102 may start acquiring speech after the customer's face or hands are recognized at the table using the recognition result of the image recognition unit 103. The speech acquisition unit 102 may then continue acquiring speech and stop acquiring speech a predetermined time after the customer's face or hands are no longer recognized. This reduces the possibility that the speech acquisition unit 102 acquires text data that becomes noise, such as speech from people other than the customer using the table. Therefore, the restaurant analysis system 100 can more accurately analyze customer speech. Furthermore, the restaurant analysis system 100 can reduce unnecessary processing for analyzing text data.

[0027] The image recognition unit 103 may start recognizing objects from the video when a customer's face or hands are recognized. Then, the image recognition unit 103 may stop recognizing objects from the video when the customer's face or hands are no longer recognized. Then, the image recognition unit 103 may recognize objects from still images taken at predetermined intervals, such as every minute. This allows the restaurant analysis system 100 to reduce unnecessary image recognition processing.

[0028] The image recognition unit 103 may recognize attributes of a customer by using the results of recognizing the customer's face. For example, the image recognition unit 103 may recognize the age or gender of the customer by using the facial features of the customer.

[0029] The detection unit 104 detects the state of the customer regarding the meal at the table based on the recognition result by the image recognition unit 103. The state of the customer detected by the detection unit 104 may include various types. The detection unit 104 may detect any one type of state from the states described below. Alternatively, the detection unit 104 may detect multiple types of states.

[0030] In one example, the detection unit 104 may detect a menu selection status by a customer as the status of the customer's meal at the table. The menu selection status indicates the status of the customer's menu selection. The menu selection status includes a status in which the customer has not started menu selection, a status in which selection is in progress, or a status in which selection is complete. Based on the results of the image recognition unit 103 recognizing the customer's hand position or gaze direction and the menu list from the image, the detection unit 104 may detect that the menu list is stored, that a booklet-type menu list is not open, or that the customer is not looking at the menu list. In these cases, the detection unit 104 detects that the customer has not started menu selection. Similarly, using the recognition results of the image recognition unit 103, the detection unit 104 may detect that a booklet-type menu list is open on the table, that the customer is touching the menu list, or that the customer is looking at the menu list. In these cases, the detection unit 104 detects that the customer is selecting a menu. Furthermore, the detection unit 104 may detect that an open menu list has been closed or that a menu has been placed on the table. If these situations are detected after detecting that a customer is selecting a menu, the detection unit 104 detects that the customer has completed the menu selection.

[0031] The detection unit 104 may detect a menu item that the customer focuses on while selecting a menu item. In this case, for example, the detection unit 104 uses the relationship between the position of the menu list and the position of the customer's hand or the direction of their gaze to detect a menu item that the customer points to or gazes at a predetermined number of times or for a predetermined period of time. The detection unit 104 may also detect a situation in which the customer is comparing two or more menu options. When two or more menu items are focused on simultaneously or within a predetermined period of time, the detection unit 104 may detect that the customer is comparing the two or more menu options.

[0032] The detection unit 104 detects featured and compared menu items based on the image recognition results, enabling the restaurant analysis system 100 to analyze customer opinions about menu items that were not ordered. A typical order management system manages order information including the receipt number, table number, number of customers at the table, ordered menu items, and order time. Restaurant employees can use the order information to obtain statistical data on menu items ordered at the restaurant. However, because menu items that were not ordered do not appear in the order information of the order management system, it can be difficult to analyze the candidate menu items. Therefore, it is advantageous for the detection unit 104 to detect the menu selection status based on the image recognition results.

[0033] The detection unit 104 may detect the customer's meal status at the table based on the text data acquired by the voice acquisition unit 102 in addition to the image recognition results. For example, the detection unit 104 may detect the menu selection status based on whether the text data contains keywords related to the menu selection status in addition to the image recognition results. Natural language processing technology is used to extract the keywords. The detection unit 104 detects that the customer is selecting a menu when the text data of the voice acquired while the menu list is open on the table contains keywords uttered during selection. Keywords uttered during selection include the menu name, "I'll take this or that," "Which one should I get," etc. The detection unit 104 may detect the menu item that the customer focused on and two or more menu items that were compared based on the keywords included in the text data. The detection unit 104 may also detect the completion of the menu selection based on the keywords included in the text data.

[0034] The detection unit 104 may detect the menu item that the customer is paying attention to by using the direction of the customer's gaze recognized from the image and information about the operation content of the tabletop terminal 30. The information about the operation content of the tabletop terminal 30 indicates the page or menu of the menu list that the customer is viewing on the tabletop terminal 30. The detection unit 104 receives the information about the operation content from the tabletop terminal 30. The detection unit 104 detects the menu item that was displayed while the customer was directing their gaze at the tabletop terminal 30 as the menu item that the customer is paying attention to. In this case, the detection unit 104 can more accurately detect the menu item that the customer is paying attention to compared to when the image recognition unit 103 uses the results of recognizing the display content of the tabletop terminal 30.

[0035] When two or more compared menu items are detected, the detection unit 104 may detect one or more menu items selected by the customer. For example, the detection unit 104 detects one or more menu items selected by the customer based on the recognition result of the image recognition unit 103 and the text data of the voice acquisition unit 102. When the menu item pointed to by the customer is recognized from the image and the text data contains a keyword indicating that the menu item has been selected, the detection unit 104 can detect the menu item pointed to by the customer as the menu item selected by the customer. An example of a keyword indicating that the menu item has been selected is "I'll order this."

[0036] Furthermore, for example, the detection unit 104 detects menu items included in order information indicating the customer's order as one or more menu items selected by the customer. The detection unit 104 may receive the order information from any device including the server of the order management system, the employee terminal 40, and the tabletop terminal 30. By using the order information, the detection unit 104 can more accurately detect which menu items were ordered and which were not ordered out of the two or more compared menu items than if the order information were not used.

[0037] Next, another example of the customer's status regarding the meal at the table, detected by the detection unit 104, will be described. In one example, the detection unit 104 may detect whether food remains as the customer's status regarding the meal at the table. The status of whether food remains indicates the customer's status regarding food consumption. The status of whether food remains includes whether the customer has not touched the food, whether the customer is currently eating, or whether the customer has finished eating. The detection unit 104 may detect whether food remains as it is using the relationship between the position of the customer's hand and the position of the food, or the amount of food on the plate. The detection unit 104 may detect that the recognized food has been left uneaten if, with food remaining on the plate, a predetermined time has elapsed since the food was first recognized, or if a predetermined time has elapsed since the food was last touched. The detection unit 104 may also detect which menu item's food has been left uneaten. Furthermore, the detection unit 104 may detect which ingredients have been left uneaten from the ingredient recognition results.

[0038] Further, other examples of the status of a customer eating at a table detected by the detection unit 104 will be described. In one example, the detection unit 104 may detect the status of food being served as the status of a customer eating at a table. The status of food being served includes a status in which the food ordered by the customer has not been served and a status in which the food has been served. The detection unit 104 may detect the status of which food has been served at the table based on the results of image recognition. Furthermore, the detection unit 104 may detect the status in which the food has not been served using order information indicating the customer's order. The detection unit 104 may detect a dish that was not recognized from the image among the ordered dishes as a dish that has not been served.

[0039] The above describes examples of customer situations detected by the detection unit 104. In addition to the above examples, the detection unit 104 may also detect customer situations related to eating at the table.

[0040] The generation unit 105 generates information indicating the customer's opinion regarding the customer's situation, based on the customer's situation regarding the meal at the table detected by the detection unit 104 and the text data acquired by the voice acquisition unit 102. The generation unit 105 generates information indicating the customer's opinion regarding the situation detected by the detection unit 104, based on the customer's utterance while the situation is occurring. In generating the information, the generation unit 105 extracts keywords indicating the customer's opinion from the text data using natural language processing technology. Then, for example, the generation unit 105 generates the extracted keywords as information indicating the customer's opinion. The keywords extracted by the generation unit 105 are set in advance. The generation unit 105 may classify the extracted keywords by type of opinion and generate the classification result as information indicating the customer's opinion.

[0041] For example, when the detection unit 104 detects a menu selection situation, the generation unit 105 generates information indicating the customer's opinion regarding the menu selection. The customer's opinion regarding the menu selection indicates the factors the customer considered when selecting the menu. The customer's opinion regarding the menu selection is an example of feedback on the food. In a more specific example, the generation unit 105 generates information indicating the customer's opinion from text data of the voice spoken when the situation in which the customer is selecting a menu to order is detected. Keywords extracted by the generation unit 105 may be preset, such as keywords related to the taste of the food, such as the spiciness or sweetness of the food, keywords related to the quantity, such as whether the food can be shared by multiple people, or keywords related to the healthiness of the food. Assume that text data including text such as "This menu has a low fat content" is acquired while the customer is selecting a menu. In this case, for example, the generation unit 105 extracts "fat" from the text data among the preset keywords. Then, based on the extracted keywords, the generation unit 105 generates information indicating that the amount of fat was taken into consideration as information indicating the customer's opinion regarding the menu selection.

[0042] When the detection unit 104 detects a menu item that has been focused on by a customer, the generation unit 105 may generate information indicating the customer's opinion as an opinion about the focused menu item. When the detection unit 104 detects two or more menu items compared by a customer and one or more menu items ordered by the customer, the generation unit 105 may generate information indicating the customer's opinion, such as a reason why one of the menu items was not selected or a reason why the ordered menu item was selected.

[0043] Furthermore, for example, when the detection unit 104 detects that some food has been left uneaten, the generation unit 105 generates information indicating the customer's opinion on the served food. The customer's opinion on the served food is an impression of the food left uneaten by the customer. Keywords related to the portion size and the seasoning of the food are preset as keywords extracted by the generation unit 105 from the text data. The customer's opinion on the food is an example of feedback on the food. In a more specific example, the generation unit 105 generates information indicating the customer's opinion from text data of a voice uttered when it is detected that a dish from the menu has been left uneaten. Assume that when it is detected that a customer has left some food uneaten, text data including text such as "It was a little too much" is acquired. In this case, for example, the generation unit 105 extracts "it was a lot" from the text data, which is one of the preset keywords. Then, the generation unit 105 generates information indicating that the portion size of the menu was felt to be too much, as information indicating the customer's opinion on the food, based on the extracted keywords. The generating unit 105 may generate the content uttered after the keyword of the menu or ingredients as information indicating the customer's opinion on the dish.

[0044] Furthermore, for example, when the detection unit 104 detects the food service status, the generation unit 105 generates information indicating the customer's opinion regarding the food service status. The customer's opinion regarding the food service status indicates the customer's thoughts on the speed and timing of food service. The customer's opinion regarding the service status is an example of feedback regarding the service. Keywords related to the speed and timing of food service are preset as keywords to be extracted from the text data by the generation unit 105. In a more specific example, the generation unit 105 generates information indicating the customer's opinion from text data of speech uttered when a situation in which the ordered food has not been served is detected. Alternatively, the generation unit 105 may generate information indicating the customer's opinion from text data of speech uttered when a situation in which the ordered food has been served is detected. Assume that when a situation in which the ordered food has not been served is detected, text data including text such as "I wonder if it has arrived yet" is acquired. In this case, for example, the generation unit 105 extracts "has not arrived" from the text data, one of the preset keywords. Then, based on the extracted keywords, the generating unit 105 generates information indicating that the customer felt that the food was being served slowly, as information indicating the customer's opinion on the food serving situation.

[0045] The generation unit 105 may generate information indicating a customer's opinion on the food service status based on text data of a voice uttered a predetermined time after the order was received. The generation unit 105 references the order information and acquires the time when the order was received. The predetermined time is set appropriately, taking into account the time required for cooking, such as 20 minutes. This allows the generation unit 105 to exclude utterances about dishes other than the ordered dish from the information generation target. Therefore, the generation unit 105 can more accurately generate information indicating a customer's opinion on the speed at which the ordered dish is served.

[0046] The above describes examples of information generated by the generation unit 105. The generation unit 105 may generate information indicating customer opinions other than the above examples, depending on the situation detected by the detection unit 104 and depending on the text included in the text data.

[0047] The output unit 106 outputs the information indicating the customer's opinion generated by the generation unit 105 to the employee terminal 40 used by the restaurant employee. That is, when the generation unit 105 generates information indicating the customer's opinion regarding the menu selection, the output unit 106 outputs the information indicating the customer's opinion regarding the menu selection. When the generation unit 105 generates information indicating the customer's opinion regarding the served food, the output unit 106 outputs the information indicating the customer's opinion regarding the served food. When the generation unit 105 generates information indicating the customer's opinion regarding the food serving status, the output unit 106 outputs the information indicating the customer's opinion regarding the food serving status. Examples of information output by the output unit 106 will be described using Figures 4, 5, and 6.

[0048] When the generation unit 105 generates information indicating the customer's opinion regarding the menu selection, the employee terminal 40 displays, for example, the table shown in FIG. 4 based on the output from the output unit 106. In FIG. 4, the information indicating the customer's opinion regarding the menu selection output by the output unit 106 is displayed as "Reason for Selection." The output unit 106 may output the information indicating the customer's opinion regarding the menu selection in association with other information detected by the detection unit 104. For example, the output unit 106 may output the information in association with a menu item that the customer focused on during selection. The output unit 106 may output two or more menu items that were compared among the menu items that the customer focused on. In FIG. 4, the menu item that the customer focused on is displayed as "Order Candidates." The output unit 106 may further output the menu item selected by the customer in association with the information. In FIG. 4, the menu item selected by the customer is displayed as "Order Details." Among the menu items that the customer focused on, a menu item that the customer actually ordered may be excluded from the "Order Candidates." The output unit 106 may output the menu compared with the menu selected by the customer and display it as "order candidates."

[0049] When the generation unit 105 generates information indicating the customer's opinion on the served food, the employee terminal 40 displays the table of FIG. 5, for example, based on the output from the output unit 106. In FIG. 5, the information indicating the customer's opinion on the food output by the output unit 106 is displayed as "reason for leaving food." The output unit 106 may output other information detected by the detection unit 104 in association with the information indicating the customer's opinion on the food. For example, the output unit 106 may output the information in association with the menu item or ingredients that were left uneaten. In FIG. 5, the menu item that was left uneaten is displayed as "order details," and the ingredients that were left uneaten are displayed as "ingredients left."

[0050] When the generation unit 105 generates information indicating the customer's opinion regarding the food serving status, the employee terminal 40 displays the screen shown in FIG. 6 based on the output from the output unit 106, for example. In FIG. 6, the word "urgent" indicates that the customer felt that the food was being served slowly, as the customer's opinion. When the screen shown in FIG. 6 is displayed on the order display terminal that displays cooking instructions, the cook can receive a reminder about the order. Furthermore, the cook can prioritize cooking the customer's food. Some customers find it difficult to request that the waiter serve them if the food is being served late. By having the output unit 106 output information indicating the customer's opinion regarding the food serving status, the customer can receive the delayed food without having to directly request it from the waiter.

[0051] The output unit 106 may output information indicating the customer's opinion in association with the customer's attributes. The customer's attributes are recognized by, for example, the image recognition unit 103. In Fig. 5, the customer's attributes are displayed.

[0052] The output unit 106 may output information indicating customer opinions in association with order information from the order management system. The order information includes a slip number, a table number, the number of customers at the table, the ordered menu item, and the time of the order. The output unit 106 obtains and outputs the order information of the customer whose opinion has been analyzed from the order management system. In FIG. 4, the order information is displayed as "date and time of use," "number of people," and "order details." In FIG. 5, the order information is displayed as "date and time of use" and "order details." In FIG. 6, the order information includes the table number, the ordered menu item, and the time of the order.

[0053] The output unit 106 may vary the timing of outputting information indicating the customer's opinion about the customer's situation depending on the type of customer situation detected by the detection unit 104. For example, the output unit 106 may output information depending on whether the information is generated by the generation unit 105 or depending on whether a request for information is received from the employee terminal 40. The restaurant analysis system 100 stores the information generated by the generation unit 105. Then, when a request is received from the employee terminal 40, the output unit 106 outputs the stored information to the employee terminal 40.

[0054] For example, when the food serving status is detected, the output unit 106 outputs the information in response to the information generated by the generation unit 105. This allows the output unit 106 to output the information at a timing when the waiter can take action regarding the serving of food to customers waiting at their tables. When the menu selection status is detected and when food is left over is detected, the output unit 106 may output the information in response to a request from the employee terminal 40. This allows the output unit 106 to output the information at a timing when the employee should consider improving the food or service. Furthermore, the output unit 106 can output information indicating multiple opinions of multiple customers all at once.

[0055] The above describes a case where output unit 106 varies the timing at which it outputs information indicating customer opinions depending on the type of customer situation. This allows restaurant analysis system 100 to assist restaurant employees in responding to customer feedback in an appropriate manner. Note that output unit 106 may also output information indicating customer opinions regarding the food serving status in response to a request from employee terminal 40. In this case, employees can later analyze whether some customers felt that the food was being served slowly.

[0056] An example of the operation of restaurant analysis system 100 according to the present disclosure will be described using FIG. 7. Restaurant analysis system 100 starts the operation of FIG. 7, for example, while the restaurant is open. In step S1, image acquisition unit 101 acquires an image of at least one of a customer sitting at a table in the restaurant or an object on the table. In step S2, voice acquisition unit 102 acquires text data obtained by converting the voice of the customer sitting at the table into text. In step S3, image recognition unit 103 recognizes a part of the customer's body or an object on the table from the image acquired by image acquisition unit 101.

[0057] In step S4, the detection unit 104 detects the state of the customer regarding the meal at the table based on the recognition result by the image recognition unit 103. In one example, the detection unit 104 detects the state of the menu selection by the customer.

[0058] In step S5, the generation unit 105 generates information indicating the customer's opinion regarding the customer's situation, based on the customer's situation regarding the meal at the table detected by the detection unit 104 and the text data acquired by the voice acquisition unit 102. In one example, the generation unit 105 generates information indicating the customer's opinion regarding the menu selection.

[0059] In step S6, output unit 106 outputs the information indicating the customer's opinion generated by generation unit 105 to employee terminal 40 used by an employee of the restaurant. With this, restaurant analysis system 100 ends the operation of FIG. 7.

[0060] According to one embodiment of restaurant analysis system 100, generation unit 105 generates information indicating customer opinions regarding the customer's situation based on the customer's situation regarding the meal at the table and text data obtained by converting the customer's voice into text. Output unit 106 then outputs the generated information indicating the customer's opinion to employee terminal 40. This eliminates the need for customers to communicate their opinions regarding the food or service to waiters. It also reduces the time waiters spend directly listening to customer opinions during busy periods. Therefore, restaurant analysis system 100 can provide restaurants with feedback that is appropriate in quantity and timing. In other words, restaurant analysis system 100 can provide a larger amount of feedback at a time that is convenient for employees to utilize, which can help improve the restaurant.

[0061] The above describes a case where restaurant analysis system 100 analyzes conversations between customers. However, restaurant analysis system 100 may also analyze conversations between staff and customers. In this case, restaurant analysis system 100 outputs information to headquarters staff without the staff having to convey the customer's opinion to headquarters staff.

[0062] [Hardware configuration] In each of the above-described embodiments, each component of the restaurant analysis system 100 represents a functional block. Some or all of the components of the restaurant analysis system 100 may be realized by any combination of a computer 500 and a program.

[0063] Fig. 8 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 8, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0064] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0065] Program 504 includes instructions for realizing each function of restaurant analysis system 100. Program 504 is stored in advance in ROM 502, RAM 503, and storage device 505. Processor 501 executes the instructions included in program 504 to realize each function of restaurant analysis system 100. RAM 503 may also store data processed in each function of restaurant analysis system 100.

[0066] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or keyboard, and receives information input from employees, etc. The output device 510 is, for example, a display, and outputs (displays) information to employees, etc. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0067] It should be noted that the hardware configuration shown in FIG. 8 is an example, and other components may be added, or some components may not be included.

[0068] There are various variations in the method of realizing restaurant analysis system 100. For example, restaurant analysis system 100 may be realized by any combination of different computers and programs for each component. Furthermore, multiple components included in restaurant analysis system 100 may be realized by any combination of a single computer and program.

[0069] Furthermore, at least a part of the restaurant analysis system 100 may be provided in a SaaS (Software as a Service) format. That is, at least a part of the functions for realizing the restaurant analysis system 100 may be executed by software executed via a network.

[0070] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0071] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0072] [Appendix 1] an image capturing means for capturing an image of at least one of a customer sitting at a table in the restaurant or an object on the table; a voice acquisition means for acquiring text data obtained by converting the voice of the customer into text; an image recognition means for recognizing a body part or an object of the customer from the image; a detection means for detecting a menu selection status by the customer based on the recognition result of the image recognition means; a generating means for generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; an output means for outputting information indicating the customer's opinion to an employee terminal used by an employee of the restaurant; A restaurant analysis system equipped with:

[0073] [Appendix 2] the detection means detects a menu item that the customer has focused on based on the hand position or gaze direction of the customer recognized by the image recognition means while the customer is selecting a menu item; The generating means generates information indicating the customer's opinion regarding the menu item of interest. Attachment 1: A restaurant analysis system.

[0074] [Appendix 3] The output means outputs information indicating the customer's opinion together with information on one or more menu items not ordered among the two or more attention menu items based on order information indicating the menu items ordered by the customer. Attachment 2: A restaurant analysis system.

[0075] [Appendix 4] The detection means detects the menu selection status based on the text data, the customer's order information, or information on the display content of a table terminal provided at the table. 4. The restaurant analysis system according to any one of appendices 1 to 3.

[0076] [Appendix 5] the image recognition means recognizes the objects including the food on the table; The detection means further detects whether any of the menu items provided to the customer is left uneaten based on the object recognition result, The generating means generates information indicating the customer's opinion on the dish based on the customer's leftovers of the menu and the text data. A restaurant analysis system according to any one of appendices 1 to 4.

[0077] [Appendix 6] the image recognition means recognizes the objects including ingredients contained in the dish, The detection means further detects any leftover ingredients based on the object recognition result, The generating means generates information indicating the customer's opinion regarding the leftover food ingredients. 6. The restaurant analysis system according to claim 5.

[0078] [Appendix 7] The detection means further detects a status of providing the ordered menu to the customer based on the object recognition result, The generating means generates information indicating the customer's opinion regarding the provision status based on the provision status to the customer and the text data. 7. A restaurant analysis system according to any one of appendices 1 to 6.

[0079] [Appendix 8] the detection means detects the customer's status regarding the meal at the table, including at least one of a status of leftover food and a status of food being served, and a status of menu selection, based on the object recognition result; the generating means generates information indicating the customer's opinion in the customer's situation based on the customer's situation and the text data; The output means outputs the information indicating the customer's opinion to the employee terminal at different times depending on the type of the customer's situation, whether the information is output in response to generation of the information or in response to a request from the employee terminal. A restaurant analysis system according to any one of appendices 1 to 7.

[0080] [Appendix 9] Acquire an image of at least one of a customer sitting at a table in the restaurant or an object on the table; Acquire text data obtained by converting the customer's voice into text; Recognizing a body part or the object of the customer from the image; Detecting a menu selection status by the customer based on the result of the recognition; generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; Information indicating the customer's opinion is output to an employee terminal used by an employee of the restaurant. Restaurant analysis method.

[0081] [Appendix 10] Acquire an image of at least one of a customer sitting at a table in the restaurant or an object on the table; Acquire text data obtained by converting the customer's voice into text; Recognizing a body part or the object of the customer from the image; Detecting a menu selection status by the customer based on the result of the recognition; generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; Information indicating the customer's opinion is output to an employee terminal used by an employee of the restaurant. A program that causes a computer to perform a process.

[0082] Some or all of the configurations described in Supplementary Notes 2-8, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 9-10 in the same dependent relationship as Supplementary Notes 2-8. Not limited to Supplementary Notes 1, 9-10, but also to various hardware, software, various recording devices for recording software, or systems, some or all of the configurations described as Supplements may be made dependent on each other within the scope of the above-mentioned embodiments. [Explanation of symbols]

[0083] 10 Camera 20. Mike 30 Tabletop Terminals 40 employee terminals 100 Restaurant Analysis System 101 Image acquisition unit 102 Voice acquisition unit 103 Image Recognition Unit 104 Detector 105 Generation part 106 Output section

Claims

1. an image capturing means for capturing an image of at least one of a customer sitting at a table in the restaurant or an object on the table; a voice acquisition means for acquiring text data obtained by converting the voice of the customer into text; an image recognition means for recognizing a body part or an object of the customer from the image; a detection means for detecting a menu selection status by the customer based on the recognition result of the image recognition means; a generating means for generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; an output means for outputting information indicating the customer's opinion to an employee terminal used by an employee of the restaurant; A restaurant analysis system equipped with:

2. the detection means detects a menu item that the customer has focused on based on the hand position or gaze direction of the customer recognized by the image recognition means while the customer is selecting a menu item; The generating means generates information indicating the customer's opinion regarding the menu item of interest. The restaurant analysis system according to claim 1 .

3. The output means outputs information indicating the customer's opinion together with information on one or more menu items not ordered among the two or more attention menu items based on order information indicating the menu items ordered by the customer. The restaurant analysis system according to claim 2 .

4. The detection means detects the menu selection status based on the text data, the customer's order information, or information on the display content of a table terminal provided at the table. The restaurant analysis system according to any one of claims 1 to 3.

5. the image recognition means recognizes the objects including the food on the table; The detection means further detects whether any of the menu items provided to the customer is left uneaten based on the object recognition result, The generating means generates information indicating the customer's opinion on the dish based on the customer's leftovers of the menu and the text data. The restaurant analysis system according to any one of claims 1 to 3.

6. the image recognition means recognizes the objects including ingredients contained in the dish, the detection means detects uneaten food ingredients based on the object recognition result, The generating means generates information indicating the customer's opinion regarding the leftover food ingredients. The restaurant analysis system according to claim 5 .

7. The detection means further detects a status of providing the ordered menu to the customer based on the object recognition result, The generating means generates information indicating the customer's opinion regarding the provision status based on the provision status to the customer and the text data. The restaurant analysis system according to any one of claims 1 to 3.

8. the detection means detects the customer's status regarding the meal at the table, including at least one of a status of leftover food and a status of food being served, and a status of menu selection, based on the object recognition result; the generating means generates information indicating the customer's opinion in the customer's situation based on the customer's situation and the text data; The output means outputs the information indicating the customer's opinion to the employee terminal at different times depending on the type of the customer's situation, whether the information is output in response to generation of the information or in response to a request from the employee terminal. The restaurant analysis system according to any one of claims 1 to 3.

9. acquiring an image of at least one of a customer sitting at a table in the restaurant or an object on the table; Obtaining text data obtained by converting the customer's voice into text; Recognizing a body part or the object of the customer from the image; Detecting a menu selection status by the customer based on the result of the recognition; generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; Outputting information indicating the customer's opinion to an employee terminal used by an employee of the restaurant. Restaurant analysis method.

10. acquiring an image of at least one of a customer sitting at a table in the restaurant or an object on the table; Obtaining text data obtained by converting the customer's voice into text; Recognizing a body part or the object of the customer from the image; Detecting a menu selection status by the customer based on the result of the recognition; generating information indicating the customer's opinion regarding the menu selection based on the menu selection status of the customer and the text data; Outputting information indicating the customer's opinion to an employee terminal used by an employee of the restaurant. A program that causes a computer to perform a process.

Images