System

Abstract

A system is provided.SOLUTION: The system includes a voice recognizing means for converting a voice inputted by a user into text data, a translating means for translating the text data into another language by a multilingual translating module, a response generating means for analyzing the translated text data and generating an appropriate response, and an speech synthesis means for converting the generated response into a voice and providing it to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the restaurant industry, particularly in large chain restaurants, labor shortages and a lack of multilingual and sign language support are serious problems. When customers make inquiries in multiple languages ​​or need to communicate in sign language, employees are unable to respond, which raises concerns about poor service and lower customer satisfaction. Furthermore, existing systems make it difficult to efficiently collect and analyze customer feedback. There is a need to solve these issues and improve operational efficiency and customer satisfaction. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a speech recognition unit that converts speech input by a user into text data, and employs a translation unit that translates the text data into another language using a multilingual translation module. Next, a system is constructed that includes a response generation unit that analyzes the translated text data and generates an appropriate response, and a speech synthesis unit that converts the generated response into speech and provides it to the user.

[0006] The system further includes a sign language analysis means for converting sign language input by a user through a camera into text data using a video analysis engine, an augmented reality menu providing means for reflecting the text data in an augmented reality menu and providing visual feedback, a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations, and a dashboard providing means for displaying the analysis results on a dashboard.

[0007] These measures will enable restaurants in the food service industry to smoothly provide multilingual and sign language support, and will also enable them to efficiently collect and analyze customer feedback, thereby improving operational efficiency and customer satisfaction.

[0008] A "voice recognition means" is a technique or device for converting a user's voice input into digital text data.

[0009] A "translation means" is a technology or module for translating text data into another language.

[0010] The "response generation means" is a technology or device for analyzing the translated text data and generating an appropriate response message.

[0011] The "voice synthesis means" is a technique or device for converting the generated response message into voice format and providing it to the user.

[0012] "Sign language analysis means" refers to a technology or device for converting sign language images input through a camera into text data.

[0013] An "augmented reality menu providing means" is a technology or device for using text data to utilize augmented reality (AR) technology to provide visual feedback.

[0014] A "feedback collection means" is a technique or device for capturing feedback from users as text data.

[0015] The "feedback analysis means" is a technology or device for analyzing the captured feedback with a natural language processing engine and generating positive and negative evaluations.

[0016] A "dashboard providing means" is a technology or device that visually displays the results of feedback analysis so that employees can easily check them. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a generative AI solution that enables multilingual and sign language support for restaurants and wait staff in the food service industry. The system for implementing the present invention consists of the following elements:

[0039] Multilingual interactive voice response system

[0040] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[0041] Examples:

[0042] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[0043] AR menu provision system that supports sign language

[0044] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[0045] Examples:

[0046] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[0047] Customer feedback collection and analysis system

[0048] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates positive and negative ratings. The results are displayed on a dashboard for employees to review.

[0049] Examples:

[0050] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[0051] Optimized menu suggestion system

[0052] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes the received data and generates optimal menu suggestions, which are presented to the customer by the terminal. The customer can then confirm and select additional orders.

[0053] Examples:

[0054] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[0055] The above is an embodiment of the present invention. This system can improve the efficiency of customer service operations in the restaurant industry and effectively solve the problem of labor shortages through functions such as multilingual support, sign language support, feedback collection and analysis, and optimized menu suggestions.

[0056] The processing flow will be explained below.

[0057] Multilingual interactive voice response system

[0058] Step 1:

[0059] The user queries the order by voice.

[0060] Examples: "Cheeseburger", "Coke"

[0061] Step 2:

[0062] The device captures the voice and converts it into text data using a voice recognition engine.

[0063] Speak: "I'd like a cheeseburger and a Coke, please."

[0064] Converted text: "I'd like a cheeseburger and a coke, please."

[0065] Step 3:

[0066] The server sends the received text data to a translation module to translate it into the customer's language.

[0067] Data sent: "I'd like a cheeseburger and a coke, please."

[0068] Translated text: "I would like a cheeseburger and a coke, please" (English)

[0069] Step 4:

[0070] The server parses the translated text data and generates an appropriate response.

[0071] Parse: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[0072] Step 5:

[0073] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[0074] Synthesized voice: "You've ordered a cheeseburger and a Coke. Is there anything else you'd like to order?"

[0075] AR menu provision system that supports sign language

[0076] Step 1:

[0077] A user enters an order using sign language.

[0078] Example: Say "hamburger" and "fries" in sign language.

[0079] Step 2:

[0080] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[0081] Captured footage: Sign language for "hamburger" and "fries"

[0082] Converted text data: "Hamburger and fries"

[0083] Step 3:

[0084] The server receives the converted text data and generates an AR menu that supports sign language.

[0085] Generated: "Hamburger and fries"

[0086] Generated AR menu: Burger and fries images, prices, and details

[0087] Step 4:

[0088] The device displays the generated AR menu and provides visual feedback to the user.

[0089] Showing: Image and details of a burger and fries

[0090] Customer feedback collection and analysis system

[0091] Step 1:

[0092] The user enters feedback.

[0093] For example: "The service was good, but the food was cold."

[0094] Step 2:

[0095] The device captures the feedback as text data and sends it to the server.

[0096] Feedback captured: "The service was good, but the food was cold."

[0097] Step 3:

[0098] The server analyzes the feedback it receives using a natural language processing engine.

[0099] Analysis results: Positive reviews (good service), negative reviews (cold food)

[0100] Step 4:

[0101] The server displays the analysis results on a dashboard.

[0102] Display: Positive and negative reviews, feedback details

[0103] Step 5:

[0104] The user (employee) checks the dashboard and considers and implements countermeasures.

[0105] Action example: Review food temperature management and further improve service

[0106] Optimized menu suggestion system

[0107] Step 1:

[0108] The user initiates the order.

[0109] Example: Select "Veggie Burger."

[0110] Step 2:

[0111] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[0112] Data sent: Current order "Veggie Burger", past order history

[0113] Step 3:

[0114] The server analyzes the received data and generates optimal menu suggestions.

[0115] Suggestion: "Would you like to try today's special smoothie?"

[0116] Step 4:

[0117] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[0118] Display content: Today's limited smoothie selection options and details

[0119] Step 5:

[0120] The user confirms the proposal and places an additional order.

[0121] Example behavior: Ordering an extra smoothie.

[0122] The above is a specific processing flow of each system in the embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In today's restaurant industry, serving customers with language barriers or communication disabilities is a major issue that leads to labor shortages and reduced operational efficiency. In particular, when multilingual and sign language support is required, the limitations of employee skills and resources become apparent. There is also a need for systems that can instantly collect and analyze customer feedback and develop appropriate countermeasures. Furthermore, it is important to be able to suggest optimal menu items based on customers' past ordering history, as this will lead to improved customer satisfaction and increased sales.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, a communication means for transmitting user speech data captured by the terminal to the server, and a playback means for playing back the generated speech response from the terminal, thereby enabling multilingual automated voice responses that overcome language barriers.

[0128] Furthermore, a system is provided that includes a sign language analysis means that converts sign language input by a user through a camera into text data using a video analysis engine, an augmented reality menu providing means that reflects the text data in an augmented reality menu and provides visual feedback, a communication means that transmits video data captured by the terminal to a server, and a display means that displays the generated augmented reality menu on the terminal, which enables smooth communication with customers who use sign language and improves the quality of service.

[0129] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback with a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, a communication means for transmitting the feedback data captured by the terminal to a server, and a confirmation means for employees to consider and implement countermeasures based on the analysis results. This makes it possible to quickly collect and analyze customer feedback and develop appropriate countermeasures.

[0130] "Speech recognition means" refers to a function for converting speech input by a user into text data.

[0131] "Translation means" refers to a function for translating text data into other languages ​​using a multilingual translation module.

[0132] "Response generation means" refers to a function for analyzing translated text data and generating an appropriate response.

[0133] "Speech synthesis means" refers to a function for converting the generated response into speech and providing it to the user.

[0134] "Communication means" refers to a function for transmitting user voice data and video data captured by the terminal to the server, and for transmitting responses and menus from the server to the terminal.

[0135] The "playback means" refers to a function for playing back the generated voice response from the terminal.

[0136] "Sign language analysis means" refers to the function for converting sign language input through a camera into text data using an image analysis engine.

[0137] The "augmented reality menu providing means" refers to a function for reflecting the converted text data in an augmented reality menu and providing visual feedback.

[0138] "Display means" refers to a function for displaying the generated augmented reality menu on the terminal.

[0139] "Feedback collection means" refers to a function for capturing feedback from users as text data.

[0140] "Feedback analysis means" refers to a function for analyzing captured feedback using a natural language processing engine and generating positive and negative evaluations.

[0141] "Dashboard provision means" refers to a function for displaying analysis results on a dashboard so that employees can check them.

[0142] "Verification means" refers to the function that allows employees to consider and implement countermeasures based on the analysis results.

[0143] Multilingual interactive voice response system

[0144] This invention is a system that automatically converts orders entered by voice into text, translates them in multiple languages, and generates voice responses. When a user enters an order by voice, the device captures the speech and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is sent to a server, which translates it into the user's native language using a multilingual translation module such as Google Cloud Translation. The server analyzes the translated text data and generates an appropriate response using a generative AI model such as GPT-4. The generated response is converted into speech using a speech synthesis engine such as Amazon Polly, and the response is finally played back by the device.

[0145] Examples:

[0146] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[0147] AR menu provision system that supports sign language

[0148] When a sign language user places an order through the camera, the device captures the video and converts it into text data using a sign language analysis engine such as Google Cloud Video Intelligence. This text data is sent to a server, which then uses an augmented reality (AR) content creation tool such as Unity to generate an AR menu that supports sign language. The generated AR menu is then displayed on the device, providing visual feedback to the user.

[0149] Examples:

[0150] When a customer orders a "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[0151] Customer feedback collection and analysis system

[0152] When a user enters feedback as text, the device captures the feedback and sends it to a server. The server receives the feedback data and analyzes it using a natural language processing engine such as GPT-4 or BERT. The analysis results in positive and negative ratings, which are displayed on a dashboard. Employees can view customer feedback through the dashboard and consider and implement appropriate measures.

[0153] Examples:

[0154] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[0155] Optimized menu suggestion system

[0156] When a user begins an order, the device sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes this data in a database such as BigQuery, generates optimal menu suggestions, and sends them to the device. The device then presents these suggestions to the user, who can then confirm and select additional orders.

[0157] Examples:

[0158] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[0159] Prompt Sentence Examples

[0160] "Please provide a detailed description of the voice commands or sign language inputs used by the user to place an order and the actions taken based on those inputs. Also, please provide a detailed description of feedback inputs and optimized menu suggestions."

[0161] The above is an embodiment of the present invention, which makes it possible to streamline customer service operations in the restaurant industry and effectively solve the problem of labor shortages through multilingual support, sign language support, feedback collection and analysis, and optimized menu proposals.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Multilingual Interactive Voice Response System Processing Steps

[0164] Step 1:

[0165] The user enters the order by voice.

[0166] Input: User speech (e.g., "I would like a cheeseburger and a coke, please")

[0167] Output: Audio data is generated on the device

[0168] Specific operation: The user speaks an order into a terminal with a microphone.

[0169] Step 2:

[0170] The device captures the audio.

[0171] Input: User's voice data

[0172] Output: Stored as digital audio data

[0173] Specific operation: The device's microphone picks up sound and records it as digital audio data.

[0174] Step 3:

[0175] The terminal transmits the audio data to the server.

[0176] Input: Digital audio data

[0177] Output: Audio data is sent to the server

[0178] Specific operation: The device sends the captured audio data to the server using a communication means.

[0179] Step 4:

[0180] The server converts the speech to text.

[0181] Input: Audio data arriving at the server

[0182] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[0183] What happens: The server converts the audio data into text using the Google Cloud Speech-to-Text API.

[0184] Step 5:

[0185] The server translates the text data.

[0186] Input: Converted text data

[0187] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[0188] Specific operation: The server uses the Google Cloud Translation API to translate the text into the user's native language.

[0189] Step 6:

[0190] The server parses the translated text data and generates a response.

[0191] Input: Translated text data

[0192] Output: Response text (e.g., "You'd like a cheeseburger and a coke").

[0193] What it does: The server uses the GPT-4 generative AI model to analyze the translated text and generate an appropriate response.

[0194] Step 7:

[0195] The server converts the response into speech.

[0196] Input: Response text

[0197] Output: Speech data (e.g., "You'd like a cheeseburger and a coke, right?")

[0198] Specific operation: The server uses a speech synthesis engine such as Amazon Polly to convert the response text into audio data.

[0199] Step 8:

[0200] The server sends the audio data to the terminal.

[0201] Input: Generated audio data

[0202] Output: Audio data is sent to the device

[0203] Specific operation: The server sends the generated voice data to the terminal.

[0204] Step 9:

[0205] The device plays a voice response.

[0206] Input: Audio data received on the device

[0207] Output: The audio played to the user

[0208] Specific operation: The device plays the received audio data through the speaker and prompts the user to confirm.

[0209] Processing steps of an AR menu provision system that supports sign language

[0210] Step 1:

[0211] A user who uses sign language enters an order through the camera.

[0212] Input: User's sign language actions

[0213] Output: Video data is generated by the camera

[0214] Specific actions: The user indicates an order in sign language in front of the camera.

[0215] Step 2:

[0216] The device captures the sign language video.

[0217] Input: Video data of the user's sign language movements

[0218] Output: Saved as digital video data

[0219] Specific operation: The device's camera captures sign language movements and records them as digital video data.

[0220] Step 3:

[0221] The terminal transmits the video data to the server.

[0222] Input: Digital video data

[0223] Output: Video data is sent to the server

[0224] Specific operation: The video data captured by the device is sent to the server using a communication means.

[0225] Step 4:

[0226] The server converts the video data into text.

[0227] Input: Video data arriving at the server

[0228] Output: Text data (e.g. "hamburger and fries")

[0229] How it works: The server uses the Google Cloud Video Intelligence API to analyze the sign language video data and convert it into text.

[0230] Step 5:

[0231] The server generates an AR menu based on the text data.

[0232] Input: Converted text data

[0233] Output: Augmented reality menu

[0234] Specific operation: The server uses an AR content creation tool such as Unity to generate an AR menu that supports sign language.

[0235] Step 6:

[0236] The server sends the generated AR menu to the device.

[0237] Input: Generated AR menu

[0238] Output: The AR menu is sent to the device.

[0239] Specific operation: The server sends the generated AR menu to the device.

[0240] Step 7:

[0241] The device will display the AR menu.

[0242] Input: AR menu received on the device

[0243] Output: Visual feedback that is displayed to the user

[0244] Specific behavior: The device displays an AR menu overlaid on the camera screen, providing visual feedback to the user.

[0245] Customer feedback collection and analysis system processing steps

[0246] Step 1:

[0247] The user enters feedback in text.

[0248] Input: User feedback text

[0249] Output: Feedback data is generated on the terminal

[0250] Specific operation: The user inputs feedback on the device screen.

[0251] Step 2:

[0252] The device captures the feedback.

[0253] Input: Feedback text

[0254] Output: Stored as digital feedback data

[0255] Specific behavior: The device captures and stores the input feedback data.

[0256] Step 3:

[0257] The terminal transmits the feedback data to the server.

[0258] Input: Digital feedback data

[0259] Output: Feedback data is sent to the server

[0260] Specific operation: The feedback data captured by the terminal is sent to the server via a communication means.

[0261] Step 4:

[0262] The server analyzes the feedback.

[0263] Input: Feedback data arriving at the server

[0264] Output: Analysis results (positive and negative evaluations)

[0265] Specific operation: The server analyzes the feedback data using a natural language processing engine (GPT-4 or BERT) and generates an evaluation.

[0266] Step 5:

[0267] The server displays the analysis results on a dashboard.

[0268] Input: Analysis results

[0269] Output: Ratings displayed on the dashboard

[0270] Specific operation: The server displays the generated evaluation results on a dashboard for employees to review.

[0271] Processing steps of an optimized menu suggestion system

[0272] Step 1:

[0273] The user initiates the order.

[0274] Input: The user's current order selection (e.g., "Veggie Burger")

[0275] Output: Order data is generated on the terminal.

[0276] Specific Actions: The user starts the ordering process at the terminal and selects an item.

[0277] Step 2:

[0278] The terminal sends past order history and current selections to the server.

[0279] Input: User's past order history and current selection data

[0280] Output: Order history data is sent to the server

[0281] Specific operation: The terminal sends the user's saved past order history and current selections to the server.

[0282] Step 3:

[0283] The server parses the order data.

[0284] Input: Order data arriving at the server

[0285] Output: Analysis results (menu suggestions)

[0286] Specific operation: The server analyzes the data using a database such as BigQuery and generates optimal menu suggestions.

[0287] Step 4:

[0288] The server generates the menu suggestions.

[0289] Input: Analysis results

[0290] Output: Generated menu suggestions (e.g., "Would you like today's special smoothie?")

[0291] Specific operation: The server forms optimal menu suggestions based on the analysis results.

[0292] Step 5:

[0293] The server sends the proposal to the device.

[0294] Input: Generated menu suggestions

[0295] Output: Menu suggestions sent to the device

[0296] Specific operation: The server sends the generated menu suggestions to the terminal.

[0297] Step 6:

[0298] The device will display suggestions.

[0299] Input: Menu suggestions received on the device

[0300] Output: Menu suggestions presented to the user

[0301] Specific operation: The terminal displays the menu suggestions sent from the server, and the user confirms and selects the additional order.

[0302] (Application example 1)

[0303] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] In the traditional restaurant industry, it is difficult to accommodate customers who speak foreign languages ​​or who are hearing impaired, and processing orders and feedback requires a great deal of time and effort. Furthermore, when customers want to order in real time using voice or sign language in multiple languages, the system that provides support is inadequate, making effective communication difficult. Furthermore, there is a lack of an efficient system for quickly and accurately processing customer feedback and taking appropriate improvement measures.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0306] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and a means for processing the user's voice order in real time using an input device of the smartphone, thereby enabling support for multiple languages ​​and sign language.

[0307] The system also includes a sign language analysis unit that converts sign language input by the user through the camera into text data using a video analysis engine, an augmented reality menu providing unit that reflects the text data in an augmented reality menu and provides visual feedback, and a unit that processes sign language orders in real time using the smartphone's camera, thereby enabling real-time sign language support for the hearing impaired.

[0308] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, and a means for obtaining feedback in real time through the smartphone application, thereby enabling feedback to be collected and analyzed quickly and accurately.

[0309] "Speech recognition means" refers to a device or software that has the function of converting voice input by a user into text data.

[0310] "Translation means" refers to a device or software that has the function of translating text data into other languages ​​using a multilingual translation module.

[0311] A "response generation means" is a device or software that has the function of analyzing the translated text data and generating an appropriate response.

[0312] The "voice synthesis means" is a device or software that has the function of converting the generated response into voice and providing it to the user.

[0313] "Smartphone input devices" is a general term for smartphone hardware and software that has the ability to receive input such as audio, video, and touch.

[0314] The "sign language analysis means" is a device or software that has the function of converting sign language input by a user through a camera into text data using an image analysis engine.

[0315] The "augmented reality menu providing means" is a device or software that has the function of visually displaying text data using augmented reality technology and providing feedback.

[0316] The "feedback collection means" is a device or software that has the function of capturing feedback from users as text data.

[0317] The "feedback analysis means" is a device or software that has the function of analyzing the captured feedback using natural language processing technology and generating positive and negative evaluations.

[0318] The "dashboard providing means" is a device or software that has the function of visually displaying the analysis results.

[0319] A "smartphone application" is application software that runs on a smartphone and provides various functions through a user interface.

[0320] The present invention provides a system for the restaurant industry that supports multiple languages ​​and sign language. This system allows customers to input orders using voice or sign language, and the system processes the orders on a server, which then responds in real time. Specific embodiments of the system are described in detail below.

[0321] Speech recognition and translation processing

[0322] When a user voice orders through an input device on a smartphone, the terminal captures the voice and converts it into text data using a speech recognition means. The text data is then sent to a server, where it is translated into another language by a translation means. A response generation means then analyzes the translated text data and generates an appropriate response. This response is then converted back into speech using a speech synthesis means and provided to the user through the terminal. Specifically, this can be achieved using the SpeechRecognition library, Googletrans library, and Pyttsx3 library.

[0323] Sign language analysis and augmented reality menus

[0324] When a user places an order using sign language through the smartphone camera, the sign language analysis means analyzes the image and converts it into text data. This data is reflected in the augmented reality menu providing means, and visual feedback is provided to the user. This allows the user to check the order details using sign language.

[0325] Collecting and analyzing feedback

[0326] When a user enters feedback through a smartphone application, the feedback is captured as text data by a feedback collection means and sent to a server. The feedback analysis means analyzes the captured data using a natural language processing engine to generate positive and negative evaluations. The analysis results are visually displayed through a dashboard provision means so that employees can check them.

[0327] For example, if a customer orders "I would like a cheeseburger and a coke, please" by voice, the system captures the speech and translates it into Japanese. The system then generates a response, "I'd like a cheeseburger and a coke, please," which is presented visually and audibly to the user. If a customer orders "hamburger and fries" by sign language, the camera captures the sign language and generates an augmented reality menu with images and detailed information about the hamburger and fries, which is presented visually to the user.

[0328] Also, if feedback such as "The service was very good, but the food was a little cold" is entered, positive and negative ratings will be generated and the results will be displayed on the dashboard.

[0329] An example of a prompt sentence to input to the generative AI model is as follows:

[0330] "Build a customer order processing system that utilizes voice recognition and multilingual translation. We need a program that can recognize and translate customer voice orders in real time, and respond with a voice response. The languages ​​used will be from English to Japanese."

[0331] Through these means, the present invention can provide real-time multilingual and sign language enabled order processing, improving customer experience in the restaurant industry.

[0332] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0333] Step 1:

[0334] The user places an order by voice through the smartphone's input device. The device captures this voice and converts it into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[0335] Step 2:

[0336] The terminal sends text data to the server, which translates the text data into another language using the Google Trans library. The input is the text data, and the output is the translated text data.

[0337] Step 3:

[0338] The server receives the translated text data and inputs it to a response generation means, which analyzes the text data and generates an appropriate response. The input is the translated text data and the output is the generated response text.

[0339] Step 4:

[0340] The server inputs the generated response text to the speech synthesis means and converts it into speech data using the Pyttsx3 library. The input is the generated response text, and the output is synthesized speech data.

[0341] Step 5:

[0342] The server sends the synthesized voice data to the terminal, and the terminal provides a voice response to the user. The input is the synthesized voice data, and the output is the voice playback.

[0343] Step 6:

[0344] The user inputs feedback through a smartphone application. The device captures this feedback and converts it into text data. The input is the voice or text of the feedback, and the output is text data.

[0345] Step 7:

[0346] The terminal transmits the feedback text data to the server, and the server analyzes the text data using the feedback analysis means to generate positive and negative evaluations. The input is the feedback text data, and the output is evaluation data.

[0347] Step 8:

[0348] The server inputs the analysis results to the dashboard providing means and generates data for visual display. The input is the evaluation data, and the output is the analysis results that are displayed visually.

[0349] Step 9:

[0350] The server provides employees with visually displayed analysis results and information for considering improvement measures. The input is the visual analysis results, and the output is information provided to employees.

[0351] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0352] The present invention provides a generative AI solution for restaurant staff and wait staff in the food service industry that combines multilingual support, sign language support, and the ability to recognize user emotions. The system for implementing the present invention consists of the following elements:

[0353] Multilingual interactive voice response system

[0354] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[0355] Adding emotion recognition functionality

[0356] The emotion engine installed on the server recognizes emotions from the user's voice and text data. The recognized emotion information is reflected in the response generation means, thereby generating a more empathetic response that is more suited to the user.

[0357] Examples:

[0358] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," if the emotion engine identifies joy in the customer's voice, the server will generate a response such as, "You'd like a cheeseburger and a coke! Enjoy!"

[0359] AR menu provision system that supports sign language

[0360] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[0361] Examples:

[0362] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[0363] Customer feedback collection and analysis system

[0364] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates a positive or negative rating. Furthermore, an emotion engine recognizes the emotion in the feedback and reflects it in the rating. The results are displayed on a dashboard for employees to check.

[0365] Examples:

[0366] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" or a "negative rating (the food was cold)." The emotion engine recognizes the customer's attitude and emotions in the feedback and reflects them in the analysis results. The analysis results are displayed on a dashboard, allowing employees to use them to make improvements.

[0367] Optimized menu suggestion system

[0368] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selection. The server analyzes the received data and uses an emotion engine to generate the optimal menu suggestion that reflects the customer's current mood, and the content is presented to the customer via the terminal. The customer can then confirm and select additional orders.

[0369] Examples:

[0370] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device will send a request to the server based on that. The server will generate a menu suggestion, "How about today's special smoothie?", and the emotion engine will take into account the customer's current emotional state (e.g., surprise) and emphasize the suggestion. The device will display the suggestion and provide a message saying, "Try our special smoothie," allowing the customer to easily place a reorder.

[0371] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[0372] The processing flow will be explained below.

[0373] Multilingual interactive voice response system

[0374] Processing flow from speech recognition to response generation

[0375] Step 1:

[0376] The user queries the order by voice.

[0377] Example: "I'd like a cheeseburger and a Coke, please."

[0378] Step 2:

[0379] The device captures the voice and converts it into text data using a voice recognition engine.

[0380] Speak: "I'd like a cheeseburger and a Coke, please."

[0381] Converted text: "I'd like a cheeseburger and a coke, please."

[0382] Step 3:

[0383] The text data received by the server is analyzed using an emotion engine to recognize the user's emotions.

[0384] Analysis result: Joy

[0385] Step 4:

[0386] The server sends the text data to the translation module and translates it into multiple languages.

[0387] Translation result: "I would like a cheeseburger and a coke, please" (English)

[0388] Step 5:

[0389] The server parses the translated text data and generates an appropriate response.

[0390] Generated response: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[0391] Step 6:

[0392] The server adjusts the response based on the user's emotional information.

[0393] Adjusted response: "Thank you for your order! You have ordered a cheeseburger and a coke. Is there anything else you would like?" (reflects the sentiment)

[0394] Step 7:

[0395] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[0396] Synthesized voice: "Thanks for your order! Cheeseburger and Coke, right? Is there anything else you'd like to order?"

[0397] AR menu provision system that supports sign language

[0398] Processing flow from sign language analysis to AR display

[0399] Step 1:

[0400] A user enters an order using sign language.

[0401] Example: Saying "hamburger and fries" in sign language

[0402] Step 2:

[0403] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[0404] Captured footage: Signing "hamburger and fries"

[0405] Converted text data: "Hamburger and fries"

[0406] Step 3:

[0407] The server receives the converted text data and generates an AR menu that supports sign language.

[0408] Generated: "Hamburger and fries"

[0409] Generated AR menu: Burger and fries images, prices, and details

[0410] Step 4:

[0411] The device displays the generated AR menu and provides visual feedback to the user.

[0412] Showing: Image and details of a burger and fries

[0413] Customer feedback collection and analysis system

[0414] Processing flow from feedback collection to analysis

[0415] Step 1:

[0416] The user enters feedback.

[0417] For example: "The service was great, but the food was a little cold."

[0418] Step 2:

[0419] The device captures the feedback as text data and sends it to the server.

[0420] Feedback captured: "The service was great, but the food was a little cold."

[0421] Step 3:

[0422] The server analyzes the feedback it receives using a natural language processing engine.

[0423] Analysis results: Positive reviews (good service), negative reviews (food was cold)

[0424] Step 4:

[0425] The server uses an emotion engine to recognize emotions in the feedback and reflects the results in the analysis.

[0426] Perceived emotions: "Satisfied", "Dissatisfied"

[0427] Final analysis results: positive evaluation, negative evaluation, emotion recognition results

[0428] Step 5:

[0429] The server displays the analysis results on a dashboard.

[0430] Shows: Positive and negative review details, feedback, and perceived sentiment

[0431] Step 6:

[0432] Employees check the dashboard and consider and implement countermeasures.

[0433] Action example: Review food temperature management and further improve service

[0434] Optimized menu suggestion system

[0435] Suggested flows based on order history and current selections

[0436] Step 1:

[0437] The user initiates the order.

[0438] Example: Select "Veggie Burger"

[0439] Step 2:

[0440] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[0441] Data sent: Current order "Veggie Burger", past order history

[0442] Step 3:

[0443] The server analyzes the received data and uses an emotion engine to recognize the current emotional state of the user.

[0444] Recognized emotion: "surprise"

[0445] Step 4:

[0446] The server takes into account the customer's past history and emotional state to generate optimal menu suggestions.

[0447] Suggestion: "How about today's special smoothie?" (reflects emotions)

[0448] Step 5:

[0449] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[0450] Display content: Today's limited smoothie selection options and details

[0451] Step 6:

[0452] The user confirms the proposal and places an additional order.

[0453] Example behavior: Ordering an extra smoothie

[0454] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[0455] Example 2

[0456] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] The current restaurant industry lacks sufficient multilingual and sign language support, and does not provide empathetic responses based on customer feelings, which can lead to a decline in customer satisfaction. It is particularly difficult to provide efficient and accurate service to customers who do not speak a foreign language or who require sign language. There is also a lack of systems in place to effectively utilize customer feedback to improve operations.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0459] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, an emotion recognition means for recognizing emotions from the user's speech and text data, and an empathetic response generation means for generating an empathetic response based on the emotion information. This makes it possible to support multiple languages ​​and sign language, and further to provide empathetic responses based on the customer's emotions, thereby improving customer satisfaction.

[0460] The "voice recognition means" is a means for capturing voice input by a user as digital data and converting the voice into text data.

[0461] The "translation means" is a means for translating the text data generated by the speech recognition means into another language.

[0462] The "response generation means" is a means for analyzing the translated text data and generating an appropriate response.

[0463] The "voice synthesis means" is a means for converting the response generated by the response generation means into voice data and providing it to the user.

[0464] The "emotion recognition means" is a means for recognizing emotions from the user's voice and text data.

[0465] The "empathic response generating means" is a means for generating an empathic response based on emotional information.

[0466] The "sign language analysis means" is a means for capturing sign language input by a user through a camera as an image and converting it into text data.

[0467] The "augmented reality menu providing means" is a means for reflecting text data in an augmented reality menu and providing visual feedback.

[0468] The "feedback collection means" is a means for capturing feedback from users as text data.

[0469] The "feedback analysis means" is a means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations.

[0470] The "dashboard providing means" is a means for visually displaying the evaluation results generated by the feedback analysis means and providing them to employees.

[0471] This invention is a system that provides generative AI solutions for the restaurant industry that integrate multilingual support, sign language support, and emotion recognition functions. This system consists of a server and a terminal, and interacts directly with users.

[0472] Multilingual interactive voice response system

[0473] When a user (customer) enters an order by voice, the device captures the voice with its microphone. The device then sends this voice data to a speech recognition engine such as Google Cloud Speech-to-Text, which converts the speech into text. The converted text data is then sent to a server where it is translated using a multilingual translation module such as Google Cloud Translation. The server then analyzes the translated text data and generates an appropriate response. The generated response is then converted into speech by a speech synthesis engine such as Amazon Polly and played back to the user from the device.

[0474] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," the device captures the voice and sends it to the server, which translates and generates a response, returning it to the user as "You would like a cheeseburger and a coke." This process enables multilingual support for customers.

[0475] Adding emotion recognition functionality

[0476] The server is equipped with an emotion recognition engine (e.g., IBM Watson Tone Analyzer) that analyzes voice and text data to recognize the user's emotions. This emotional information is reflected in the response generation means, which generates an empathetic response.

[0477] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," and the emotion recognition engine detects "happiness," the server will generate an empathetic response such as, "You ordered a cheeseburger and a coke! Enjoy!" and play it back as audio.

[0478] Example prompt sentence:

[0479] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[0480] AR menu provision system that supports sign language

[0481] When a sign language user enters an order through the camera, the device captures the image and sends it to a sign language analysis engine such as Microsoft Kinect to convert it into text data. The server receives the converted text data and generates an augmented reality (AR) menu, which the device displays.

[0482] For example, when a user signs for "hamburger and fries," the device's camera captures the image, and the server converts it into text to generate an AR menu. The menu includes images of the hamburger and fries, along with detailed information, and the device provides visual feedback to the user.

[0483] Customer feedback collection and analysis system

[0484] When a user enters feedback into the device, it is captured as text data and sent to a server. The server then analyzes the feedback data using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative ratings. An emotion recognition engine then analyzes the emotional information in the feedback and displays the results on a dashboard.

[0485] For example, if a user enters feedback such as "The service was good, but the food was a little cold," the server analyzes it and generates a "positive rating (good in terms of service)" and a "negative rating (the food was cold)," which are then displayed on a dashboard where employees can view them.

[0486] Optimized menu suggestion system

[0487] When a user begins an order, the terminal sends a menu suggestion request to the server based on past order history and current selections. The server then analyzes the data using a personalization engine such as Amazon Personalize to generate optimal menu suggestions that take customer sentiment into account. These suggestions are then presented to the user by the terminal, allowing them to easily place additional orders.

[0488] For example, if a user has frequently ordered a "veggie burger" in the past, when they select it again, the server will generate a menu suggestion such as "How about today's special smoothie?", and the emotion engine will highlight the suggestion taking into account the user's current emotion. The device will then display the suggestion, allowing the user to easily place an additional order.

[0489] With these functions, this system will improve the efficiency of customer service and customer satisfaction at restaurants and wait staff in the food service industry. In addition, by combining it with an emotion engine, it will be possible to provide even more personalized services.

[0490] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0491] Processing flow of a multilingual interactive voice response system

[0492] Step 1: Capture audio

[0493] The device captures the user's voice with a microphone. The input is the user's voice, and the output is the captured voice data. This voice data is temporarily stored in the device's storage.

[0494] Specific behavior:

[0495] When a user speaks something like "I would like a cheeseburger and a coke, please," the device's microphone captures and stores the speech as digital data.

[0496] Step 2: Speech to text

[0497] The device sends the captured voice data to the Google Cloud Speech-to-Text API. The input is the captured voice data and the output is text data. The Google Cloud Speech-to-Text API converts the voice data into text data.

[0498] Specific behavior:

[0499] The device sends the voice data to Google Cloud Speech-to-Text, which returns the text, "I would like a cheeseburger and a coke, please."

[0500] Step 3: Translate the text

[0501] The server sends the text data received from the device to the Google Cloud Translation API. The input is text data, and the output is translated text data. The Google Cloud Translation API translates the text data into the specified other language.

[0502] Specific behavior:

[0503] The server sends the text "I would like a cheeseburger and a coke, please" to the Google Cloud Translation API and receives the translated text "I would like a cheeseburger and a coke, please."

[0504] Step 4: Generate a response

[0505] The server analyzes the translated text data and generates an appropriate response. The input is the translated text data and the output is the response text data. The server uses an algorithm to generate the response text.

[0506] Specific behavior:

[0507] Based on the translation result of "Please give me a cheeseburger and a coke", the server generates a response text such as "You've ordered a cheeseburger and a coke."

[0508] Step 5: Synthesize and play the audio

[0509] The server sends the response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data. The input is the response text, and the output is voice data. The generated voice data is sent to the device and played back.

[0510] Specific behavior:

[0511] The server sends the text "You'd like a cheeseburger and a coke" to Amazon Polly, which generates a voice file. The device plays the voice and responds to the user.

[0512] Process flow for adding emotion recognition functionality

[0513] Step 1: Acquiring audio and text

[0514] The server receives the user's voice and text data, which are used for further processing.

[0515] Specific behavior:

[0516] When a user places an order by saying, "I would like a cheeseburger and a coke, please," the voice and text data are sent to the server.

[0517] Step 2: Sentiment Analysis

[0518] The server sends the voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to obtain emotion information. The input is the voice data, and the output is the emotion information.

[0519] Specific behavior:

[0520] The server sends the voice data to the IBM Watson Tone Analyzer, which returns emotional information such as "joy."

[0521] Step 3: Generating an empathetic response

[0522] The server generates an empathetic response based on the emotional information and sends it to a speech synthesis engine. The input is the emotional information and the response text, and the output is voice data as the empathetic response.

[0523] Specific behavior:

[0524] If the emotion engine identifies "joy," the server generates an empathetic response text such as "You ordered a cheeseburger and a Coke! Enjoy!" and converts it into speech to play.

[0525] Example prompt sentence:

[0526] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[0527] Processing flow of the sign language compatible AR menu provision system

[0528] Step 1: Capture sign language video

[0529] The device captures the user's sign language with a camera. The input is video data of the sign language, and the output is the captured video data.

[0530] Specific behavior:

[0531] When the user signs for "hamburger and fries," the device's camera captures the movement.

[0532] Step 2: Translating sign language to text

[0533] The device sends the captured video data to a sign language analysis engine (e.g., Microsoft Kinect) and converts it into text data. The input is video data, and the output is text data.

[0534] Specific behavior:

[0535] The device sends sign language video to a Microsoft Kinect, which responds with the text "hamburger and fries."

[0536] Step 3: Generate the AR menu

[0537] The server generates an AR menu based on the text data. The input is the text data, and the output is the generated AR menu.

[0538] Specific behavior:

[0539] The server receives the text data for "hamburger and fries" and generates a corresponding AR menu.

[0540] Step 4: Display the AR menu

[0541] The device displays the generated AR menu to the user. The input is the AR menu, and the output is the displayed AR menu.

[0542] Specific behavior:

[0543] The device displays an AR menu, providing the user with images and details about the burger and fries.

[0544] Customer feedback collection and analysis system process flow

[0545] Step 1: Enter your feedback

[0546] The user inputs feedback and the device captures it. The input is the feedback content and the output is text data.

[0547] Specific behavior:

[0548] The user enters feedback saying, "The service was good, but the food was cold."

[0549] Step 2: Submit your feedback

[0550] The terminal sends the captured feedback data to the server. The input is text data, and the output is the feedback data transferred to the server.

[0551] Specific behavior:

[0552] The terminal sends the feedback text to the server.

[0553] Step 3: Analyze the feedback

[0554] The server analyzes the received feedback using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative evaluations. The input is the feedback text, and the output is the evaluation data as the analysis result.

[0555] Specific behavior:

[0556] The server analyzes the feedback text "The service was good, but the food was cold" and generates a positive rating (the service was good) and a negative rating (the food was cold).

[0557] Step 4: View the evaluation results

[0558] The server displays the analysis results on a dashboard. The input is the evaluation data, and the output is the displayed dashboard.

[0559] Specific behavior:

[0560] The server displays the evaluation results on a dashboard, where employees can view them.

[0561] Optimized menu suggestion system processing flow

[0562] Step 1: Initiate your order

[0563] When a user initiates an order, the terminal sends the user's past order history and current selections to the server. The input is the order data, and the output is the request data to the server.

[0564] Specific behavior:

[0565] When the user selects "veggie burger," the terminal sends the data along with past order history to the server.

[0566] Step 2: Generate menu suggestions

[0567] The server uses a personalization engine (e.g., Amazon Personalize) to analyze the data and generate optimal menu suggestions. The input is the request data, and the output is the suggested menu.

[0568] Specific behavior:

[0569] The server analyzes the user's past order history and current selections and generates a menu suggestion such as, "How about today's special smoothie?"

[0570] Step 3: Viewing Proposals

[0571] The terminal displays the generated menu suggestions to the user. The input is the suggested menu and the output is the suggestion displayed on the user's screen.

[0572] Specific behavior:

[0573] The device will display a suggestion to the user, "How about today's special smoothie?", encouraging them to order more.

[0574] The above is the specific processing flow of each system, which will enable the restaurant industry to improve customer service efficiency and customer satisfaction.

[0575] (Application example 2)

[0576] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0577] In today's brick-and-mortar stores, multilingual support is required to accommodate the increasing number of foreign tourists and diversifying customer needs, but language barriers hinder the efficiency of customer service and improve customer satisfaction. Dealing with customers who use sign language and reading their emotions to respond appropriately are also challenges. A system is needed to resolve these issues and improve the customer experience.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0579] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation. This not only enables multilingual communication across language barriers, but also makes it possible to provide personalized responses according to sign language and emotions.

[0580] A "voice recognition means" is a device or system that converts voice input by a user into text data.

[0581] A "multilingual translation module" is a device or software for translating text data into different languages.

[0582] "Translation means" is a function that converts text data into other languages ​​using a multilingual translation module.

[0583] The "response generation means" is a function that analyzes the translated text data and generates an appropriate response.

[0584] A "voice synthesis means" is a device or software that converts the generated response into speech and provides it to the user.

[0585] The "emotion recognition means" is a function that recognizes emotions from the user's voice and reflects the emotion data in response generation.

[0586] The "sign language analysis means" is a function that converts sign language input by the user through the camera into text data using an image analysis engine.

[0587] The "augmented reality menu providing means" is a function that reflects the text data obtained by the sign language analysis means in an augmented reality menu and provides visual feedback.

[0588] A "feedback collection means" is a device or software that captures feedback from users as text data.

[0589] The "feedback analysis means" is a function that analyzes the captured feedback using a natural language processing engine and generates positive and negative evaluations.

[0590] The "dashboard provision means" is a function that displays the results of feedback analysis on a dashboard so that employees can check them.

[0591] The system programs required to implement the present invention will now be described.

[0592] The server includes: a speech recognition means for converting speech input by the user into text data; a multilingual translation module for translating the text data into other languages; a response generation means for analyzing the translated text data and generating an appropriate response; a speech synthesis means for converting the generated response into speech and providing it to the user; an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation; a sign language analysis means for converting sign language input by the user through a camera into text data using a video analysis engine; an augmented reality menu providing means for reflecting the text data obtained by the sign language analysis means in an augmented reality menu and providing visual feedback; a feedback collection means for capturing feedback from the user as text data; a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations; and a dashboard providing means for displaying the results of the feedback analysis on a dashboard so that employees can check it.

[0593] The speech recognition means uses, for example, the speech_recognition library. This library captures the speech input by the user and converts it into text data. The converted text data is sent to a multilingual translation module (for example, Google Trans New API) and translated into another language. The translated text data is analyzed by the server, and an appropriate response is generated. This response is converted into audio data using a speech synthesis means (for example, pyttsx3) and provided to the user by the terminal.

[0594] The emotion recognition means uses the EmotionRecognition model. When the user's voice is captured, the emotion recognition means analyzes the voice data and detects emotions. The detected emotional data is reflected in the responses generated by the response generation means, resulting in more personalized responses.

[0595] The sign language analysis means uses the SignLanguageRecognition model. When a user inputs sign language through the camera, this model analyzes the video data and converts it into corresponding text data. The converted text data is sent to the augmented reality menu providing means, which provides visual feedback. This allows users who use sign language to order smoothly.

[0596] The feedback collection means includes a function for capturing text data. When feedback is entered by a user, it is captured as text data. The feedback analysis means then analyzes the feedback using a natural language processing engine (such as spaCy), and positive or negative evaluations are generated. These evaluation results are displayed in a form that can be checked by employees using the dashboard provision means. The dashboard is implemented using tools such as dash or Tableau.

[0597] Specific examples

[0598] Suppose a customer places an order by saying, "I would like a cheeseburger and a coke, please." At this time, the speech recognition means converts the speech into text, which is then translated by the multilingual translation module. If the emotion recognition means identifies joy from the customer's speech, the response generation means generates a response such as, "You'd like a cheeseburger and a coke! Enjoy!" The speech synthesis means converts this response into speech and provides it to the customer from the terminal.

[0599] Prompt Sentence Examples

[0600] Input prompt:

[0601] User: "I would like a cheeseburger and a coke, please"

[0602] System: "You ordered a cheeseburger and a Coke. Is there anything else I can help you with?"

[0603] Response prompt:

[0604] User: "No, that's all for now."

[0605] System: "Thank you. Your order has been placed. Please wait a moment."

[0606] As described above, this system not only enables multilingual communication across language barriers, but also provides personalized responses based on emotions and sign language support, which is expected to improve customer satisfaction in brick-and-mortar stores.

[0607] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0608] Step 1:

[0609] The user enters the order by voice.

[0610] Input: Speech data (e.g., "I would like a cheeseburger and a coke, please")

[0611] What happens: The device's microphone captures audio data.

[0612] Output: Captured audio data

[0613] Step 2:

[0614] A speech recognition means converts the captured speech data into text data.

[0615] Input: Audio data

[0616] Data processing: A speech recognition engine (e.g., speech_recognition library) analyzes the voice data and converts it into text data.

[0617] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[0618] Step 3:

[0619] A multilingual translation module translates the text data into other languages.

[0620] Input: Text data

[0621] Data calculation: A translation engine (e.g., Google Trans New API) translates the input text data into the specified language.

[0622] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[0623] Step 4:

[0624] An emotion recognition means recognizes emotions from the translated text data and voice data.

[0625] Input: Translated text data, audio data

[0626] Data computation: An emotion recognition engine (e.g., EmotionRecognition model) analyzes the speech and text and generates emotion data.

[0627] Output: Emotion data (e.g., "joy")

[0628] Step 5:

[0629] A response generation means generates an appropriate response based on the translated text data and the emotion data.

[0630] Input: Translated text data, emotion data

[0631] Data computation: A response generation engine generates empathetic responses based on input data.

[0632] Output: Response text data (e.g., "You ordered a cheeseburger and a Coke! Enjoy!")

[0633] Step 6:

[0634] A speech synthesis means converts the response text data into speech.

[0635] Input: Response text data

[0636] Data processing: A speech synthesis engine (e.g., pyttsx3) converts text data into speech data.

[0637] Output: Response audio data

[0638] Step 7:

[0639] The terminal provides the response voice data to the user.

[0640] Input: Response audio data

[0641] Action: The device speaker plays the audio data.

[0642] Output: Speech played to the user (e.g., "You've ordered a cheeseburger and a Coke! Enjoy!")

[0643] Step 8:

[0644] When a user inputs an order in sign language through the camera, the sign language analysis means captures the video data.

[0645] Input: Video data (e.g., sign language for "cheeseburger and coke")

[0646] Action: The device's camera captures video data.

[0647] Output: Captured video data

[0648] Step 9:

[0649] A sign language analysis means converts the captured video data into text data.

[0650] Input: Video data

[0651] Data processing: A sign language analysis engine (e.g., SignLanguageRecognition model) analyzes the video data and converts it into text.

[0652] Output: Text data corresponding to the sign language (e.g., "Cheeseburger and Coke")

[0653] Step 10:

[0654] The augmented reality menu providing means generates an augmented reality menu based on the converted text data and provides visual feedback.

[0655] Input: Text data obtained from sign language analysis

[0656] Data processing: The augmented reality engine generates a menu based on the text data and displays it on the screen.

[0657] Output: Augmented reality menu displayed to the user (e.g., image of a cheeseburger and a coke with details)

[0658] Step 11:

[0659] Feedback from users is captured as text data by a feedback collection means.

[0660] Input: Text data (e.g., "The service was very good, but the food was a little cold.")

[0661] Operation: Feedback data is sent from the device to the server.

[0662] Output: Captured feedback text data

[0663] Step 12:

[0664] A feedback analysis means analyzes the captured feedback using a natural language processing engine to generate positive and negative evaluations.

[0665] Input: Feedback text data

[0666] Data calculation: A natural language processing engine (e.g., spaCy) analyzes the feedback data and generates positive and negative ratings.

[0667] Output: Rating data (e.g., "The service was good" or "The food was cold")

[0668] Step 13:

[0669] The dashboard provider displays the analysis results on a dashboard, which helps employees make improvements.

[0670] Input: Parsed evaluation data

[0671] How it works: A dashboard system (e.g., dash or Tableau) visually displays the assessment data.

[0672] Output: The dashboard to be displayed (e.g., positive and negative feedback ratings)

[0673] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0674] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0675] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0676] [Second embodiment]

[0677] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0678] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0679] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0680] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0681] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0682] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0683] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0684] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0685] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0686] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0687] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0688] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0689] The present invention provides a generative AI solution that enables multilingual and sign language support for restaurants and wait staff in the food service industry. The system for implementing the present invention consists of the following elements:

[0690] Multilingual interactive voice response system

[0691] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[0692] Examples:

[0693] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[0694] AR menu provision system that supports sign language

[0695] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[0696] Examples:

[0697] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[0698] Customer feedback collection and analysis system

[0699] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates positive and negative ratings. The results are displayed on a dashboard for employees to review.

[0700] Examples:

[0701] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[0702] Optimized menu suggestion system

[0703] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes the received data and generates optimal menu suggestions, which are presented to the customer by the terminal. The customer can then confirm and select additional orders.

[0704] Examples:

[0705] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[0706] The above is an embodiment of the present invention. This system can improve the efficiency of customer service operations in the restaurant industry and effectively solve the problem of labor shortages through functions such as multilingual support, sign language support, feedback collection and analysis, and optimized menu suggestions.

[0707] The processing flow will be explained below.

[0708] Multilingual interactive voice response system

[0709] Step 1:

[0710] The user queries the order by voice.

[0711] Examples: "Cheeseburger", "Coke"

[0712] Step 2:

[0713] The device captures the voice and converts it into text data using a voice recognition engine.

[0714] Speak: "I'd like a cheeseburger and a Coke, please."

[0715] Converted text: "I'd like a cheeseburger and a coke, please."

[0716] Step 3:

[0717] The server sends the received text data to a translation module to translate it into the customer's language.

[0718] Data sent: "I'd like a cheeseburger and a coke, please."

[0719] Translated text: "I would like a cheeseburger and a coke, please" (English)

[0720] Step 4:

[0721] The server parses the translated text data and generates an appropriate response.

[0722] Parse: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[0723] Step 5:

[0724] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[0725] Synthesized voice: "You've ordered a cheeseburger and a Coke. Is there anything else you'd like to order?"

[0726] AR menu provision system that supports sign language

[0727] Step 1:

[0728] A user enters an order using sign language.

[0729] Example: Say "hamburger" and "fries" in sign language.

[0730] Step 2:

[0731] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[0732] Captured footage: Sign language for "hamburger" and "fries"

[0733] Converted text data: "Hamburger and fries"

[0734] Step 3:

[0735] The server receives the converted text data and generates an AR menu that supports sign language.

[0736] Generated: "Hamburger and fries"

[0737] Generated AR menu: Burger and fries images, prices, and details

[0738] Step 4:

[0739] The device displays the generated AR menu and provides visual feedback to the user.

[0740] Showing: Image and details of a burger and fries

[0741] Customer feedback collection and analysis system

[0742] Step 1:

[0743] The user enters feedback.

[0744] For example: "The service was good, but the food was cold."

[0745] Step 2:

[0746] The device captures the feedback as text data and sends it to the server.

[0747] Feedback captured: "The service was good, but the food was cold."

[0748] Step 3:

[0749] The server analyzes the feedback it receives using a natural language processing engine.

[0750] Analysis results: Positive reviews (good service), negative reviews (cold food)

[0751] Step 4:

[0752] The server displays the analysis results on a dashboard.

[0753] Display: Positive and negative reviews, feedback details

[0754] Step 5:

[0755] The user (employee) checks the dashboard and considers and implements countermeasures.

[0756] Action example: Review food temperature management and further improve service

[0757] Optimized menu suggestion system

[0758] Step 1:

[0759] The user initiates the order.

[0760] Example: Select "Veggie Burger."

[0761] Step 2:

[0762] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[0763] Data sent: Current order "Veggie Burger", past order history

[0764] Step 3:

[0765] The server analyzes the received data and generates optimal menu suggestions.

[0766] Suggestion: "Would you like to try today's special smoothie?"

[0767] Step 4:

[0768] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[0769] Display content: Today's limited smoothie selection options and details

[0770] Step 5:

[0771] The user confirms the proposal and places an additional order.

[0772] Example behavior: Ordering an extra smoothie.

[0773] The above is a specific processing flow of each system in the embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry.

[0774] Example 1

[0775] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] In today's restaurant industry, serving customers with language barriers or communication disabilities is a major issue that leads to labor shortages and reduced operational efficiency. In particular, when multilingual and sign language support is required, the limitations of employee skills and resources become apparent. There is also a need for systems that can instantly collect and analyze customer feedback and develop appropriate countermeasures. Furthermore, it is important to be able to suggest optimal menu items based on customers' past ordering history, as this will lead to improved customer satisfaction and increased sales.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0778] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, a communication means for transmitting user speech data captured by the terminal to the server, and a playback means for playing back the generated speech response from the terminal, thereby enabling multilingual automated voice responses that overcome language barriers.

[0779] Furthermore, a system is provided that includes a sign language analysis means that converts sign language input by a user through a camera into text data using a video analysis engine, an augmented reality menu providing means that reflects the text data in an augmented reality menu and provides visual feedback, a communication means that transmits video data captured by the terminal to a server, and a display means that displays the generated augmented reality menu on the terminal, which enables smooth communication with customers who use sign language and improves the quality of service.

[0780] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback with a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, a communication means for transmitting the feedback data captured by the terminal to a server, and a confirmation means for employees to consider and implement countermeasures based on the analysis results. This makes it possible to quickly collect and analyze customer feedback and develop appropriate countermeasures.

[0781] "Speech recognition means" refers to a function for converting speech input by a user into text data.

[0782] "Translation means" refers to a function for translating text data into other languages ​​using a multilingual translation module.

[0783] "Response generation means" refers to a function for analyzing translated text data and generating an appropriate response.

[0784] "Speech synthesis means" refers to a function for converting the generated response into speech and providing it to the user.

[0785] "Communication means" refers to a function for transmitting user voice data and video data captured by the terminal to the server, and for transmitting responses and menus from the server to the terminal.

[0786] The "playback means" refers to a function for playing back the generated voice response from the terminal.

[0787] "Sign language analysis means" refers to the function for converting sign language input through a camera into text data using an image analysis engine.

[0788] The "augmented reality menu providing means" refers to a function for reflecting the converted text data in an augmented reality menu and providing visual feedback.

[0789] "Display means" refers to a function for displaying the generated augmented reality menu on the terminal.

[0790] "Feedback collection means" refers to a function for capturing feedback from users as text data.

[0791] "Feedback analysis means" refers to a function for analyzing captured feedback using a natural language processing engine and generating positive and negative evaluations.

[0792] "Dashboard provision means" refers to a function for displaying analysis results on a dashboard so that employees can check them.

[0793] "Verification means" refers to the function that allows employees to consider and implement countermeasures based on the analysis results.

[0794] Multilingual interactive voice response system

[0795] This invention is a system that automatically converts orders entered by voice into text, translates them in multiple languages, and generates voice responses. When a user enters an order by voice, the device captures the speech and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is sent to a server, which translates it into the user's native language using a multilingual translation module such as Google Cloud Translation. The server analyzes the translated text data and generates an appropriate response using a generative AI model such as GPT-4. The generated response is converted into speech using a speech synthesis engine such as Amazon Polly, and the response is finally played back by the device.

[0796] Examples:

[0797] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[0798] AR menu provision system that supports sign language

[0799] When a sign language user places an order through the camera, the device captures the video and converts it into text data using a sign language analysis engine such as Google Cloud Video Intelligence. This text data is sent to a server, which then uses an augmented reality (AR) content creation tool such as Unity to generate an AR menu that supports sign language. The generated AR menu is then displayed on the device, providing visual feedback to the user.

[0800] Examples:

[0801] When a customer orders a "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[0802] Customer feedback collection and analysis system

[0803] When a user enters feedback as text, the device captures the feedback and sends it to a server. The server receives the feedback data and analyzes it using a natural language processing engine such as GPT-4 or BERT. The analysis results in positive and negative ratings, which are displayed on a dashboard. Employees can view customer feedback through the dashboard and consider and implement appropriate measures.

[0804] Examples:

[0805] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[0806] Optimized menu suggestion system

[0807] When a user begins an order, the device sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes this data in a database such as BigQuery, generates optimal menu suggestions, and sends them to the device. The device then presents these suggestions to the user, who can then confirm and select additional orders.

[0808] Examples:

[0809] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[0810] Prompt Sentence Examples

[0811] "Please provide a detailed description of the voice commands or sign language inputs used by the user to place an order and the actions taken based on those inputs. Also, please provide a detailed description of feedback inputs and optimized menu suggestions."

[0812] The above is an embodiment of the present invention, which makes it possible to streamline customer service operations in the restaurant industry and effectively solve the problem of labor shortages through multilingual support, sign language support, feedback collection and analysis, and optimized menu proposals.

[0813] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0814] Multilingual Interactive Voice Response System Processing Steps

[0815] Step 1:

[0816] The user enters the order by voice.

[0817] Input: User speech (e.g., "I would like a cheeseburger and a coke, please")

[0818] Output: Audio data is generated on the device

[0819] Specific operation: The user speaks an order into a terminal with a microphone.

[0820] Step 2:

[0821] The device captures the audio.

[0822] Input: User's voice data

[0823] Output: Stored as digital audio data

[0824] Specific operation: The device's microphone picks up sound and records it as digital audio data.

[0825] Step 3:

[0826] The terminal transmits the audio data to the server.

[0827] Input: Digital audio data

[0828] Output: Audio data is sent to the server

[0829] Specific operation: The device sends the captured audio data to the server using a communication means.

[0830] Step 4:

[0831] The server converts the speech to text.

[0832] Input: Audio data arriving at the server

[0833] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[0834] What happens: The server converts the audio data into text using the Google Cloud Speech-to-Text API.

[0835] Step 5:

[0836] The server translates the text data.

[0837] Input: Converted text data

[0838] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[0839] Specific operation: The server uses the Google Cloud Translation API to translate the text into the user's native language.

[0840] Step 6:

[0841] The server parses the translated text data and generates a response.

[0842] Input: Translated text data

[0843] Output: Response text (e.g., "You'd like a cheeseburger and a coke").

[0844] What it does: The server uses the GPT-4 generative AI model to analyze the translated text and generate an appropriate response.

[0845] Step 7:

[0846] The server converts the response into speech.

[0847] Input: Response text

[0848] Output: Speech data (e.g., "You'd like a cheeseburger and a coke, right?")

[0849] Specific operation: The server uses a speech synthesis engine such as Amazon Polly to convert the response text into audio data.

[0850] Step 8:

[0851] The server sends the audio data to the terminal.

[0852] Input: Generated audio data

[0853] Output: Audio data is sent to the device

[0854] Specific operation: The server sends the generated voice data to the terminal.

[0855] Step 9:

[0856] The device plays a voice response.

[0857] Input: Audio data received on the device

[0858] Output: The audio played to the user

[0859] Specific operation: The device plays the received audio data through the speaker and prompts the user to confirm.

[0860] Processing steps of an AR menu provision system that supports sign language

[0861] Step 1:

[0862] A user who uses sign language enters an order through the camera.

[0863] Input: User's sign language actions

[0864] Output: Video data is generated by the camera

[0865] Specific actions: The user indicates an order in sign language in front of the camera.

[0866] Step 2:

[0867] The device captures the sign language video.

[0868] Input: Video data of the user's sign language movements

[0869] Output: Saved as digital video data

[0870] Specific operation: The device's camera captures sign language movements and records them as digital video data.

[0871] Step 3:

[0872] The terminal transmits the video data to the server.

[0873] Input: Digital video data

[0874] Output: Video data is sent to the server

[0875] Specific operation: The video data captured by the device is sent to the server using a communication means.

[0876] Step 4:

[0877] The server converts the video data into text.

[0878] Input: Video data arriving at the server

[0879] Output: Text data (e.g. "hamburger and fries")

[0880] How it works: The server uses the Google Cloud Video Intelligence API to analyze the sign language video data and convert it into text.

[0881] Step 5:

[0882] The server generates an AR menu based on the text data.

[0883] Input: Converted text data

[0884] Output: Augmented reality menu

[0885] Specific operation: The server uses an AR content creation tool such as Unity to generate an AR menu that supports sign language.

[0886] Step 6:

[0887] The server sends the generated AR menu to the device.

[0888] Input: Generated AR menu

[0889] Output: The AR menu is sent to the device.

[0890] Specific operation: The server sends the generated AR menu to the device.

[0891] Step 7:

[0892] The device will display the AR menu.

[0893] Input: AR menu received on the device

[0894] Output: Visual feedback that is displayed to the user

[0895] Specific behavior: The device displays an AR menu overlaid on the camera screen, providing visual feedback to the user.

[0896] Customer feedback collection and analysis system processing steps

[0897] Step 1:

[0898] The user enters feedback in text.

[0899] Input: User feedback text

[0900] Output: Feedback data is generated on the terminal

[0901] Specific operation: The user inputs feedback on the device screen.

[0902] Step 2:

[0903] The device captures the feedback.

[0904] Input: Feedback text

[0905] Output: Stored as digital feedback data

[0906] Specific behavior: The device captures and stores the input feedback data.

[0907] Step 3:

[0908] The terminal transmits the feedback data to the server.

[0909] Input: Digital feedback data

[0910] Output: Feedback data is sent to the server

[0911] Specific operation: The feedback data captured by the terminal is sent to the server via a communication means.

[0912] Step 4:

[0913] The server analyzes the feedback.

[0914] Input: Feedback data arriving at the server

[0915] Output: Analysis results (positive and negative evaluations)

[0916] Specific operation: The server analyzes the feedback data using a natural language processing engine (GPT-4 or BERT) and generates an evaluation.

[0917] Step 5:

[0918] The server displays the analysis results on a dashboard.

[0919] Input: Analysis results

[0920] Output: Ratings displayed on the dashboard

[0921] Specific operation: The server displays the generated evaluation results on a dashboard for employees to review.

[0922] Processing steps of an optimized menu suggestion system

[0923] Step 1:

[0924] The user initiates the order.

[0925] Input: The user's current order selection (e.g., "Veggie Burger")

[0926] Output: Order data is generated on the terminal.

[0927] Specific Actions: The user starts the ordering process at the terminal and selects an item.

[0928] Step 2:

[0929] The terminal sends past order history and current selections to the server.

[0930] Input: User's past order history and current selection data

[0931] Output: Order history data is sent to the server

[0932] Specific operation: The terminal sends the user's saved past order history and current selections to the server.

[0933] Step 3:

[0934] The server parses the order data.

[0935] Input: Order data arriving at the server

[0936] Output: Analysis results (menu suggestions)

[0937] Specific operation: The server analyzes the data using a database such as BigQuery and generates optimal menu suggestions.

[0938] Step 4:

[0939] The server generates the menu suggestions.

[0940] Input: Analysis results

[0941] Output: Generated menu suggestions (e.g., "Would you like today's special smoothie?")

[0942] Specific operation: The server forms optimal menu suggestions based on the analysis results.

[0943] Step 5:

[0944] The server sends the proposal to the device.

[0945] Input: Generated menu suggestions

[0946] Output: Menu suggestions sent to the device

[0947] Specific operation: The server sends the generated menu suggestions to the terminal.

[0948] Step 6:

[0949] The device will display suggestions.

[0950] Input: Menu suggestions received on the device

[0951] Output: Menu suggestions presented to the user

[0952] Specific operation: The terminal displays the menu suggestions sent from the server, and the user confirms and selects the additional order.

[0953] (Application example 1)

[0954] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] In the traditional restaurant industry, it is difficult to accommodate customers who speak foreign languages ​​or who are hearing impaired, and processing orders and feedback requires a great deal of time and effort. Furthermore, when customers want to order in real time using voice or sign language in multiple languages, the system that provides support is inadequate, making effective communication difficult. Furthermore, there is a lack of an efficient system for quickly and accurately processing customer feedback and taking appropriate improvement measures.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0957] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and a means for processing the user's voice order in real time using an input device of the smartphone, thereby enabling support for multiple languages ​​and sign language.

[0958] The system also includes a sign language analysis unit that converts sign language input by the user through the camera into text data using a video analysis engine, an augmented reality menu providing unit that reflects the text data in an augmented reality menu and provides visual feedback, and a unit that processes sign language orders in real time using the smartphone's camera, thereby enabling real-time sign language support for the hearing impaired.

[0959] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, and a means for obtaining feedback in real time through the smartphone application, thereby enabling feedback to be collected and analyzed quickly and accurately.

[0960] "Speech recognition means" refers to a device or software that has the function of converting voice input by a user into text data.

[0961] "Translation means" refers to a device or software that has the function of translating text data into other languages ​​using a multilingual translation module.

[0962] A "response generation means" is a device or software that has the function of analyzing the translated text data and generating an appropriate response.

[0963] The "voice synthesis means" is a device or software that has the function of converting the generated response into voice and providing it to the user.

[0964] "Smartphone input devices" is a general term for smartphone hardware and software that has the ability to receive input such as audio, video, and touch.

[0965] The "sign language analysis means" is a device or software that has the function of converting sign language input by a user through a camera into text data using an image analysis engine.

[0966] The "augmented reality menu providing means" is a device or software that has the function of visually displaying text data using augmented reality technology and providing feedback.

[0967] The "feedback collection means" is a device or software that has the function of capturing feedback from users as text data.

[0968] The "feedback analysis means" is a device or software that has the function of analyzing the captured feedback using natural language processing technology and generating positive and negative evaluations.

[0969] The "dashboard providing means" is a device or software that has the function of visually displaying the analysis results.

[0970] A "smartphone application" is application software that runs on a smartphone and provides various functions through a user interface.

[0971] The present invention provides a system for the restaurant industry that supports multiple languages ​​and sign language. This system allows customers to input orders using voice or sign language, and the system processes the orders on a server, which then responds in real time. Specific embodiments of the system are described in detail below.

[0972] Speech recognition and translation processing

[0973] When a user voice orders through an input device on a smartphone, the terminal captures the voice and converts it into text data using a speech recognition means. The text data is then sent to a server, where it is translated into another language by a translation means. A response generation means then analyzes the translated text data and generates an appropriate response. This response is then converted back into speech using a speech synthesis means and provided to the user through the terminal. Specifically, this can be achieved using the SpeechRecognition library, Googletrans library, and Pyttsx3 library.

[0974] Sign language analysis and augmented reality menus

[0975] When a user places an order using sign language through the smartphone camera, the sign language analysis means analyzes the image and converts it into text data. This data is reflected in the augmented reality menu providing means, and visual feedback is provided to the user. This allows the user to check the order details using sign language.

[0976] Collecting and analyzing feedback

[0977] When a user enters feedback through a smartphone application, the feedback is captured as text data by a feedback collection means and sent to a server. The feedback analysis means analyzes the captured data using a natural language processing engine to generate positive and negative evaluations. The analysis results are visually displayed through a dashboard provision means so that employees can check them.

[0978] For example, if a customer orders "I would like a cheeseburger and a coke, please" by voice, the system captures the speech and translates it into Japanese. The system then generates a response, "I'd like a cheeseburger and a coke, please," which is presented visually and audibly to the user. If a customer orders "hamburger and fries" by sign language, the camera captures the sign language and generates an augmented reality menu with images and detailed information about the hamburger and fries, which is presented visually to the user.

[0979] Also, if feedback such as "The service was very good, but the food was a little cold" is entered, positive and negative ratings will be generated and the results will be displayed on the dashboard.

[0980] An example of a prompt sentence to input to the generative AI model is as follows:

[0981] "Build a customer order processing system that utilizes voice recognition and multilingual translation. We need a program that can recognize and translate customer voice orders in real time, and respond with a voice response. The languages ​​used will be from English to Japanese."

[0982] Through these means, the present invention can provide real-time multilingual and sign language enabled order processing, improving customer experience in the restaurant industry.

[0983] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0984] Step 1:

[0985] The user places an order by voice through the smartphone's input device. The device captures this voice and converts it into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[0986] Step 2:

[0987] The terminal sends text data to the server, which translates the text data into another language using the Google Trans library. The input is the text data, and the output is the translated text data.

[0988] Step 3:

[0989] The server receives the translated text data and inputs it to a response generation means, which analyzes the text data and generates an appropriate response. The input is the translated text data and the output is the generated response text.

[0990] Step 4:

[0991] The server inputs the generated response text to the speech synthesis means and converts it into speech data using the Pyttsx3 library. The input is the generated response text, and the output is synthesized speech data.

[0992] Step 5:

[0993] The server sends the synthesized voice data to the terminal, and the terminal provides a voice response to the user. The input is the synthesized voice data, and the output is the voice playback.

[0994] Step 6:

[0995] The user inputs feedback through a smartphone application. The device captures this feedback and converts it into text data. The input is the voice or text of the feedback, and the output is text data.

[0996] Step 7:

[0997] The terminal transmits the feedback text data to the server, and the server analyzes the text data using the feedback analysis means to generate positive and negative evaluations. The input is the feedback text data, and the output is evaluation data.

[0998] Step 8:

[0999] The server inputs the analysis results to the dashboard providing means and generates data for visual display. The input is the evaluation data, and the output is the analysis results that are displayed visually.

[1000] Step 9:

[1001] The server provides employees with visually displayed analysis results and information for considering improvement measures. The input is the visual analysis results, and the output is information provided to employees.

[1002] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1003] The present invention provides a generative AI solution for restaurant staff and wait staff in the food service industry that combines multilingual support, sign language support, and the ability to recognize user emotions. The system for implementing the present invention consists of the following elements:

[1004] Multilingual interactive voice response system

[1005] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[1006] Adding emotion recognition functionality

[1007] The emotion engine installed on the server recognizes emotions from the user's voice and text data. The recognized emotion information is reflected in the response generation means, thereby generating a more empathetic response that is more suited to the user.

[1008] Examples:

[1009] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," if the emotion engine identifies joy in the customer's voice, the server will generate a response such as, "You'd like a cheeseburger and a coke! Enjoy!"

[1010] AR menu provision system that supports sign language

[1011] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[1012] Examples:

[1013] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[1014] Customer feedback collection and analysis system

[1015] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates a positive or negative rating. Furthermore, an emotion engine recognizes the emotion in the feedback and reflects it in the rating. The results are displayed on a dashboard for employees to check.

[1016] Examples:

[1017] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" or a "negative rating (the food was cold)." The emotion engine recognizes the customer's attitude and emotions in the feedback and reflects them in the analysis results. The analysis results are displayed on a dashboard, allowing employees to use them to make improvements.

[1018] Optimized menu suggestion system

[1019] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selection. The server analyzes the received data and uses an emotion engine to generate the optimal menu suggestion that reflects the customer's current mood, and the content is presented to the customer via the terminal. The customer can then confirm and select additional orders.

[1020] Examples:

[1021] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device will send a request to the server based on that. The server will generate a menu suggestion, "How about today's special smoothie?", and the emotion engine will take into account the customer's current emotional state (e.g., surprise) and emphasize the suggestion. The device will display the suggestion and provide a message saying, "Try our special smoothie," allowing the customer to easily place a reorder.

[1022] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[1023] The processing flow will be explained below.

[1024] Multilingual interactive voice response system

[1025] Processing flow from speech recognition to response generation

[1026] Step 1:

[1027] The user queries the order by voice.

[1028] Example: "I'd like a cheeseburger and a Coke, please."

[1029] Step 2:

[1030] The device captures the voice and converts it into text data using a voice recognition engine.

[1031] Speak: "I'd like a cheeseburger and a Coke, please."

[1032] Converted text: "I'd like a cheeseburger and a coke, please."

[1033] Step 3:

[1034] The text data received by the server is analyzed using an emotion engine to recognize the user's emotions.

[1035] Analysis result: Joy

[1036] Step 4:

[1037] The server sends the text data to the translation module and translates it into multiple languages.

[1038] Translation result: "I would like a cheeseburger and a coke, please" (English)

[1039] Step 5:

[1040] The server parses the translated text data and generates an appropriate response.

[1041] Generated response: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[1042] Step 6:

[1043] The server adjusts the response based on the user's emotional information.

[1044] Adjusted response: "Thank you for your order! You have ordered a cheeseburger and a coke. Is there anything else you would like?" (reflects the sentiment)

[1045] Step 7:

[1046] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[1047] Synthesized voice: "Thanks for your order! Cheeseburger and Coke, right? Is there anything else you'd like to order?"

[1048] AR menu provision system that supports sign language

[1049] Processing flow from sign language analysis to AR display

[1050] Step 1:

[1051] A user enters an order using sign language.

[1052] Example: Saying "hamburger and fries" in sign language

[1053] Step 2:

[1054] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[1055] Captured footage: Signing "hamburger and fries"

[1056] Converted text data: "Hamburger and fries"

[1057] Step 3:

[1058] The server receives the converted text data and generates an AR menu that supports sign language.

[1059] Generated: "Hamburger and fries"

[1060] Generated AR menu: Burger and fries images, prices, and details

[1061] Step 4:

[1062] The device displays the generated AR menu and provides visual feedback to the user.

[1063] Showing: Image and details of a burger and fries

[1064] Customer feedback collection and analysis system

[1065] Processing flow from feedback collection to analysis

[1066] Step 1:

[1067] The user enters feedback.

[1068] For example: "The service was great, but the food was a little cold."

[1069] Step 2:

[1070] The device captures the feedback as text data and sends it to the server.

[1071] Feedback captured: "The service was great, but the food was a little cold."

[1072] Step 3:

[1073] The server analyzes the feedback it receives using a natural language processing engine.

[1074] Analysis results: Positive reviews (good service), negative reviews (food was cold)

[1075] Step 4:

[1076] The server uses an emotion engine to recognize emotions in the feedback and reflects the results in the analysis.

[1077] Perceived emotions: "Satisfied", "Dissatisfied"

[1078] Final analysis results: positive evaluation, negative evaluation, emotion recognition results

[1079] Step 5:

[1080] The server displays the analysis results on a dashboard.

[1081] Shows: Positive and negative review details, feedback, and perceived sentiment

[1082] Step 6:

[1083] Employees check the dashboard and consider and implement countermeasures.

[1084] Action example: Review food temperature management and further improve service

[1085] Optimized menu suggestion system

[1086] Suggested flows based on order history and current selections

[1087] Step 1:

[1088] The user initiates the order.

[1089] Example: Select "Veggie Burger"

[1090] Step 2:

[1091] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[1092] Data sent: Current order "Veggie Burger", past order history

[1093] Step 3:

[1094] The server analyzes the received data and uses an emotion engine to recognize the current emotional state of the user.

[1095] Recognized emotion: "surprise"

[1096] Step 4:

[1097] The server takes into account the customer's past history and emotional state to generate optimal menu suggestions.

[1098] Suggestion: "How about today's special smoothie?" (reflects emotions)

[1099] Step 5:

[1100] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[1101] Display content: Today's limited smoothie selection options and details

[1102] Step 6:

[1103] The user confirms the proposal and places an additional order.

[1104] Example behavior: Ordering an extra smoothie

[1105] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[1106] Example 2

[1107] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1108] The current restaurant industry lacks sufficient multilingual and sign language support, and does not provide empathetic responses based on customer feelings, which can lead to a decline in customer satisfaction. It is particularly difficult to provide efficient and accurate service to customers who do not speak a foreign language or who require sign language. There is also a lack of systems in place to effectively utilize customer feedback to improve operations.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1110] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, an emotion recognition means for recognizing emotions from the user's speech and text data, and an empathetic response generation means for generating an empathetic response based on the emotion information. This makes it possible to support multiple languages ​​and sign language, and further to provide empathetic responses based on the customer's emotions, thereby improving customer satisfaction.

[1111] The "voice recognition means" is a means for capturing voice input by a user as digital data and converting the voice into text data.

[1112] The "translation means" is a means for translating the text data generated by the speech recognition means into another language.

[1113] The "response generation means" is a means for analyzing the translated text data and generating an appropriate response.

[1114] The "voice synthesis means" is a means for converting the response generated by the response generation means into voice data and providing it to the user.

[1115] The "emotion recognition means" is a means for recognizing emotions from the user's voice and text data.

[1116] The "empathic response generating means" is a means for generating an empathic response based on emotional information.

[1117] The "sign language analysis means" is a means for capturing sign language input by a user through a camera as an image and converting it into text data.

[1118] The "augmented reality menu providing means" is a means for reflecting text data in an augmented reality menu and providing visual feedback.

[1119] The "feedback collection means" is a means for capturing feedback from users as text data.

[1120] The "feedback analysis means" is a means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations.

[1121] The "dashboard providing means" is a means for visually displaying the evaluation results generated by the feedback analysis means and providing them to employees.

[1122] This invention is a system that provides generative AI solutions for the restaurant industry that integrate multilingual support, sign language support, and emotion recognition functions. This system consists of a server and a terminal, and interacts directly with users.

[1123] Multilingual interactive voice response system

[1124] When a user (customer) enters an order by voice, the device captures the voice with its microphone. The device then sends this voice data to a speech recognition engine such as Google Cloud Speech-to-Text, which converts the speech into text. The converted text data is then sent to a server where it is translated using a multilingual translation module such as Google Cloud Translation. The server then analyzes the translated text data and generates an appropriate response. The generated response is then converted into speech by a speech synthesis engine such as Amazon Polly and played back to the user from the device.

[1125] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," the device captures the voice and sends it to the server, which translates and generates a response, returning it to the user as "You would like a cheeseburger and a coke." This process enables multilingual support for customers.

[1126] Adding emotion recognition functionality

[1127] The server is equipped with an emotion recognition engine (e.g., IBM Watson Tone Analyzer) that analyzes voice and text data to recognize the user's emotions. This emotional information is reflected in the response generation means, which generates an empathetic response.

[1128] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," and the emotion recognition engine detects "happiness," the server will generate an empathetic response such as, "You ordered a cheeseburger and a coke! Enjoy!" and play it back as audio.

[1129] Example prompt sentence:

[1130] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[1131] AR menu provision system that supports sign language

[1132] When a sign language user enters an order through the camera, the device captures the image and sends it to a sign language analysis engine such as Microsoft Kinect to convert it into text data. The server receives the converted text data and generates an augmented reality (AR) menu, which the device displays.

[1133] For example, when a user signs for "hamburger and fries," the device's camera captures the image, and the server converts it into text to generate an AR menu. The menu includes images of the hamburger and fries, along with detailed information, and the device provides visual feedback to the user.

[1134] Customer feedback collection and analysis system

[1135] When a user enters feedback into the device, it is captured as text data and sent to a server. The server then analyzes the feedback data using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative ratings. An emotion recognition engine then analyzes the emotional information in the feedback and displays the results on a dashboard.

[1136] For example, if a user enters feedback such as "The service was good, but the food was a little cold," the server analyzes it and generates a "positive rating (good in terms of service)" and a "negative rating (the food was cold)," which are then displayed on a dashboard where employees can view them.

[1137] Optimized menu suggestion system

[1138] When a user begins an order, the terminal sends a menu suggestion request to the server based on past order history and current selections. The server then analyzes the data using a personalization engine such as Amazon Personalize to generate optimal menu suggestions that take customer sentiment into account. These suggestions are then presented to the user by the terminal, allowing them to easily place additional orders.

[1139] For example, if a user has frequently ordered a "veggie burger" in the past, when they select it again, the server will generate a menu suggestion such as "How about today's special smoothie?", and the emotion engine will highlight the suggestion taking into account the user's current emotion. The device will then display the suggestion, allowing the user to easily place an additional order.

[1140] With these functions, this system will improve the efficiency of customer service and customer satisfaction at restaurants and wait staff in the food service industry. In addition, by combining it with an emotion engine, it will be possible to provide even more personalized services.

[1141] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1142] Processing flow of a multilingual interactive voice response system

[1143] Step 1: Capture audio

[1144] The device captures the user's voice with a microphone. The input is the user's voice, and the output is the captured voice data. This voice data is temporarily stored in the device's storage.

[1145] Specific behavior:

[1146] When a user speaks something like "I would like a cheeseburger and a coke, please," the device's microphone captures and stores the speech as digital data.

[1147] Step 2: Speech to text

[1148] The device sends the captured voice data to the Google Cloud Speech-to-Text API. The input is the captured voice data and the output is text data. The Google Cloud Speech-to-Text API converts the voice data into text data.

[1149] Specific behavior:

[1150] The device sends the voice data to Google Cloud Speech-to-Text, which returns the text, "I would like a cheeseburger and a coke, please."

[1151] Step 3: Translate the text

[1152] The server sends the text data received from the device to the Google Cloud Translation API. The input is text data, and the output is translated text data. The Google Cloud Translation API translates the text data into the specified other language.

[1153] Specific behavior:

[1154] The server sends the text "I would like a cheeseburger and a coke, please" to the Google Cloud Translation API and receives the translated text "I would like a cheeseburger and a coke, please."

[1155] Step 4: Generate a response

[1156] The server analyzes the translated text data and generates an appropriate response. The input is the translated text data and the output is the response text data. The server uses an algorithm to generate the response text.

[1157] Specific behavior:

[1158] Based on the translation result of "Please give me a cheeseburger and a coke", the server generates a response text such as "You've ordered a cheeseburger and a coke."

[1159] Step 5: Synthesize and play the audio

[1160] The server sends the response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data. The input is the response text, and the output is voice data. The generated voice data is sent to the device and played back.

[1161] Specific behavior:

[1162] The server sends the text "You'd like a cheeseburger and a coke" to Amazon Polly, which generates a voice file. The device plays the voice and responds to the user.

[1163] Process flow for adding emotion recognition functionality

[1164] Step 1: Acquiring audio and text

[1165] The server receives the user's voice and text data, which are used for further processing.

[1166] Specific behavior:

[1167] When a user places an order by saying, "I would like a cheeseburger and a coke, please," the voice and text data are sent to the server.

[1168] Step 2: Sentiment Analysis

[1169] The server sends the voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to obtain emotion information. The input is the voice data, and the output is the emotion information.

[1170] Specific behavior:

[1171] The server sends the voice data to the IBM Watson Tone Analyzer, which returns emotional information such as "joy."

[1172] Step 3: Generating an empathetic response

[1173] The server generates an empathetic response based on the emotional information and sends it to a speech synthesis engine. The input is the emotional information and the response text, and the output is voice data as the empathetic response.

[1174] Specific behavior:

[1175] If the emotion engine identifies "joy," the server generates an empathetic response text such as "You ordered a cheeseburger and a Coke! Enjoy!" and converts it into speech to play.

[1176] Example prompt sentence:

[1177] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[1178] Processing flow of the sign language compatible AR menu provision system

[1179] Step 1: Capture sign language video

[1180] The device captures the user's sign language with a camera. The input is video data of the sign language, and the output is the captured video data.

[1181] Specific behavior:

[1182] When the user signs for "hamburger and fries," the device's camera captures the movement.

[1183] Step 2: Translating sign language to text

[1184] The device sends the captured video data to a sign language analysis engine (e.g., Microsoft Kinect) and converts it into text data. The input is video data, and the output is text data.

[1185] Specific behavior:

[1186] The device sends sign language video to a Microsoft Kinect, which responds with the text "hamburger and fries."

[1187] Step 3: Generate the AR menu

[1188] The server generates an AR menu based on the text data. The input is the text data, and the output is the generated AR menu.

[1189] Specific behavior:

[1190] The server receives the text data for "hamburger and fries" and generates a corresponding AR menu.

[1191] Step 4: Display the AR menu

[1192] The device displays the generated AR menu to the user. The input is the AR menu, and the output is the displayed AR menu.

[1193] Specific behavior:

[1194] The device displays an AR menu, providing the user with images and details about the burger and fries.

[1195] Customer feedback collection and analysis system process flow

[1196] Step 1: Enter your feedback

[1197] The user inputs feedback and the device captures it. The input is the feedback content and the output is text data.

[1198] Specific behavior:

[1199] The user enters feedback saying, "The service was good, but the food was cold."

[1200] Step 2: Submit your feedback

[1201] The terminal sends the captured feedback data to the server. The input is text data, and the output is the feedback data transferred to the server.

[1202] Specific behavior:

[1203] The terminal sends the feedback text to the server.

[1204] Step 3: Analyze the feedback

[1205] The server analyzes the received feedback using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative evaluations. The input is the feedback text, and the output is the evaluation data as the analysis result.

[1206] Specific behavior:

[1207] The server analyzes the feedback text "The service was good, but the food was cold" and generates a positive rating (the service was good) and a negative rating (the food was cold).

[1208] Step 4: View the evaluation results

[1209] The server displays the analysis results on a dashboard. The input is the evaluation data, and the output is the displayed dashboard.

[1210] Specific behavior:

[1211] The server displays the evaluation results on a dashboard, where employees can view them.

[1212] Optimized menu suggestion system processing flow

[1213] Step 1: Initiate your order

[1214] When a user initiates an order, the terminal sends the user's past order history and current selections to the server. The input is the order data, and the output is the request data to the server.

[1215] Specific behavior:

[1216] When the user selects "veggie burger," the terminal sends the data along with past order history to the server.

[1217] Step 2: Generate menu suggestions

[1218] The server uses a personalization engine (e.g., Amazon Personalize) to analyze the data and generate optimal menu suggestions. The input is the request data, and the output is the suggested menu.

[1219] Specific behavior:

[1220] The server analyzes the user's past order history and current selections and generates a menu suggestion such as, "How about today's special smoothie?"

[1221] Step 3: Viewing Proposals

[1222] The terminal displays the generated menu suggestions to the user. The input is the suggested menu and the output is the suggestion displayed on the user's screen.

[1223] Specific behavior:

[1224] The device will display a suggestion to the user, "How about today's special smoothie?", encouraging them to order more.

[1225] The above is the specific processing flow of each system, which will enable the restaurant industry to improve customer service efficiency and customer satisfaction.

[1226] (Application example 2)

[1227] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1228] In today's brick-and-mortar stores, multilingual support is required to accommodate the increasing number of foreign tourists and diversifying customer needs, but language barriers hinder the efficiency of customer service and improve customer satisfaction. Dealing with customers who use sign language and reading their emotions to respond appropriately are also challenges. A system is needed to resolve these issues and improve the customer experience.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1230] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation. This not only enables multilingual communication across language barriers, but also makes it possible to provide personalized responses according to sign language and emotions.

[1231] A "voice recognition means" is a device or system that converts voice input by a user into text data.

[1232] A "multilingual translation module" is a device or software for translating text data into different languages.

[1233] "Translation means" is a function that converts text data into other languages ​​using a multilingual translation module.

[1234] The "response generation means" is a function that analyzes the translated text data and generates an appropriate response.

[1235] A "voice synthesis means" is a device or software that converts the generated response into speech and provides it to the user.

[1236] The "emotion recognition means" is a function that recognizes emotions from the user's voice and reflects the emotion data in response generation.

[1237] The "sign language analysis means" is a function that converts sign language input by the user through the camera into text data using an image analysis engine.

[1238] The "augmented reality menu providing means" is a function that reflects the text data obtained by the sign language analysis means in an augmented reality menu and provides visual feedback.

[1239] A "feedback collection means" is a device or software that captures feedback from users as text data.

[1240] The "feedback analysis means" is a function that analyzes the captured feedback using a natural language processing engine and generates positive and negative evaluations.

[1241] The "dashboard provision means" is a function that displays the results of feedback analysis on a dashboard so that employees can check them.

[1242] The system programs required to implement the present invention will now be described.

[1243] The server includes: a speech recognition means for converting speech input by the user into text data; a multilingual translation module for translating the text data into other languages; a response generation means for analyzing the translated text data and generating an appropriate response; a speech synthesis means for converting the generated response into speech and providing it to the user; an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation; a sign language analysis means for converting sign language input by the user through a camera into text data using a video analysis engine; an augmented reality menu providing means for reflecting the text data obtained by the sign language analysis means in an augmented reality menu and providing visual feedback; a feedback collection means for capturing feedback from the user as text data; a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations; and a dashboard providing means for displaying the results of the feedback analysis on a dashboard so that employees can check it.

[1244] The speech recognition means uses, for example, the speech_recognition library. This library captures the speech input by the user and converts it into text data. The converted text data is sent to a multilingual translation module (for example, Google Trans New API) and translated into another language. The translated text data is analyzed by the server, and an appropriate response is generated. This response is converted into audio data using a speech synthesis means (for example, pyttsx3) and provided to the user by the terminal.

[1245] The emotion recognition means uses the EmotionRecognition model. When the user's voice is captured, the emotion recognition means analyzes the voice data and detects emotions. The detected emotional data is reflected in the responses generated by the response generation means, resulting in more personalized responses.

[1246] The sign language analysis means uses the SignLanguageRecognition model. When a user inputs sign language through the camera, this model analyzes the video data and converts it into corresponding text data. The converted text data is sent to the augmented reality menu providing means, which provides visual feedback. This allows users who use sign language to order smoothly.

[1247] The feedback collection means includes a function for capturing text data. When feedback is entered by a user, it is captured as text data. The feedback analysis means then analyzes the feedback using a natural language processing engine (such as spaCy), and positive or negative evaluations are generated. These evaluation results are displayed in a form that can be checked by employees using the dashboard provision means. The dashboard is implemented using tools such as dash or Tableau.

[1248] Specific examples

[1249] Suppose a customer places an order by saying, "I would like a cheeseburger and a coke, please." At this time, the speech recognition means converts the speech into text, which is then translated by the multilingual translation module. If the emotion recognition means identifies joy from the customer's speech, the response generation means generates a response such as, "You'd like a cheeseburger and a coke! Enjoy!" The speech synthesis means converts this response into speech and provides it to the customer from the terminal.

[1250] Prompt Sentence Examples

[1251] Input prompt:

[1252] User: "I would like a cheeseburger and a coke, please"

[1253] System: "You ordered a cheeseburger and a Coke. Is there anything else I can help you with?"

[1254] Response prompt:

[1255] User: "No, that's all for now."

[1256] System: "Thank you. Your order has been placed. Please wait a moment."

[1257] As described above, this system not only enables multilingual communication across language barriers, but also provides personalized responses based on emotions and sign language support, which is expected to improve customer satisfaction in brick-and-mortar stores.

[1258] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1259] Step 1:

[1260] The user enters the order by voice.

[1261] Input: Speech data (e.g., "I would like a cheeseburger and a coke, please")

[1262] What happens: The device's microphone captures audio data.

[1263] Output: Captured audio data

[1264] Step 2:

[1265] A speech recognition means converts the captured speech data into text data.

[1266] Input: Audio data

[1267] Data processing: A speech recognition engine (e.g., speech_recognition library) analyzes the voice data and converts it into text data.

[1268] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[1269] Step 3:

[1270] A multilingual translation module translates the text data into other languages.

[1271] Input: Text data

[1272] Data calculation: A translation engine (e.g., Google Trans New API) translates the input text data into the specified language.

[1273] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[1274] Step 4:

[1275] An emotion recognition means recognizes emotions from the translated text data and voice data.

[1276] Input: Translated text data, audio data

[1277] Data computation: An emotion recognition engine (e.g., EmotionRecognition model) analyzes the speech and text and generates emotion data.

[1278] Output: Emotion data (e.g., "joy")

[1279] Step 5:

[1280] A response generation means generates an appropriate response based on the translated text data and the emotion data.

[1281] Input: Translated text data, emotion data

[1282] Data computation: A response generation engine generates empathetic responses based on input data.

[1283] Output: Response text data (e.g., "You ordered a cheeseburger and a Coke! Enjoy!")

[1284] Step 6:

[1285] A speech synthesis means converts the response text data into speech.

[1286] Input: Response text data

[1287] Data processing: A speech synthesis engine (e.g., pyttsx3) converts text data into speech data.

[1288] Output: Response audio data

[1289] Step 7:

[1290] The terminal provides the response voice data to the user.

[1291] Input: Response audio data

[1292] Action: The device speaker plays the audio data.

[1293] Output: Speech played to the user (e.g., "You've ordered a cheeseburger and a Coke! Enjoy!")

[1294] Step 8:

[1295] When a user inputs an order in sign language through the camera, the sign language analysis means captures the video data.

[1296] Input: Video data (e.g., sign language for "cheeseburger and coke")

[1297] Action: The device's camera captures video data.

[1298] Output: Captured video data

[1299] Step 9:

[1300] A sign language analysis means converts the captured video data into text data.

[1301] Input: Video data

[1302] Data processing: A sign language analysis engine (e.g., SignLanguageRecognition model) analyzes the video data and converts it into text.

[1303] Output: Text data corresponding to the sign language (e.g., "Cheeseburger and Coke")

[1304] Step 10:

[1305] The augmented reality menu providing means generates an augmented reality menu based on the converted text data and provides visual feedback.

[1306] Input: Text data obtained from sign language analysis

[1307] Data processing: The augmented reality engine generates a menu based on the text data and displays it on the screen.

[1308] Output: Augmented reality menu displayed to the user (e.g., image of a cheeseburger and a coke with details)

[1309] Step 11:

[1310] Feedback from users is captured as text data by a feedback collection means.

[1311] Input: Text data (e.g., "The service was very good, but the food was a little cold.")

[1312] Operation: Feedback data is sent from the device to the server.

[1313] Output: Captured feedback text data

[1314] Step 12:

[1315] A feedback analysis means analyzes the captured feedback using a natural language processing engine to generate positive and negative evaluations.

[1316] Input: Feedback text data

[1317] Data calculation: A natural language processing engine (e.g., spaCy) analyzes the feedback data and generates positive and negative ratings.

[1318] Output: Rating data (e.g., "The service was good" or "The food was cold")

[1319] Step 13:

[1320] The dashboard provider displays the analysis results on a dashboard, which helps employees make improvements.

[1321] Input: Parsed evaluation data

[1322] How it works: A dashboard system (e.g., dash or Tableau) visually displays the assessment data.

[1323] Output: The dashboard to be displayed (e.g., positive and negative feedback ratings)

[1324] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1325] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1326] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1327] [Third embodiment]

[1328] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1329] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1330] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1331] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1332] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1333] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1334] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1335] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1336] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1337] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1338] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1339] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1340] The present invention provides a generative AI solution that enables multilingual and sign language support for restaurants and wait staff in the food service industry. The system for implementing the present invention consists of the following elements:

[1341] Multilingual interactive voice response system

[1342] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[1343] Examples:

[1344] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[1345] AR menu provision system that supports sign language

[1346] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[1347] Examples:

[1348] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[1349] Customer feedback collection and analysis system

[1350] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates positive and negative ratings. The results are displayed on a dashboard for employees to review.

[1351] Examples:

[1352] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[1353] Optimized menu suggestion system

[1354] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes the received data and generates optimal menu suggestions, which are presented to the customer by the terminal. The customer can then confirm and select additional orders.

[1355] Examples:

[1356] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[1357] The above is an embodiment of the present invention. This system can improve the efficiency of customer service operations in the restaurant industry and effectively solve the problem of labor shortages through functions such as multilingual support, sign language support, feedback collection and analysis, and optimized menu suggestions.

[1358] The processing flow will be explained below.

[1359] Multilingual interactive voice response system

[1360] Step 1:

[1361] The user queries the order by voice.

[1362] Examples: "Cheeseburger", "Coke"

[1363] Step 2:

[1364] The device captures the voice and converts it into text data using a voice recognition engine.

[1365] Speak: "I'd like a cheeseburger and a Coke, please."

[1366] Converted text: "I'd like a cheeseburger and a coke, please."

[1367] Step 3:

[1368] The server sends the received text data to a translation module to translate it into the customer's language.

[1369] Data sent: "I'd like a cheeseburger and a coke, please."

[1370] Translated text: "I would like a cheeseburger and a coke, please" (English)

[1371] Step 4:

[1372] The server parses the translated text data and generates an appropriate response.

[1373] Parse: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[1374] Step 5:

[1375] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[1376] Synthesized voice: "You've ordered a cheeseburger and a Coke. Is there anything else you'd like to order?"

[1377] AR menu provision system that supports sign language

[1378] Step 1:

[1379] A user enters an order using sign language.

[1380] Example: Say "hamburger" and "fries" in sign language.

[1381] Step 2:

[1382] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[1383] Captured footage: Sign language for "hamburger" and "fries"

[1384] Converted text data: "Hamburger and fries"

[1385] Step 3:

[1386] The server receives the converted text data and generates an AR menu that supports sign language.

[1387] Generated: "Hamburger and fries"

[1388] Generated AR menu: Burger and fries images, prices, and details

[1389] Step 4:

[1390] The device displays the generated AR menu and provides visual feedback to the user.

[1391] Showing: Image and details of a burger and fries

[1392] Customer feedback collection and analysis system

[1393] Step 1:

[1394] The user enters feedback.

[1395] For example: "The service was good, but the food was cold."

[1396] Step 2:

[1397] The device captures the feedback as text data and sends it to the server.

[1398] Feedback captured: "The service was good, but the food was cold."

[1399] Step 3:

[1400] The server analyzes the feedback it receives using a natural language processing engine.

[1401] Analysis results: Positive reviews (good service), negative reviews (cold food)

[1402] Step 4:

[1403] The server displays the analysis results on a dashboard.

[1404] Display: Positive and negative reviews, feedback details

[1405] Step 5:

[1406] The user (employee) checks the dashboard and considers and implements countermeasures.

[1407] Action example: Review food temperature management and further improve service

[1408] Optimized menu suggestion system

[1409] Step 1:

[1410] The user initiates the order.

[1411] Example: Select "Veggie Burger."

[1412] Step 2:

[1413] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[1414] Data sent: Current order "Veggie Burger", past order history

[1415] Step 3:

[1416] The server analyzes the received data and generates optimal menu suggestions.

[1417] Suggestion: "Would you like to try today's special smoothie?"

[1418] Step 4:

[1419] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[1420] Display content: Today's limited smoothie selection options and details

[1421] Step 5:

[1422] The user confirms the proposal and places an additional order.

[1423] Example behavior: Ordering an extra smoothie.

[1424] The above is a specific processing flow of each system in the embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry.

[1425] Example 1

[1426] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1427] In today's restaurant industry, serving customers with language barriers or communication disabilities is a major issue that leads to labor shortages and reduced operational efficiency. In particular, when multilingual and sign language support is required, the limitations of employee skills and resources become apparent. There is also a need for systems that can instantly collect and analyze customer feedback and develop appropriate countermeasures. Furthermore, it is important to be able to suggest optimal menu items based on customers' past ordering history, as this will lead to improved customer satisfaction and increased sales.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1429] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, a communication means for transmitting user speech data captured by the terminal to the server, and a playback means for playing back the generated speech response from the terminal, thereby enabling multilingual automated voice responses that overcome language barriers.

[1430] Furthermore, a system is provided that includes a sign language analysis means that converts sign language input by a user through a camera into text data using a video analysis engine, an augmented reality menu providing means that reflects the text data in an augmented reality menu and provides visual feedback, a communication means that transmits video data captured by the terminal to a server, and a display means that displays the generated augmented reality menu on the terminal, which enables smooth communication with customers who use sign language and improves the quality of service.

[1431] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback with a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, a communication means for transmitting the feedback data captured by the terminal to a server, and a confirmation means for employees to consider and implement countermeasures based on the analysis results. This makes it possible to quickly collect and analyze customer feedback and develop appropriate countermeasures.

[1432] "Speech recognition means" refers to a function for converting speech input by a user into text data.

[1433] "Translation means" refers to a function for translating text data into other languages ​​using a multilingual translation module.

[1434] "Response generation means" refers to a function for analyzing translated text data and generating an appropriate response.

[1435] "Speech synthesis means" refers to a function for converting the generated response into speech and providing it to the user.

[1436] "Communication means" refers to a function for transmitting user voice data and video data captured by the terminal to the server, and for transmitting responses and menus from the server to the terminal.

[1437] The "playback means" refers to a function for playing back the generated voice response from the terminal.

[1438] "Sign language analysis means" refers to the function for converting sign language input through a camera into text data using an image analysis engine.

[1439] The "augmented reality menu providing means" refers to a function for reflecting the converted text data in an augmented reality menu and providing visual feedback.

[1440] "Display means" refers to a function for displaying the generated augmented reality menu on the terminal.

[1441] "Feedback collection means" refers to a function for capturing feedback from users as text data.

[1442] "Feedback analysis means" refers to a function for analyzing captured feedback using a natural language processing engine and generating positive and negative evaluations.

[1443] "Dashboard provision means" refers to a function for displaying analysis results on a dashboard so that employees can check them.

[1444] "Verification means" refers to the function that allows employees to consider and implement countermeasures based on the analysis results.

[1445] Multilingual interactive voice response system

[1446] This invention is a system that automatically converts orders entered by voice into text, translates them in multiple languages, and generates voice responses. When a user enters an order by voice, the device captures the speech and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is sent to a server, which translates it into the user's native language using a multilingual translation module such as Google Cloud Translation. The server analyzes the translated text data and generates an appropriate response using a generative AI model such as GPT-4. The generated response is converted into speech using a speech synthesis engine such as Amazon Polly, and the response is finally played back by the device.

[1447] Examples:

[1448] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[1449] AR menu provision system that supports sign language

[1450] When a sign language user places an order through the camera, the device captures the video and converts it into text data using a sign language analysis engine such as Google Cloud Video Intelligence. This text data is sent to a server, which then uses an augmented reality (AR) content creation tool such as Unity to generate an AR menu that supports sign language. The generated AR menu is then displayed on the device, providing visual feedback to the user.

[1451] Examples:

[1452] When a customer orders a "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[1453] Customer feedback collection and analysis system

[1454] When a user enters feedback as text, the device captures the feedback and sends it to a server. The server receives the feedback data and analyzes it using a natural language processing engine such as GPT-4 or BERT. The analysis results in positive and negative ratings, which are displayed on a dashboard. Employees can view customer feedback through the dashboard and consider and implement appropriate measures.

[1455] Examples:

[1456] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[1457] Optimized menu suggestion system

[1458] When a user begins an order, the device sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes this data in a database such as BigQuery, generates optimal menu suggestions, and sends them to the device. The device then presents these suggestions to the user, who can then confirm and select additional orders.

[1459] Examples:

[1460] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[1461] Prompt Sentence Examples

[1462] "Please provide a detailed description of the voice commands or sign language inputs used by the user to place an order and the actions taken based on those inputs. Also, please provide a detailed description of feedback inputs and optimized menu suggestions."

[1463] The above is an embodiment of the present invention, which makes it possible to streamline customer service operations in the restaurant industry and effectively solve the problem of labor shortages through multilingual support, sign language support, feedback collection and analysis, and optimized menu proposals.

[1464] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1465] Multilingual Interactive Voice Response System Processing Steps

[1466] Step 1:

[1467] The user enters the order by voice.

[1468] Input: User speech (e.g., "I would like a cheeseburger and a coke, please")

[1469] Output: Audio data is generated on the device

[1470] Specific operation: The user speaks an order into a terminal with a microphone.

[1471] Step 2:

[1472] The device captures the audio.

[1473] Input: User's voice data

[1474] Output: Stored as digital audio data

[1475] Specific operation: The device's microphone picks up sound and records it as digital audio data.

[1476] Step 3:

[1477] The terminal transmits the audio data to the server.

[1478] Input: Digital audio data

[1479] Output: Audio data is sent to the server

[1480] Specific operation: The device sends the captured audio data to the server using a communication means.

[1481] Step 4:

[1482] The server converts the speech to text.

[1483] Input: Audio data arriving at the server

[1484] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[1485] What happens: The server converts the audio data into text using the Google Cloud Speech-to-Text API.

[1486] Step 5:

[1487] The server translates the text data.

[1488] Input: Converted text data

[1489] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[1490] Specific operation: The server uses the Google Cloud Translation API to translate the text into the user's native language.

[1491] Step 6:

[1492] The server parses the translated text data and generates a response.

[1493] Input: Translated text data

[1494] Output: Response text (e.g., "You'd like a cheeseburger and a coke").

[1495] What it does: The server uses the GPT-4 generative AI model to analyze the translated text and generate an appropriate response.

[1496] Step 7:

[1497] The server converts the response into speech.

[1498] Input: Response text

[1499] Output: Speech data (e.g., "You'd like a cheeseburger and a coke, right?")

[1500] Specific operation: The server uses a speech synthesis engine such as Amazon Polly to convert the response text into audio data.

[1501] Step 8:

[1502] The server sends the audio data to the terminal.

[1503] Input: Generated audio data

[1504] Output: Audio data is sent to the device

[1505] Specific operation: The server sends the generated voice data to the terminal.

[1506] Step 9:

[1507] The device plays a voice response.

[1508] Input: Audio data received on the device

[1509] Output: The audio played to the user

[1510] Specific operation: The device plays the received audio data through the speaker and prompts the user to confirm.

[1511] Processing steps of an AR menu provision system that supports sign language

[1512] Step 1:

[1513] A user who uses sign language enters an order through the camera.

[1514] Input: User's sign language actions

[1515] Output: Video data is generated by the camera

[1516] Specific actions: The user indicates an order in sign language in front of the camera.

[1517] Step 2:

[1518] The device captures the sign language video.

[1519] Input: Video data of the user's sign language movements

[1520] Output: Saved as digital video data

[1521] Specific operation: The device's camera captures sign language movements and records them as digital video data.

[1522] Step 3:

[1523] The terminal transmits the video data to the server.

[1524] Input: Digital video data

[1525] Output: Video data is sent to the server

[1526] Specific operation: The video data captured by the device is sent to the server using a communication means.

[1527] Step 4:

[1528] The server converts the video data into text.

[1529] Input: Video data arriving at the server

[1530] Output: Text data (e.g. "hamburger and fries")

[1531] How it works: The server uses the Google Cloud Video Intelligence API to analyze the sign language video data and convert it into text.

[1532] Step 5:

[1533] The server generates an AR menu based on the text data.

[1534] Input: Converted text data

[1535] Output: Augmented reality menu

[1536] Specific operation: The server uses an AR content creation tool such as Unity to generate an AR menu that supports sign language.

[1537] Step 6:

[1538] The server sends the generated AR menu to the device.

[1539] Input: Generated AR menu

[1540] Output: The AR menu is sent to the device.

[1541] Specific operation: The server sends the generated AR menu to the device.

[1542] Step 7:

[1543] The device will display the AR menu.

[1544] Input: AR menu received on the device

[1545] Output: Visual feedback that is displayed to the user

[1546] Specific behavior: The device displays an AR menu overlaid on the camera screen, providing visual feedback to the user.

[1547] Customer feedback collection and analysis system processing steps

[1548] Step 1:

[1549] The user enters feedback in text.

[1550] Input: User feedback text

[1551] Output: Feedback data is generated on the terminal

[1552] Specific operation: The user inputs feedback on the device screen.

[1553] Step 2:

[1554] The device captures the feedback.

[1555] Input: Feedback text

[1556] Output: Stored as digital feedback data

[1557] Specific behavior: The device captures and stores the input feedback data.

[1558] Step 3:

[1559] The terminal transmits the feedback data to the server.

[1560] Input: Digital feedback data

[1561] Output: Feedback data is sent to the server

[1562] Specific operation: The feedback data captured by the terminal is sent to the server via a communication means.

[1563] Step 4:

[1564] The server analyzes the feedback.

[1565] Input: Feedback data arriving at the server

[1566] Output: Analysis results (positive and negative evaluations)

[1567] Specific operation: The server analyzes the feedback data using a natural language processing engine (GPT-4 or BERT) and generates an evaluation.

[1568] Step 5:

[1569] The server displays the analysis results on a dashboard.

[1570] Input: Analysis results

[1571] Output: Ratings displayed on the dashboard

[1572] Specific operation: The server displays the generated evaluation results on a dashboard for employees to review.

[1573] Processing steps of an optimized menu suggestion system

[1574] Step 1:

[1575] The user initiates the order.

[1576] Input: The user's current order selection (e.g., "Veggie Burger")

[1577] Output: Order data is generated on the terminal.

[1578] Specific Actions: The user starts the ordering process at the terminal and selects an item.

[1579] Step 2:

[1580] The terminal sends past order history and current selections to the server.

[1581] Input: User's past order history and current selection data

[1582] Output: Order history data is sent to the server

[1583] Specific operation: The terminal sends the user's saved past order history and current selections to the server.

[1584] Step 3:

[1585] The server parses the order data.

[1586] Input: Order data arriving at the server

[1587] Output: Analysis results (menu suggestions)

[1588] Specific operation: The server analyzes the data using a database such as BigQuery and generates optimal menu suggestions.

[1589] Step 4:

[1590] The server generates the menu suggestions.

[1591] Input: Analysis results

[1592] Output: Generated menu suggestions (e.g., "Would you like today's special smoothie?")

[1593] Specific operation: The server forms optimal menu suggestions based on the analysis results.

[1594] Step 5:

[1595] The server sends the proposal to the device.

[1596] Input: Generated menu suggestions

[1597] Output: Menu suggestions sent to the device

[1598] Specific operation: The server sends the generated menu suggestions to the terminal.

[1599] Step 6:

[1600] The device will display suggestions.

[1601] Input: Menu suggestions received on the device

[1602] Output: Menu suggestions presented to the user

[1603] Specific operation: The terminal displays the menu suggestions sent from the server, and the user confirms and selects the additional order.

[1604] (Application example 1)

[1605] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1606] In the traditional restaurant industry, it is difficult to accommodate customers who speak foreign languages ​​or who are hearing impaired, and processing orders and feedback requires a great deal of time and effort. Furthermore, when customers want to order in real time using voice or sign language in multiple languages, the system that provides support is inadequate, making effective communication difficult. Furthermore, there is a lack of an efficient system for quickly and accurately processing customer feedback and taking appropriate improvement measures.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1608] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and a means for processing the user's voice order in real time using an input device of the smartphone, thereby enabling support for multiple languages ​​and sign language.

[1609] The system also includes a sign language analysis unit that converts sign language input by the user through the camera into text data using a video analysis engine, an augmented reality menu providing unit that reflects the text data in an augmented reality menu and provides visual feedback, and a unit that processes sign language orders in real time using the smartphone's camera, thereby enabling real-time sign language support for the hearing impaired.

[1610] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, and a means for obtaining feedback in real time through the smartphone application, thereby enabling feedback to be collected and analyzed quickly and accurately.

[1611] "Speech recognition means" refers to a device or software that has the function of converting voice input by a user into text data.

[1612] "Translation means" refers to a device or software that has the function of translating text data into other languages ​​using a multilingual translation module.

[1613] A "response generation means" is a device or software that has the function of analyzing the translated text data and generating an appropriate response.

[1614] The "voice synthesis means" is a device or software that has the function of converting the generated response into voice and providing it to the user.

[1615] "Smartphone input devices" is a general term for smartphone hardware and software that has the ability to receive input such as audio, video, and touch.

[1616] The "sign language analysis means" is a device or software that has the function of converting sign language input by a user through a camera into text data using an image analysis engine.

[1617] The "augmented reality menu providing means" is a device or software that has the function of visually displaying text data using augmented reality technology and providing feedback.

[1618] The "feedback collection means" is a device or software that has the function of capturing feedback from users as text data.

[1619] The "feedback analysis means" is a device or software that has the function of analyzing the captured feedback using natural language processing technology and generating positive and negative evaluations.

[1620] The "dashboard providing means" is a device or software that has the function of visually displaying the analysis results.

[1621] A "smartphone application" is application software that runs on a smartphone and provides various functions through a user interface.

[1622] The present invention provides a system for the restaurant industry that supports multiple languages ​​and sign language. This system allows customers to input orders using voice or sign language, and the system processes the orders on a server, which then responds in real time. Specific embodiments of the system are described in detail below.

[1623] Speech recognition and translation processing

[1624] When a user voice orders through an input device on a smartphone, the terminal captures the voice and converts it into text data using a speech recognition means. The text data is then sent to a server, where it is translated into another language by a translation means. A response generation means then analyzes the translated text data and generates an appropriate response. This response is then converted back into speech using a speech synthesis means and provided to the user through the terminal. Specifically, this can be achieved using the SpeechRecognition library, Googletrans library, and Pyttsx3 library.

[1625] Sign language analysis and augmented reality menus

[1626] When a user places an order using sign language through the smartphone camera, the sign language analysis means analyzes the image and converts it into text data. This data is reflected in the augmented reality menu providing means, and visual feedback is provided to the user. This allows the user to check the order details using sign language.

[1627] Collecting and analyzing feedback

[1628] When a user enters feedback through a smartphone application, the feedback is captured as text data by a feedback collection means and sent to a server. The feedback analysis means analyzes the captured data using a natural language processing engine to generate positive and negative evaluations. The analysis results are visually displayed through a dashboard provision means so that employees can check them.

[1629] For example, if a customer orders "I would like a cheeseburger and a coke, please" by voice, the system captures the speech and translates it into Japanese. The system then generates a response, "I'd like a cheeseburger and a coke, please," which is presented visually and audibly to the user. If a customer orders "hamburger and fries" by sign language, the camera captures the sign language and generates an augmented reality menu with images and detailed information about the hamburger and fries, which is presented visually to the user.

[1630] Also, if feedback such as "The service was very good, but the food was a little cold" is entered, positive and negative ratings will be generated and the results will be displayed on the dashboard.

[1631] An example of a prompt sentence to input to the generative AI model is as follows:

[1632] "Build a customer order processing system that utilizes voice recognition and multilingual translation. We need a program that can recognize and translate customer voice orders in real time, and respond with a voice response. The languages ​​used will be from English to Japanese."

[1633] Through these means, the present invention can provide real-time multilingual and sign language enabled order processing, improving customer experience in the restaurant industry.

[1634] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1635] Step 1:

[1636] The user places an order by voice through the smartphone's input device. The device captures this voice and converts it into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[1637] Step 2:

[1638] The terminal sends text data to the server, which translates the text data into another language using the Google Trans library. The input is the text data, and the output is the translated text data.

[1639] Step 3:

[1640] The server receives the translated text data and inputs it to a response generation means, which analyzes the text data and generates an appropriate response. The input is the translated text data and the output is the generated response text.

[1641] Step 4:

[1642] The server inputs the generated response text to the speech synthesis means and converts it into speech data using the Pyttsx3 library. The input is the generated response text, and the output is synthesized speech data.

[1643] Step 5:

[1644] The server sends the synthesized voice data to the terminal, and the terminal provides a voice response to the user. The input is the synthesized voice data, and the output is the voice playback.

[1645] Step 6:

[1646] The user inputs feedback through a smartphone application. The device captures this feedback and converts it into text data. The input is the voice or text of the feedback, and the output is text data.

[1647] Step 7:

[1648] The terminal transmits the feedback text data to the server, and the server analyzes the text data using the feedback analysis means to generate positive and negative evaluations. The input is the feedback text data, and the output is evaluation data.

[1649] Step 8:

[1650] The server inputs the analysis results to the dashboard providing means and generates data for visual display. The input is the evaluation data, and the output is the analysis results that are displayed visually.

[1651] Step 9:

[1652] The server provides employees with visually displayed analysis results and information for considering improvement measures. The input is the visual analysis results, and the output is information provided to employees.

[1653] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1654] The present invention provides a generative AI solution for restaurant staff and wait staff in the food service industry that combines multilingual support, sign language support, and the ability to recognize user emotions. The system for implementing the present invention consists of the following elements:

[1655] Multilingual interactive voice response system

[1656] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[1657] Adding emotion recognition functionality

[1658] The emotion engine installed on the server recognizes emotions from the user's voice and text data. The recognized emotion information is reflected in the response generation means, thereby generating a more empathetic response that is more suited to the user.

[1659] Examples:

[1660] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," if the emotion engine identifies joy in the customer's voice, the server will generate a response such as, "You'd like a cheeseburger and a coke! Enjoy!"

[1661] AR menu provision system that supports sign language

[1662] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[1663] Examples:

[1664] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[1665] Customer feedback collection and analysis system

[1666] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates a positive or negative rating. Furthermore, an emotion engine recognizes the emotion in the feedback and reflects it in the rating. The results are displayed on a dashboard for employees to check.

[1667] Examples:

[1668] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" or a "negative rating (the food was cold)." The emotion engine recognizes the customer's attitude and emotions in the feedback and reflects them in the analysis results. The analysis results are displayed on a dashboard, allowing employees to use them to make improvements.

[1669] Optimized menu suggestion system

[1670] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selection. The server analyzes the received data and uses an emotion engine to generate the optimal menu suggestion that reflects the customer's current mood, and the content is presented to the customer via the terminal. The customer can then confirm and select additional orders.

[1671] Examples:

[1672] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device will send a request to the server based on that. The server will generate a menu suggestion, "How about today's special smoothie?", and the emotion engine will take into account the customer's current emotional state (e.g., surprise) and emphasize the suggestion. The device will display the suggestion and provide a message saying, "Try our special smoothie," allowing the customer to easily place a reorder.

[1673] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[1674] The processing flow will be explained below.

[1675] Multilingual interactive voice response system

[1676] Processing flow from speech recognition to response generation

[1677] Step 1:

[1678] The user queries the order by voice.

[1679] Example: "I'd like a cheeseburger and a Coke, please."

[1680] Step 2:

[1681] The device captures the voice and converts it into text data using a voice recognition engine.

[1682] Speak: "I'd like a cheeseburger and a Coke, please."

[1683] Converted text: "I'd like a cheeseburger and a coke, please."

[1684] Step 3:

[1685] The text data received by the server is analyzed using an emotion engine to recognize the user's emotions.

[1686] Analysis result: Joy

[1687] Step 4:

[1688] The server sends the text data to the translation module and translates it into multiple languages.

[1689] Translation result: "I would like a cheeseburger and a coke, please" (English)

[1690] Step 5:

[1691] The server parses the translated text data and generates an appropriate response.

[1692] Generated response: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[1693] Step 6:

[1694] The server adjusts the response based on the user's emotional information.

[1695] Adjusted response: "Thank you for your order! You have ordered a cheeseburger and a coke. Is there anything else you would like?" (reflects the sentiment)

[1696] Step 7:

[1697] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[1698] Synthesized voice: "Thanks for your order! Cheeseburger and Coke, right? Is there anything else you'd like to order?"

[1699] AR menu provision system that supports sign language

[1700] Processing flow from sign language analysis to AR display

[1701] Step 1:

[1702] A user enters an order using sign language.

[1703] Example: Saying "hamburger and fries" in sign language

[1704] Step 2:

[1705] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[1706] Captured footage: Signing "hamburger and fries"

[1707] Converted text data: "Hamburger and fries"

[1708] Step 3:

[1709] The server receives the converted text data and generates an AR menu that supports sign language.

[1710] Generated: "Hamburger and fries"

[1711] Generated AR menu: Burger and fries images, prices, and details

[1712] Step 4:

[1713] The device displays the generated AR menu and provides visual feedback to the user.

[1714] Showing: Image and details of a burger and fries

[1715] Customer feedback collection and analysis system

[1716] Processing flow from feedback collection to analysis

[1717] Step 1:

[1718] The user enters feedback.

[1719] For example: "The service was great, but the food was a little cold."

[1720] Step 2:

[1721] The device captures the feedback as text data and sends it to the server.

[1722] Feedback captured: "The service was great, but the food was a little cold."

[1723] Step 3:

[1724] The server analyzes the feedback it receives using a natural language processing engine.

[1725] Analysis results: Positive reviews (good service), negative reviews (food was cold)

[1726] Step 4:

[1727] The server uses an emotion engine to recognize emotions in the feedback and reflects the results in the analysis.

[1728] Perceived emotions: "Satisfied", "Dissatisfied"

[1729] Final analysis results: positive evaluation, negative evaluation, emotion recognition results

[1730] Step 5:

[1731] The server displays the analysis results on a dashboard.

[1732] Shows: Positive and negative review details, feedback, and perceived sentiment

[1733] Step 6:

[1734] Employees check the dashboard and consider and implement countermeasures.

[1735] Action example: Review food temperature management and further improve service

[1736] Optimized menu suggestion system

[1737] Suggested flows based on order history and current selections

[1738] Step 1:

[1739] The user initiates the order.

[1740] Example: Select "Veggie Burger"

[1741] Step 2:

[1742] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[1743] Data sent: Current order "Veggie Burger", past order history

[1744] Step 3:

[1745] The server analyzes the received data and uses an emotion engine to recognize the current emotional state of the user.

[1746] Recognized emotion: "surprise"

[1747] Step 4:

[1748] The server takes into account the customer's past history and emotional state to generate optimal menu suggestions.

[1749] Suggestion: "How about today's special smoothie?" (reflects emotions)

[1750] Step 5:

[1751] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[1752] Display content: Today's limited smoothie selection options and details

[1753] Step 6:

[1754] The user confirms the proposal and places an additional order.

[1755] Example behavior: Ordering an extra smoothie

[1756] The above is a specific processing flow for each system in an embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry. Furthermore, by adding an emotion engine, it becomes possible to provide even more personalized services.

[1757] Example 2

[1758] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1759] The current restaurant industry lacks sufficient multilingual and sign language support, and does not provide empathetic responses based on customer feelings, which can lead to a decline in customer satisfaction. It is particularly difficult to provide efficient and accurate service to customers who do not speak a foreign language or who require sign language. There is also a lack of systems in place to effectively utilize customer feedback to improve operations.

[1760] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1761] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, an emotion recognition means for recognizing emotions from the user's speech and text data, and an empathetic response generation means for generating an empathetic response based on the emotion information. This makes it possible to support multiple languages ​​and sign language, and further to provide empathetic responses based on the customer's emotions, thereby improving customer satisfaction.

[1762] The "voice recognition means" is a means for capturing voice input by a user as digital data and converting the voice into text data.

[1763] The "translation means" is a means for translating the text data generated by the speech recognition means into another language.

[1764] The "response generation means" is a means for analyzing the translated text data and generating an appropriate response.

[1765] The "voice synthesis means" is a means for converting the response generated by the response generation means into voice data and providing it to the user.

[1766] The "emotion recognition means" is a means for recognizing emotions from the user's voice and text data.

[1767] The "empathic response generating means" is a means for generating an empathic response based on emotional information.

[1768] The "sign language analysis means" is a means for capturing sign language input by a user through a camera as an image and converting it into text data.

[1769] The "augmented reality menu providing means" is a means for reflecting text data in an augmented reality menu and providing visual feedback.

[1770] The "feedback collection means" is a means for capturing feedback from users as text data.

[1771] The "feedback analysis means" is a means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations.

[1772] The "dashboard providing means" is a means for visually displaying the evaluation results generated by the feedback analysis means and providing them to employees.

[1773] This invention is a system that provides generative AI solutions for the restaurant industry that integrate multilingual support, sign language support, and emotion recognition functions. This system consists of a server and a terminal, and interacts directly with users.

[1774] Multilingual interactive voice response system

[1775] When a user (customer) enters an order by voice, the device captures the voice with its microphone. The device then sends this voice data to a speech recognition engine such as Google Cloud Speech-to-Text, which converts the speech into text. The converted text data is then sent to a server where it is translated using a multilingual translation module such as Google Cloud Translation. The server then analyzes the translated text data and generates an appropriate response. The generated response is then converted into speech by a speech synthesis engine such as Amazon Polly and played back to the user from the device.

[1776] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," the device captures the voice and sends it to the server, which translates and generates a response, returning it to the user as "You would like a cheeseburger and a coke." This process enables multilingual support for customers.

[1777] Adding emotion recognition functionality

[1778] The server is equipped with an emotion recognition engine (e.g., IBM Watson Tone Analyzer) that analyzes voice and text data to recognize the user's emotions. This emotional information is reflected in the response generation means, which generates an empathetic response.

[1779] For example, if a user places an order by saying, "I would like a cheeseburger and a coke, please," and the emotion recognition engine detects "happiness," the server will generate an empathetic response such as, "You ordered a cheeseburger and a coke! Enjoy!" and play it back as audio.

[1780] Example prompt sentence:

[1781] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[1782] AR menu provision system that supports sign language

[1783] When a sign language user enters an order through the camera, the device captures the image and sends it to a sign language analysis engine such as Microsoft Kinect to convert it into text data. The server receives the converted text data and generates an augmented reality (AR) menu, which the device displays.

[1784] For example, when a user signs for "hamburger and fries," the device's camera captures the image, and the server converts it into text to generate an AR menu. The menu includes images of the hamburger and fries, along with detailed information, and the device provides visual feedback to the user.

[1785] Customer feedback collection and analysis system

[1786] When a user enters feedback into the device, it is captured as text data and sent to a server. The server then analyzes the feedback data using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative ratings. An emotion recognition engine then analyzes the emotional information in the feedback and displays the results on a dashboard.

[1787] For example, if a user enters feedback such as "The service was good, but the food was a little cold," the server analyzes it and generates a "positive rating (good in terms of service)" and a "negative rating (the food was cold)," which are then displayed on a dashboard where employees can view them.

[1788] Optimized menu suggestion system

[1789] When a user begins an order, the terminal sends a menu suggestion request to the server based on past order history and current selections. The server then analyzes the data using a personalization engine such as Amazon Personalize to generate optimal menu suggestions that take customer sentiment into account. These suggestions are then presented to the user by the terminal, allowing them to easily place additional orders.

[1790] For example, if a user has frequently ordered a "veggie burger" in the past, when they select it again, the server will generate a menu suggestion such as "How about today's special smoothie?", and the emotion engine will highlight the suggestion taking into account the user's current emotion. The device will then display the suggestion, allowing the user to easily place an additional order.

[1791] With these functions, this system will improve the efficiency of customer service and customer satisfaction at restaurants and wait staff in the food service industry. In addition, by combining it with an emotion engine, it will be possible to provide even more personalized services.

[1792] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1793] Processing flow of a multilingual interactive voice response system

[1794] Step 1: Capture audio

[1795] The device captures the user's voice with a microphone. The input is the user's voice, and the output is the captured voice data. This voice data is temporarily stored in the device's storage.

[1796] Specific behavior:

[1797] When a user speaks something like "I would like a cheeseburger and a coke, please," the device's microphone captures and stores the speech as digital data.

[1798] Step 2: Speech to text

[1799] The device sends the captured voice data to the Google Cloud Speech-to-Text API. The input is the captured voice data and the output is text data. The Google Cloud Speech-to-Text API converts the voice data into text data.

[1800] Specific behavior:

[1801] The device sends the voice data to Google Cloud Speech-to-Text, which returns the text, "I would like a cheeseburger and a coke, please."

[1802] Step 3: Translate the text

[1803] The server sends the text data received from the device to the Google Cloud Translation API. The input is text data, and the output is translated text data. The Google Cloud Translation API translates the text data into the specified other language.

[1804] Specific behavior:

[1805] The server sends the text "I would like a cheeseburger and a coke, please" to the Google Cloud Translation API and receives the translated text "I would like a cheeseburger and a coke, please."

[1806] Step 4: Generate a response

[1807] The server analyzes the translated text data and generates an appropriate response. The input is the translated text data and the output is the response text data. The server uses an algorithm to generate the response text.

[1808] Specific behavior:

[1809] Based on the translation result of "Please give me a cheeseburger and a coke", the server generates a response text such as "You've ordered a cheeseburger and a coke."

[1810] Step 5: Synthesize and play the audio

[1811] The server sends the response text to a speech synthesis engine (e.g., Amazon Polly) to generate voice data. The input is the response text, and the output is voice data. The generated voice data is sent to the device and played back.

[1812] Specific behavior:

[1813] The server sends the text "You'd like a cheeseburger and a coke" to Amazon Polly, which generates a voice file. The device plays the voice and responds to the user.

[1814] Process flow for adding emotion recognition functionality

[1815] Step 1: Acquiring audio and text

[1816] The server receives the user's voice and text data, which are used for further processing.

[1817] Specific behavior:

[1818] When a user places an order by saying, "I would like a cheeseburger and a coke, please," the voice and text data are sent to the server.

[1819] Step 2: Sentiment Analysis

[1820] The server sends the voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to obtain emotion information. The input is the voice data, and the output is the emotion information.

[1821] Specific behavior:

[1822] The server sends the voice data to the IBM Watson Tone Analyzer, which returns emotional information such as "joy."

[1823] Step 3: Generating an empathetic response

[1824] The server generates an empathetic response based on the emotional information and sends it to a speech synthesis engine. The input is the emotional information and the response text, and the output is voice data as the empathetic response.

[1825] Specific behavior:

[1826] If the emotion engine identifies "joy," the server generates an empathetic response text such as "You ordered a cheeseburger and a Coke! Enjoy!" and converts it into speech to play.

[1827] Example prompt sentence:

[1828] "Describe a program that recognizes emotions from speech and generates an empathetic response based on that emotion."

[1829] Processing flow of the sign language compatible AR menu provision system

[1830] Step 1: Capture sign language video

[1831] The device captures the user's sign language with a camera. The input is video data of the sign language, and the output is the captured video data.

[1832] Specific behavior:

[1833] When the user signs for "hamburger and fries," the device's camera captures the movement.

[1834] Step 2: Translating sign language to text

[1835] The device sends the captured video data to a sign language analysis engine (e.g., Microsoft Kinect) and converts it into text data. The input is video data, and the output is text data.

[1836] Specific behavior:

[1837] The device sends sign language video to a Microsoft Kinect, which responds with the text "hamburger and fries."

[1838] Step 3: Generate the AR menu

[1839] The server generates an AR menu based on the text data. The input is the text data, and the output is the generated AR menu.

[1840] Specific behavior:

[1841] The server receives the text data for "hamburger and fries" and generates a corresponding AR menu.

[1842] Step 4: Display the AR menu

[1843] The device displays the generated AR menu to the user. The input is the AR menu, and the output is the displayed AR menu.

[1844] Specific behavior:

[1845] The device displays an AR menu, providing the user with images and details about the burger and fries.

[1846] Customer feedback collection and analysis system process flow

[1847] Step 1: Enter your feedback

[1848] The user inputs feedback and the device captures it. The input is the feedback content and the output is text data.

[1849] Specific behavior:

[1850] The user enters feedback saying, "The service was good, but the food was cold."

[1851] Step 2: Submit your feedback

[1852] The terminal sends the captured feedback data to the server. The input is text data, and the output is the feedback data transferred to the server.

[1853] Specific behavior:

[1854] The terminal sends the feedback text to the server.

[1855] Step 3: Analyze the feedback

[1856] The server analyzes the received feedback using a natural language processing engine (e.g., Google Cloud Natural Language) to generate positive and negative evaluations. The input is the feedback text, and the output is the evaluation data as the analysis result.

[1857] Specific behavior:

[1858] The server analyzes the feedback text "The service was good, but the food was cold" and generates a positive rating (the service was good) and a negative rating (the food was cold).

[1859] Step 4: View the evaluation results

[1860] The server displays the analysis results on a dashboard. The input is the evaluation data, and the output is the displayed dashboard.

[1861] Specific behavior:

[1862] The server displays the evaluation results on a dashboard, where employees can view them.

[1863] Optimized menu suggestion system processing flow

[1864] Step 1: Initiate your order

[1865] When a user initiates an order, the terminal sends the user's past order history and current selections to the server. The input is the order data, and the output is the request data to the server.

[1866] Specific behavior:

[1867] When the user selects "veggie burger," the terminal sends the data along with past order history to the server.

[1868] Step 2: Generate menu suggestions

[1869] The server uses a personalization engine (e.g., Amazon Personalize) to analyze the data and generate optimal menu suggestions. The input is the request data, and the output is the suggested menu.

[1870] Specific behavior:

[1871] The server analyzes the user's past order history and current selections and generates a menu suggestion such as, "How about today's special smoothie?"

[1872] Step 3: Viewing Proposals

[1873] The terminal displays the generated menu suggestions to the user. The input is the suggested menu and the output is the suggestion displayed on the user's screen.

[1874] Specific behavior:

[1875] The device will display a suggestion to the user, "How about today's special smoothie?", encouraging them to order more.

[1876] The above is the specific processing flow of each system, which will enable the restaurant industry to improve customer service efficiency and customer satisfaction.

[1877] (Application example 2)

[1878] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1879] In today's brick-and-mortar stores, multilingual support is required to accommodate the increasing number of foreign tourists and diversifying customer needs, but language barriers hinder the efficiency of customer service and improve customer satisfaction. Dealing with customers who use sign language and reading their emotions to respond appropriately are also challenges. A system is needed to resolve these issues and improve the customer experience.

[1880] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1881] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation. This not only enables multilingual communication across language barriers, but also makes it possible to provide personalized responses according to sign language and emotions.

[1882] A "voice recognition means" is a device or system that converts voice input by a user into text data.

[1883] A "multilingual translation module" is a device or software for translating text data into different languages.

[1884] "Translation means" is a function that converts text data into other languages ​​using a multilingual translation module.

[1885] The "response generation means" is a function that analyzes the translated text data and generates an appropriate response.

[1886] A "voice synthesis means" is a device or software that converts the generated response into speech and provides it to the user.

[1887] The "emotion recognition means" is a function that recognizes emotions from the user's voice and reflects the emotion data in response generation.

[1888] The "sign language analysis means" is a function that converts sign language input by the user through the camera into text data using an image analysis engine.

[1889] The "augmented reality menu providing means" is a function that reflects the text data obtained by the sign language analysis means in an augmented reality menu and provides visual feedback.

[1890] A "feedback collection means" is a device or software that captures feedback from users as text data.

[1891] The "feedback analysis means" is a function that analyzes the captured feedback using a natural language processing engine and generates positive and negative evaluations.

[1892] The "dashboard provision means" is a function that displays the results of feedback analysis on a dashboard so that employees can check them.

[1893] The system programs required to implement the present invention will now be described.

[1894] The server includes: a speech recognition means for converting speech input by the user into text data; a multilingual translation module for translating the text data into other languages; a response generation means for analyzing the translated text data and generating an appropriate response; a speech synthesis means for converting the generated response into speech and providing it to the user; an emotion recognition means for recognizing emotions from the user's speech and reflecting the emotion data in response generation; a sign language analysis means for converting sign language input by the user through a camera into text data using a video analysis engine; an augmented reality menu providing means for reflecting the text data obtained by the sign language analysis means in an augmented reality menu and providing visual feedback; a feedback collection means for capturing feedback from the user as text data; a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations; and a dashboard providing means for displaying the results of the feedback analysis on a dashboard so that employees can check it.

[1895] The speech recognition means uses, for example, the speech_recognition library. This library captures the speech input by the user and converts it into text data. The converted text data is sent to a multilingual translation module (for example, Google Trans New API) and translated into another language. The translated text data is analyzed by the server, and an appropriate response is generated. This response is converted into audio data using a speech synthesis means (for example, pyttsx3) and provided to the user by the terminal.

[1896] The emotion recognition means uses the EmotionRecognition model. When the user's voice is captured, the emotion recognition means analyzes the voice data and detects emotions. The detected emotional data is reflected in the responses generated by the response generation means, resulting in more personalized responses.

[1897] The sign language analysis means uses the SignLanguageRecognition model. When a user inputs sign language through the camera, this model analyzes the video data and converts it into corresponding text data. The converted text data is sent to the augmented reality menu providing means, which provides visual feedback. This allows users who use sign language to order smoothly.

[1898] The feedback collection means includes a function for capturing text data. When feedback is entered by a user, it is captured as text data. The feedback analysis means then analyzes the feedback using a natural language processing engine (such as spaCy), and positive or negative evaluations are generated. These evaluation results are displayed in a form that can be checked by employees using the dashboard provision means. The dashboard is implemented using tools such as dash or Tableau.

[1899] Specific examples

[1900] Suppose a customer places an order by saying, "I would like a cheeseburger and a coke, please." At this time, the speech recognition means converts the speech into text, which is then translated by the multilingual translation module. If the emotion recognition means identifies joy from the customer's speech, the response generation means generates a response such as, "You'd like a cheeseburger and a coke! Enjoy!" The speech synthesis means converts this response into speech and provides it to the customer from the terminal.

[1901] Prompt Sentence Examples

[1902] Input prompt:

[1903] User: "I would like a cheeseburger and a coke, please"

[1904] System: "You ordered a cheeseburger and a Coke. Is there anything else I can help you with?"

[1905] Response prompt:

[1906] User: "No, that's all for now."

[1907] System: "Thank you. Your order has been placed. Please wait a moment."

[1908] As described above, this system not only enables multilingual communication across language barriers, but also provides personalized responses based on emotions and sign language support, which is expected to improve customer satisfaction in brick-and-mortar stores.

[1909] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1910] Step 1:

[1911] The user enters the order by voice.

[1912] Input: Speech data (e.g., "I would like a cheeseburger and a coke, please")

[1913] What happens: The device's microphone captures audio data.

[1914] Output: Captured audio data

[1915] Step 2:

[1916] A speech recognition means converts the captured speech data into text data.

[1917] Input: Audio data

[1918] Data processing: A speech recognition engine (e.g., speech_recognition library) analyzes the voice data and converts it into text data.

[1919] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[1920] Step 3:

[1921] A multilingual translation module translates the text data into other languages.

[1922] Input: Text data

[1923] Data calculation: A translation engine (e.g., Google Trans New API) translates the input text data into the specified language.

[1924] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[1925] Step 4:

[1926] An emotion recognition means recognizes emotions from the translated text data and voice data.

[1927] Input: Translated text data, audio data

[1928] Data computation: An emotion recognition engine (e.g., EmotionRecognition model) analyzes the speech and text and generates emotion data.

[1929] Output: Emotion data (e.g., "joy")

[1930] Step 5:

[1931] A response generation means generates an appropriate response based on the translated text data and the emotion data.

[1932] Input: Translated text data, emotion data

[1933] Data computation: A response generation engine generates empathetic responses based on input data.

[1934] Output: Response text data (e.g., "You ordered a cheeseburger and a Coke! Enjoy!")

[1935] Step 6:

[1936] A speech synthesis means converts the response text data into speech.

[1937] Input: Response text data

[1938] Data processing: A speech synthesis engine (e.g., pyttsx3) converts text data into speech data.

[1939] Output: Response audio data

[1940] Step 7:

[1941] The terminal provides the response voice data to the user.

[1942] Input: Response audio data

[1943] Action: The device speaker plays the audio data.

[1944] Output: Speech played to the user (e.g., "You've ordered a cheeseburger and a Coke! Enjoy!")

[1945] Step 8:

[1946] When a user inputs an order in sign language through the camera, the sign language analysis means captures the video data.

[1947] Input: Video data (e.g., sign language for "cheeseburger and coke")

[1948] Action: The device's camera captures video data.

[1949] Output: Captured video data

[1950] Step 9:

[1951] A sign language analysis means converts the captured video data into text data.

[1952] Input: Video data

[1953] Data processing: A sign language analysis engine (e.g., SignLanguageRecognition model) analyzes the video data and converts it into text.

[1954] Output: Text data corresponding to the sign language (e.g., "Cheeseburger and Coke")

[1955] Step 10:

[1956] The augmented reality menu providing means generates an augmented reality menu based on the converted text data and provides visual feedback.

[1957] Input: Text data obtained from sign language analysis

[1958] Data processing: The augmented reality engine generates a menu based on the text data and displays it on the screen.

[1959] Output: Augmented reality menu displayed to the user (e.g., image of a cheeseburger and a coke with details)

[1960] Step 11:

[1961] Feedback from users is captured as text data by a feedback collection means.

[1962] Input: Text data (e.g., "The service was very good, but the food was a little cold.")

[1963] Operation: Feedback data is sent from the device to the server.

[1964] Output: Captured feedback text data

[1965] Step 12:

[1966] A feedback analysis means analyzes the captured feedback using a natural language processing engine to generate positive and negative evaluations.

[1967] Input: Feedback text data

[1968] Data calculation: A natural language processing engine (e.g., spaCy) analyzes the feedback data and generates positive and negative ratings.

[1969] Output: Rating data (e.g., "The service was good" or "The food was cold")

[1970] Step 13:

[1971] The dashboard provider displays the analysis results on a dashboard, which helps employees make improvements.

[1972] Input: Parsed evaluation data

[1973] How it works: A dashboard system (e.g., dash or Tableau) visually displays the assessment data.

[1974] Output: The dashboard to be displayed (e.g., positive and negative feedback ratings)

[1975] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1976] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1977] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1978] [Fourth embodiment]

[1979] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1980] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1981] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1982] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1983] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1984] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1985] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1986] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1987] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1988] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1989] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1990] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1991] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1992] The present invention provides a generative AI solution that enables multilingual and sign language support for restaurants and wait staff in the food service industry. The system for implementing the present invention consists of the following elements:

[1993] Multilingual interactive voice response system

[1994] When a user (customer) enters an order by voice, the device captures the voice and converts it into text using a speech recognition engine. The server then sends the text data to a multilingual translation module, which translates it into text appropriate for the customer's language. The server analyzes the translated text data and generates an appropriate response. This response is converted into speech using a speech synthesis engine and played back to the user from the device.

[1995] Examples:

[1996] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[1997] AR menu provision system that supports sign language

[1998] When a customer who uses sign language inputs an order through the camera, the device captures the video and converts it into text using a sign language analysis engine. The server receives the converted text and generates a sign-language-compatible AR menu, which provides visual feedback to the customer.

[1999] Examples:

[2000] When a customer orders "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[2001] Customer feedback collection and analysis system

[2002] When a user (customer) enters feedback, the device captures the feedback as text data and sends it to the server. The server analyzes the received feedback using a natural language processing engine and generates positive and negative ratings. The results are displayed on a dashboard for employees to review.

[2003] Examples:

[2004] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[2005] Optimized menu suggestion system

[2006] When a user (customer) starts an order, the terminal sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes the received data and generates optimal menu suggestions, which are presented to the customer by the terminal. The customer can then confirm and select additional orders.

[2007] Examples:

[2008] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[2009] The above is an embodiment of the present invention. This system can improve the efficiency of customer service operations in the restaurant industry and effectively solve the problem of labor shortages through functions such as multilingual support, sign language support, feedback collection and analysis, and optimized menu suggestions.

[2010] The processing flow will be explained below.

[2011] Multilingual interactive voice response system

[2012] Step 1:

[2013] The user queries the order by voice.

[2014] Examples: "Cheeseburger", "Coke"

[2015] Step 2:

[2016] The device captures the voice and converts it into text data using a voice recognition engine.

[2017] Speak: "I'd like a cheeseburger and a Coke, please."

[2018] Converted text: "I'd like a cheeseburger and a coke, please."

[2019] Step 3:

[2020] The server sends the received text data to a translation module to translate it into the customer's language.

[2021] Data sent: "I'd like a cheeseburger and a coke, please."

[2022] Translated text: "I would like a cheeseburger and a coke, please" (English)

[2023] Step 4:

[2024] The server parses the translated text data and generates an appropriate response.

[2025] Parse: "You have ordered a cheeseburger and a coke. Is there anything else you would like?"

[2026] Step 5:

[2027] The terminal converts the generated response into speech using a speech synthesis engine and provides it to the user.

[2028] Synthesized voice: "You've ordered a cheeseburger and a Coke. Is there anything else you'd like to order?"

[2029] AR menu provision system that supports sign language

[2030] Step 1:

[2031] A user enters an order using sign language.

[2032] Example: Say "hamburger" and "fries" in sign language.

[2033] Step 2:

[2034] The device captures sign language through its camera and converts it into text data using a video analysis engine.

[2035] Captured footage: Sign language for "hamburger" and "fries"

[2036] Converted text data: "Hamburger and fries"

[2037] Step 3:

[2038] The server receives the converted text data and generates an AR menu that supports sign language.

[2039] Generated: "Hamburger and fries"

[2040] Generated AR menu: Burger and fries images, prices, and details

[2041] Step 4:

[2042] The device displays the generated AR menu and provides visual feedback to the user.

[2043] Showing: Image and details of a burger and fries

[2044] Customer feedback collection and analysis system

[2045] Step 1:

[2046] The user enters feedback.

[2047] For example: "The service was good, but the food was cold."

[2048] Step 2:

[2049] The device captures the feedback as text data and sends it to the server.

[2050] Feedback captured: "The service was good, but the food was cold."

[2051] Step 3:

[2052] The server analyzes the feedback it receives using a natural language processing engine.

[2053] Analysis results: Positive reviews (good service), negative reviews (cold food)

[2054] Step 4:

[2055] The server displays the analysis results on a dashboard.

[2056] Display: Positive and negative reviews, feedback details

[2057] Step 5:

[2058] The user (employee) checks the dashboard and considers and implements countermeasures.

[2059] Action example: Review food temperature management and further improve service

[2060] Optimized menu suggestion system

[2061] Step 1:

[2062] The user initiates the order.

[2063] Example: Select "Veggie Burger."

[2064] Step 2:

[2065] The terminal sends a menu suggestion request to the server based on the customer's past order history and current selections.

[2066] Data sent: Current order "Veggie Burger", past order history

[2067] Step 3:

[2068] The server analyzes the received data and generates optimal menu suggestions.

[2069] Suggestion: "Would you like to try today's special smoothie?"

[2070] Step 4:

[2071] The terminal displays the generated proposal on the screen, allowing the user to easily place an additional order.

[2072] Display content: Today's limited smoothie selection options and details

[2073] Step 5:

[2074] The user confirms the proposal and places an additional order.

[2075] Example behavior: Ordering an extra smoothie.

[2076] The above is a specific processing flow of each system in the embodiment of the present invention. By using this system, it is possible to improve the efficiency of customer service operations and customer satisfaction in the restaurant industry.

[2077] Example 1

[2078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2079] In today's restaurant industry, serving customers with language barriers or communication disabilities is a major issue that leads to labor shortages and reduced operational efficiency. In particular, when multilingual and sign language support is required, the limitations of employee skills and resources become apparent. There is also a need for systems that can instantly collect and analyze customer feedback and develop appropriate countermeasures. Furthermore, it is important to be able to suggest optimal menu items based on customers' past ordering history, as this will lead to improved customer satisfaction and increased sales.

[2080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2081] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, a communication means for transmitting user speech data captured by the terminal to the server, and a playback means for playing back the generated speech response from the terminal, thereby enabling multilingual automated voice responses that overcome language barriers.

[2082] Furthermore, a system is provided that includes a sign language analysis means that converts sign language input by a user through a camera into text data using a video analysis engine, an augmented reality menu providing means that reflects the text data in an augmented reality menu and provides visual feedback, a communication means that transmits video data captured by the terminal to a server, and a display means that displays the generated augmented reality menu on the terminal, which enables smooth communication with customers who use sign language and improves the quality of service.

[2083] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback with a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, a communication means for transmitting the feedback data captured by the terminal to a server, and a confirmation means for employees to consider and implement countermeasures based on the analysis results. This makes it possible to quickly collect and analyze customer feedback and develop appropriate countermeasures.

[2084] "Speech recognition means" refers to a function for converting speech input by a user into text data.

[2085] "Translation means" refers to a function for translating text data into other languages ​​using a multilingual translation module.

[2086] "Response generation means" refers to a function for analyzing translated text data and generating an appropriate response.

[2087] "Speech synthesis means" refers to a function for converting the generated response into speech and providing it to the user.

[2088] "Communication means" refers to a function for transmitting user voice data and video data captured by the terminal to the server, and for transmitting responses and menus from the server to the terminal.

[2089] The "playback means" refers to a function for playing back the generated voice response from the terminal.

[2090] "Sign language analysis means" refers to the function for converting sign language input through a camera into text data using an image analysis engine.

[2091] The "augmented reality menu providing means" refers to a function for reflecting the converted text data in an augmented reality menu and providing visual feedback.

[2092] "Display means" refers to a function for displaying the generated augmented reality menu on the terminal.

[2093] "Feedback collection means" refers to a function for capturing feedback from users as text data.

[2094] "Feedback analysis means" refers to a function for analyzing captured feedback using a natural language processing engine and generating positive and negative evaluations.

[2095] "Dashboard provision means" refers to a function for displaying analysis results on a dashboard so that employees can check them.

[2096] "Verification means" refers to the function that allows employees to consider and implement countermeasures based on the analysis results.

[2097] Multilingual interactive voice response system

[2098] This invention is a system that automatically converts orders entered by voice into text, translates them in multiple languages, and generates voice responses. When a user enters an order by voice, the device captures the speech and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is sent to a server, which translates it into the user's native language using a multilingual translation module such as Google Cloud Translation. The server analyzes the translated text data and generates an appropriate response using a generative AI model such as GPT-4. The generated response is converted into speech using a speech synthesis engine such as Amazon Polly, and the response is finally played back by the device.

[2099] Examples:

[2100] When a customer places an order by saying, "I would like a cheeseburger and a coke, please," the device captures this speech and sends it to the server. The server translates this as "I would like a cheeseburger and a coke, please," and generates the response, "You would like a cheeseburger and a coke, right?" The device then plays back this response aloud for the customer to confirm.

[2101] AR menu provision system that supports sign language

[2102] When a sign language user places an order through the camera, the device captures the video and converts it into text data using a sign language analysis engine such as Google Cloud Video Intelligence. This text data is sent to a server, which then uses an augmented reality (AR) content creation tool such as Unity to generate an AR menu that supports sign language. The generated AR menu is then displayed on the device, providing visual feedback to the user.

[2103] Examples:

[2104] When a customer orders a "hamburger and fries" using sign language, the device's camera captures the sign and sends it to the server. The server analyzes it and generates an AR menu containing images and detailed information about the hamburger and fries. The device then displays the AR menu, providing the customer with visual feedback.

[2105] Customer feedback collection and analysis system

[2106] When a user enters feedback as text, the device captures the feedback and sends it to a server. The server receives the feedback data and analyzes it using a natural language processing engine such as GPT-4 or BERT. The analysis results in positive and negative ratings, which are displayed on a dashboard. Employees can view customer feedback through the dashboard and consider and implement appropriate measures.

[2107] Examples:

[2108] When a customer enters feedback such as "The service was very good, but the food was a little cold," the device captures it and sends it to the server. The server analyzes the feedback and generates a "positive rating (good service)" and a "negative rating (the food was cold)." The analysis results are displayed on a dashboard, and employees consider countermeasures.

[2109] Optimized menu suggestion system

[2110] When a user begins an order, the device sends a menu suggestion request to the server based on the customer's past order history and current selections. The server analyzes this data in a database such as BigQuery, generates optimal menu suggestions, and sends them to the device. The device then presents these suggestions to the user, who can then confirm and select additional orders.

[2111] Examples:

[2112] If a customer who frequently ordered "veggie burgers" in the past selects "veggie burgers" again, the device sends a request to the server based on that selection. The server generates a menu suggestion, "How about today's special smoothie?" and sends it to the device. The device displays the suggestion and provides a message saying, "Try our special smoothie," allowing the customer to easily place an additional order.

[2113] Prompt Sentence Examples

[2114] "Please provide a detailed description of the voice commands or sign language inputs used by the user to place an order and the actions taken based on those inputs. Also, please provide a detailed description of feedback inputs and optimized menu suggestions."

[2115] The above is an embodiment of the present invention, which makes it possible to streamline customer service operations in the restaurant industry and effectively solve the problem of labor shortages through multilingual support, sign language support, feedback collection and analysis, and optimized menu proposals.

[2116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2117] Multilingual Interactive Voice Response System Processing Steps

[2118] Step 1:

[2119] The user enters the order by voice.

[2120] Input: User speech (e.g., "I would like a cheeseburger and a coke, please")

[2121] Output: Audio data is generated on the device

[2122] Specific operation: The user speaks an order into a terminal with a microphone.

[2123] Step 2:

[2124] The device captures the audio.

[2125] Input: User's voice data

[2126] Output: Stored as digital audio data

[2127] Specific operation: The device's microphone picks up sound and records it as digital audio data.

[2128] Step 3:

[2129] The terminal transmits the audio data to the server.

[2130] Input: Digital audio data

[2131] Output: Audio data is sent to the server

[2132] Specific operation: The device sends the captured audio data to the server using a communication means.

[2133] Step 4:

[2134] The server converts the speech to text.

[2135] Input: Audio data arriving at the server

[2136] Output: Text data (e.g., "I would like a cheeseburger and a coke, please")

[2137] What happens: The server converts the audio data into text using the Google Cloud Speech-to-Text API.

[2138] Step 5:

[2139] The server translates the text data.

[2140] Input: Converted text data

[2141] Output: Translated text data (e.g., "I'd like a cheeseburger and a coke, please.")

[2142] Specific operation: The server uses the Google Cloud Translation API to translate the text into the user's native language.

[2143] Step 6:

[2144] The server parses the translated text data and generates a response.

[2145] Input: Translated text data

[2146] Output: Response text (e.g., "You'd like a cheeseburger and a coke").

[2147] What it does: The server uses the GPT-4 generative AI model to analyze the translated text and generate an appropriate response.

[2148] Step 7:

[2149] The server converts the response into speech.

[2150] Input: Response text

[2151] Output: Speech data (e.g., "You'd like a cheeseburger and a coke, right?")

[2152] Specific operation: The server uses a speech synthesis engine such as Amazon Polly to convert the response text into audio data.

[2153] Step 8:

[2154] The server sends the audio data to the terminal.

[2155] Input: Generated audio data

[2156] Output: Audio data is sent to the device

[2157] Specific operation: The server sends the generated voice data to the terminal.

[2158] Step 9:

[2159] The device plays a voice response.

[2160] Input: Audio data received on the device

[2161] Output: The audio played to the user

[2162] Specific operation: The device plays the received audio data through the speaker and prompts the user to confirm.

[2163] Processing steps of an AR menu provision system that supports sign language

[2164] Step 1:

[2165] A user who uses sign language enters an order through the camera.

[2166] Input: User's sign language actions

[2167] Output: Video data is generated by the camera

[2168] Specific actions: The user indicates an order in sign language in front of the camera.

[2169] Step 2:

[2170] The device captures the sign language video.

[2171] Input: Video data of the user's sign language movements

[2172] Output: Saved as digital video data

[2173] Specific operation: The device's camera captures sign language movements and records them as digital video data.

[2174] Step 3:

[2175] The terminal transmits the video data to the server.

[2176] Input: Digital video data

[2177] Output: Video data is sent to the server

[2178] Specific operation: The video data captured by the device is sent to the server using a communication means.

[2179] Step 4:

[2180] The server converts the video data into text.

[2181] Input: Video data arriving at the server

[2182] Output: Text data (e.g. "hamburger and fries")

[2183] How it works: The server uses the Google Cloud Video Intelligence API to analyze the sign language video data and convert it into text.

[2184] Step 5:

[2185] The server generates an AR menu based on the text data.

[2186] Input: Converted text data

[2187] Output: Augmented reality menu

[2188] Specific operation: The server uses an AR content creation tool such as Unity to generate an AR menu that supports sign language.

[2189] Step 6:

[2190] The server sends the generated AR menu to the device.

[2191] Input: Generated AR menu

[2192] Output: The AR menu is sent to the device.

[2193] Specific operation: The server sends the generated AR menu to the device.

[2194] Step 7:

[2195] The device will display the AR menu.

[2196] Input: AR menu received on the device

[2197] Output: Visual feedback that is displayed to the user

[2198] Specific behavior: The device displays an AR menu overlaid on the camera screen, providing visual feedback to the user.

[2199] Customer feedback collection and analysis system processing steps

[2200] Step 1:

[2201] The user enters feedback in text.

[2202] Input: User feedback text

[2203] Output: Feedback data is generated on the terminal

[2204] Specific operation: The user inputs feedback on the device screen.

[2205] Step 2:

[2206] The device captures the feedback.

[2207] Input: Feedback text

[2208] Output: Stored as digital feedback data

[2209] Specific behavior: The device captures and stores the input feedback data.

[2210] Step 3:

[2211] The terminal transmits the feedback data to the server.

[2212] Input: Digital feedback data

[2213] Output: Feedback data is sent to the server

[2214] Specific operation: The feedback data captured by the terminal is sent to the server via a communication means.

[2215] Step 4:

[2216] The server analyzes the feedback.

[2217] Input: Feedback data arriving at the server

[2218] Output: Analysis results (positive and negative evaluations)

[2219] Specific operation: The server analyzes the feedback data using a natural language processing engine (GPT-4 or BERT) and generates an evaluation.

[2220] Step 5:

[2221] The server displays the analysis results on a dashboard.

[2222] Input: Analysis results

[2223] Output: Ratings displayed on the dashboard

[2224] Specific operation: The server displays the generated evaluation results on a dashboard for employees to review.

[2225] Processing steps of an optimized menu suggestion system

[2226] Step 1:

[2227] The user initiates the order.

[2228] Input: The user's current order selection (e.g., "Veggie Burger")

[2229] Output: Order data is generated on the terminal.

[2230] Specific Actions: The user starts the ordering process at the terminal and selects an item.

[2231] Step 2:

[2232] The terminal sends past order history and current selections to the server.

[2233] Input: User's past order history and current selection data

[2234] Output: Order history data is sent to the server

[2235] Specific operation: The terminal sends the user's saved past order history and current selections to the server.

[2236] Step 3:

[2237] The server parses the order data.

[2238] Input: Order data arriving at the server

[2239] Output: Analysis results (menu suggestions)

[2240] Specific operation: The server analyzes the data using a database such as BigQuery and generates optimal menu suggestions.

[2241] Step 4:

[2242] The server generates the menu suggestions.

[2243] Input: Analysis results

[2244] Output: Generated menu suggestions (e.g., "Would you like today's special smoothie?")

[2245] Specific operation: The server forms optimal menu suggestions based on the analysis results.

[2246] Step 5:

[2247] The server sends the proposal to the device.

[2248] Input: Generated menu suggestions

[2249] Output: Menu suggestions sent to the device

[2250] Specific operation: The server sends the generated menu suggestions to the terminal.

[2251] Step 6:

[2252] The device will display suggestions.

[2253] Input: Menu suggestions received on the device

[2254] Output: Menu suggestions presented to the user

[2255] Specific operation: The terminal displays the menu suggestions sent from the server, and the user confirms and selects the additional order.

[2256] (Application example 1)

[2257] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2258] In the traditional restaurant industry, it is difficult to accommodate customers who speak foreign languages ​​or who are hearing impaired, and processing orders and feedback requires a great deal of time and effort. Furthermore, when customers want to order in real time using voice or sign language in multiple languages, the system that provides support is inadequate, making effective communication difficult. Furthermore, there is a lack of an efficient system for quickly and accurately processing customer feedback and taking appropriate improvement measures.

[2259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2260] In this invention, the server includes a speech recognition means for converting speech input by a user into text data, a translation means for translating the text data into another language using a multilingual translation module, a response generation means for analyzing the translated text data and generating an appropriate response, a speech synthesis means for converting the generated response into speech and providing it to the user, and a means for processing the user's voice order in real time using an input device of the smartphone, thereby enabling support for multiple languages ​​and sign language.

[2261] The system also includes a sign language analysis unit that converts sign language input by the user through the camera into text data using a video analysis engine, an augmented reality menu providing unit that reflects the text data in an augmented reality menu and provides visual feedback, and a unit that processes sign language orders in real time using the smartphone's camera, thereby enabling real-time sign language support for the hearing impaired.

[2262] The system further includes a feedback collection means for capturing feedback from users as text data, a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations, a dashboard provision means for displaying the analysis results on a dashboard, and a means for obtaining feedback in real time through the smartphone application, thereby enabling feedback to be collected and analyzed quickly and accurately.

[2263] "Speech recognition means" refers to a device or software that has the function of converting voice input by a user into text data.

[2264] "Translation means" refers to a device or software that has the function of translating text data into other languages ​​using a multilingual translation module.

[2265] A "response generation means" is a device or software that has the function of analyzing the translated text data and generating an appropriate response.

[2266] The "voice synthesis means" is a device or software that has the function of converting the generated response into voice and providing it to the user.

[2267] "Smartphone input devices" is a general term for smartphone hardware and software that has the ability to receive input such as audio, video, and touch.

[2268] The "sign language analysis means" is a device or software that has the function of converting sign language input by a user through a camera into text data using an image analysis engine.

[2269] The "augmented reality menu providing means" is a device or software that has the function of visually displaying text data using augmented reality technology and providing feedback.

[2270] The "feedback collection means" is a device or software that has the function of capturing feedback from users as text data.

[2271] The "feedback analysis means" is a device or software that has the function of analyzing the captured feedback using natural language processing technology and generating positive and negative evaluations.

[2272] The "dashboard providing means" is a device or software that has the function of visually displaying the analysis results.

[2273] A "smartphone application" is application software that runs on a smartphone and provides various functions through a user interface.

[2274] The present invention provides a system for the restaurant industry that supports multiple languages ​​and sign language. This system allows customers to input orders using voice or sign language, and the system processes the orders on a server, which then responds in real time. Specific embodiments of the system are described in detail below.

[2275] Speech recognition and translation processing

[2276] When a user voice orders through an input device on a smartphone, the terminal captures the voice and converts it into text data using a speech recognition means. The text data is then sent to a server, where it is translated into another language by a translation means. A response generation means then analyzes the translated text data and generates an appropriate response. This response is then converted back into speech using a speech synthesis means and provided to the user through the terminal. Specifically, this can be achieved using the SpeechRecognition library, Googletrans library, and Pyttsx3 library.

[2277] Sign language analysis and augmented reality menus

[2278] When a user places an order using sign language through the smartphone camera, the sign language analysis means analyzes the image and converts it into text data. This data is reflected in the augmented reality menu providing means, and visual feedback is provided to the user. This allows the user to check the order details using sign language.

[2279] Collecting and analyzing feedback

[2280] When a user enters feedback through a smartphone application, the feedback is captured as text data by a feedback collection means and sent to a server. The feedback analysis means analyzes the captured data using a natural language processing engine to generate positive and negative evaluations. The analysis results are visually displayed through a dashboard provision means so that employees can check them.

[2281] For example, if a customer orders "I would like a cheeseburger and a coke, please" by voice, the system captures the speech and translates it into Japanese. The system then generates a response, "I'd like a cheeseburger and a coke, please," which is presented visually and audibly to the user. If a customer orders "hamburger and fries" by sign language, the camera captures the sign language and generates an augmented reality menu with images and detailed information about the hamburger and fries, which is presented visually to the ...

Claims

1. a speech recognition means for converting a speech input by a user into text data; a translation means for translating the text data into another language using a multilingual translation module; a response generation means for analyzing the translated text data and generating an appropriate response; a speech synthesis means for converting the generated response into speech and providing it to the user; A system including:

2. A sign language analysis means for converting sign language input by a user through a camera into text data using a video analysis engine; an augmented reality menu providing means for reflecting the text data in an augmented reality menu and providing visual feedback; The system of claim 1 , comprising:

3. a feedback collection means for capturing feedback from users as text data; a feedback analysis means for analyzing the captured feedback using a natural language processing engine and generating positive and negative evaluations; a dashboard providing means for displaying the analysis results on a dashboard; The system of claim 1 , comprising:

Images