System

Abstract

A system is provided.SOLUTION: A means for acquiring image data of a menu, a means for extracting character information from the acquired image data, a means for automatically translating the extracted character information, a means for acquiring detailed information of a dish based on the translated information, a means for displaying the acquired detailed information and a translation result on a user interface, a means for receiving a voice input of a user, a means for translating the received voice input into another language, a means for displaying the translation result to a staff, a means for transmitting order information of a dish selected by the user to an order system of a restaurant, and a means for performing settlement based on the order information; Means for notifying the user of the payment result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Restaurants face the problem of difficulty in providing foreign language support to tourists and foreign customers who speak different languages, as it can be difficult to understand the menu, communicate questions, and process orders and payments. This problem is particularly pronounced in independently owned restaurants, which lack sufficient foreign language support resources. Furthermore, customers often take time to understand the menu, which can make ordering and payment difficult, placing a burden on both customers and restaurants. [Means for solving the problem]

[0005] The present invention provides a system that acquires image data of a menu, extracts text information from the acquired image data, automatically translates the extracted text information, acquires detailed information about dishes based on the translated information, and displays the acquired detailed information and the translation result on a user interface. It also includes a system that receives a user's voice input, translates the received voice input into another language, displays the translation result to staff, transmits order information for the dishes selected by the user to the restaurant's ordering system, performs payment based on the order information, and notifies the user of the payment result. In this way, the entire process from understanding the menu to asking questions, ordering, and payment can be carried out consistently and smoothly, reducing the burden of foreign language support.

[0006] "Means for obtaining image data of menus" refers to a system for capturing restaurant menus as image data using devices such as cameras or scanners.

[0007] "Means for extracting character information from acquired image data" refers to a technology that uses optical character recognition (OCR) technology to analyze the character information contained in the image and extract it as text data.

[0008] "Means for automatically translating extracted text information" refers to an automatic translation system that converts extracted text information into a specified language.

[0009] "Means for obtaining detailed information about dishes based on translated information" refers to a system for obtaining detailed information about the ingredients, taste, appearance, etc. of dishes from a database based on translated menu information.

[0010] "Means for displaying the acquired detailed information and translation results on a user interface" refers to a display device or application for visually presenting the acquired detailed information and translation results to the user.

[0011] "Means for receiving user voice input" refers to a microphone or voice recognition system that captures voice data verbally input by the user.

[0012] "Means for translating received voice input into another language" refers to a speech translation system that translates a user's voice input into another specified language in real time.

[0013] "Means for displaying translation results to staff" refers to a display system or device for showing translated audio data in text format to staff.

[0014] "Means for sending order information for the dishes selected by the user to the restaurant's ordering system" refers to technology that digitally sends the order details of the dishes selected by the user to the restaurant's order management system.

[0015] "Means for making payments based on order information" refers to an electronic payment system for processing payments according to the user's order details.

[0016] "Means for notifying users of payment results" refers to a system for notifying users of the success or failure of payment results in real time. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The invention described in this specification is a system that performs a complete process from scanning the menu, translating it, displaying detailed information, translating voice input, ordering, and payment. Specific embodiments of this system are described below.

[0039] System configuration

[0040] This system is composed of multiple components, including the user's device, the restaurant's server, and a database. The device corresponds to the user's smartphone or tablet and is equipped with a camera, microphone, and display. The server provides image processing, translation, database access, voice recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more.

[0041] Program Overview

[0042] The system's program consists of a series of processes that provide image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment functions. Each step is explained below with concrete examples.

[0043] Menu scanning and translation

[0044] Example 1: Menu scanning and character recognition

[0045] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0046] 2. The device acquires the scanned image data and sends it to the server.

[0047] 3. The server receives the image data, uses OCR technology to extract the text information in the menu, and processes it as text data.

[0048] Example 2: Analysis and translation of text data

[0049] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish.

[0050] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts this information into a format for display on the user interface.

[0051] 6. The device displays the translation result and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0052] Receiving and translating voice input

[0053] Example 3: Simultaneous audio interpretation

[0054] 7. When a user has a question for a staff member, they can type it into their smartphone using voice input: "What kind of fish is used in this sushi?"

[0055] 8. The device captures the audio data and sends it to the server.

[0056] 9. The server receives the voice data and uses generative AI to translate English into Japanese in real time.

[0057] 10. The server sends the translation results to the terminal, which displays the message, "What kind of fish is used in this sushi?"

[0058] 11. Once the staff member responds, the content is sent back to the server, where it is translated into English in real time by the generating AI, and the message "This sushi is made with tuna" is displayed on the device.

[0059] Ordering and Payment

[0060] Example 4: Food ordering and payment

[0061] 12. The user selects their favorite dish and confirms the order via their smartphone.

[0062] 13. The terminal sends the order details to the server.

[0063] 14. The server receives the order information and sends it to the restaurant's ordering system.

[0064] 15. The terminal displays the payment screen to the user, for example, by selecting an electronic payment service and making the payment.

[0065] 16. User selects payment method and enters payment information.

[0066] 17. The server sends a payment request to the electronic payment service and receives the result.

[0067] 18. The terminal displays a payment completion screen to the user, completing the process.

[0068] As described above, this system significantly improves user convenience by providing a comprehensive process including multilingual menu display, voice interpretation, ordering, and payment, thereby reducing the burden of foreign language support in restaurants.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] A user scans a restaurant menu with their smartphone camera.

[0072] Step 2:

[0073] The terminal acquires image data of the scanned menu and sends the image data to the server for transfer.

[0074] Step 3:

[0075] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[0076] Step 4:

[0077] The server analyzes the text data obtained by OCR, divides it into food items, and identifies the name, price, description, etc. of each item.

[0078] Step 5:

[0079] The server automatically translates each identified dish item into the specified foreign language using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[0080] Step 6:

[0081] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from a database based on the translated dish items.

[0082] Step 7:

[0083] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[0084] Step 8:

[0085] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0086] Step 9:

[0087] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0088] Step 10:

[0089] The device captures the audio data and sends it to the server.

[0090] Step 11:

[0091] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into the specified language by the AI ​​generator. For example, an English question can be translated into Japanese.

[0092] Step 12:

[0093] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[0094] Step 13:

[0095] When the staff member answers the question, the answer is entered into the terminal, which then sends the answer back to the server.

[0096] Step 14:

[0097] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[0098] Step 15:

[0099] The device receives the translation results and displays them to the user.

[0100] Step 16:

[0101] The user selects the dish and places the order via smartphone.

[0102] Step 17:

[0103] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[0104] Step 18:

[0105] The terminal transitions to the payment screen, where the user selects the desired payment method. For example, the user selects an electronic payment service.

[0106] Step 19:

[0107] The user enters payment information and confirms the payment.

[0108] Step 20:

[0109] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[0110] Step 21:

[0111] The server receives the payment result and notifies the terminal.

[0112] Step 22:

[0113] The terminal displays a payment completion screen to the user, completing the series of operations.

[0114] These are the specific steps involved in the process, from scanning the menu to translating, asking questions, ordering, and payment. This ensures smooth communication and service between the user and the restaurant.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Conventional multilingual systems for restaurants handle each process separately: menu translation, detailed information display, voice interpretation, ordering, and payment. This resulted in low user convenience and a heavy burden on the restaurant. In addition, it was difficult to consistently process real-time voice interpretation, order confirmation, and payment, leading to a demand for improved user experience and operational efficiency.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for sending order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for using OCR technology and natural language processing technology, means for acquiring information from a database based on the translation result, and means for translating voice data in real time using a generative AI model. This allows for smooth multilingual support, making it possible to provide a system that is efficient and convenient for both users and restaurants.

[0120] "Image data" refers to visual information captured by a device such as a camera or scanner and stored in digital form.

[0121] "Textual information" refers to information that is displayed as text, typically expressed in language or characters.

[0122] "Translation" is the process of transcribing text written in one language into a different language.

[0123] "Detailed information" refers to more specific and detailed information about a particular item, including ingredients, images, and flavor descriptions.

[0124] A "user interface" is an interface that exchanges information between a computer system and a user, and typically includes a display, buttons, a touch panel, etc.

[0125] "Voice input" is the process of capturing a user's voice as digital data using a microphone device.

[0126] "Another language" means a language other than the original language.

[0127] "Staff" refers to employees and service providers working at restaurants and other establishments.

[0128] "Order Information" refers to detailed data about the products or services selected by the user, including the quantity and selections.

[0129] "Ordering system" refers to an order management system used in restaurants and stores to receive, process, and manage order information.

[0130] "Payment" refers to the process of paying for goods and services, and includes credit cards, electronic money, cashless payments, etc.

[0131] "OCR technology" stands for optical character recognition technology, which refers to the technology that converts character information from paper documents, image data, etc. into digital text.

[0132] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and generate human language, and includes translation, speech recognition, and dialogue systems.

[0133] A "generative AI model" is a model that uses artificial intelligence to generate data, and is used for tasks such as natural language generation and image generation.

[0134] The present invention is a system that utilizes a user's terminal, a store's server, and a database to provide a series of processes from scanning the menu to translating it, displaying detailed information, translating voice input, ordering, and payment. The specific configuration and operation for implementing the present invention are described in detail below.

[0135] System configuration

[0136] This system consists of a user-held device (such as a smartphone or tablet), a restaurant server, and a database. The device is equipped with a camera, microphone, and display, and is used to scan menus and input information by voice. The server provides functions using image processing, translation, database access, voice recognition, and generative AI models. The database stores detailed information about dishes, translation data, order history, etc.

[0137] Program Overview

[0138] The program of this system consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment. The specific processing content of each step is explained below.

[0139] Menu scanning and translation

[0140] Example 1: Menu scanning and character recognition

[0141] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0142] 2. The device acquires the scanned image data and sends it to the server. At this time, the device encodes the image data in JPEG format and uploads it to the server.

[0143] 3. The server receives the image data and extracts the text information from the menu using OCR technology (e.g., Google Cloud Vision API). The extracted text data is stored in a temporary database.

[0144] Example 2: Analysis and translation of text data

[0145] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API).

[0146] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts it into a format for display on the user interface.

[0147] 6. The device displays the translation result and detailed information to the user. For example, the information "Sushi - Raw fish on rice, typically served with soy sauce and wasabi" is displayed on the screen.

[0148] Receiving and translating voice input

[0149] Example 3: Simultaneous audio interpretation

[0150] 1. When a user wants to ask a question to a staff member, they type it into their smartphone: "What kind of fish is used in this sushi?"

[0151] 2. The device captures the audio data and sends it to the server, where it encodes it in FLAC format and uploads it to the server.

[0152] 3. The server receives the voice data, analyzes it using a speech recognition API (e.g., Google Cloud Speech-to-Text), and extracts text data.

[0153] 4. The server translates the resulting text using a generative AI model (e.g., GPT-3).

[0154] 5. The device displays the translation result to the user, asking, "What kind of fish is used in this sushi?"

[0155] 6. Once the staff member responds, their voice is sent back to the server, and the translation is converted into English in real time. "This sushi is made with tuna" is displayed on the device.

[0156] Ordering and Payment

[0157] Example 4: Food ordering and payment

[0158] 1. The user selects their favorite dish and confirms the order via smartphone.

[0159] 2. The terminal sends the order details to the server. The order data is encoded in JSON format.

[0160] 3. The server receives the order information and sends it to the restaurant's ordering system.

[0161] 4. The terminal displays the payment screen to the user, for example, using an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.).

[0162] 5. The user selects a payment method and enters payment information.

[0163] 6. The server sends a payment request to the electronic payment service and receives the result.

[0164] 7. The terminal displays a payment completion screen to the user, completing the process.

[0165] Examples of prompt statements

[0166] Here are some examples of input prompts for generative AI models:

[0167] 1. Prompt the user as they scan the menu:

[0168] "Please take a picture of the Japanese menu. After taking the picture, the image will be sent to the server."

[0169] 2. Prompt the user when using voice input:

[0170] "Please speak your question in English. When you're done, press the send button."

[0171] 3. Prompt the user to confirm the order:

[0172] Please confirm your selection and confirm your order.

[0173] 4. Prompt user when making payment:

[0174] Select your preferred payment method and enter your payment information.

[0175] As a result, this system handles everything from menu translation to ordering and payment, improving the user experience while reducing the burden on the store.

[0176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0177] Program processing flow

[0178] Menu scanning and translation

[0179] Step 1:

[0180] A user enters a restaurant and scans a Japanese menu with their smartphone camera. The user launches a camera app and takes a picture of the menu. The input is image data of the menu, and the output is a JPEG image file saved on the device.

[0181] Step 2:

[0182] The device acquires the scanned image data and sends it to the server. The device encodes the image data in JPEG format and uploads it to the server using an HTTP POST request. The input is a JPEG image file, and the output is an HTTP request to the server.

[0183] Step 3:

[0184] The server receives the image data and uses OCR technology (e.g., Google Cloud Vision API) to extract the text information in the menu. The server passes the image data to the API and receives text data as the analysis result. The input is JPEG image data, and the output is the extracted text data.

[0185] Step 4:

[0186] The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API). The server passes the text data to the translation API and receives the translation results as text data. The input is the text data extracted by OCR, and the output is the translated text data.

[0187] Step 5:

[0188] Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, flavor descriptions, etc.) from the database and converts it into a format to display on the user interface. The server queries the database using the translated text data as a key, retrieves the relevant detailed information, and formats it. The input is the translated text data, and the output is formatted data including the detailed information.

[0189] Step 6:

[0190] The terminal displays the translation results and detailed information received from the server to the user. The terminal receives the formatted data and displays it on the display. The input is the formatted data sent from the server, and the output is the display content that the user visually perceives.

[0191] Receiving and translating voice input

[0192] Step 7:

[0193] When a user wants to ask a question to a staff member, they input it into their smartphone: "What kind of fish is used in this sushi?" The user taps the voice input button and then asks the question. The input is the user's voice data, and the output is the voice data captured on the device.

[0194] Step 8:

[0195] The device captures audio data and sends it to the server. The device encodes the audio data in FLAC format and uploads it to the server using an HTTP POST request. The input is the captured audio data, and the output is an HTTP request to the server.

[0196] Step 9:

[0197] The server analyzes the received audio data using a speech recognition API (e.g., Google Cloud Speech-to-Text) and extracts it as text data. The server passes the audio data to the API and receives text data as the analysis result. The input is audio data in FLAC format, and the output is the extracted text data.

[0198] Step 10:

[0199] The server translates the obtained text using a generative AI model (e.g., GPT-3). The server passes the text data to the translation model and receives the translation result as text data. The input is text data extracted by speech recognition, and the output is translated text data.

[0200] Step 11:

[0201] The terminal displays the translation result to the user. It displays "What kind of fish is used in this sushi?" The terminal receives the translated text data and displays it on the display. The input is the translated text data sent from the server, and the output is the display content that the user sees.

[0202] Step 12:

[0203] When the staff member responds, their voice is sent back to the server, and the translation result is converted into English in real time. "This sushi is made with tuna" is displayed on the terminal. The input is the staff member's voice data and the results of the translation model, and the output is what the user sees visually.

[0204] Ordering and Payment

[0205] Step 13:

[0206] The user selects a dish and confirms the order via their smartphone. The user taps the order confirmation button to submit the order. The input is the order details selected by the user, and the output is the order data stored on the device.

[0207] Step 14:

[0208] The terminal sends the order details to the server. The terminal encodes the order data in JSON format and uploads it to the server using an HTTP POST request. The input is the order details data, and the output is an HTTP request to the server.

[0209] Step 15:

[0210] The server receives the order information and sends it to the restaurant's ordering system. The server sends the order data to the ordering system and waits for a response. The input is the order data in JSON format, and the output is the result sent to the ordering system.

[0211] Step 16:

[0212] The terminal displays the payment screen to the user. The user selects an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.). The input is the user's confirmation action, and the output is the content displayed on the payment screen.

[0213] Step 17:

[0214] The user selects a payment method and enters payment information. The user may use a credit card or an electronic payment app. The input is the user's payment method and payment information, and the output is the data entered into the terminal.

[0215] Step 18:

[0216] The server sends a payment request to the electronic payment service and receives the result. The server sends a payment request and receives a success or failure response. The input is the user's payment information, and the output is the payment result.

[0217] Step 19:

[0218] The terminal displays a payment completion screen to the user. The terminal receives the payment result and displays "Payment completed." The input is the payment result data, and the output is the display content that the user sees.

[0219] The above is the specific processing flow of this system.

[0220] (Application example 1)

[0221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] This system solves the problem of users who cannot understand foreign languages ​​having difficulty accurately understanding products and menus at restaurants and retail stores, and smoothly ordering and paying. Stores also often struggle with multilingual support, making it difficult for them to serve foreign customers. Furthermore, conventional systems often have low translation accuracy or are unable to provide real-time translations or display detailed information. This can result in a poor user experience and have a negative impact on store sales and customer satisfaction.

[0223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0224] In this invention, the server includes a means for acquiring menu image data, a means for extracting text information from the acquired image data, a means for automatically translating the extracted text information, a means for acquiring detailed content information based on the translated information, and a means for displaying the acquired detailed information and translation results on a user interface. This allows users to easily understand foreign language menus and smoothly order and pay. The system also includes a means for receiving user voice input, a means for translating the received voice input into another language, a means for notifying the translation results, a means for transmitting user-selected order information to the system, a means for making a payment based on the order information, and a means for notifying the user of the payment results, thereby efficiently managing complex dialogues, ordering, and payment processes. The system also includes a means for generating highly accurate translations and detailed information using a generative AI model, and a means for acquiring prompts from the user in advance and suggesting appropriate prompts based on the order status. This provides a system that is easy to use for both users and stores, and has simple and accurate foreign language support.

[0225] - "Menu image data" refers to image data of paper or digital menus of restaurants or retail stores that list items and prices, obtained using the camera or scanner on the user's device.

[0226] "Text information" is text data extracted from the image data of the menu, and includes information such as the name of the dish, the price, and a description.

[0227] An "automated translation tool" is software or a service that mechanically converts textual information obtained by a user into a different language, typically using a translation engine or a generative AI model.

[0228] "Content Details" is additional information about the translated menu item, including ingredients, allergens, cooking instructions, and pairings.

[0229] A "user interface" is an interactive screen or display device that allows a user to interact with a system, such as the display of a smartphone or tablet.

[0230] "Voice input" refers to the voice information that a user speaks into a terminal, and is input in the form of commands, questions, comments, etc.

[0231] "Means for notifying the translation result" refers to a method for visually or audibly communicating the translated information to the user or store staff.

[0232] "Order information" refers to data including order details such as the dishes and products selected by the user, quantities, and special notes.

[0233] "Means for sending to the system" refers to a method for sending order information from the user's terminal to the store's server or ordering system.

[0234] "Payment method" means the method by which a user electronically pays for the items they have ordered, such as by using a credit card or electronic payment service.

[0235] "Payment result" is information indicating that the payment has been completed, and includes the status of success or failure.

[0236] A "generative AI model" is an artificial intelligence model that has been trained to perform tasks such as natural language processing, image recognition, and translation.

[0237] A "prompt" is a concise, clear instruction or question entered by a user that is used as input to a generative AI model.

[0238] MODE FOR CARRYING OUT THE INVENTION

[0239] The invention described in this specification is a system that uses a user's terminal, a store's server, and a database to consistently display multilingual menus, perform voice interpretation, and handle ordering and payment. Specific embodiments for implementing this system are described below.

[0240] System configuration

[0241] 1. User's Device

[0242] This refers to the user's smartphone or tablet, which is equipped with a camera, microphone, and display, allowing for menu scanning and voice input. This allows the user to capture image data of the menu using the smartphone's camera and send that data to a server.

[0243] 2. Server

[0244] The server includes the following main modules to efficiently perform multiple processes:

[0245] Image Recognition Module: Using OCR technology, the server extracts text information from the image data sent by the user. Specifically, it uses Tesseract OCR.

[0246] Translation module: Uses the Google Translate API and generative AI models to translate extracted text into multiple languages, enabling natural-sounding, highly accurate translations.

[0247] Detailed information acquisition module: Based on the translation results, detailed information about the corresponding dish (ingredients, allergen information, images, etc.) is acquired from the database.

[0248] Speech Recognition Module: Captures user voice input and transcribes it using the Google Speech-to-Text API, while also translating it in real time using a generative AI model.

[0249] Payment module: Receives order information and processes payments through electronic payment services such as Stripe.

[0250] 3. Database

[0251] It connects to a server and stores detailed information about dishes, translation data, order history, etc. The database can use a common SQL database such as MySQL or PostgreSQL.

[0252] Example

[0253] Menu scanning and translation

[0254] A user enters a restaurant and scans the menu with their smartphone camera. The image data is sent to a server, where text information is extracted using OCR technology. The extracted text information is translated using a generative AI model, and the results are displayed on the user's device.

[0255] Receiving and translating voice input

[0256] When a user wants to ask a question to a staff member, they type it into their smartphone. Questions such as "What kind of fish is used in this sushi?" are sent to the server and recognized. The generative AI model translates the question in real time, and the results are displayed on the device.

[0257] Ordering and Payment

[0258] Based on the translation and detailed information, the user selects their favorite dish and confirms the order via smartphone. The order details are sent to the restaurant's ordering system via the server, and at the same time, a payment screen is displayed on the user's smartphone. The user selects an electronic payment service to make the payment, and the server notifies the user of the payment result.

[0259] Prompt Sentence Examples

[0260] Here are some examples of prompts:

[0261] User scans the menu: "Scan the menu."

[0262] The user wants to know more about a dish: "Tell me more about this dish."

[0263] The user asks aloud: "What kind of fish is used in this sushi?"

[0264] User places an order: "I'd like to order this."

[0265] User pays: "I'll pay with my credit card."

[0266] This makes the system easy to use for both users and stores, and enables the provision of simple, highly accurate services in foreign languages.

[0267] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0268] Step 1:

[0269] A user uses the camera on their smartphone to acquire image data of a menu. The user takes a picture of a restaurant menu with the camera and inputs the image data. The device then sends this image data to a server. The input is the image data of the menu, and the output is the transmission of the image data to the server.

[0270] Step 2:

[0271] The server uses OCR technology to extract text from the image data. Specifically, it uses Tesseract OCR to convert the text in the image into text data. The input is the image data, and the output is the extracted text.

[0272] Step 3:

[0273] The server translates the extracted text using a generative AI model. Here, a language translation service (e.g., Google Translate API) is used to convert the text into another language. The input is text, and the output is translated text data.

[0274] Step 4:

[0275] The server retrieves detailed information from the database based on the translated text data. Specifically, it searches for detailed data such as ingredient information and allergen information based on the translated dish name, etc. The input is the translated text data, and the output is the detailed information.

[0276] Step 5:

[0277] The server converts the detailed information and translation results into a format that can be displayed on the user interface and sends it to the terminal. Here, the information is integrated into a user-friendly format and displayed on the screen. The input is the detailed information and translation results, and the output is the display data on the user interface.

[0278] Step 6:

[0279] The user uses the microphone of the device to input voice, for example, to ask a question about cooking. The device sends the captured voice data to the server. The input is the voice data, and the output is the transmission of the voice data to the server.

[0280] Step 7:

[0281] The server transcribes the audio data and translates it in real time using a generative AI model. Specifically, it converts the audio data into text using the Google Speech-to-Text API and then translates the text. The input is audio data, and the output is translated text data.

[0282] Step 8:

[0283] The server sends the translation results to the terminal and displays them. The user can check the displayed translation results and get a response to their question from the staff. The input is the translated text data, and the output is the data displayed on the terminal.

[0284] Step 9:

[0285] The user confirms the order using the terminal. The user specifies the dish they have selected from the menu and presses the order button. The terminal sends the order information to the server. The input is the order information, and the output is the transmission of the order information to the server.

[0286] Step 10:

[0287] The server receives the order information and sends it to the restaurant's ordering system. Specifically, it uses the ordering system's API to link the order details. The input is the order information, and the output is the information sent to the ordering system.

[0288] Step 11:

[0289] The server generates a payment screen and sends it to the terminal. The user checks the payment screen and selects a payment method. The input is order information, and the output is the payment screen.

[0290] Step 12:

[0291] The user selects a payment method and enters the necessary information. The terminal sends the payment information to the server. The input is the payment information, and the output is the transmission of the payment information to the server.

[0292] Step 13:

[0293] The server connects to the electronic payment system to execute the payment. Specifically, it sends a request and receives the result using an electronic payment service such as Stripe. The input is the payment information, and the output is the payment result.

[0294] Step 14:

[0295] The server notifies the terminal of the payment result. The user confirms the completion of the payment on the terminal. The input is the payment result, and the output is the notification to the terminal.

[0296] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0297] The invention described in this specification is a system that provides more personalized services by combining menu scanning and translation, detailed information display, voice input and translation, ordering and payment, as well as an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[0298] System configuration

[0299] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[0300] Program Overview

[0301] The system's program consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, emotion recognition, user interface, and electronic payment. Each step is explained below with concrete examples.

[0302] Menu scanning and translation

[0303] Example 1: Menu scanning and character recognition

[0304] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0305] 2. The device acquires the scanned image data and sends it to the server.

[0306] 3. The server receives the image data and uses an OCR (optical character recognition) engine to extract the text information in the menu and process it as text data.

[0307] Example 2: Analysis and translation of text data

[0308] 4. The server analyzes the text data obtained by OCR, divides it into each dish item, and automatically translates the name, price, description, etc. of each item.

[0309] 5. Based on the translation results, the server retrieves detailed information about each dish (e.g., ingredients, taste description, images, etc.) from the database.

[0310] 6. The server formats the detailed information and translation results it has obtained into a data format to be displayed on the user interface and sends it to the terminal.

[0311] 7. The device displays the translation and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0312] Receiving and translating voice input

[0313] Example 3: Simultaneous audio interpretation

[0314] 8. If the user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0315] 9. The device captures the audio data and sends it to the server.

[0316] 10. The server receives the voice data and converts it into text data using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[0317] 11. The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" becomes "What kind of fish is used in this sushi?"

[0318] Emotion Recognition and Response Personalization

[0319] Example 4: Recognizing user emotions

[0320] 12. The device uses voice data and image data obtained from the camera to recognize the user's emotions through an emotion engine. For example, it can determine whether the user is feeling anxious based on the tone of their voice and facial expression.

[0321] 13. The server adjusts translation results and user interface display content in real time based on the recognized emotion data. For example, if the user is feeling anxious, the translation results will be changed to more friendly and understandable expressions.

[0322] 14. The device will display adjusted translation results and display content to the user, allowing the user to use the service comfortably.

[0323] Ordering and Payment

[0324] Example 5: Food ordering and payment

[0325] 15. The user selects the food and places the order via smartphone.

[0326] 16. The terminal sends the order details of the selected dishes to the server, and the server forwards the order information to the restaurant's order management system.

[0327] 17. The terminal transitions to the payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[0328] 18. The user enters payment information and confirms the payment.

[0329] 19. The terminal sends the payment information to the server, and the server sends a payment request to the specified payment service.

[0330] 20. The server receives the payment result and notifies the terminal.

[0331] 21. The terminal displays a payment completion screen to the user, completing the series of operations.

[0332] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[0333] The processing flow will be explained below.

[0334] Step 1:

[0335] A user scans a restaurant menu with their smartphone camera.

[0336] Step 2:

[0337] The terminal acquires image data of the scanned menu and transmits the image data to the server.

[0338] Step 3:

[0339] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[0340] Step 4:

[0341] The server analyzes the text data obtained by OCR, divides it into each dish item, and identifies the name, price, description, etc. of each item.

[0342] Step 5:

[0343] The server automatically translates each identified dish item into multiple languages ​​using natural language processing (NLP) algorithms, for example, from Japanese to English.

[0344] Step 6:

[0345] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from the database based on the translated dish items.

[0346] Step 7:

[0347] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[0348] Step 8:

[0349] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0350] Step 9:

[0351] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0352] Step 10:

[0353] The device captures the audio data and sends it to the server.

[0354] Step 11:

[0355] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[0356] Step 12:

[0357] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[0358] Step 13:

[0359] The device or camera uses an emotion engine to analyze the user's facial expressions and tone of voice, for example, to determine whether the user is feeling anxious.

[0360] Step 14:

[0361] The server adjusts the wording of the translation based on the results of the emotion engine. For example, if the user is feeling anxious, instead of a terse response like "This sushi is made with tuna," it will change it to a more friendly and reassuring response like "This sushi is made with fresh tuna, which is a favorite among many of our guests. Please enjoy it!"

[0362] Step 15:

[0363] The device displays the adjusted translation results to both the user and staff.

[0364] Step 16:

[0365] The staff member responds to the user and enters the response into the terminal again, which then sends the response back to the server.

[0366] Step 17:

[0367] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[0368] Step 18:

[0369] The terminal receives the translation result and displays it to the user.

[0370] Step 19:

[0371] The user selects the dish and places the order via smartphone.

[0372] Step 20:

[0373] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[0374] Step 21:

[0375] The terminal transitions to a payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[0376] Step 22:

[0377] The user enters payment information and confirms the payment.

[0378] Step 23:

[0379] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[0380] Step 24:

[0381] The server receives the payment result and notifies the terminal.

[0382] Step 25:

[0383] The terminal displays a payment completion screen to the user, completing the series of operations.

[0384] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[0385] Example 2

[0386] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] At restaurants, foreign customers face language barriers when it comes to understanding the menu, ordering, and paying. Furthermore, language and cultural differences can make users feel anxious or confused, potentially leading to a decline in service quality. For this reason, it's important to provide not just a translation service, but a personalized experience that takes users' emotions into consideration.

[0388] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0389] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for transmitting order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for recognizing the user's emotions, and means for adjusting the display content of the user interface based on the emotion data. This enables smooth ordering and payment that transcends language barriers and also enables personalized service that takes user emotions into consideration.

[0390] "Means for obtaining image data of menus" refers to a function that allows users to take a photo of a restaurant menu using the camera on their smartphone or tablet, and then save and transmit the image as digital data.

[0391] The "means for extracting text information" refers to a function that includes optical character recognition (OCR) technology for identifying text information from acquired image data and converting it into digital text.

[0392] The "means for automatic translation" is a function that uses extracted text information to translate into a selected different language in real time.

[0393] The "means for obtaining detailed information about a dish" is a function that obtains additional information about the dish (ingredients, description of the taste, images, etc.) from a database based on the translated information.

[0394] "Means for displaying on the user interface" refers to the screen or application functions that display the acquired detailed information and translation results in a format that is easy for the user to visually confirm.

[0395] The "means for receiving voice input" is a function for capturing what the user says into the microphone as voice data and processing that data.

[0396] The "means for translating into another language" is a function for analyzing received voice data, converting it into text, and then automatically translating the text into a different language.

[0397] "Means for displaying to staff" refers to a function for displaying translated information on a display or monitor so that restaurant staff can check it.

[0398] The "means for sending order information to the restaurant's ordering system" is a function for sending information about the dishes selected by the user to the restaurant's in-house order management system.

[0399] "Means for making payments" refers to a function for carrying out electronic payment processing based on the payment method selected by the user.

[0400] "Means for notifying the user of the settlement result" is a function for notifying the user of the result, such as whether the settlement was successful or unsuccessful.

[0401] "Means for recognizing emotions" is a function for analyzing the user's tone of voice and facial expressions to determine the user's emotional state.

[0402] "Means for adjusting display content" refers to a function that changes the translation results and the display content of the user interface based on recognized user emotional data, allowing users to use the service more comfortably.

[0403] This invention is a system that helps foreign customers at restaurants understand the menu, overcome language barriers when placing orders and making payments, and provides personalized services that respond to the customer's emotions. Specific embodiments of the system are described below.

[0404] System configuration

[0405] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[0406] Program Overview

[0407] The system's program consists of a series of processes that provide image recognition, natural language processing, speech translation, emotion recognition, user interface, and electronic payment functions.

[0408] The server uses an OCR (Optical Character Recognition) engine (e.g., Tesseract OCR) to extract text information from the image data. The extracted text information is automatically translated by a translation engine (e.g., Google Translate API). The translation result is combined with the dish details before being displayed in the user interface. The details are retrieved from the database and then displayed on the device in a format that the user can visually confirm.

[0409] When a user speaks, the device sends the captured voice data to the server, which converts the voice data into text using speech recognition technology (e.g., Google Cloud Speech-to-Text API) and translates the text into other languages ​​using a generative AI model (e.g., OpenAI GPT-4). The translation results are sent to the device and displayed to the user and staff.

[0410] The emotion engine uses deep learning models (e.g., Emotion API) to analyze the user's tone of voice and facial expressions. Based on the emotion data, the server can adjust the translation results and user interface display. For example, if it determines that the user is anxious, it can add a more helpful explanation.

[0411] When the user selects a dish and enters the order information, the terminal sends that information to the server. The server then forwards the order information to the restaurant's ordering system, allowing the user to select their preferred payment method on the payment screen. The payment information is sent via the server to the specified payment service (e.g., Stripe API), and once the payment is complete, the result is notified to the user.

[0412] Specific use cases

[0413] A user enters a restaurant and scans the Japanese menu with their smartphone camera. The device then sends the image data to the server, where an OCR engine extracts text information from the image. The extracted text information is translated into English or other languages ​​by a translation engine, and the translation result is displayed on the device along with detailed information about the dish. When the user asks, "What kind of fish is used in this sushi?", their voice is converted to text and translated into Japanese. The translated question is then displayed to the staff. Furthermore, if the user is feeling unsure, the system can display an additional reassuring message such as, "Don't worry, this sushi is made with fresh tuna." Once the user confirms their order and pays, they are instantly notified of the results.

[0414] Example prompts for generative AI models

[0415] "Translate the Japanese menu into English: 'Sushi - raw fish on rice, usually served with soy sauce and wasabi.'"

[0416] This system will enable smooth ordering and payment processes that transcend language barriers, and will also enable personalized services that respond to the user's emotions.

[0417] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0418] Step 1:

[0419] The user scans a Japanese menu with their smartphone camera. The user launches the camera app and takes a clear picture of the entire menu. Image data of the menu is generated as input. The image data is saved on the device and sent to the server.

[0420] Step 2:

[0421] The terminal sends scanned image data to the server. The image data sent from the terminal as input is sent to the server, which receives the data. The server then passes the received image data to the next process.

[0422] Step 3:

[0423] The server uses an OCR engine to extract text information from image data. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to identify text in the image and extract it as digital text. The input is the image data, and the output is the extracted text.

[0424] Step 4:

[0425] The server analyzes the extracted text and separates it into each dish item. For example, it performs an operation to extract the dish name, price, and description individually. The input is the text from the OCR engine, and the output is the separated text data.

[0426] Step 5:

[0427] The server sends the split text data to a translation engine, which automatically translates it into another language (e.g., English). The server uses a translation engine (e.g., Google Translate API) to convert the text data into the specified language. The input is text data split into dish name, price, and description, and the output is the translated text data.

[0428] Step 6:

[0429] The server retrieves detailed information about each dish (e.g., ingredients, flavor descriptions, images, etc.) from the database based on the translated text data. The server sends a query to the database to retrieve the corresponding records. The input is the translated text data, and the output is the data containing the detailed information.

[0430] Step 7:

[0431] The server formats the detailed information and translation results it has acquired and converts them into a data format to be displayed on the user interface. Specifically, it compiles the dish name, price, description, and image and converts them into JSON format data or HTML format. The input is the detailed information and translation results, and the output is formatted data.

[0432] Step 8:

[0433] The terminal receives the formatted data from the server and displays it in a format that is easy for the user to view. For example, it displays "Sushi: Rice topped with raw fish, typically served with soy sauce and wasabi" along with an image of the dish. The input is the formatted data from the server, and the output is the information that is displayed to the user.

[0434] Step 9:

[0435] When a user wants to ask a staff member a question, they speak into the microphone on their smartphone to input the question. For example, they might say, "What kind of fish is used in this sushi?" The input is the user's voice data.

[0436] Step 10:

[0437] The device captures the audio data and sends it to the server. The input is the audio data and the output is the data sent to the server.

[0438] Step 11:

[0439] The server receives the voice data and converts it into text data using voice recognition technology. The server then uses a voice recognition API (e.g., Google Cloud Speech-to-Text API) to convert the voice into text. The input is voice data, and the output is text data.

[0440] Step 12:

[0441] The server uses a generative AI model to translate text data into another specified language. The server uses generative AI (e.g., OpenAI GPT-4) to translate the text data. The input is the text data obtained from speech recognition, and the output is the translated text data.

[0442] Step 13:

[0443] The server sends the translation result back to the terminal, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?" The input is the text data of the translation result, and the output is the information displayed to the user or staff.

[0444] Step 14:

[0445] The device sends the user's voice data and image data obtained from the camera to the emotion engine. The input is voice data and image data. The emotion engine uses this data to recognize the user's emotion. The output is the recognized emotion data.

[0446] Step 15:

[0447] The server adjusts the translation results and display content based on the recognized emotion data. For example, if it determines that the user is feeling anxious, it adds an explanatory text and changes the expression to be more friendly. The input is emotion data and translation results, and the output is the adjusted display content.

[0448] Step 16:

[0449] The device displays the adjusted translation results and display content to the user, allowing the user to use the service comfortably. The input is the adjusted display content, and the output is the information displayed to the user.

[0450] Step 17:

[0451] The user selects a dish and presses the order button. The input is the user's selection. The terminal sends the information about the selected dish to the server.

[0452] Step 18:

[0453] The terminal sends the order details of the selected dishes to the server. The input is the dish information, and the output is the data sent to the server.

[0454] Step 19:

[0455] The server transfers the order information to the restaurant's ordering system and sends a confirmation of receipt of the order information to the terminal. The input is the order information, and the output is the transfer result to the ordering system and confirmation data to the terminal.

[0456] Step 20:

[0457] The terminal transitions to the payment screen, where the user can select the desired payment method, such as credit card, electronic money, etc. The input is the user's selection of payment method.

[0458] Step 21:

[0459] The user inputs payment information and confirms the payment. The input is payment information, which the terminal sends to the server.

[0460] Step 22:

[0461] The terminal sends payment information to the server, and the server sends a payment request to the specified payment service (e.g., Stripe API). The input is payment information, and the output is request data for the payment service.

[0462] Step 23:

[0463] The server receives the payment result and notifies the terminal. The input is the payment result from the payment service, and the output is notification data.

[0464] Step 24:

[0465] The terminal displays a payment completion screen to the user, completing the series of operations. The input is the payment result notification data, and the output is a payment completion message that is displayed to the user.

[0466] (Application example 2)

[0467] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0468] Conventional restaurant menu scanning and translation systems simply translate the menu contents into other languages ​​and do not provide personalized service based on the user's emotions or questions. Furthermore, the user's emotional state is not taken into account when asking a question or placing an order, which can result in a decline in service quality. Furthermore, the payment process is not smooth, which can cause inconvenience to users. A system that can solve these problems and provide a more comfortable and personalized service is needed.

[0469] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0470] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for recognizing emotions from user voice data and image data, and means for adjusting the display content and translation result based on the recognized emotion. This allows users not only to scan the menu and obtain translated information, but also to translate voice-input questions and orders in real time, enabling personalized responses according to the user's emotions and smooth payment.

[0471] The "means for acquiring image data of a menu" is a function that allows a user to use the camera of the device they own to scan a restaurant menu and acquire the image as data.

[0472] The "means for extracting text information" is a function for recognizing text from the acquired menu image and extracting it as digital text.

[0473] The "means for automatic translation" is a function for automatically translating extracted text information into another language.

[0474] The "means for obtaining detailed information about dishes" is a function for obtaining detailed information about each dish, such as ingredients and descriptions, from a database based on the translated menu information.

[0475] "Means for displaying on the user interface" refers to a function for displaying detailed information about the acquired dish and the translation results on the device screen in a format that is easy for the user to understand.

[0476] The "means for receiving user voice input" is a function for receiving voice data input by the user using a microphone.

[0477] "Means for translating voice input into another language" refers to a function for converting received voice data into text and translating the text into another language.

[0478] "Means for displaying translation results to staff" is a function for displaying translated questions and instructions to store staff.

[0479] The "means for sending order information to the store's ordering system" is a function for sending order information for the dishes selected by the user to the order management system in the store.

[0480] "Payment method" is a function for making payment using the payment method selected by the user based on the order information.

[0481] "Means for notifying the user of the payment result" is a function for notifying the user that the payment has been completed.

[0482] "Means for recognizing emotions" is a function for analyzing and recognizing the emotional state of a user from their voice data and image data.

[0483] The "means for adjusting the display content and translation results" is a function for adjusting the display content and translation results of the user interface based on the recognized emotion to adapt to the user's emotion.

[0484] The following describes the mode for carrying out the present invention: This system integrates menu scanning and translation, voice input and emotion recognition, ordering and payment, and is composed of the following components:

[0485] System configuration

[0486] 1. User device: This includes smartphones and tablets, which are equipped with a camera, microphone, and display.

[0487] 2. Server: Provides functionality for image processing, natural language processing (NLP), speech recognition, generative AI, and emotion recognition engines (e.g., Google Cloud Vision API, Microsoft Translator API, OpenAI GPT-4, Affectiva SDK).

[0488] 3. Database: Stores detailed information about dishes, translation data, order history, etc.

[0489] Hardware and software usage

[0490] First, when a user enters a restaurant, they scan the menu with their smartphone camera. The device acquires this image data and sends it to a server. The server extracts text information from the image data using an OCR engine (e.g., Google Cloud Vision API) and automatically translates that information using a translation engine (e.g., Microsoft Translator API). The translation results, along with detailed information about the dish retrieved from a database, are sent to the device and displayed on the user interface.

[0491] When a user asks a question by voice input, the smartphone microphone captures the voice data. The device sends this voice data to a server, which converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text) and translates the text into another language using a translation engine (e.g., Microsoft Translator API). The translation result is sent back to the device and displayed in the user interface.

[0492] Additionally, an emotion recognition engine (e.g., Affectiva SDK) is used to analyze emotions from the user's voice data and camera images. Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) is used to generate a response according to the user's emotion and display it in the user interface.

[0493] The user then enters the order information for the selected dishes via their smartphone and sends it to the server. The server then sends the information to the restaurant's order management system, which confirms the order. As for payment, the user completes the payment via an electronic payment service (e.g., Stripe API), and the payment result is notified to the terminal.

[0494] Specific examples

[0495] As a concrete example, consider a scenario in which a tourist named Tanaka uses this system at a restaurant in Japan. Tanaka scans the menu with his smartphone, which is automatically translated into English. If Tanaka voice-inputs a question in English, it is translated into Japanese and displayed. If Tanaka looks anxious, the emotion recognition engine detects this, and the generation AI generates and displays a kind sentence to reassure the user.

[0496] Prompt Sentence Examples

[0497] User Question: "What is the main ingredient of this dish?"

[0498] Prompt to spawn AI:

[0499] "Translate the following question to Japanese: 'What is the main ingredient of this dish?' and considering user shows signs of anxiety, frame your answer politely and informatively."

[0500] This allows Tanaka to order food with peace of mind and complete payment smoothly.

[0501] The above is a specific embodiment for carrying out the present invention. This system enables communication that transcends language barriers and provides personalized services.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] A user enters a store and scans the menu with their smartphone camera. The device captures this image data and sends it to the server. In this step, the menu image data is the input, and the image data sent to the server is the output.

[0505] Step 2:

[0506] The server analyzes the received image data using an OCR engine (e.g., Google Cloud Vision API) and extracts text data. In this step, the image data sent to the server is the input, and the extracted text information is the output.

[0507] Step 3:

[0508] The server automatically translates the extracted text into other languages ​​using a translation engine (e.g., Microsoft Translator API). In this step, the extracted text is the input, and the translated text is the output.

[0509] Step 4:

[0510] The server retrieves detailed information about the dish (e.g., ingredients, descriptions) from the database based on the translated text data. In this step, the translated text data is the input, and the retrieved detailed information about the dish is the output.

[0511] Step 5:

[0512] The server formats the acquired dish details and translation results into a format suitable for the user interface and sends them to the terminal. In this step, the input is the translated text data and dish details, and the output is data formatted for the user interface.

[0513] Step 6:

[0514] The terminal receives the formatted data and displays it to the user. The user reads the menu and asks questions via voice input. At this step, we have the formatted data as input and the menu information that the user views as output.

[0515] Step 7:

[0516] The user's voice input is captured by the smartphone's microphone, and the device sends the voice data to the server. In this step, the user's voice input data is the input, and the voice data sent to the server is the output.

[0517] Step 8:

[0518] The server receives the voice data, converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), and then translates it into another language using a translation engine (e.g., Microsoft Translator API). In this step, the input is the voice data, and the output is the translated text data.

[0519] Step 9:

[0520] The server sends the translated question to the terminal, which displays it on the user interface. In this step, the input is the translated text data, and the output is the translation result that is displayed to the user and staff.

[0521] Step 10:

[0522] The device sends the user's voice data and camera images to an emotion recognition engine (e.g., Affectiva SDK), and the server analyzes the data to recognize the user's emotional state. In this step, the input is voice data and camera images, and the output is recognized emotion data.

[0523] Step 11:

[0524] Based on the recognized emotion data, the server uses generative AI (e.g., OpenAI GPT-4) to generate emotion-based responses and adjust them for the user interface. In this step, the input is emotion data and the output is adjusted response data.

[0525] Step 12:

[0526] The terminal displays the adjusted response data to the user and provides a response in a form that is easy for the user to understand. In this step, there is the adjusted response data as input and a specific response that is displayed to the user as output.

[0527] Step 13:

[0528] The user selects a dish and enters the order information through their smartphone. The device sends the order information to the server. In this step, the user's order information is the input, and the order information sent to the server is the output.

[0529] Step 14:

[0530] The server sends the order information to the store's ordering system, and the order is confirmed. In this step, the input is the order information sent to the server, and the output is the data sent to the store's ordering system.

[0531] Step 15:

[0532] The terminal displays an order confirmation screen to the user, and the user selects the desired payment method. The user makes the payment through an electronic payment service (e.g., Stripe API). This step has the user's selected payment method and payment information as input, and the payment information sent to the server as output.

[0533] Step 16:

[0534] The server receives the payment result, notifies the terminal, and displays a payment completion screen to the user. In this step, the input is payment result data, and the output is a payment completion notification that is displayed to the user.

[0535] Through these steps, the system can realize the entire process from menu scanning and translation, to voice input queries and emotion recognition, to ordering and payment.

[0536] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0537] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0538] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0539] [Second embodiment]

[0540] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0541] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0542] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0543] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0544] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0545] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0546] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0547] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0548] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0549] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0550] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0551] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0552] The invention described in this specification is a system that performs a complete process from scanning the menu, translating it, displaying detailed information, translating voice input, ordering, and payment. Specific embodiments of this system are described below.

[0553] System configuration

[0554] This system is composed of multiple components, including the user's device, the restaurant's server, and a database. The device corresponds to the user's smartphone or tablet and is equipped with a camera, microphone, and display. The server provides image processing, translation, database access, voice recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more.

[0555] Program Overview

[0556] The system's program consists of a series of processes that provide image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment functions. Each step is explained below with concrete examples.

[0557] Menu scanning and translation

[0558] Example 1: Menu scanning and character recognition

[0559] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0560] 2. The device acquires the scanned image data and sends it to the server.

[0561] 3. The server receives the image data, uses OCR technology to extract the text information in the menu, and processes it as text data.

[0562] Example 2: Analysis and translation of text data

[0563] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish.

[0564] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts this information into a format for display on the user interface.

[0565] 6. The device displays the translation result and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0566] Receiving and translating voice input

[0567] Example 3: Simultaneous audio interpretation

[0568] 7. When a user has a question for a staff member, they can type it into their smartphone using voice input: "What kind of fish is used in this sushi?"

[0569] 8. The device captures the audio data and sends it to the server.

[0570] 9. The server receives the voice data and uses generative AI to translate English into Japanese in real time.

[0571] 10. The server sends the translation results to the terminal, which displays the message, "What kind of fish is used in this sushi?"

[0572] 11. Once the staff member responds, the content is sent back to the server, where it is translated into English in real time by the generating AI, and the message "This sushi is made with tuna" is displayed on the device.

[0573] Ordering and Payment

[0574] Example 4: Food ordering and payment

[0575] 12. The user selects their favorite dish and confirms the order via their smartphone.

[0576] 13. The terminal sends the order details to the server.

[0577] 14. The server receives the order information and sends it to the restaurant's ordering system.

[0578] 15. The terminal displays the payment screen to the user, for example, by selecting an electronic payment service and making the payment.

[0579] 16. User selects payment method and enters payment information.

[0580] 17. The server sends a payment request to the electronic payment service and receives the result.

[0581] 18. The terminal displays a payment completion screen to the user, completing the process.

[0582] As described above, this system significantly improves user convenience by providing a comprehensive process including multilingual menu display, voice interpretation, ordering, and payment, thereby reducing the burden of foreign language support in restaurants.

[0583] The processing flow will be explained below.

[0584] Step 1:

[0585] A user scans a restaurant menu with their smartphone camera.

[0586] Step 2:

[0587] The terminal acquires image data of the scanned menu and sends the image data to the server for transfer.

[0588] Step 3:

[0589] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[0590] Step 4:

[0591] The server analyzes the text data obtained by OCR, divides it into food items, and identifies the name, price, description, etc. of each item.

[0592] Step 5:

[0593] The server automatically translates each identified dish item into the specified foreign language using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[0594] Step 6:

[0595] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from a database based on the translated dish items.

[0596] Step 7:

[0597] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[0598] Step 8:

[0599] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0600] Step 9:

[0601] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0602] Step 10:

[0603] The device captures the audio data and sends it to the server.

[0604] Step 11:

[0605] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into the specified language by the AI ​​generator. For example, an English question can be translated into Japanese.

[0606] Step 12:

[0607] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[0608] Step 13:

[0609] When the staff member answers the question, the answer is entered into the terminal, which then sends the answer back to the server.

[0610] Step 14:

[0611] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[0612] Step 15:

[0613] The device receives the translation results and displays them to the user.

[0614] Step 16:

[0615] The user selects the dish and places the order via smartphone.

[0616] Step 17:

[0617] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[0618] Step 18:

[0619] The terminal transitions to the payment screen, where the user selects the desired payment method. For example, the user selects an electronic payment service.

[0620] Step 19:

[0621] The user enters payment information and confirms the payment.

[0622] Step 20:

[0623] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[0624] Step 21:

[0625] The server receives the payment result and notifies the terminal.

[0626] Step 22:

[0627] The terminal displays a payment completion screen to the user, completing the series of operations.

[0628] These are the specific steps involved in the process, from scanning the menu to translating, asking questions, ordering, and payment. This ensures smooth communication and service between the user and the restaurant.

[0629] Example 1

[0630] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] Conventional multilingual systems for restaurants handle each process separately: menu translation, detailed information display, voice interpretation, ordering, and payment. This resulted in low user convenience and a heavy burden on the restaurant. In addition, it was difficult to consistently process real-time voice interpretation, order confirmation, and payment, leading to a demand for improved user experience and operational efficiency.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0633] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for sending order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for using OCR technology and natural language processing technology, means for acquiring information from a database based on the translation result, and means for translating voice data in real time using a generative AI model. This allows for smooth multilingual support, making it possible to provide a system that is efficient and convenient for both users and restaurants.

[0634] "Image data" refers to visual information captured by a device such as a camera or scanner and stored in digital form.

[0635] "Textual information" refers to information that is displayed as text, typically expressed in language or characters.

[0636] "Translation" is the process of transcribing text written in one language into a different language.

[0637] "Detailed information" refers to more specific and detailed information about a particular item, including ingredients, images, and flavor descriptions.

[0638] A "user interface" is an interface that exchanges information between a computer system and a user, and typically includes a display, buttons, a touch panel, etc.

[0639] "Voice input" is the process of capturing a user's voice as digital data using a microphone device.

[0640] "Another language" means a language other than the original language.

[0641] "Staff" refers to employees and service providers working at restaurants and other establishments.

[0642] "Order Information" refers to detailed data about the products or services selected by the user, including the quantity and selections.

[0643] "Ordering system" refers to an order management system used in restaurants and stores to receive, process, and manage order information.

[0644] "Payment" refers to the process of paying for goods and services, and includes credit cards, electronic money, cashless payments, etc.

[0645] "OCR technology" stands for optical character recognition technology, which refers to the technology that converts character information from paper documents, image data, etc. into digital text.

[0646] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and generate human language, and includes translation, speech recognition, and dialogue systems.

[0647] A "generative AI model" is a model that uses artificial intelligence to generate data, and is used for tasks such as natural language generation and image generation.

[0648] The present invention is a system that utilizes a user's terminal, a store's server, and a database to provide a series of processes from scanning the menu to translating it, displaying detailed information, translating voice input, ordering, and payment. The specific configuration and operation for implementing the present invention are described in detail below.

[0649] System configuration

[0650] This system consists of a user-held device (such as a smartphone or tablet), a restaurant server, and a database. The device is equipped with a camera, microphone, and display, and is used to scan menus and input information by voice. The server provides functions using image processing, translation, database access, voice recognition, and generative AI models. The database stores detailed information about dishes, translation data, order history, etc.

[0651] Program Overview

[0652] The program of this system consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment. The specific processing content of each step is explained below.

[0653] Menu scanning and translation

[0654] Example 1: Menu scanning and character recognition

[0655] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0656] 2. The device acquires the scanned image data and sends it to the server. At this time, the device encodes the image data in JPEG format and uploads it to the server.

[0657] 3. The server receives the image data and extracts the text information from the menu using OCR technology (e.g., Google Cloud Vision API). The extracted text data is stored in a temporary database.

[0658] Example 2: Analysis and translation of text data

[0659] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API).

[0660] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts it into a format for display on the user interface.

[0661] 6. The device displays the translation result and detailed information to the user. For example, the information "Sushi - Raw fish on rice, typically served with soy sauce and wasabi" is displayed on the screen.

[0662] Receiving and translating voice input

[0663] Example 3: Simultaneous audio interpretation

[0664] 1. When a user wants to ask a question to a staff member, they type it into their smartphone: "What kind of fish is used in this sushi?"

[0665] 2. The device captures the audio data and sends it to the server, where it encodes it in FLAC format and uploads it to the server.

[0666] 3. The server receives the voice data, analyzes it using a speech recognition API (e.g., Google Cloud Speech-to-Text), and extracts text data.

[0667] 4. The server translates the resulting text using a generative AI model (e.g., GPT-3).

[0668] 5. The device displays the translation result to the user, asking, "What kind of fish is used in this sushi?"

[0669] 6. Once the staff member responds, their voice is sent back to the server, and the translation is converted into English in real time. "This sushi is made with tuna" is displayed on the device.

[0670] Ordering and Payment

[0671] Example 4: Food ordering and payment

[0672] 1. The user selects their favorite dish and confirms the order via smartphone.

[0673] 2. The terminal sends the order details to the server. The order data is encoded in JSON format.

[0674] 3. The server receives the order information and sends it to the restaurant's ordering system.

[0675] 4. The terminal displays the payment screen to the user, for example, using an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.).

[0676] 5. The user selects a payment method and enters payment information.

[0677] 6. The server sends a payment request to the electronic payment service and receives the result.

[0678] 7. The terminal displays a payment completion screen to the user, completing the process.

[0679] Examples of prompt statements

[0680] Here are some examples of input prompts for generative AI models:

[0681] 1. Prompt the user as they scan the menu:

[0682] "Please take a picture of the Japanese menu. After taking the picture, the image will be sent to the server."

[0683] 2. Prompt the user when using voice input:

[0684] "Please speak your question in English. When you're done, press the send button."

[0685] 3. Prompt the user to confirm the order:

[0686] Please confirm your selection and confirm your order.

[0687] 4. Prompt user when making payment:

[0688] Select your preferred payment method and enter your payment information.

[0689] As a result, this system handles everything from menu translation to ordering and payment, improving the user experience while reducing the burden on the store.

[0690] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0691] Program processing flow

[0692] Menu scanning and translation

[0693] Step 1:

[0694] A user enters a restaurant and scans a Japanese menu with their smartphone camera. The user launches a camera app and takes a picture of the menu. The input is image data of the menu, and the output is a JPEG image file saved on the device.

[0695] Step 2:

[0696] The device acquires the scanned image data and sends it to the server. The device encodes the image data in JPEG format and uploads it to the server using an HTTP POST request. The input is a JPEG image file, and the output is an HTTP request to the server.

[0697] Step 3:

[0698] The server receives the image data and uses OCR technology (e.g., Google Cloud Vision API) to extract the text information in the menu. The server passes the image data to the API and receives text data as the analysis result. The input is JPEG image data, and the output is the extracted text data.

[0699] Step 4:

[0700] The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API). The server passes the text data to the translation API and receives the translation results as text data. The input is the text data extracted by OCR, and the output is the translated text data.

[0701] Step 5:

[0702] Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, flavor descriptions, etc.) from the database and converts it into a format to display on the user interface. The server queries the database using the translated text data as a key, retrieves the relevant detailed information, and formats it. The input is the translated text data, and the output is formatted data including the detailed information.

[0703] Step 6:

[0704] The terminal displays the translation results and detailed information received from the server to the user. The terminal receives the formatted data and displays it on the display. The input is the formatted data sent from the server, and the output is the display content that the user visually perceives.

[0705] Receiving and translating voice input

[0706] Step 7:

[0707] When a user wants to ask a question to a staff member, they input it into their smartphone: "What kind of fish is used in this sushi?" The user taps the voice input button and then asks the question. The input is the user's voice data, and the output is the voice data captured on the device.

[0708] Step 8:

[0709] The device captures audio data and sends it to the server. The device encodes the audio data in FLAC format and uploads it to the server using an HTTP POST request. The input is the captured audio data, and the output is an HTTP request to the server.

[0710] Step 9:

[0711] The server analyzes the received audio data using a speech recognition API (e.g., Google Cloud Speech-to-Text) and extracts it as text data. The server passes the audio data to the API and receives text data as the analysis result. The input is audio data in FLAC format, and the output is the extracted text data.

[0712] Step 10:

[0713] The server translates the obtained text using a generative AI model (e.g., GPT-3). The server passes the text data to the translation model and receives the translation result as text data. The input is text data extracted by speech recognition, and the output is translated text data.

[0714] Step 11:

[0715] The terminal displays the translation result to the user. It displays "What kind of fish is used in this sushi?" The terminal receives the translated text data and displays it on the display. The input is the translated text data sent from the server, and the output is the display content that the user sees.

[0716] Step 12:

[0717] When the staff member responds, their voice is sent back to the server, and the translation result is converted into English in real time. "This sushi is made with tuna" is displayed on the terminal. The input is the staff member's voice data and the results of the translation model, and the output is what the user sees visually.

[0718] Ordering and Payment

[0719] Step 13:

[0720] The user selects a dish and confirms the order via their smartphone. The user taps the order confirmation button to submit the order. The input is the order details selected by the user, and the output is the order data stored on the device.

[0721] Step 14:

[0722] The terminal sends the order details to the server. The terminal encodes the order data in JSON format and uploads it to the server using an HTTP POST request. The input is the order details data, and the output is an HTTP request to the server.

[0723] Step 15:

[0724] The server receives the order information and sends it to the restaurant's ordering system. The server sends the order data to the ordering system and waits for a response. The input is the order data in JSON format, and the output is the result sent to the ordering system.

[0725] Step 16:

[0726] The terminal displays the payment screen to the user. The user selects an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.). The input is the user's confirmation action, and the output is the content displayed on the payment screen.

[0727] Step 17:

[0728] The user selects a payment method and enters payment information. The user may use a credit card or an electronic payment app. The input is the user's payment method and payment information, and the output is the data entered into the terminal.

[0729] Step 18:

[0730] The server sends a payment request to the electronic payment service and receives the result. The server sends a payment request and receives a success or failure response. The input is the user's payment information, and the output is the payment result.

[0731] Step 19:

[0732] The terminal displays a payment completion screen to the user. The terminal receives the payment result and displays "Payment completed." The input is the payment result data, and the output is the display content that the user sees.

[0733] The above is the specific processing flow of this system.

[0734] (Application example 1)

[0735] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0736] This system solves the problem of users who cannot understand foreign languages ​​having difficulty accurately understanding products and menus at restaurants and retail stores, and smoothly ordering and paying. Stores also often struggle with multilingual support, making it difficult for them to serve foreign customers. Furthermore, conventional systems often have low translation accuracy or are unable to provide real-time translations or display detailed information. This can result in a poor user experience and have a negative impact on store sales and customer satisfaction.

[0737] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0738] In this invention, the server includes a means for acquiring menu image data, a means for extracting text information from the acquired image data, a means for automatically translating the extracted text information, a means for acquiring detailed content information based on the translated information, and a means for displaying the acquired detailed information and translation results on a user interface. This allows users to easily understand foreign language menus and smoothly order and pay. The system also includes a means for receiving user voice input, a means for translating the received voice input into another language, a means for notifying the translation results, a means for transmitting user-selected order information to the system, a means for making a payment based on the order information, and a means for notifying the user of the payment results, thereby efficiently managing complex dialogues, ordering, and payment processes. The system also includes a means for generating highly accurate translations and detailed information using a generative AI model, and a means for acquiring prompts from the user in advance and suggesting appropriate prompts based on the order status. This provides a system that is easy to use for both users and stores, and has simple and accurate foreign language support.

[0739] - "Menu image data" refers to image data of paper or digital menus of restaurants or retail stores that list items and prices, obtained using the camera or scanner on the user's device.

[0740] "Text information" is text data extracted from the image data of the menu, and includes information such as the name of the dish, the price, and a description.

[0741] An "automated translation tool" is software or a service that mechanically converts textual information obtained by a user into a different language, typically using a translation engine or a generative AI model.

[0742] "Content Details" is additional information about the translated menu item, including ingredients, allergens, cooking instructions, pairings, etc.

[0743] A "user interface" is an interactive screen or display device that allows a user to interact with a system, such as the display of a smartphone or tablet.

[0744] "Voice input" refers to the voice information that a user speaks into a terminal, and is input in the form of commands, questions, comments, etc.

[0745] "Means for notifying the translation result" refers to a method for visually or audibly communicating the translated information to the user or store staff.

[0746] "Order information" refers to data including order details such as the dishes and products selected by the user, quantities, and special notes.

[0747] "Means for sending to the system" refers to a method for sending order information from the user's terminal to the store's server or ordering system.

[0748] "Payment method" means the method by which a user electronically pays for the items they have ordered, such as by using a credit card or electronic payment service.

[0749] "Payment result" is information indicating that the payment has been completed, and includes the status of success or failure.

[0750] A "generative AI model" is an artificial intelligence model that has been trained to perform tasks such as natural language processing, image recognition, and translation.

[0751] A "prompt" is a concise, clear instruction or question entered by a user that is used as input to a generative AI model.

[0752] MODE FOR CARRYING OUT THE INVENTION

[0753] The invention described in this specification is a system that uses a user's terminal, a store's server, and a database to consistently display multilingual menus, perform voice interpretation, and handle ordering and payment. Specific embodiments for implementing this system are described below.

[0754] System configuration

[0755] 1. User's Device

[0756] This refers to the user's smartphone or tablet, which is equipped with a camera, microphone, and display, allowing for menu scanning and voice input. This allows the user to capture image data of the menu using the smartphone's camera and send that data to a server.

[0757] 2. Server

[0758] The server includes the following main modules to efficiently perform multiple processes:

[0759] Image Recognition Module: Using OCR technology, the server extracts text information from the image data sent by the user. Specifically, it uses Tesseract OCR.

[0760] Translation module: Uses the Google Translate API and generative AI models to translate extracted text into multiple languages, enabling natural-sounding, highly accurate translations.

[0761] Detailed information acquisition module: Based on the translation results, detailed information about the corresponding dish (ingredients, allergen information, images, etc.) is acquired from the database.

[0762] Speech Recognition Module: Captures user voice input and transcribes it using the Google Speech-to-Text API, while also translating it in real time using a generative AI model.

[0763] Payment module: Receives order information and processes payments through electronic payment services such as Stripe.

[0764] 3. Database

[0765] It connects to a server and stores detailed information about dishes, translation data, order history, etc. The database can use a common SQL database such as MySQL or PostgreSQL.

[0766] Example

[0767] Menu scanning and translation

[0768] A user enters a restaurant and scans the menu with their smartphone camera. The image data is sent to a server, where text information is extracted using OCR technology. The extracted text information is translated using a generative AI model, and the results are displayed on the user's device.

[0769] Receiving and translating voice input

[0770] When a user wants to ask a question to a staff member, they type it into their smartphone. Questions such as "What kind of fish is used in this sushi?" are sent to the server and recognized. The generative AI model translates the question in real time, and the results are displayed on the device.

[0771] Ordering and Payment

[0772] Based on the translation and detailed information, the user selects their favorite dish and confirms the order via smartphone. The order details are sent to the restaurant's ordering system via the server, and at the same time, a payment screen is displayed on the user's smartphone. The user selects an electronic payment service to make the payment, and the server notifies the user of the payment result.

[0773] Prompt Sentence Examples

[0774] Here are some examples of prompts:

[0775] User scans the menu: "Scan the menu."

[0776] The user wants to know more about a dish: "Tell me more about this dish."

[0777] The user asks aloud: "What kind of fish is used in this sushi?"

[0778] User places an order: "I'd like to order this."

[0779] User pays: "I'll pay with my credit card."

[0780] This makes the system easy to use for both users and stores, and enables the provision of simple, highly accurate services in foreign languages.

[0781] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0782] Step 1:

[0783] A user uses the camera on their smartphone to acquire image data of a menu. The user takes a picture of a restaurant menu with the camera and inputs the image data. The device then sends this image data to a server. The input is the image data of the menu, and the output is the transmission of the image data to the server.

[0784] Step 2:

[0785] The server uses OCR technology to extract text from the image data. Specifically, it uses Tesseract OCR to convert the text in the image into text data. The input is the image data, and the output is the extracted text.

[0786] Step 3:

[0787] The server translates the extracted text using a generative AI model. Here, a language translation service (e.g., Google Translate API) is used to convert the text into another language. The input is text, and the output is translated text data.

[0788] Step 4:

[0789] The server retrieves detailed information from the database based on the translated text data. Specifically, it searches for detailed data such as ingredient information and allergen information based on the translated dish name, etc. The input is the translated text data, and the output is the detailed information.

[0790] Step 5:

[0791] The server converts the detailed information and translation results into a format that can be displayed on the user interface and sends it to the terminal. Here, the information is integrated into a user-friendly format and displayed on the screen. The input is the detailed information and translation results, and the output is the display data on the user interface.

[0792] Step 6:

[0793] The user uses the microphone of the device to input voice, for example, to ask a question about cooking. The device sends the captured voice data to the server. The input is the voice data, and the output is the transmission of the voice data to the server.

[0794] Step 7:

[0795] The server transcribes the audio data and translates it in real time using a generative AI model. Specifically, it converts the audio data into text using the Google Speech-to-Text API, and then translates the text. The input is audio data, and the output is translated text data.

[0796] Step 8:

[0797] The server sends the translation results to the terminal and displays them. The user can check the displayed translation results and get a response to the question they had from the staff. The input is the translated text data, and the output is the data displayed on the terminal.

[0798] Step 9:

[0799] The user confirms the order using the terminal. The user specifies the dish they have selected from the menu and presses the order button. The terminal sends the order information to the server. The input is the order information, and the output is the transmission of the order information to the server.

[0800] Step 10:

[0801] The server receives the order information and sends it to the restaurant's ordering system. Specifically, it uses the ordering system's API to link the order details. The input is the order information, and the output is the information sent to the ordering system.

[0802] Step 11:

[0803] The server generates a payment screen and sends it to the terminal. The user checks the payment screen and selects a payment method. The input is order information, and the output is the payment screen.

[0804] Step 12:

[0805] The user selects a payment method and enters the necessary information. The terminal sends the payment information to the server. The input is the payment information, and the output is the transmission of the payment information to the server.

[0806] Step 13:

[0807] The server connects to the electronic payment system to execute the payment. Specifically, it sends a request and receives the result using an electronic payment service such as Stripe. The input is the payment information, and the output is the payment result.

[0808] Step 14:

[0809] The server notifies the terminal of the payment result. The user confirms the completion of the payment on the terminal. The input is the payment result, and the output is the notification to the terminal.

[0810] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0811] The invention described in this specification is a system that provides more personalized services by combining menu scanning and translation, detailed information display, voice input and translation, ordering and payment, as well as an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[0812] System configuration

[0813] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[0814] Program Overview

[0815] The system's program consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, emotion recognition, user interface, and electronic payment. Each step is explained below with concrete examples.

[0816] Menu scanning and translation

[0817] Example 1: Menu scanning and character recognition

[0818] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[0819] 2. The device acquires the scanned image data and sends it to the server.

[0820] 3. The server receives the image data and uses an OCR (optical character recognition) engine to extract the text information in the menu and process it as text data.

[0821] Example 2: Analysis and translation of text data

[0822] 4. The server analyzes the text data obtained by OCR, divides it into each dish item, and automatically translates the name, price, description, etc. of each item.

[0823] 5. Based on the translation results, the server retrieves detailed information about each dish (e.g., ingredients, taste description, images, etc.) from the database.

[0824] 6. The server formats the detailed information and translation results it has obtained into a data format to be displayed on the user interface and sends it to the terminal.

[0825] 7. The device displays the translation and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0826] Receiving and translating voice input

[0827] Example 3: Simultaneous audio interpretation

[0828] 8. If the user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0829] 9. The device captures the audio data and sends it to the server.

[0830] 10. The server receives the voice data and converts it into text data using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[0831] 11. The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" becomes "What kind of fish is used in this sushi?"

[0832] Emotion Recognition and Response Personalization

[0833] Example 4: Recognizing user emotions

[0834] 12. The device uses voice data and image data obtained from the camera to recognize the user's emotions through an emotion engine. For example, it can determine whether the user is feeling anxious based on the tone of their voice and facial expression.

[0835] 13. The server adjusts translation results and user interface display content in real time based on the recognized emotion data. For example, if the user is feeling anxious, the translation results will be changed to more friendly and understandable expressions.

[0836] 14. The device will display adjusted translation results and display content to the user, allowing the user to use the service comfortably.

[0837] Ordering and Payment

[0838] Example 5: Food ordering and payment

[0839] 15. The user selects the food and places the order via smartphone.

[0840] 16. The terminal sends the order details of the selected dishes to the server, and the server forwards the order information to the restaurant's order management system.

[0841] 17. The terminal transitions to the payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[0842] 18. The user enters payment information and confirms the payment.

[0843] 19. The terminal sends the payment information to the server, and the server sends a payment request to the specified payment service.

[0844] 20. The server receives the payment result and notifies the terminal.

[0845] 21. The terminal displays a payment completion screen to the user, completing the series of operations.

[0846] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[0847] The processing flow will be explained below.

[0848] Step 1:

[0849] A user scans a restaurant menu with their smartphone camera.

[0850] Step 2:

[0851] The terminal acquires image data of the scanned menu and transmits the image data to the server.

[0852] Step 3:

[0853] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[0854] Step 4:

[0855] The server analyzes the text data obtained by OCR, divides it into each dish item, and identifies the name, price, description, etc. of each item.

[0856] Step 5:

[0857] The server automatically translates each identified dish item into multiple languages ​​using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[0858] Step 6:

[0859] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from the database based on the translated dish items.

[0860] Step 7:

[0861] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[0862] Step 8:

[0863] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[0864] Step 9:

[0865] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[0866] Step 10:

[0867] The device captures the audio data and sends it to the server.

[0868] Step 11:

[0869] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[0870] Step 12:

[0871] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[0872] Step 13:

[0873] The device or camera uses an emotion engine to analyze the user's facial expressions and tone of voice, for example, to determine whether the user is feeling anxious.

[0874] Step 14:

[0875] The server adjusts the wording of the translation based on the results of the emotion engine. For example, if the user is feeling anxious, instead of a terse response like "This sushi is made with tuna," it will change it to a more friendly and reassuring response like "This sushi is made with fresh tuna, which is a favorite among many of our guests. Please enjoy it!"

[0876] Step 15:

[0877] The device displays the adjusted translation results to both the user and staff.

[0878] Step 16:

[0879] The staff member responds to the user and enters the response into the terminal again, which then sends the response back to the server.

[0880] Step 17:

[0881] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[0882] Step 18:

[0883] The terminal receives the translation result and displays it to the user.

[0884] Step 19:

[0885] The user selects the dish and places the order via smartphone.

[0886] Step 20:

[0887] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[0888] Step 21:

[0889] The terminal transitions to a payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[0890] Step 22:

[0891] The user enters payment information and confirms the payment.

[0892] Step 23:

[0893] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[0894] Step 24:

[0895] The server receives the payment result and notifies the terminal.

[0896] Step 25:

[0897] The terminal displays a payment completion screen to the user, completing the series of operations.

[0898] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[0899] Example 2

[0900] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0901] At restaurants, foreign customers face language barriers when it comes to understanding the menu, ordering, and paying. Furthermore, language and cultural differences can make users feel anxious or confused, potentially leading to a decline in service quality. For this reason, it's important to provide not just a translation service, but a personalized experience that takes users' emotions into consideration.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0903] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for transmitting order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for recognizing the user's emotions, and means for adjusting the display content of the user interface based on the emotion data. This enables smooth ordering and payment that transcends language barriers and also enables personalized service that takes user emotions into consideration.

[0904] "Means for obtaining image data of menus" refers to a function that allows users to take a photo of a restaurant menu using the camera on their smartphone or tablet, and then save and transmit the image as digital data.

[0905] The "means for extracting text information" refers to a function that includes optical character recognition (OCR) technology for identifying text information from acquired image data and converting it into digital text.

[0906] The "means for automatic translation" is a function that uses extracted text information to translate into a selected different language in real time.

[0907] The "means for obtaining detailed information about a dish" is a function that obtains additional information about the dish (ingredients, description of the taste, images, etc.) from a database based on the translated information.

[0908] "Means for displaying on the user interface" refers to the screen or application functions that display the acquired detailed information and translation results in a format that is easy for the user to visually confirm.

[0909] The "means for receiving voice input" is a function for capturing what the user says into the microphone as voice data and processing that data.

[0910] The "means for translating into another language" is a function for analyzing received voice data, converting it into text, and then automatically translating the text into a different language.

[0911] "Means for displaying to staff" refers to a function for displaying translated information on a display or monitor so that restaurant staff can check it.

[0912] The "means for sending order information to the restaurant's ordering system" is a function for sending information about the dishes selected by the user to the restaurant's in-house order management system.

[0913] "Payment means" refers to a function for executing electronic payment processing based on the payment method selected by the user.

[0914] "Means for notifying the user of the settlement result" is a function for notifying the user of the result, such as whether the settlement was successful or unsuccessful.

[0915] "Means for recognizing emotions" is a function for analyzing the user's tone of voice and facial expressions to determine the user's emotional state.

[0916] "Means for adjusting display content" refers to a function that changes the display content of the translation results and user interface based on recognized user emotional data, allowing users to use the service more comfortably.

[0917] This invention is a system that helps foreign customers at restaurants understand the menu, overcome language barriers when placing orders and making payments, and provides personalized services that respond to the customer's emotions. Specific embodiments of the system are described below.

[0918] System configuration

[0919] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[0920] Program Overview

[0921] The system's program consists of a series of processes that provide image recognition, natural language processing, speech translation, emotion recognition, user interface, and electronic payment functions.

[0922] The server uses an OCR (Optical Character Recognition) engine (e.g., Tesseract OCR) to extract text information from the image data. The extracted text information is automatically translated by a translation engine (e.g., Google Translate API). The translation result is combined with the dish details before being displayed in the user interface. The details are retrieved from the database and then displayed on the device in a format that the user can visually confirm.

[0923] When a user speaks, the device sends the captured voice data to the server, which converts the voice data into text using speech recognition technology (e.g., Google Cloud Speech-to-Text API) and translates the text into other languages ​​using a generative AI model (e.g., OpenAI GPT-4). The translation results are sent to the device and displayed to the user and staff.

[0924] The emotion engine uses deep learning models (e.g., Emotion API) to analyze the user's tone of voice and facial expressions. Based on the emotion data, the server can adjust the translation results and user interface display. For example, if it determines that the user is anxious, it can add a more helpful explanation.

[0925] When the user selects a dish and enters the order information, the terminal sends that information to the server. The server then forwards the order information to the restaurant's ordering system, allowing the user to select their preferred payment method on the payment screen. The payment information is sent via the server to the specified payment service (e.g., Stripe API), and once the payment is complete, the result is notified to the user.

[0926] Specific use cases

[0927] A user enters a restaurant and scans the Japanese menu with their smartphone camera. The device then sends the image data to the server, where an OCR engine extracts text information from the image. The extracted text information is translated into English or other languages ​​by a translation engine, and the translation result is displayed on the device along with detailed information about the dish. When the user asks, "What kind of fish is used in this sushi?", their voice is converted to text and translated into Japanese. The translated question is then displayed to the staff. Furthermore, if the user is feeling unsure, the system can display an additional reassuring message such as, "Don't worry, this sushi is made with fresh tuna." Once the user confirms their order and pays, they are instantly notified of the results.

[0928] Example prompts for generative AI models

[0929] "Translate the Japanese menu into English: 'Sushi - raw fish on rice, usually served with soy sauce and wasabi.'"

[0930] This system will enable smooth ordering and payment processes that transcend language barriers, and will also enable personalized services that respond to the user's emotions.

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Step 1:

[0933] The user scans a Japanese menu with their smartphone camera. The user launches the camera app and takes a clear picture of the entire menu. Image data of the menu is generated as input. The image data is saved on the device and sent to the server.

[0934] Step 2:

[0935] The terminal sends scanned image data to the server. The image data sent from the terminal as input is sent to the server, which receives the data. The server then passes the received image data to the next process.

[0936] Step 3:

[0937] The server uses an OCR engine to extract text information from image data. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to identify text in the image and extract it as digital text. The input is the image data, and the output is the extracted text.

[0938] Step 4:

[0939] The server analyzes the extracted text and separates it into each dish item. For example, it performs an operation to extract the dish name, price, and description separately. The input is the text from the OCR engine, and the output is the separated text data.

[0940] Step 5:

[0941] The server sends the split text data to a translation engine, which automatically translates it into another language (e.g., English). The server uses a translation engine (e.g., Google Translate API) to convert the text data into the specified language. The input is text data split into dish name, price, and description, and the output is the translated text data.

[0942] Step 6:

[0943] The server retrieves detailed information about each dish (e.g., ingredients, flavor descriptions, images, etc.) from the database based on the translated text data. The server sends a query to the database to retrieve the corresponding records. The input is the translated text data, and the output is the data containing the detailed information.

[0944] Step 7:

[0945] The server formats the detailed information and translation results it has acquired and converts them into a data format to be displayed on the user interface. Specifically, it compiles the dish name, price, description, and image and converts them into JSON format data or HTML format. The input is the detailed information and translation results, and the output is formatted data.

[0946] Step 8:

[0947] The terminal receives the formatted data from the server and displays it in a format that is easy for the user to view. For example, it displays "Sushi: Rice topped with raw fish, typically served with soy sauce and wasabi" along with an image of the dish. The input is the formatted data from the server, and the output is the information that is displayed to the user.

[0948] Step 9:

[0949] When a user wants to ask a staff member a question, they speak into the microphone on their smartphone to input the question. For example, they might say, "What kind of fish is used in this sushi?" The input is the user's voice data.

[0950] Step 10:

[0951] The device captures the audio data and sends it to the server. The input is the audio data and the output is the data sent to the server.

[0952] Step 11:

[0953] The server receives the voice data and converts it into text data using voice recognition technology. The server then uses a voice recognition API (e.g., Google Cloud Speech-to-Text API) to convert the voice into text. The input is voice data, and the output is text data.

[0954] Step 12:

[0955] The server uses a generative AI model to translate text data into another specified language. The server uses generative AI (e.g., OpenAI GPT-4) to translate the text data. The input is the text data obtained from speech recognition, and the output is the translated text data.

[0956] Step 13:

[0957] The server sends the translation result back to the terminal, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?" The input is the text data of the translation result, and the output is the information displayed to the user or staff.

[0958] Step 14:

[0959] The device sends the user's voice data and image data obtained from the camera to the emotion engine. The input is voice data and image data. The emotion engine uses this data to recognize the user's emotion. The output is the recognized emotion data.

[0960] Step 15:

[0961] The server adjusts the translation results and display content based on the recognized emotion data. For example, if it determines that the user is feeling anxious, it adds an explanatory text and changes the expression to be more friendly. The input is emotion data and translation results, and the output is the adjusted display content.

[0962] Step 16:

[0963] The device displays the adjusted translation results and display content to the user, allowing the user to use the service comfortably. The input is the adjusted display content, and the output is the information displayed to the user.

[0964] Step 17:

[0965] The user selects a dish and presses the order button. The input is the user's selection. The terminal sends the information about the selected dish to the server.

[0966] Step 18:

[0967] The terminal sends the order details of the selected dishes to the server. The input is the dish information, and the output is the data sent to the server.

[0968] Step 19:

[0969] The server transfers the order information to the restaurant's ordering system and sends a confirmation of receipt of the order information to the terminal. The input is the order information, and the output is the transfer result to the ordering system and confirmation data to the terminal.

[0970] Step 20:

[0971] The terminal transitions to the payment screen, where the user can select the desired payment method, such as credit card, electronic money, etc. The input is the user's selection of payment method.

[0972] Step 21:

[0973] The user inputs payment information and confirms the payment. The input is payment information, which the terminal sends to the server.

[0974] Step 22:

[0975] The terminal sends payment information to the server, and the server sends a payment request to the specified payment service (e.g., Stripe API). The input is payment information, and the output is request data for the payment service.

[0976] Step 23:

[0977] The server receives the payment result and notifies the terminal. The input is the payment result from the payment service, and the output is notification data.

[0978] Step 24:

[0979] The terminal displays a payment completion screen to the user, completing the series of operations. The input is the payment result notification data, and the output is a payment completion message that is displayed to the user.

[0980] (Application example 2)

[0981] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0982] Conventional restaurant menu scanning and translation systems simply translate the menu contents into other languages ​​and do not provide personalized service based on the user's emotions or questions. Furthermore, the user's emotional state is not taken into account when asking a question or placing an order, which can result in a decline in service quality. Furthermore, the payment process is not smooth, which can cause inconvenience to users. A system that can solve these problems and provide a more comfortable and personalized service is needed.

[0983] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0984] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for recognizing emotions from user voice data and image data, and means for adjusting the display content and translation result based on the recognized emotion. This allows users not only to scan the menu and obtain translated information, but also to translate voice-input questions and orders in real time, enabling personalized responses according to the user's emotions and smooth payment.

[0985] The "means for acquiring image data of a menu" is a function that allows a user to use the camera of the device they own to scan a restaurant menu and acquire the image as data.

[0986] The "means for extracting text information" is a function for recognizing text from the acquired menu image and extracting it as digital text.

[0987] The "means for automatic translation" is a function for automatically translating extracted text information into another language.

[0988] The "means for obtaining detailed information about dishes" is a function for obtaining detailed information about each dish, such as ingredients and descriptions, from a database based on the translated menu information.

[0989] "Means for displaying on the user interface" refers to a function for displaying detailed information about the acquired dish and the translation results on the device screen in a format that is easy for the user to understand.

[0990] The "means for receiving user voice input" is a function for receiving voice data input by the user using a microphone.

[0991] The "means for translating voice input into another language" is a function for converting received voice data into text and translating the text into another language.

[0992] "Means for displaying translation results to staff" is a function for displaying translated questions and instructions to store staff.

[0993] The "means for sending order information to the store's ordering system" is a function for sending order information for the dishes selected by the user to the order management system in the store.

[0994] "Payment method" is a function for making payment using the payment method selected by the user based on the order information.

[0995] "Means for notifying the user of the payment result" is a function for notifying the user that the payment has been completed.

[0996] "Means for recognizing emotions" is a function for analyzing and recognizing the emotional state of a user from their voice data and image data.

[0997] The "means for adjusting the display content and translation results" is a function for adjusting the display content and translation results of the user interface based on the recognized emotion to adapt to the user's emotion.

[0998] The following describes an embodiment of the present invention: This system integrates menu scanning and translation, voice input and emotion recognition, ordering and payment, and is composed of the following components:

[0999] System configuration

[1000] 1. User's device: This includes smartphones and tablets, which are equipped with a camera, microphone, and display.

[1001] 2. Server: Provides functionality for image processing, natural language processing (NLP), speech recognition, generative AI, and emotion recognition engines (e.g., Google Cloud Vision API, Microsoft Translator API, OpenAI GPT-4, Affectiva SDK).

[1002] 3. Database: Stores detailed information about dishes, translation data, order history, etc.

[1003] Hardware and software usage

[1004] First, when a user enters a restaurant, they scan the menu with their smartphone camera. The device acquires this image data and sends it to a server. The server extracts text information from the image data using an OCR engine (e.g., Google Cloud Vision API) and automatically translates that information using a translation engine (e.g., Microsoft Translator API). The translation results, along with detailed information about the dish retrieved from a database, are sent to the device and displayed on the user interface.

[1005] When a user asks a question by voice input, the smartphone microphone captures the voice data. The device sends this voice data to a server, which converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text) and translates the text into another language using a translation engine (e.g., Microsoft Translator API). The translation result is sent back to the device and displayed in the user interface.

[1006] Additionally, an emotion recognition engine (e.g., Affectiva SDK) is used to analyze emotions from the user's voice data and camera images. Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) is used to generate a response according to the user's emotion and display it in the user interface.

[1007] The user then enters the order information for the selected dishes via their smartphone and sends it to the server. The server then sends the information to the restaurant's order management system, which confirms the order. As for payment, the user completes the payment via an electronic payment service (e.g., Stripe API), and the payment result is notified to the terminal.

[1008] Specific examples

[1009] As a concrete example, consider a scenario in which a tourist named Tanaka uses this system at a restaurant in Japan. Tanaka scans the menu with his smartphone, which is automatically translated into English. If Tanaka voice-inputs a question in English, it is translated into Japanese and displayed. If Tanaka looks anxious, the emotion recognition engine detects this, and the generation AI generates and displays a kind sentence to reassure the user.

[1010] Prompt Sentence Examples

[1011] User Question: "What is the main ingredient of this dish?"

[1012] Prompt to spawn AI:

[1013] "Translate the following question to Japanese: 'What is the main ingredient of this dish?' and considering user shows signs of anxiety, frame your answer politely and informatively."

[1014] This allows Tanaka to order food with peace of mind and complete payment smoothly.

[1015] The above is a specific embodiment for carrying out the present invention. This system enables communication that transcends language barriers and provides personalized services.

[1016] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1017] Step 1:

[1018] A user enters a store and scans the menu with their smartphone camera. The device captures this image data and sends it to the server. In this step, the menu image data is the input, and the image data sent to the server is the output.

[1019] Step 2:

[1020] The server analyzes the received image data using an OCR engine (e.g., Google Cloud Vision API) and extracts text data. In this step, the image data sent to the server is the input, and the extracted text information is the output.

[1021] Step 3:

[1022] The server automatically translates the extracted text into other languages ​​using a translation engine (e.g., Microsoft Translator API). In this step, the extracted text is the input, and the translated text is the output.

[1023] Step 4:

[1024] The server retrieves detailed information about the dish (e.g., ingredients, descriptions) from the database based on the translated text data. In this step, the translated text data is the input, and the retrieved detailed information about the dish is the output.

[1025] Step 5:

[1026] The server formats the acquired dish details and translation results into a format suitable for the user interface and sends them to the terminal. In this step, the input is the translated text data and dish details, and the output is data formatted for the user interface.

[1027] Step 6:

[1028] The terminal receives the formatted data and displays it to the user. The user reads the menu and asks questions via voice input. At this step, we have the formatted data as input and the menu information that the user views as output.

[1029] Step 7:

[1030] The user's voice input is captured by the smartphone's microphone, and the device sends the voice data to the server. In this step, the user's voice input data is the input, and the voice data sent to the server is the output.

[1031] Step 8:

[1032] The server receives the voice data, converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), and then translates it into another language using a translation engine (e.g., Microsoft Translator API). In this step, the input is the voice data, and the output is the translated text data.

[1033] Step 9:

[1034] The server sends the translated question to the terminal, which displays it on the user interface. In this step, the input is the translated text data, and the output is the translation result that is displayed to the user and staff.

[1035] Step 10:

[1036] The device sends the user's voice data and camera images to an emotion recognition engine (e.g., Affectiva SDK), and the server analyzes the data to recognize the user's emotional state. In this step, the input is voice data and camera images, and the output is recognized emotion data.

[1037] Step 11:

[1038] Based on the recognized emotion data, the server uses generative AI (e.g., OpenAI GPT-4) to generate emotion-based responses and adjust them for the user interface. In this step, the input is emotion data and the output is adjusted response data.

[1039] Step 12:

[1040] The terminal displays the adjusted response data to the user and provides a response in a form that is easy for the user to understand. In this step, there is the adjusted response data as input and a specific response that is displayed to the user as output.

[1041] Step 13:

[1042] The user selects a dish and enters the order information through their smartphone. The device sends the order information to the server. In this step, the user's order information is the input, and the order information sent to the server is the output.

[1043] Step 14:

[1044] The server sends the order information to the store's ordering system, and the order is confirmed. In this step, the input is the order information sent to the server, and the output is the data sent to the store's ordering system.

[1045] Step 15:

[1046] The terminal displays an order confirmation screen to the user, and the user selects the desired payment method. The user makes the payment through an electronic payment service (e.g., Stripe API). This step has the user's selected payment method and payment information as input, and the payment information sent to the server as output.

[1047] Step 16:

[1048] The server receives the payment result, notifies the terminal, and displays a payment completion screen to the user. In this step, the input is payment result data, and the output is a payment completion notification that is displayed to the user.

[1049] Through these steps, the system can complete the entire process, from menu scanning and translation, to voice input queries and emotion recognition, to ordering and payment.

[1050] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1051] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1052] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1053] [Third embodiment]

[1054] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1055] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1056] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1057] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1058] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1059] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1060] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1061] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1062] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1063] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1064] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1065] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1066] The invention described in this specification is a system that performs a complete process from scanning the menu, translating it, displaying detailed information, translating voice input, ordering, and payment. Specific embodiments of this system are described below.

[1067] System configuration

[1068] This system is composed of multiple components, including the user's device, the restaurant's server, and a database. The device corresponds to the user's smartphone or tablet and is equipped with a camera, microphone, and display. The server provides image processing, translation, database access, voice recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more.

[1069] Program Overview

[1070] The system's program consists of a series of processes that provide image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment functions. Each step is explained below with concrete examples.

[1071] Menu scanning and translation

[1072] Example 1: Menu scanning and character recognition

[1073] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1074] 2. The device acquires the scanned image data and sends it to the server.

[1075] 3. The server receives the image data, uses OCR technology to extract the text information in the menu, and processes it as text data.

[1076] Example 2: Analysis and translation of text data

[1077] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish.

[1078] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts this information into a format for display on the user interface.

[1079] 6. The device displays the translation result and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1080] Receiving and translating voice input

[1081] Example 3: Simultaneous audio interpretation

[1082] 7. When a user has a question for a staff member, they can type it into their smartphone using voice input: "What kind of fish is used in this sushi?"

[1083] 8. The device captures the audio data and sends it to the server.

[1084] 9. The server receives the voice data and uses generative AI to translate English into Japanese in real time.

[1085] 10. The server sends the translation results to the terminal, which displays the message, "What kind of fish is used in this sushi?"

[1086] 11. Once the staff member responds, the content is sent back to the server, where it is translated into English in real time by the generating AI, and the message "This sushi is made with tuna" is displayed on the device.

[1087] Ordering and Payment

[1088] Example 4: Food ordering and payment

[1089] 12. The user selects their favorite dish and confirms the order via their smartphone.

[1090] 13. The terminal sends the order details to the server.

[1091] 14. The server receives the order information and sends it to the restaurant's ordering system.

[1092] 15. The terminal displays the payment screen to the user, for example, by selecting an electronic payment service and making the payment.

[1093] 16. User selects payment method and enters payment information.

[1094] 17. The server sends a payment request to the electronic payment service and receives the result.

[1095] 18. The terminal displays a payment completion screen to the user, completing the process.

[1096] As described above, this system significantly improves user convenience by providing a comprehensive process including multilingual menu display, voice interpretation, ordering, and payment, thereby reducing the burden of foreign language support in restaurants.

[1097] The processing flow will be explained below.

[1098] Step 1:

[1099] A user scans a restaurant menu with their smartphone camera.

[1100] Step 2:

[1101] The terminal acquires image data of the scanned menu and sends the image data to the server for transfer.

[1102] Step 3:

[1103] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[1104] Step 4:

[1105] The server analyzes the text data obtained by OCR, divides it into food items, and identifies the name, price, description, etc. of each item.

[1106] Step 5:

[1107] The server automatically translates each identified dish item into the specified foreign language using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[1108] Step 6:

[1109] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from a database based on the translated dish items.

[1110] Step 7:

[1111] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[1112] Step 8:

[1113] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1114] Step 9:

[1115] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[1116] Step 10:

[1117] The device captures the audio data and sends it to the server.

[1118] Step 11:

[1119] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into the specified language by the AI ​​generator. For example, an English question can be translated into Japanese.

[1120] Step 12:

[1121] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[1122] Step 13:

[1123] When the staff member answers the question, the answer is entered into the terminal, which then sends the answer back to the server.

[1124] Step 14:

[1125] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[1126] Step 15:

[1127] The device receives the translation results and displays them to the user.

[1128] Step 16:

[1129] The user selects the dish and places the order via smartphone.

[1130] Step 17:

[1131] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[1132] Step 18:

[1133] The terminal transitions to the payment screen, where the user selects the desired payment method. For example, the user selects an electronic payment service.

[1134] Step 19:

[1135] The user enters payment information and confirms the payment.

[1136] Step 20:

[1137] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[1138] Step 21:

[1139] The server receives the payment result and notifies the terminal.

[1140] Step 22:

[1141] The terminal displays a payment completion screen to the user, completing the series of operations.

[1142] These are the specific steps involved in the process, from scanning the menu to translating, asking questions, ordering, and payment. This ensures smooth communication and service between the user and the restaurant.

[1143] Example 1

[1144] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] Conventional multilingual systems for restaurants handle each process separately: menu translation, detailed information display, voice interpretation, ordering, and payment. This resulted in low user convenience and a heavy burden on the restaurant. In addition, it was difficult to consistently process real-time voice interpretation, order confirmation, and payment, leading to a demand for improved user experience and operational efficiency.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1147] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for sending order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for using OCR technology and natural language processing technology, means for acquiring information from a database based on the translation result, and means for translating voice data in real time using a generative AI model. This allows for smooth multilingual support, making it possible to provide a system that is efficient and convenient for both users and restaurants.

[1148] "Image data" refers to visual information captured by a device such as a camera or scanner and stored in digital form.

[1149] "Textual information" refers to information that is displayed as text, typically expressed in language or characters.

[1150] "Translation" is the process of transcribing text written in one language into a different language.

[1151] "Detailed information" refers to more specific and detailed information about a particular item, including ingredients, images, and flavor descriptions.

[1152] A "user interface" is an interface that exchanges information between a computer system and a user, and typically includes a display, buttons, a touch panel, etc.

[1153] "Voice input" is the process of capturing a user's voice as digital data using a microphone device.

[1154] "Another language" means a language other than the original language.

[1155] "Staff" refers to employees and service providers working at restaurants and other establishments.

[1156] "Order Information" refers to detailed data about the products or services selected by the user, including the quantity and selections.

[1157] "Ordering system" refers to an order management system used in restaurants and stores to receive, process, and manage order information.

[1158] "Payment" refers to the process of paying for goods and services, and includes credit cards, electronic money, cashless payments, etc.

[1159] "OCR technology" stands for optical character recognition technology, which refers to the technology that converts character information from paper documents, image data, etc. into digital text.

[1160] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and generate human language, and includes translation, speech recognition, and dialogue systems.

[1161] A "generative AI model" is a model that uses artificial intelligence to generate data, and is used for tasks such as natural language generation and image generation.

[1162] The present invention is a system that utilizes a user's terminal, a store's server, and a database to provide a series of processes from scanning the menu to translating it, displaying detailed information, translating voice input, ordering, and payment. The specific configuration and operation for implementing the present invention are described in detail below.

[1163] System configuration

[1164] This system consists of a user-held device (such as a smartphone or tablet), a restaurant server, and a database. The device is equipped with a camera, microphone, and display, and is used to scan menus and input information by voice. The server provides functions using image processing, translation, database access, voice recognition, and generative AI models. The database stores detailed information about dishes, translation data, order history, etc.

[1165] Program Overview

[1166] The program of this system consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment. The specific processing content of each step is explained below.

[1167] Menu scanning and translation

[1168] Example 1: Menu scanning and character recognition

[1169] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1170] 2. The device acquires the scanned image data and sends it to the server. At this time, the device encodes the image data in JPEG format and uploads it to the server.

[1171] 3. The server receives the image data and extracts the text information from the menu using OCR technology (e.g., Google Cloud Vision API). The extracted text data is stored in a temporary database.

[1172] Example 2: Analysis and translation of text data

[1173] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API).

[1174] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts it into a format for display on the user interface.

[1175] 6. The device displays the translation result and detailed information to the user. For example, the information "Sushi - Raw fish on rice, typically served with soy sauce and wasabi" is displayed on the screen.

[1176] Receiving and translating voice input

[1177] Example 3: Simultaneous audio interpretation

[1178] 1. When a user wants to ask a question to a staff member, they type it into their smartphone: "What kind of fish is used in this sushi?"

[1179] 2. The device captures the audio data and sends it to the server, where it encodes it in FLAC format and uploads it to the server.

[1180] 3. The server receives the voice data, analyzes it using a speech recognition API (e.g., Google Cloud Speech-to-Text), and extracts text data.

[1181] 4. The server translates the resulting text using a generative AI model (e.g., GPT-3).

[1182] 5. The device displays the translation result to the user, asking, "What kind of fish is used in this sushi?"

[1183] 6. Once the staff member responds, their voice is sent back to the server, and the translation is converted into English in real time. "This sushi is made with tuna" is displayed on the device.

[1184] Ordering and Payment

[1185] Example 4: Food ordering and payment

[1186] 1. The user selects their favorite dish and confirms the order via smartphone.

[1187] 2. The terminal sends the order details to the server. The order data is encoded in JSON format.

[1188] 3. The server receives the order information and sends it to the restaurant's ordering system.

[1189] 4. The terminal displays the payment screen to the user, for example, using an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.).

[1190] 5. The user selects a payment method and enters payment information.

[1191] 6. The server sends a payment request to the electronic payment service and receives the result.

[1192] 7. The terminal displays a payment completion screen to the user, completing the process.

[1193] Examples of prompt statements

[1194] Here are some examples of input prompts for generative AI models:

[1195] 1. Prompt the user as they scan the menu:

[1196] "Please take a picture of the Japanese menu. After taking the picture, the image will be sent to the server."

[1197] 2. Prompt the user when using voice input:

[1198] "Please speak your question in English. When you're done, press the send button."

[1199] 3. Prompt the user to confirm the order:

[1200] Please confirm your selection and confirm your order.

[1201] 4. Prompt user when making payment:

[1202] Select your preferred payment method and enter your payment information.

[1203] As a result, this system handles everything from menu translation to ordering and payment, improving the user experience while reducing the burden on the store.

[1204] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1205] Program processing flow

[1206] Menu scanning and translation

[1207] Step 1:

[1208] A user enters a restaurant and scans a Japanese menu with their smartphone camera. The user launches a camera app and takes a picture of the menu. The input is image data of the menu, and the output is a JPEG image file saved on the device.

[1209] Step 2:

[1210] The device acquires the scanned image data and sends it to the server. The device encodes the image data in JPEG format and uploads it to the server using an HTTP POST request. The input is a JPEG image file, and the output is an HTTP request to the server.

[1211] Step 3:

[1212] The server receives the image data and uses OCR technology (e.g., Google Cloud Vision API) to extract the text information in the menu. The server passes the image data to the API and receives text data as the analysis result. The input is JPEG image data, and the output is the extracted text data.

[1213] Step 4:

[1214] The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API). The server passes the text data to the translation API and receives the translation results as text data. The input is the text data extracted by OCR, and the output is the translated text data.

[1215] Step 5:

[1216] Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, flavor descriptions, etc.) from the database and converts it into a format to display on the user interface. The server queries the database using the translated text data as a key, retrieves the relevant detailed information, and formats it. The input is the translated text data, and the output is formatted data including the detailed information.

[1217] Step 6:

[1218] The terminal displays the translation results and detailed information received from the server to the user. The terminal receives the formatted data and displays it on the display. The input is the formatted data sent from the server, and the output is the display content that the user visually perceives.

[1219] Receiving and translating voice input

[1220] Step 7:

[1221] When a user wants to ask a question to a staff member, they input it into their smartphone: "What kind of fish is used in this sushi?" The user taps the voice input button and then asks the question. The input is the user's voice data, and the output is the voice data captured on the device.

[1222] Step 8:

[1223] The device captures audio data and sends it to the server. The device encodes the audio data in FLAC format and uploads it to the server using an HTTP POST request. The input is the captured audio data, and the output is an HTTP request to the server.

[1224] Step 9:

[1225] The server analyzes the received audio data using a speech recognition API (e.g., Google Cloud Speech-to-Text) and extracts it as text data. The server passes the audio data to the API and receives text data as the analysis result. The input is audio data in FLAC format, and the output is the extracted text data.

[1226] Step 10:

[1227] The server translates the obtained text using a generative AI model (e.g., GPT-3). The server passes the text data to the translation model and receives the translation result as text data. The input is text data extracted by speech recognition, and the output is translated text data.

[1228] Step 11:

[1229] The terminal displays the translation result to the user. It displays "What kind of fish is used in this sushi?" The terminal receives the translated text data and displays it on the display. The input is the translated text data sent from the server, and the output is the display content that the user sees.

[1230] Step 12:

[1231] When the staff member responds, their voice is sent back to the server, and the translation result is converted into English in real time. "This sushi is made with tuna" is displayed on the terminal. The input is the staff member's voice data and the results of the translation model, and the output is what the user sees visually.

[1232] Ordering and Payment

[1233] Step 13:

[1234] The user selects a dish and confirms the order via their smartphone. The user taps the order confirmation button to submit the order. The input is the order details selected by the user, and the output is the order data stored on the device.

[1235] Step 14:

[1236] The terminal sends the order details to the server. The terminal encodes the order data in JSON format and uploads it to the server using an HTTP POST request. The input is the order details data, and the output is an HTTP request to the server.

[1237] Step 15:

[1238] The server receives the order information and sends it to the restaurant's ordering system. The server sends the order data to the ordering system and waits for a response. The input is the order data in JSON format, and the output is the result sent to the ordering system.

[1239] Step 16:

[1240] The terminal displays the payment screen to the user. The user selects an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.). The input is the user's confirmation action, and the output is the content displayed on the payment screen.

[1241] Step 17:

[1242] The user selects a payment method and enters payment information. The user may use a credit card or an electronic payment app. The input is the user's payment method and payment information, and the output is the data entered into the terminal.

[1243] Step 18:

[1244] The server sends a payment request to the electronic payment service and receives the result. The server sends a payment request and receives a success or failure response. The input is the user's payment information, and the output is the payment result.

[1245] Step 19:

[1246] The terminal displays a payment completion screen to the user. The terminal receives the payment result and displays "Payment completed." The input is the payment result data, and the output is the display content that the user sees.

[1247] The above is the specific processing flow of this system.

[1248] (Application example 1)

[1249] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1250] This system solves the problem of users who cannot understand foreign languages ​​having difficulty accurately understanding products and menus at restaurants and retail stores, and smoothly ordering and paying. Stores also often struggle with multilingual support, making it difficult for them to serve foreign customers. Furthermore, conventional systems often have low translation accuracy or are unable to provide real-time translations or display detailed information. This can result in a poor user experience and have a negative impact on store sales and customer satisfaction.

[1251] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1252] In this invention, the server includes a means for acquiring menu image data, a means for extracting text information from the acquired image data, a means for automatically translating the extracted text information, a means for acquiring detailed content information based on the translated information, and a means for displaying the acquired detailed information and translation results on a user interface. This allows users to easily understand foreign language menus and smoothly order and pay. The system also includes a means for receiving user voice input, a means for translating the received voice input into another language, a means for notifying the translation results, a means for transmitting user-selected order information to the system, a means for making a payment based on the order information, and a means for notifying the user of the payment results, thereby efficiently managing complex dialogues, ordering, and payment processes. The system also includes a means for generating highly accurate translations and detailed information using a generative AI model, and a means for acquiring prompts from the user in advance and suggesting appropriate prompts based on the order status. This provides a system that is easy to use for both users and stores, and has simple and accurate foreign language support.

[1253] - "Menu image data" refers to image data of paper or digital menus of restaurants or retail stores that list items and prices, obtained using the camera or scanner on the user's device.

[1254] "Text information" is text data extracted from the image data of the menu, and includes information such as the name of the dish, the price, and a description.

[1255] An "automated translation tool" is software or a service that mechanically converts textual information obtained by a user into a different language, typically using a translation engine or a generative AI model.

[1256] "Content Details" is additional information about the translated menu item, including ingredients, allergens, cooking instructions, pairings, etc.

[1257] A "user interface" is an interactive screen or display device that allows a user to interact with a system, such as the display of a smartphone or tablet.

[1258] "Voice input" refers to the voice information that a user speaks into a terminal, and is input in the form of commands, questions, comments, etc.

[1259] "Means for notifying the translation result" refers to a method for visually or audibly communicating the translated information to the user or store staff.

[1260] "Order information" refers to data including order details such as the dishes and products selected by the user, quantities, and special notes.

[1261] "Means for sending to the system" refers to a method for sending order information from the user's terminal to the store's server or ordering system.

[1262] "Payment method" means the method by which a user electronically pays for the items they have ordered, such as by using a credit card or electronic payment service.

[1263] "Payment result" is information indicating that the payment has been completed, and includes the status of success or failure.

[1264] A "generative AI model" is an artificial intelligence model that has been trained to perform tasks such as natural language processing, image recognition, and translation.

[1265] A "prompt" is a concise, clear instruction or question entered by a user that is used as input to a generative AI model.

[1266] MODE FOR CARRYING OUT THE INVENTION

[1267] The invention described in this specification is a system that uses a user's terminal, a store's server, and a database to consistently display multilingual menus, perform voice interpretation, and handle ordering and payment. Specific embodiments for implementing this system are described below.

[1268] System configuration

[1269] 1. User's Device

[1270] This refers to the user's smartphone or tablet, which is equipped with a camera, microphone, and display, allowing for menu scanning and voice input. This allows the user to capture image data of the menu using the smartphone's camera and send that data to a server.

[1271] 2. Server

[1272] The server includes the following main modules to efficiently perform multiple processes:

[1273] Image Recognition Module: Using OCR technology, the server extracts text information from the image data sent by the user. Specifically, it uses Tesseract OCR.

[1274] Translation module: Uses the Google Translate API and generative AI models to translate extracted text into multiple languages, enabling natural-sounding, highly accurate translations.

[1275] Detailed information acquisition module: Based on the translation results, detailed information about the corresponding dish (ingredients, allergen information, images, etc.) is acquired from the database.

[1276] Speech Recognition Module: Captures user voice input and transcribes it using the Google Speech-to-Text API, while also translating it in real time using a generative AI model.

[1277] Payment module: Receives order information and processes payments through electronic payment services such as Stripe.

[1278] 3. Database

[1279] It connects to a server and stores detailed information about dishes, translation data, order history, etc. The database can use a common SQL database such as MySQL or PostgreSQL.

[1280] Example

[1281] Menu scanning and translation

[1282] A user enters a restaurant and scans the menu with their smartphone camera. The image data is sent to a server, where text information is extracted using OCR technology. The extracted text information is translated using a generative AI model, and the results are displayed on the user's device.

[1283] Receiving and translating voice input

[1284] When a user wants to ask a question to a staff member, they type it into their smartphone. Questions such as "What kind of fish is used in this sushi?" are sent to the server and recognized. The generative AI model translates the question in real time, and the results are displayed on the device.

[1285] Ordering and Payment

[1286] Based on the translation and detailed information, the user selects their favorite dish and confirms the order via smartphone. The order details are sent to the restaurant's ordering system via the server, and at the same time, a payment screen is displayed on the user's smartphone. The user selects an electronic payment service to make the payment, and the server notifies the user of the payment result.

[1287] Prompt Sentence Examples

[1288] Here are some examples of prompts:

[1289] User scans the menu: "Scan the menu."

[1290] The user wants to know more about a dish: "Tell me more about this dish."

[1291] The user asks aloud: "What kind of fish is used in this sushi?"

[1292] User places an order: "I'd like to order this."

[1293] User pays: "I'll pay with my credit card."

[1294] This makes the system easy to use for both users and stores, and enables the provision of simple, highly accurate services in foreign languages.

[1295] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1296] Step 1:

[1297] A user uses the camera on their smartphone to acquire image data of a menu. The user takes a picture of a restaurant menu with the camera and inputs the image data. The device then sends this image data to a server. The input is the image data of the menu, and the output is the transmission of the image data to the server.

[1298] Step 2:

[1299] The server uses OCR technology to extract text from the image data. Specifically, it uses Tesseract OCR to convert the text in the image into text data. The input is the image data, and the output is the extracted text.

[1300] Step 3:

[1301] The server translates the extracted text using a generative AI model. Here, a language translation service (e.g., Google Translate API) is used to convert the text into another language. The input is text, and the output is translated text data.

[1302] Step 4:

[1303] The server retrieves detailed information from the database based on the translated text data. Specifically, it searches for detailed data such as ingredient information and allergen information based on the translated dish name, etc. The input is the translated text data, and the output is the detailed information.

[1304] Step 5:

[1305] The server converts the detailed information and translation results into a format that can be displayed on the user interface and sends it to the terminal. Here, the information is integrated into a user-friendly format and displayed on the screen. The input is the detailed information and translation results, and the output is the display data on the user interface.

[1306] Step 6:

[1307] The user uses the microphone of the device to input voice, for example, to ask a question about cooking. The device sends the captured voice data to the server. The input is the voice data, and the output is the transmission of the voice data to the server.

[1308] Step 7:

[1309] The server transcribes the audio data and translates it in real time using a generative AI model. Specifically, it converts the audio data into text using the Google Speech-to-Text API, and then translates the text. The input is audio data, and the output is translated text data.

[1310] Step 8:

[1311] The server sends the translation results to the terminal and displays them. The user can check the displayed translation results and get a response to the question they had from the staff. The input is the translated text data, and the output is the data displayed on the terminal.

[1312] Step 9:

[1313] The user confirms the order using the terminal. The user specifies the dish they have selected from the menu and presses the order button. The terminal sends the order information to the server. The input is the order information, and the output is the transmission of the order information to the server.

[1314] Step 10:

[1315] The server receives the order information and sends it to the restaurant's ordering system. Specifically, it uses the ordering system's API to link the order details. The input is the order information, and the output is the information sent to the ordering system.

[1316] Step 11:

[1317] The server generates a payment screen and sends it to the terminal. The user checks the payment screen and selects a payment method. The input is order information, and the output is the payment screen.

[1318] Step 12:

[1319] The user selects a payment method and enters the necessary information. The terminal sends the payment information to the server. The input is the payment information, and the output is the transmission of the payment information to the server.

[1320] Step 13:

[1321] The server connects to the electronic payment system to execute the payment. Specifically, it sends a request and receives the result using an electronic payment service such as Stripe. The input is the payment information, and the output is the payment result.

[1322] Step 14:

[1323] The server notifies the terminal of the payment result. The user confirms the completion of the payment on the terminal. The input is the payment result, and the output is the notification to the terminal.

[1324] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1325] The invention described in this specification is a system that provides more personalized services by combining menu scanning and translation, detailed information display, voice input and translation, ordering and payment, as well as an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[1326] System configuration

[1327] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[1328] Program Overview

[1329] The system's program consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, emotion recognition, user interface, and electronic payment. Each step is explained below with concrete examples.

[1330] Menu scanning and translation

[1331] Example 1: Menu scanning and character recognition

[1332] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1333] 2. The device acquires the scanned image data and sends it to the server.

[1334] 3. The server receives the image data and uses an OCR (optical character recognition) engine to extract the text information in the menu and process it as text data.

[1335] Example 2: Analysis and translation of text data

[1336] 4. The server analyzes the text data obtained by OCR, divides it into each dish item, and automatically translates the name, price, description, etc. of each item.

[1337] 5. Based on the translation results, the server retrieves detailed information about each dish (e.g., ingredients, taste description, images, etc.) from the database.

[1338] 6. The server formats the detailed information and translation results it has obtained into a data format to be displayed on the user interface and sends it to the terminal.

[1339] 7. The device displays the translation and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1340] Receiving and translating voice input

[1341] Example 3: Simultaneous audio interpretation

[1342] 8. If the user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[1343] 9. The device captures the audio data and sends it to the server.

[1344] 10. The server receives the voice data and converts it into text data using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[1345] 11. The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" becomes "What kind of fish is used in this sushi?"

[1346] Emotion Recognition and Response Personalization

[1347] Example 4: Recognizing user emotions

[1348] 12. The device uses voice data and image data obtained from the camera to recognize the user's emotions through an emotion engine. For example, it can determine whether the user is feeling anxious based on the tone of their voice and facial expression.

[1349] 13. The server adjusts translation results and user interface display content in real time based on the recognized emotion data. For example, if the user is feeling anxious, the translation results will be changed to more friendly and understandable expressions.

[1350] 14. The device will display adjusted translation results and display content to the user, allowing the user to use the service comfortably.

[1351] Ordering and Payment

[1352] Example 5: Food ordering and payment

[1353] 15. The user selects the food and places the order via smartphone.

[1354] 16. The terminal sends the order details of the selected dishes to the server, and the server forwards the order information to the restaurant's order management system.

[1355] 17. The terminal transitions to the payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[1356] 18. The user enters payment information and confirms the payment.

[1357] 19. The terminal sends the payment information to the server, and the server sends a payment request to the specified payment service.

[1358] 20. The server receives the payment result and notifies the terminal.

[1359] 21. The terminal displays a payment completion screen to the user, completing the series of operations.

[1360] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[1361] The processing flow will be explained below.

[1362] Step 1:

[1363] A user scans a restaurant menu with their smartphone camera.

[1364] Step 2:

[1365] The terminal acquires image data of the scanned menu and transmits the image data to the server.

[1366] Step 3:

[1367] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[1368] Step 4:

[1369] The server analyzes the text data obtained by OCR, divides it into each dish item, and identifies the name, price, description, etc. of each item.

[1370] Step 5:

[1371] The server automatically translates each identified dish item into multiple languages ​​using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[1372] Step 6:

[1373] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from the database based on the translated dish items.

[1374] Step 7:

[1375] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[1376] Step 8:

[1377] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1378] Step 9:

[1379] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[1380] Step 10:

[1381] The device captures the audio data and sends it to the server.

[1382] Step 11:

[1383] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into another language by generative AI. For example, an English question can be translated into Japanese.

[1384] Step 12:

[1385] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[1386] Step 13:

[1387] The device or camera uses an emotion engine to analyze the user's facial expressions and tone of voice, for example, to determine whether the user is feeling anxious.

[1388] Step 14:

[1389] The server adjusts the wording of the translation based on the results of the emotion engine. For example, if the user is feeling anxious, instead of a terse response like "This sushi is made with tuna," it will change it to a more friendly and reassuring response like "This sushi is made with fresh tuna, which is a favorite among many of our guests. Please enjoy it!"

[1390] Step 15:

[1391] The device displays the adjusted translation results to both the user and staff.

[1392] Step 16:

[1393] The staff member responds to the user and enters the response into the terminal again, which then sends the response back to the server.

[1394] Step 17:

[1395] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi uses tuna" is translated into "This sushi is made with tuna."

[1396] Step 18:

[1397] The terminal receives the translation result and displays it to the user.

[1398] Step 19:

[1399] The user selects the dish and places the order via smartphone.

[1400] Step 20:

[1401] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[1402] Step 21:

[1403] The terminal transitions to a payment screen, where the user selects the desired payment method, for example, an electronic payment service.

[1404] Step 22:

[1405] The user enters payment information and confirms the payment.

[1406] Step 23:

[1407] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[1408] Step 24:

[1409] The server receives the payment result and notifies the terminal.

[1410] Step 25:

[1411] The terminal displays a payment completion screen to the user, completing the series of operations.

[1412] These are the specific processing steps, from scanning the menu to translating, asking questions, recognizing emotions, ordering, and payment. By combining the emotion engine, personalized services can be provided according to the user's emotions, resulting in a comfortable and smooth experience.

[1413] Example 2

[1414] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1415] At restaurants, foreign customers face language barriers when it comes to understanding the menu, ordering, and paying. Furthermore, language and cultural differences can make users feel anxious or confused, potentially leading to a decline in service quality. For this reason, it's important to provide not just a translation service, but a personalized experience that takes users' emotions into consideration.

[1416] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1417] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for transmitting order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for recognizing the user's emotions, and means for adjusting the display content of the user interface based on the emotion data. This enables smooth ordering and payment that transcends language barriers and also enables personalized service that takes user emotions into consideration.

[1418] "Means for obtaining image data of menus" refers to a function that allows users to take a photo of a restaurant menu using the camera on their smartphone or tablet, and then save and transmit the image as digital data.

[1419] The "means for extracting text information" refers to a function that includes optical character recognition (OCR) technology for identifying text information from acquired image data and converting it into digital text.

[1420] The "means for automatic translation" is a function that uses extracted text information to translate into a selected different language in real time.

[1421] The "means for obtaining detailed information about a dish" is a function that obtains additional information about the dish (ingredients, description of the taste, images, etc.) from a database based on the translated information.

[1422] "Means for displaying on the user interface" refers to the screen or application functions that display the acquired detailed information and translation results in a format that is easy for the user to visually confirm.

[1423] The "means for receiving voice input" is a function for capturing what the user says into the microphone as voice data and processing that data.

[1424] The "means for translating into another language" is a function for analyzing received voice data, converting it into text, and then automatically translating the text into a different language.

[1425] "Means for displaying to staff" refers to a function for displaying translated information on a display or monitor so that restaurant staff can check it.

[1426] The "means for sending order information to the restaurant's ordering system" is a function for sending information about the dishes selected by the user to the restaurant's in-house order management system.

[1427] "Payment means" refers to a function for executing electronic payment processing based on the payment method selected by the user.

[1428] "Means for notifying the user of the settlement result" is a function for notifying the user of the result, such as whether the settlement was successful or unsuccessful.

[1429] "Means for recognizing emotions" is a function for analyzing the user's tone of voice and facial expressions to determine the user's emotional state.

[1430] "Means for adjusting display content" refers to a function that changes the display content of the translation results and user interface based on recognized user emotional data, allowing users to use the service more comfortably.

[1431] This invention is a system that helps foreign customers at restaurants understand the menu, overcome language barriers when placing orders and making payments, and provides personalized services that respond to the customer's emotions. Specific embodiments of the system are described below.

[1432] System configuration

[1433] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[1434] Program Overview

[1435] The system's program consists of a series of processes that provide image recognition, natural language processing, speech translation, emotion recognition, user interface, and electronic payment functions.

[1436] The server uses an OCR (Optical Character Recognition) engine (e.g., Tesseract OCR) to extract text information from the image data. The extracted text information is automatically translated by a translation engine (e.g., Google Translate API). The translation result is combined with the dish details before being displayed in the user interface. The details are retrieved from the database and then displayed on the device in a format that the user can visually confirm.

[1437] When a user speaks, the device sends the captured voice data to the server, which converts the voice data into text using speech recognition technology (e.g., Google Cloud Speech-to-Text API) and translates the text into other languages ​​using a generative AI model (e.g., OpenAI GPT-4). The translation results are sent to the device and displayed to the user and staff.

[1438] The emotion engine uses deep learning models (e.g., Emotion API) to analyze the user's tone of voice and facial expressions. Based on the emotion data, the server can adjust the translation results and user interface display. For example, if it determines that the user is anxious, it can add a more helpful explanation.

[1439] When the user selects a dish and enters the order information, the terminal sends that information to the server. The server then forwards the order information to the restaurant's ordering system, allowing the user to select their preferred payment method on the payment screen. The payment information is sent via the server to the specified payment service (e.g., Stripe API), and once the payment is complete, the result is notified to the user.

[1440] Specific use cases

[1441] A user enters a restaurant and scans the Japanese menu with their smartphone camera. The device then sends the image data to the server, where an OCR engine extracts text information from the image. The extracted text information is translated into English or other languages ​​by a translation engine, and the translation result is displayed on the device along with detailed information about the dish. When the user asks, "What kind of fish is used in this sushi?", their voice is converted to text and translated into Japanese. The translated question is then displayed to the staff. Furthermore, if the user is feeling unsure, the system can display an additional reassuring message such as, "Don't worry, this sushi is made with fresh tuna." Once the user confirms their order and pays, they are instantly notified of the results.

[1442] Example prompts for generative AI models

[1443] "Translate the Japanese menu into English: 'Sushi - raw fish on rice, usually served with soy sauce and wasabi.'"

[1444] This system will enable smooth ordering and payment processes that transcend language barriers, and will also enable personalized services that respond to the user's emotions.

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] Step 1:

[1447] The user scans a Japanese menu with their smartphone camera. The user launches the camera app and takes a clear picture of the entire menu. Image data of the menu is generated as input. The image data is saved on the device and sent to the server.

[1448] Step 2:

[1449] The terminal sends scanned image data to the server. The image data sent from the terminal as input is sent to the server, which receives the data. The server then passes the received image data to the next process.

[1450] Step 3:

[1451] The server uses an OCR engine to extract text information from image data. Specifically, the server uses an OCR engine (e.g., Tesseract OCR) to identify text in the image and extract it as digital text. The input is the image data, and the output is the extracted text.

[1452] Step 4:

[1453] The server analyzes the extracted text and separates it into each dish item. For example, it performs an operation to extract the dish name, price, and description separately. The input is the text from the OCR engine, and the output is the separated text data.

[1454] Step 5:

[1455] The server sends the split text data to a translation engine, which automatically translates it into another language (e.g., English). The server uses a translation engine (e.g., Google Translate API) to convert the text data into the specified language. The input is text data split into dish name, price, and description, and the output is the translated text data.

[1456] Step 6:

[1457] The server retrieves detailed information about each dish (e.g., ingredients, flavor descriptions, images, etc.) from the database based on the translated text data. The server sends a query to the database to retrieve the corresponding records. The input is the translated text data, and the output is the data containing the detailed information.

[1458] Step 7:

[1459] The server formats the detailed information and translation results it has acquired and converts them into a data format to be displayed on the user interface. Specifically, it compiles the dish name, price, description, and image and converts them into JSON format data or HTML format. The input is the detailed information and translation results, and the output is formatted data.

[1460] Step 8:

[1461] The terminal receives the formatted data from the server and displays it in a format that is easy for the user to view. For example, it displays "Sushi: Rice topped with raw fish, typically served with soy sauce and wasabi" along with an image of the dish. The input is the formatted data from the server, and the output is the information that is displayed to the user.

[1462] Step 9:

[1463] When a user wants to ask a staff member a question, they speak into the microphone on their smartphone to input the question. For example, they might say, "What kind of fish is used in this sushi?" The input is the user's voice data.

[1464] Step 10:

[1465] The device captures the audio data and sends it to the server. The input is the audio data and the output is the data sent to the server.

[1466] Step 11:

[1467] The server receives the voice data and converts it into text data using voice recognition technology. The server then uses a voice recognition API (e.g., Google Cloud Speech-to-Text API) to convert the voice into text. The input is voice data, and the output is text data.

[1468] Step 12:

[1469] The server uses a generative AI model to translate text data into another specified language. The server uses generative AI (e.g., OpenAI GPT-4) to translate the text data. The input is the text data obtained from speech recognition, and the output is the translated text data.

[1470] Step 13:

[1471] The server sends the translation result back to the terminal, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?" The input is the text data of the translation result, and the output is the information displayed to the user or staff.

[1472] Step 14:

[1473] The device sends the user's voice data and image data obtained from the camera to the emotion engine. The input is voice data and image data. The emotion engine uses this data to recognize the user's emotion. The output is the recognized emotion data.

[1474] Step 15:

[1475] The server adjusts the translation results and display content based on the recognized emotion data. For example, if it determines that the user is feeling anxious, it adds an explanatory text and changes the expression to be more friendly. The input is emotion data and translation results, and the output is the adjusted display content.

[1476] Step 16:

[1477] The device displays the adjusted translation results and display content to the user, allowing the user to use the service comfortably. The input is the adjusted display content, and the output is the information displayed to the user.

[1478] Step 17:

[1479] The user selects a dish and presses the order button. The input is the user's selection. The terminal sends the information about the selected dish to the server.

[1480] Step 18:

[1481] The terminal sends the order details of the selected dishes to the server. The input is the dish information, and the output is the data sent to the server.

[1482] Step 19:

[1483] The server transfers the order information to the restaurant's ordering system and sends a confirmation of receipt of the order information to the terminal. The input is the order information, and the output is the transfer result to the ordering system and confirmation data to the terminal.

[1484] Step 20:

[1485] The terminal transitions to the payment screen, where the user can select the desired payment method, such as credit card, electronic money, etc. The input is the user's selection of payment method.

[1486] Step 21:

[1487] The user inputs payment information and confirms the payment. The input is payment information, which the terminal sends to the server.

[1488] Step 22:

[1489] The terminal sends payment information to the server, and the server sends a payment request to the specified payment service (e.g., Stripe API). The input is payment information, and the output is request data for the payment service.

[1490] Step 23:

[1491] The server receives the payment result and notifies the terminal. The input is the payment result from the payment service, and the output is notification data.

[1492] Step 24:

[1493] The terminal displays a payment completion screen to the user, completing the series of operations. The input is the payment result notification data, and the output is a payment completion message that is displayed to the user.

[1494] (Application example 2)

[1495] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1496] Conventional restaurant menu scanning and translation systems simply translate the menu contents into other languages ​​and do not provide personalized service based on the user's emotions or questions. Furthermore, the user's emotional state is not taken into account when asking a question or placing an order, which can result in a decline in service quality. Furthermore, the payment process is not smooth, which can cause inconvenience to users. A system that can solve these problems and provide a more comfortable and personalized service is needed.

[1497] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1498] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for recognizing emotions from user voice data and image data, and means for adjusting the display content and translation result based on the recognized emotion. This allows users not only to scan the menu and obtain translated information, but also to translate voice-input questions and orders in real time, enabling personalized responses according to the user's emotions and smooth payment.

[1499] The "means for acquiring image data of a menu" is a function that allows a user to use the camera of the device they own to scan a restaurant menu and acquire the image as data.

[1500] The "means for extracting text information" is a function for recognizing text from the acquired menu image and extracting it as digital text.

[1501] The "means for automatic translation" is a function for automatically translating extracted text information into another language.

[1502] The "means for obtaining detailed information about dishes" is a function for obtaining detailed information about each dish, such as ingredients and descriptions, from a database based on the translated menu information.

[1503] "Means for displaying on the user interface" refers to a function for displaying detailed information about the acquired dish and the translation results on the device screen in a format that is easy for the user to understand.

[1504] The "means for receiving user voice input" is a function for receiving voice data input by the user using a microphone.

[1505] The "means for translating voice input into another language" is a function for converting received voice data into text and translating the text into another language.

[1506] "Means for displaying translation results to staff" is a function for displaying translated questions and instructions to store staff.

[1507] The "means for sending order information to the store's ordering system" is a function for sending order information for the dishes selected by the user to the order management system in the store.

[1508] "Payment method" is a function for making payment using the payment method selected by the user based on the order information.

[1509] "Means for notifying the user of the payment result" is a function for notifying the user that the payment has been completed.

[1510] "Means for recognizing emotions" is a function for analyzing and recognizing the emotional state of a user from their voice data and image data.

[1511] The "means for adjusting the display content and translation results" is a function for adjusting the display content and translation results of the user interface based on the recognized emotion to adapt to the user's emotion.

[1512] The following describes an embodiment of the present invention: This system integrates menu scanning and translation, voice input and emotion recognition, ordering and payment, and is composed of the following components:

[1513] System configuration

[1514] 1. User's device: This includes smartphones and tablets, which are equipped with a camera, microphone, and display.

[1515] 2. Server: Provides functionality for image processing, natural language processing (NLP), speech recognition, generative AI, and emotion recognition engines (e.g., Google Cloud Vision API, Microsoft Translator API, OpenAI GPT-4, Affectiva SDK).

[1516] 3. Database: Stores detailed information about dishes, translation data, order history, etc.

[1517] Hardware and software usage

[1518] First, when a user enters a restaurant, they scan the menu with their smartphone camera. The device acquires this image data and sends it to a server. The server extracts text information from the image data using an OCR engine (e.g., Google Cloud Vision API) and automatically translates that information using a translation engine (e.g., Microsoft Translator API). The translation results, along with detailed information about the dish retrieved from a database, are sent to the device and displayed on the user interface.

[1519] When a user asks a question by voice input, the smartphone microphone captures the voice data. The device sends this voice data to a server, which converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text) and translates the text into another language using a translation engine (e.g., Microsoft Translator API). The translation result is sent back to the device and displayed in the user interface.

[1520] Additionally, an emotion recognition engine (e.g., Affectiva SDK) is used to analyze emotions from the user's voice data and camera images. Based on the analysis results, a generative AI (e.g., OpenAI GPT-4) is used to generate a response according to the user's emotion and display it in the user interface.

[1521] The user then enters the order information for the selected dishes via their smartphone and sends it to the server. The server then sends the information to the restaurant's order management system, which confirms the order. As for payment, the user completes the payment via an electronic payment service (e.g., Stripe API), and the payment result is notified to the terminal.

[1522] Specific examples

[1523] As a concrete example, consider a scenario in which a tourist named Tanaka uses this system at a restaurant in Japan. Tanaka scans the menu with his smartphone, which is automatically translated into English. If Tanaka voice-inputs a question in English, it is translated into Japanese and displayed. If Tanaka looks anxious, the emotion recognition engine detects this, and the generation AI generates and displays a kind sentence to reassure the user.

[1524] Prompt Sentence Examples

[1525] User Question: "What is the main ingredient of this dish?"

[1526] Prompt to spawn AI:

[1527] "Translate the following question to Japanese: 'What is the main ingredient of this dish?' and considering user shows signs of anxiety, frame your answer politely and informatively."

[1528] This allows Tanaka to order food with peace of mind and complete payment smoothly.

[1529] The above is a specific embodiment for carrying out the present invention. This system enables communication that transcends language barriers and provides personalized services.

[1530] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1531] Step 1:

[1532] A user enters a store and scans the menu with their smartphone camera. The device captures this image data and sends it to the server. In this step, the menu image data is the input, and the image data sent to the server is the output.

[1533] Step 2:

[1534] The server analyzes the received image data using an OCR engine (e.g., Google Cloud Vision API) and extracts text data. In this step, the image data sent to the server is the input, and the extracted text information is the output.

[1535] Step 3:

[1536] The server automatically translates the extracted text into other languages ​​using a translation engine (e.g., Microsoft Translator API). In this step, the extracted text is the input, and the translated text is the output.

[1537] Step 4:

[1538] The server retrieves detailed information about the dish (e.g., ingredients, descriptions) from the database based on the translated text data. In this step, the translated text data is the input, and the retrieved detailed information about the dish is the output.

[1539] Step 5:

[1540] The server formats the acquired dish details and translation results into a format suitable for the user interface and sends them to the terminal. In this step, the input is the translated text data and dish details, and the output is data formatted for the user interface.

[1541] Step 6:

[1542] The terminal receives the formatted data and displays it to the user. The user reads the menu and asks questions via voice input. At this step, we have the formatted data as input and the menu information that the user views as output.

[1543] Step 7:

[1544] The user's voice input is captured by the smartphone's microphone, and the device sends the voice data to the server. In this step, the user's voice input data is the input, and the voice data sent to the server is the output.

[1545] Step 8:

[1546] The server receives the voice data, converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text), and then translates it into another language using a translation engine (e.g., Microsoft Translator API). In this step, the input is the voice data, and the output is the translated text data.

[1547] Step 9:

[1548] The server sends the translated question to the terminal, which displays it on the user interface. In this step, the input is the translated text data, and the output is the translation result that is displayed to the user and staff.

[1549] Step 10:

[1550] The device sends the user's voice data and camera images to an emotion recognition engine (e.g., Affectiva SDK), and the server analyzes the data to recognize the user's emotional state. In this step, the input is voice data and camera images, and the output is recognized emotion data.

[1551] Step 11:

[1552] Based on the recognized emotion data, the server uses generative AI (e.g., OpenAI GPT-4) to generate emotion-based responses and adjust them for the user interface. In this step, the input is emotion data and the output is adjusted response data.

[1553] Step 12:

[1554] The terminal displays the adjusted response data to the user and provides a response in a form that is easy for the user to understand. In this step, there is the adjusted response data as input and a specific response that is displayed to the user as output.

[1555] Step 13:

[1556] The user selects a dish and enters the order information through their smartphone. The device sends the order information to the server. In this step, the user's order information is the input, and the order information sent to the server is the output.

[1557] Step 14:

[1558] The server sends the order information to the store's ordering system, and the order is confirmed. In this step, the input is the order information sent to the server, and the output is the data sent to the store's ordering system.

[1559] Step 15:

[1560] The terminal displays an order confirmation screen to the user, and the user selects the desired payment method. The user makes the payment through an electronic payment service (e.g., Stripe API). This step has the user's selected payment method and payment information as input, and the payment information sent to the server as output.

[1561] Step 16:

[1562] The server receives the payment result, notifies the terminal, and displays a payment completion screen to the user. In this step, the input is payment result data, and the output is a payment completion notification that is displayed to the user.

[1563] Through these steps, the system can complete the entire process, from menu scanning and translation, to voice input queries and emotion recognition, to ordering and payment.

[1564] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1565] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1566] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1567] [Fourth embodiment]

[1568] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1569] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1570] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1571] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1572] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1573] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1574] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1575] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1576] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1577] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1578] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1579] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1580] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1581] The invention described in this specification is a system that performs a complete process from scanning the menu, translating it, displaying detailed information, translating voice input, ordering, and payment. Specific embodiments of this system are described below.

[1582] System configuration

[1583] This system is composed of multiple components, including the user's device, the restaurant's server, and a database. The device corresponds to the user's smartphone or tablet and is equipped with a camera, microphone, and display. The server provides image processing, translation, database access, voice recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more.

[1584] Program Overview

[1585] The system's program consists of a series of processes that provide image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment functions. Each step is explained below with specific examples.

[1586] Menu scanning and translation

[1587] Example 1: Menu scanning and character recognition

[1588] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1589] 2. The device acquires the scanned image data and sends it to the server.

[1590] 3. The server receives the image data, uses OCR technology to extract the text information in the menu, and processes it as text data.

[1591] Example 2: Analysis and translation of text data

[1592] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish.

[1593] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts this information into a format for display on the user interface.

[1594] 6. The device displays the translation result and detailed information to the user, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1595] Receiving and translating voice input

[1596] Example 3: Simultaneous audio interpretation

[1597] 7. When a user has a question for a staff member, they can type it into their smartphone using voice input: "What kind of fish is used in this sushi?"

[1598] 8. The device captures the audio data and sends it to the server.

[1599] 9. The server receives the voice data and uses generative AI to translate English into Japanese in real time.

[1600] 10. The server sends the translation results to the terminal, which displays the message, "What kind of fish is used in this sushi?"

[1601] 11. Once the staff member responds, the content is sent back to the server, where it is translated into English in real time by the generating AI, and the message "This sushi is made with tuna" is displayed on the device.

[1602] Ordering and Payment

[1603] Example 4: Food ordering and payment

[1604] 12. The user selects their favorite dish and confirms the order via their smartphone.

[1605] 13. The terminal sends the order details to the server.

[1606] 14. The server receives the order information and sends it to the restaurant's ordering system.

[1607] 15. The terminal displays the payment screen to the user, for example, by selecting an electronic payment service and making the payment.

[1608] 16. User selects payment method and enters payment information.

[1609] 17. The server sends a payment request to the electronic payment service and receives the result.

[1610] 18. The terminal displays a payment completion screen to the user, completing the process.

[1611] As described above, this system significantly improves user convenience by providing a comprehensive process including multilingual menu display, voice interpretation, ordering, and payment, thereby reducing the burden of foreign language support in restaurants.

[1612] The processing flow will be explained below.

[1613] Step 1:

[1614] A user scans a restaurant menu with their smartphone camera.

[1615] Step 2:

[1616] The terminal acquires image data of the scanned menu and sends the image data to the server for transfer.

[1617] Step 3:

[1618] The image data received by the server is passed through an OCR (optical character recognition) engine to extract the character information within the image and convert it into text data.

[1619] Step 4:

[1620] The server analyzes the text data obtained by OCR, divides it into food items, and identifies the name, price, description, etc. of each item.

[1621] Step 5:

[1622] The server automatically translates each identified dish item into the specified foreign language using natural language processing (NLP) algorithms, for example, translating from Japanese to English.

[1623] Step 6:

[1624] The server retrieves detailed information (e.g., ingredients, flavor descriptions, images, etc.) for each dish from a database based on the translated dish items.

[1625] Step 7:

[1626] The server combines the detailed information and translated dish items it has obtained and formats them into a data format for display on the user interface.

[1627] Step 8:

[1628] The device receives the formatted translation data and detailed information and displays it in the user interface, for example, "Sushi - Raw fish on rice, typically served with soy sauce and wasabi."

[1629] Step 9:

[1630] If a user wants to ask a staff member a question, they can speak their question into the microphone on their smartphone.

[1631] Step 10:

[1632] The device captures the audio data and sends it to the server.

[1633] Step 11:

[1634] The server receives the voice data and converts it into text using speech recognition technology. The text data is then instantly translated into the specified language by the AI ​​generator. For example, an English question can be translated into Japanese.

[1635] Step 12:

[1636] The server sends the translation back to the device, which then displays the translated question on the screen. For example, "What kind of fish is used in this sushi?" is displayed as "What kind of fish is used in this sushi?"

[1637] Step 13:

[1638] When the staff member answers the question, the answer is entered into the terminal, which then sends the answer back to the server.

[1639] Step 14:

[1640] The server receives the staff's Japanese responses and translates them in real time using generative AI. For example, "This sushi is made with tuna" is translated into "This sushi is made with tuna."

[1641] Step 15:

[1642] The device receives the translation results and displays them to the user.

[1643] Step 16:

[1644] The user selects the dish and places the order via smartphone.

[1645] Step 17:

[1646] The terminal transmits the order details of the selected dishes to the server, and the server transfers the order information to the order management system of the restaurant.

[1647] Step 18:

[1648] The terminal transitions to the payment screen, where the user selects the desired payment method. For example, the user selects an electronic payment service.

[1649] Step 19:

[1650] The user enters payment information and confirms the payment.

[1651] Step 20:

[1652] The terminal transmits payment information to the server, and the server sends a payment request to the specified payment service.

[1653] Step 21:

[1654] The server receives the payment result and notifies the terminal.

[1655] Step 22:

[1656] The terminal displays a payment completion screen to the user, completing the series of operations.

[1657] These are the specific steps in the process, from scanning the menu to translating, asking questions, ordering, and paying, which will ensure smooth communication and service between the user and the restaurant.

[1658] Example 1

[1659] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1660] Conventional multilingual systems for restaurants handle each process separately: menu translation, detailed information display, voice interpretation, ordering, and payment. This resulted in low user convenience and a heavy burden on the restaurant. In addition, it was difficult to consistently process real-time voice interpretation, order confirmation, and payment, leading to a demand for improved user experience and operational efficiency.

[1661] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1662] In this invention, the server includes means for acquiring image data of a menu, means for extracting text information from the acquired image data, means for automatically translating the extracted text information, means for acquiring detailed information about a dish based on the translated information, means for displaying the acquired detailed information and the translation result on a user interface, means for receiving a user's voice input, means for translating the received voice input into another language, means for displaying the translation result to a staff member, means for sending order information about a dish selected by the user to an ordering system of the restaurant, means for making a payment based on the order information, means for notifying the user of the payment result, means for using OCR technology and natural language processing technology, means for acquiring information from a database based on the translation result, and means for translating voice data in real time using a generative AI model. This allows for smooth multilingual support, making it possible to provide a system that is efficient and convenient for both users and restaurants.

[1663] "Image data" refers to visual information captured by a device such as a camera or scanner and stored in digital form.

[1664] "Textual information" refers to information that is displayed as text, typically expressed in language or characters.

[1665] "Translation" is the process of transcribing text written in one language into a different language.

[1666] "Detailed information" refers to more specific and detailed information about a particular item, including ingredients, images, and flavor descriptions.

[1667] A "user interface" is an interface that exchanges information between a computer system and a user, and typically includes a display, buttons, a touch panel, etc.

[1668] "Voice input" is the process of capturing a user's voice as digital data using a microphone device.

[1669] "Another language" means a language other than the original language.

[1670] "Staff" refers to employees and service providers working at restaurants and other establishments.

[1671] "Order Information" refers to detailed data about the products or services selected by the user, including the quantity and selections.

[1672] "Ordering system" refers to an order management system used in restaurants and stores to receive, process, and manage order information.

[1673] "Payment" refers to the process of paying for goods and services, and includes credit cards, electronic money, cashless payments, etc.

[1674] "OCR technology" stands for optical character recognition technology, which refers to the technology that converts character information from paper documents, image data, etc. into digital text.

[1675] "Natural language processing technology" refers to technology that enables computers to understand, interpret, and generate human language, and includes translation, speech recognition, and dialogue systems.

[1676] A "generative AI model" is a model that uses artificial intelligence to generate data, and is used for tasks such as natural language generation and image generation.

[1677] The present invention is a system that utilizes a user's terminal, a store's server, and a database to provide a series of processes from scanning the menu to translating it, displaying detailed information, translating voice input, ordering, and payment. The specific configuration and operation for implementing the present invention are described in detail below.

[1678] System configuration

[1679] This system consists of a user-held device (such as a smartphone or tablet), a restaurant server, and a database. The device is equipped with a camera, microphone, and display, and is used to scan menus and input information by voice. The server provides functions using image processing, translation, database access, voice recognition, and generative AI models. The database stores detailed information about dishes, translation data, order history, etc.

[1680] Program Overview

[1681] The program of this system consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, user interface, and electronic payment. The specific processing content of each step is explained below.

[1682] Menu scanning and translation

[1683] Example 1: Menu scanning and character recognition

[1684] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1685] 2. The device acquires the scanned image data and sends it to the server. At this time, the device encodes the image data in JPEG format and uploads it to the server.

[1686] 3. The server receives the image data and extracts the text information from the menu using OCR technology (e.g., Google Cloud Vision API). The extracted text data is stored in a temporary database.

[1687] Example 2: Analysis and translation of text data

[1688] 4. The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API).

[1689] 5. Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, taste descriptions, etc.) from the database and converts it into a format for display on the user interface.

[1690] 6. The device displays the translation result and detailed information to the user. For example, the information "Sushi - Raw fish on rice, typically served with soy sauce and wasabi" is displayed on the screen.

[1691] Receiving and translating voice input

[1692] Example 3: Simultaneous audio interpretation

[1693] 1. When a user wants to ask a question to a staff member, they type it into their smartphone: "What kind of fish is used in this sushi?"

[1694] 2. The device captures the audio data and sends it to the server, where it encodes it in FLAC format and uploads it to the server.

[1695] 3. The server receives the voice data, analyzes it using a speech recognition API (e.g., Google Cloud Speech-to-Text), and extracts text data.

[1696] 4. The server translates the resulting text using a generative AI model (e.g., GPT-3).

[1697] 5. The device displays the translation result to the user, asking, "What kind of fish is used in this sushi?"

[1698] 6. Once the staff member responds, their voice is sent back to the server, and the translation is converted into English in real time. "This sushi is made with tuna" is displayed on the device.

[1699] Ordering and Payment

[1700] Example 4: Food ordering and payment

[1701] 1. The user selects their favorite dish and confirms the order via smartphone.

[1702] 2. The terminal sends the order details to the server. The order data is encoded in JSON format.

[1703] 3. The server receives the order information and sends it to the restaurant's ordering system.

[1704] 4. The terminal displays the payment screen to the user, for example, using an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.).

[1705] 5. The user selects a payment method and enters payment information.

[1706] 6. The server sends a payment request to the electronic payment service and receives the result.

[1707] 7. The terminal displays a payment completion screen to the user, completing the process.

[1708] Examples of prompt statements

[1709] Here are some examples of input prompts for generative AI models:

[1710] 1. Prompt the user as they scan the menu:

[1711] "Please take a picture of the Japanese menu. After taking the picture, the image will be sent to the server."

[1712] 2. Prompt the user when using voice input:

[1713] "Please speak your question in English. When you're done, press the send button."

[1714] 3. Prompt the user to confirm the order:

[1715] Please confirm your selection and confirm your order.

[1716] 4. Prompt user when making payment:

[1717] Select your preferred payment method and enter your payment information.

[1718] As a result, this system handles everything from menu translation to ordering and payment, improving the user experience while reducing the burden on the store.

[1719] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1720] Program processing flow

[1721] Menu scanning and translation

[1722] Step 1:

[1723] A user enters a restaurant and scans a Japanese menu with their smartphone camera. The user launches a camera app and takes a picture of the menu. The input is image data of the menu, and the output is a JPEG image file saved on the device.

[1724] Step 2:

[1725] The device acquires the scanned image data and sends it to the server. The device encodes the image data in JPEG format and uploads it to the server using an HTTP POST request. The input is a JPEG image file, and the output is an HTTP request to the server.

[1726] Step 3:

[1727] The server receives the image data and uses OCR technology (e.g., Google Cloud Vision API) to extract the text information in the menu. The server passes the image data to the API and receives text data as the analysis result. The input is JPEG image data, and the output is the extracted text data.

[1728] Step 4:

[1729] The server analyzes the text data obtained by OCR and automatically translates the names and descriptions of each dish (e.g., Google Cloud Translate API). The server passes the text data to the translation API and receives the translation results as text data. The input is the text data extracted by OCR, and the output is the translated text data.

[1730] Step 5:

[1731] Based on the translation results, the server retrieves detailed information about each dish (ingredients, images, flavor descriptions, etc.) from the database and converts it into a format to display on the user interface. The server queries the database using the translated text data as a key, retrieves the relevant detailed information, and formats it. The input is the translated text data, and the output is formatted data including the detailed information.

[1732] Step 6:

[1733] The terminal displays the translation results and detailed information received from the server to the user. The terminal receives the formatted data and displays it on the display. The input is the formatted data sent from the server, and the output is the display content that the user visually perceives.

[1734] Receiving and translating voice input

[1735] Step 7:

[1736] When a user wants to ask a question to a staff member, they input it into their smartphone: "What kind of fish is used in this sushi?" The user taps the voice input button and then asks the question. The input is the user's voice data, and the output is the voice data captured on the device.

[1737] Step 8:

[1738] The device captures audio data and sends it to the server. The device encodes the audio data in FLAC format and uploads it to the server using an HTTP POST request. The input is the captured audio data, and the output is an HTTP request to the server.

[1739] Step 9:

[1740] The server analyzes the received audio data using a speech recognition API (e.g., Google Cloud Speech-to-Text) and extracts it as text data. The server passes the audio data to the API and receives text data as the analysis result. The input is audio data in FLAC format, and the output is the extracted text data.

[1741] Step 10:

[1742] The server translates the obtained text using a generative AI model (e.g., GPT-3). The server passes the text data to the translation model and receives the translation result as text data. The input is text data extracted by speech recognition, and the output is translated text data.

[1743] Step 11:

[1744] The terminal displays the translation result to the user. It displays "What kind of fish is used in this sushi?" The terminal receives the translated text data and displays it on the display. The input is the translated text data sent from the server, and the output is the display content that the user sees.

[1745] Step 12:

[1746] When the staff member responds, their voice is sent back to the server, and the translation result is converted into English in real time. "This sushi is made with tuna" is displayed on the terminal. The input is the staff member's voice data and the results of the translation model, and the output is what the user sees visually.

[1747] Ordering and Payment

[1748] Step 13:

[1749] The user selects a dish and confirms the order via their smartphone. The user taps the order confirmation button to submit the order. The input is the order details selected by the user, and the output is the order data stored on the device.

[1750] Step 14:

[1751] The terminal sends the order details to the server. The terminal encodes the order data in JSON format and uploads it to the server using an HTTP POST request. The input is the order details data, and the output is an HTTP request to the server.

[1752] Step 15:

[1753] The server receives the order information and sends it to the restaurant's ordering system. The server sends the order data to the ordering system and waits for a response. The input is the order data in JSON format, and the output is the result sent to the ordering system.

[1754] Step 16:

[1755] The terminal displays the payment screen to the user. The user selects an electronic payment service (e.g., PayPay, Apple Pay, Google Pay, etc.). The input is the user's confirmation action, and the output is the content displayed on the payment screen.

[1756] Step 17:

[1757] The user selects a payment method and enters payment information. The user may use a credit card or an electronic payment app. The input is the user's payment method and payment information, and the output is the data entered into the terminal.

[1758] Step 18:

[1759] The server sends a payment request to the electronic payment service and receives the result. The server sends a payment request and receives a success or failure response. The input is the user's payment information, and the output is the payment result.

[1760] Step 19:

[1761] The terminal displays a payment completion screen to the user. The terminal receives the payment result and displays "Payment completed." The input is the payment result data, and the output is the display content that the user sees.

[1762] The above is the specific processing flow of this system.

[1763] (Application example 1)

[1764] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1765] This system solves the problem of users who cannot understand foreign languages ​​having difficulty accurately understanding products and menus at restaurants and retail stores, and smoothly ordering and paying. Stores also often struggle with multilingual support, making it difficult for them to serve foreign customers. Furthermore, conventional systems often have low translation accuracy or are unable to provide real-time translations or display detailed information. This can result in a poor user experience and have a negative impact on store sales and customer satisfaction.

[1766] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1767] In this invention, the server includes a means for acquiring menu image data, a means for extracting text information from the acquired image data, a means for automatically translating the extracted text information, a means for acquiring detailed content information based on the translated information, and a means for displaying the acquired detailed information and translation results on a user interface. This allows users to easily understand foreign language menus and smoothly order and pay. The system also includes a means for receiving user voice input, a means for translating the received voice input into another language, a means for notifying the translation results, a means for transmitting user-selected order information to the system, a means for making a payment based on the order information, and a means for notifying the user of the payment results, thereby efficiently managing complex dialogues, ordering, and payment processes. The system also includes a means for generating highly accurate translations and detailed information using a generative AI model, and a means for acquiring prompts from the user in advance and suggesting appropriate prompts based on the order status. This provides a system that is easy to use for both users and stores, and has simple and accurate foreign language support.

[1768] - "Menu image data" refers to image data of paper or digital menus of restaurants or retail stores that list items and prices, obtained using the camera or scanner on the user's device.

[1769] "Text information" is text data extracted from the image data of the menu, and includes information such as the name of the dish, the price, and a description.

[1770] An "automated translation tool" is software or a service that mechanically converts textual information obtained by a user into a different language, typically using a translation engine or a generative AI model.

[1771] "Content Details" is additional information about the translated menu item, including ingredients, allergens, cooking instructions, pairings, etc.

[1772] A "user interface" is an interactive screen or display device that allows a user to interact with a system, such as the display of a smartphone or tablet.

[1773] "Voice input" refers to the voice information that a user speaks into a terminal, and is input in the form of commands, questions, comments, etc.

[1774] "Means for notifying the translation result" refers to a method for visually or audibly communicating the translated information to the user or store staff.

[1775] "Order information" refers to data including order details such as the dishes and products selected by the user, quantities, and special notes.

[1776] "Means for sending to the system" refers to a method for sending order information from the user's terminal to the store's server or ordering system.

[1777] "Payment method" means the method by which a user electronically pays for the items they have ordered, such as by using a credit card or electronic payment service.

[1778] "Payment result" is information indicating that the payment has been completed, and includes the status of success or failure.

[1779] A "generative AI model" is an artificial intelligence model that has been trained to perform tasks such as natural language processing, image recognition, and translation.

[1780] A "prompt" is a concise, clear instruction or question entered by a user that is used as input to a generative AI model.

[1781] MODE FOR CARRYING OUT THE INVENTION

[1782] The invention described in this specification is a system that uses a user's terminal, a store's server, and a database to consistently display multilingual menus, perform voice interpretation, and handle ordering and payment. Specific embodiments for implementing this system are described below.

[1783] System configuration

[1784] 1. User's Device

[1785] This refers to the user's smartphone or tablet, which is equipped with a camera, microphone, and display, allowing for menu scanning and voice input. This allows the user to capture image data of the menu using the smartphone's camera and send that data to a server.

[1786] 2. Server

[1787] The server includes the following main modules to efficiently perform multiple processes:

[1788] Image Recognition Module: Using OCR technology, the server extracts text information from the image data sent by the user. Specifically, it uses Tesseract OCR.

[1789] Translation module: Uses the Google Translate API and generative AI models to translate extracted text into multiple languages, enabling natural-sounding, highly accurate translations.

[1790] Detailed information acquisition module: Based on the translation results, detailed information about the corresponding dish (ingredients, allergen information, images, etc.) is acquired from the database.

[1791] Speech Recognition Module: Captures user voice input and transcribes it using the Google Speech-to-Text API, while also translating it in real time using a generative AI model.

[1792] Payment module: Receives order information and processes payments through electronic payment services such as Stripe.

[1793] 3. Database

[1794] It connects to a server and stores detailed information about dishes, translation data, order history, etc. The database can use a common SQL database such as MySQL or PostgreSQL.

[1795] Example

[1796] Menu scanning and translation

[1797] A user enters a restaurant and scans the menu with their smartphone camera. The image data is sent to a server, where text information is extracted using OCR technology. The extracted text information is translated using a generative AI model, and the results are displayed on the user's device.

[1798] Receiving and translating voice input

[1799] When a user wants to ask a question to a staff member, they type it into their smartphone. Questions such as "What kind of fish is used in this sushi?" are sent to the server and recognized. The generative AI model translates the question in real time, and the results are displayed on the device.

[1800] Ordering and Payment

[1801] Based on the translation and detailed information, the user selects their favorite dish and confirms the order via smartphone. The order details are sent to the restaurant's ordering system via the server, and at the same time, a payment screen is displayed on the user's smartphone. The user selects an electronic payment service to make the payment, and the server notifies the user of the payment result.

[1802] Prompt Sentence Examples

[1803] Here are some examples of prompts:

[1804] User scans the menu: "Scan the menu."

[1805] The user wants to know more about a dish: "Tell me more about this dish."

[1806] The user asks aloud: "What kind of fish is used in this sushi?"

[1807] User places an order: "I'd like to order this."

[1808] User pays: "I'll pay with my credit card."

[1809] This makes the system easy to use for both users and stores, and enables the provision of simple, highly accurate services in foreign languages.

[1810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1811] Step 1:

[1812] A user uses the camera on their smartphone to acquire image data of a menu. The user takes a picture of a restaurant menu with the camera and inputs the image data. The device then sends this image data to a server. The input is the image data of the menu, and the output is the transmission of the image data to the server.

[1813] Step 2:

[1814] The server uses OCR technology to extract text from the image data. Specifically, it uses Tesseract OCR to convert the text in the image into text data. The input is the image data, and the output is the extracted text.

[1815] Step 3:

[1816] The server translates the extracted text using a generative AI model. Here, a language translation service (e.g., Google Translate API) is used to convert the text into another language. The input is text, and the output is translated text data.

[1817] Step 4:

[1818] The server retrieves detailed information from the database based on the translated text data. Specifically, it searches for detailed data such as ingredient information and allergen information based on the translated dish name, etc. The input is the translated text data, and the output is the detailed information.

[1819] Step 5:

[1820] The server converts the detailed information and translation results into a format that can be displayed on the user interface and sends it to the terminal. Here, the information is integrated into a user-friendly format and displayed on the screen. The input is the detailed information and translation results, and the output is the display data on the user interface.

[1821] Step 6:

[1822] The user uses the microphone of the device to input voice, for example, to ask a question about cooking. The device sends the captured voice data to the server. The input is the voice data, and the output is the transmission of the voice data to the server.

[1823] Step 7:

[1824] The server transcribes the audio data and translates it in real time using a generative AI model. Specifically, it converts the audio data into text using the Google Speech-to-Text API, and then translates the text. The input is audio data, and the output is translated text data.

[1825] Step 8:

[1826] The server sends the translation results to the terminal and displays them. The user can check the displayed translation results and get a response to the question they had from the staff. The input is the translated text data, and the output is the data displayed on the terminal.

[1827] Step 9:

[1828] The user confirms the order using the terminal. The user specifies the dish they have selected from the menu and presses the order button. The terminal sends the order information to the server. The input is the order information, and the output is the transmission of the order information to the server.

[1829] Step 10:

[1830] The server receives the order information and sends it to the restaurant's ordering system. Specifically, it uses the ordering system's API to link the order details. The input is the order information, and the output is the information sent to the ordering system.

[1831] Step 11:

[1832] The server generates a payment screen and sends it to the terminal. The user checks the payment screen and selects a payment method. The input is order information, and the output is the payment screen.

[1833] Step 12:

[1834] The user selects a payment method and enters the necessary information. The terminal sends the payment information to the server. The input is the payment information, and the output is the transmission of the payment information to the server.

[1835] Step 13:

[1836] The server connects to the electronic payment system to execute the payment. Specifically, it sends a request and receives the result using an electronic payment service such as Stripe. The input is the payment information, and the output is the payment result.

[1837] Step 14:

[1838] The server notifies the terminal of the payment result. The user confirms the completion of the payment on the terminal. The input is the payment result, and the output is the notification to the terminal.

[1839] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1840] The invention described in this specification is a system that provides more personalized services by combining menu scanning and translation, detailed information display, voice input and translation, ordering and payment, as well as an emotion engine that recognizes user emotions. Specific embodiments of the system are described below.

[1841] System configuration

[1842] This system is composed of multiple components, including the user's device, the restaurant's server, a database, and an emotion engine. The device is the user's smartphone or tablet, equipped with a camera, microphone, and display. The server provides image processing, natural language processing (NLP), speech recognition, and generative AI functions. The database stores detailed information about dishes, translation data, order history, and more. The emotion engine recognizes emotions from the user's voice input and facial expressions, and provides services based on those emotions.

[1843] Program Overview

[1844] The system's program consists of a series of processes that provide functions such as image recognition, natural language processing (NLP), speech translation, emotion recognition, user interface, and electronic payment. Each step is explained below with concrete examples.

[1845] Menu scanning and translation

[1846] Example 1: Menu scanning and character recognition

[1847] 1. The user enters a restaurant and scans the Japanese menu with their smartphone camera.

[1848] 2. The device acquires the scanned image data and sends it to the server.

[1849] 3. The se...

Claims

1. A means for obtaining image data of the menu; means for extracting character information from the acquired image data; A means for automatically translating the extracted text information; A means for obtaining detailed information about a dish based on the translated information; a means for displaying the obtained details and translation results in a user interface; a means for receiving user voice input; means for translating the received voice input into another language; A means of displaying the translation results to staff; A means for transmitting order information of the food selected by the user to an ordering system of the restaurant; a means for making a payment based on the order information; A means of notifying users of payment results A system including:

2. 2. The system according to claim 1, further comprising means for displaying the translated menu information including images of the dishes.

3. 2. The system of claim 1, further comprising means for translating the user's voice input in real time and displaying it to the staff.

Images