System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving voice input, a means for converting voice data into text, a means for analyzing the text data to specify order contents, a means for feeding back the generated order contents by voice, and a means for registering the finally confirmed order contents in a database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Operating tablets and touchscreen ticket vending machines in restaurants can be difficult, especially for the elderly, visually impaired, foreigners, and those unfamiliar with the devices. This can cause inconvenience to customers when ordering and often leads to questions or requests for assistance from staff. This situation can further burden the restaurant and staff, potentially impacting overall operational efficiency. The present invention aims to resolve these issues and improve overall customer satisfaction. [Means for solving the problem]

[0005] The present invention provides a system that allows users to place orders through voice input by incorporating voice recognition technology and generation AI into the ordering system for restaurants. Specifically, the system comprises the following means:

[0006] 1. A means for receiving audio input.

[0007] 2. A means of converting voice data into text.

[0008] 3. A means of analyzing text data to identify order details.

[0009] 4. A means of providing voice feedback on the generated order.

[0010] 5. A means of registering the final confirmed order details in the database.

[0011] The system may also include means for playing back voice feedback to the user and re-analyzing the user's confirmation response, means for identifying multiple languages ​​and providing feedback in a specified language, and means for analyzing voice data and text data to collect and store preference data. This eliminates the difficulty of touch panel operation for elderly people, visually impaired people, foreigners, and people who are not accustomed to operating touch panels, making ordering procedures easier. It also reduces the burden on stores and store staff and improves work efficiency.

[0012] "Voice input" is a means by which a user communicates information or instructions to a system using their voice.

[0013] "Voice Data" refers to a digital recording of a user's voice input.

[0014] "Text conversion" refers to the process of analyzing the content of audio data and converting it into character data (text).

[0015] "Order details" refers to information that the user uses voice input to inform the system of the selection of desired products and services.

[0016] "Generative AI" refers to systems or software that use artificial intelligence techniques to generate natural-looking language or speech.

[0017] "Feedback" refers to the system responding to or confirming a user's input or actions.

[0018] "Final confirmation" is a process of finally confirming that the order details entered by the user are accurate.

[0019] A "database" is an electronic system for organizing and storing information.

[0020] "Voice recognition technology" refers to the technology that analyzes a user's voice and converts it into text data.

[0021] "Preference data" refers to information related to a user's past ordering history and preferences.

[0022] "Multilingual feedback" refers to the system providing responses and confirmations in the language specified by the user.

[0023] "Terminal" refers to a device that is directly operated by a user.

[0024] A "server" refers to a computer that manages the processing and data of the entire system on a network. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] The present invention is a system that combines voice recognition technology and generative AI, allowing users to place orders through voice input. Below, the program processing of this system is explained in natural language, and an embodiment is explained using specific examples.

[0047] Server-side processing

[0048] The server receives the voice data sent from the terminal. This is the first step in recognizing what the user wants to order. Once the voice data is received, the server uses a speech recognition engine to convert the received voice data into text. For example, the server can use the Google® Speech-to-Text API or other voice recognition services.

[0049] The server then uses generative AI to analyze the order details from the converted text data. For example, it extracts specific order details such as "one cup of coffee, no milk." Based on this analysis, the server generates a confirmation message to provide feedback to the user. The generated message is in a natural conversational format, such as "One cup of coffee, no milk, okay?"

[0050] The server converts this confirmation message into voice data and sends the generated voice data to the terminal. The server then receives additional input from the user, such as a confirmation response of "yes" or "no," analyzes the content, and performs a final confirmation. The final confirmed order details are then registered in a database, ensuring that the order is processed accurately.

[0051] Finally, the server generates a voice message to notify the user that the order has been completed and sends the voice message to the terminal, allowing the user to confirm that the order has been successfully received.

[0052] Terminal side processing

[0053] The device detects when the user starts voice input and activates recording mode. Voice input begins when the user taps the microphone icon or speaks a voice input command. The device records the user's speech and sends the voice data to the server.

[0054] When the terminal receives the feedback voice data from the server, it plays this voice data to the user. The user checks the feedback content and responds again if necessary. The terminal also receives a notification voice indicating that the order has been completed and plays it to the user.

[0055] User operations

[0056] The user speaks into the device's microphone to input the specific details of their order. For example, they might say, "I'd like a beer and a hamburger, please." When the feedback is returned, the user checks the details and responds "Yes" if there are no problems. If corrections are needed, they can say "No, not that one," followed by the new order details.

[0057] Once the final confirmation is complete, the user receives a notification from the terminal that the order has been completed, knowing that the order has been successfully accepted and can wait for the product in peace.

[0058] Specific examples

[0059] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. It then converts the confirmation message "One cup of coffee, no milk, okay?" into voice data and sends it to the device. If the user responds "Yes," the server performs a final confirmation, registers the order details in the database, and generates a notification of order completion that is sent to the device.

[0060] This system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation, which improves store operational efficiency and overall user satisfaction.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[0064] Step 2:

[0065] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[0066] Step 3:

[0067] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[0068] Step 4:

[0069] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[0070] Step 5:

[0071] The server generates a feedback message based on the parsed order details, using generative AI to create a natural conversational message, such as "Would you like a beer and a hamburger?"

[0072] Step 6:

[0073] The server converts the generated feedback message into voice data and transmits it to the terminal.

[0074] Step 7:

[0075] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[0076] Step 8:

[0077] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[0078] Step 9:

[0079] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[0080] Step 10:

[0081] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[0082] Step 11:

[0083] The server registers the final confirmed order details in the database, which officially accepts the order.

[0084] Step 12:

[0085] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal.

[0086] Step 13:

[0087] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[0088] This process flow provides a natural ordering experience using voice, making it easy for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation of the system to order, improving store operational efficiency and overall user satisfaction.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In conventional voice input systems, the accuracy of voice recognition and the generation of natural conversational formats were issues in the process of accurately understanding a user's order and providing feedback. Furthermore, when a user speaks multiple languages, it was difficult to switch languages ​​and adapt feedback accordingly. This resulted in problems that reduced user satisfaction for users who were unfamiliar with the system, such as the elderly, the visually impaired, and foreigners.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order details, means for analyzing the order details using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order details, and means for registering the final confirmed order details in a database. This allows the user to accurately communicate the order details through voice input and receive feedback in a natural conversational style. Furthermore, appropriate feedback can be provided even when the user speaks multiple languages.

[0094] "Means for receiving voice input" refers to technology for capturing and recording voice data spoken by a user into the terminal.

[0095] "Means for converting voice data to text" refers to a technique that uses a voice recognition engine to convert received voice data into corresponding text data.

[0096] The "means for identifying the order details by analyzing the text data" refers to a technology for analyzing the converted text data and extracting the specific order details that the user is trying to convey.

[0097] "Means of analyzing order content using a generative AI model" refers to technology that utilizes generative AI to accurately read the user's intended order content from secured text data.

[0098] The "means for generating a feedback message in a natural conversational style based on the analysis results" refers to a technology for generating a confirmation message to the user in a natural conversational style based on the analyzed order details.

[0099] The "means for providing voice feedback on the generated order details" refers to a technology for converting the generated confirmation message into voice data and providing it to the user.

[0100] "Means for registering the final confirmed order details in a database" refers to a technology for saving the order details finally confirmed by the user in a database and managing them as order processing information.

[0101] "Means for playing back audio feedback to the user" refers to a technique for playing back the generated audio data at the terminal and providing feedback to the user.

[0102] The "means for receiving and re-analyzing a confirmation response from the user" refers to a technology for re-analyzing the user's response voice and determining the next action based on the content of the analysis.

[0103] "Means for identifying multiple languages ​​and providing feedback in a specified language" refers to technology that automatically determines the language used by a user and provides appropriate feedback in that language.

[0104] This invention is a system that allows users to place orders through voice input using a system that combines voice recognition technology and a generative AI model. This system is mainly composed of a server, a terminal, and a user.

[0105] The server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order contents, means for analyzing the order contents using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order contents, means for registering the final confirmed order contents in a database, and means for identifying multiple languages ​​and providing feedback in a specified language.

[0106] The terminal includes means for detecting a user's voice input, activating a recording mode, and playing feedback voice data from the server. The user provides voice input through a microphone of the terminal and transmits the voice data to the server.

[0107] The server converts the received voice data into text using the Google Speech-to-Text API or other voice recognition services. It then uses a generative AI model (e.g., GPT-3 (registered trademark)) to analyze the order details from the converted text data. Based on the analysis results, it generates a feedback message in a natural conversational style, such as "Would you like a cup of coffee, without milk?", converts it into voice data, and sends it to the device.

[0108] The terminal plays a feedback voice to the user, and the user responds with a confirmation. The user's response voice is also sent from the terminal to the server. The server again performs voice recognition and analysis, and registers the final confirmed order details in the database. This ensures that the order is processed accurately.

[0109] As a concrete example, consider the case where a domestic customer says, "One hamburger, one glass of coke." When the user speaks, the device records this voice and sends it to the server. The server converts the voice data into text and analyzes the order using a generative AI model. Based on the analysis results, it generates a feedback message saying, "One hamburger, one glass of coke, would you like?" and sends it as voice data to the device. If the user responds "Yes," the server analyzes again, registers the order in the database, and notifies the user that "the order has been completed."

[0110] For example, by inputting an example of a prompt sentence such as "Please explain the detailed steps of the process in which the user says 'one hamburger, one glass of coke'", the detailed process described above is generated.

[0111] The system provides an intuitive and easy-to-use interface, particularly for elderly people, visually impaired people, foreigners, and users unfamiliar with the operation, improving store operational efficiency and overall user satisfaction.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user begins voice input. The user speaks into the device's microphone, saying, "A cup of hot coffee, no sugar." This becomes the base data for subsequent processing.

[0115] Step 2:

[0116] The device detects the user's speech and activates recording mode. The device's microphone captures and records the audio data. The input is audio data, and the output is a recorded audio file.

[0117] Step 3:

[0118] The device sends the recorded audio data to the server via Wi-Fi or mobile data network. The input is the recorded audio data, and the output is the audio data sent as an HTTP request to the server.

[0119] Step 4:

[0120] The server converts the received voice data into text. This is done using speech recognition software such as the Google Speech-to-Text API. The input is voice data, and the output is text data generated based on that voice data. Specifically, the generated text might be something like "A cup of hot coffee, no sugar."

[0121] Step 5:

[0122] The server uses a generative AI model to analyze the order. During analysis, it uses text data as input to identify each element of the order (drink, temperature, whether sugar is added, etc.). The input is text data, and the output is a data structure of the analyzed order. For example, the drink might be "coffee," the temperature "warm," and the sugar "no."

[0123] Step 6:

[0124] The server generates a feedback message in a natural conversational format based on the analysis results. It uses a generative AI model to generate a message such as "Would you like a cup of coffee, no sugar?" and converts it into voice data. The input is the analysis results, and the output is the generated voice message.

[0125] Step 7:

[0126] The server sends the generated feedback voice data to the device. The communication method is again Wi-Fi or mobile data network. The input is voice data, and the output is voice data sent as an HTTP response to the device.

[0127] Step 8:

[0128] The terminal plays back the feedback audio data received from the server. The user can hear the audio feedback, "Would you like a cup of coffee, no sugar?" The input is the audio data received from the server, and the output is the audio playback.

[0129] Step 9:

[0130] The user confirms the feedback voice and responds again with voice, for example, saying "Yes, that's fine." This provides the user with final confirmation.

[0131] Step 10:

[0132] The terminal re-records the user's confirmation response and sends it to the server. The input is the re-recorded voice data, and the output is the response voice data to be sent as an HTTP request to the server.

[0133] Step 11:

[0134] The server converts the user's confirmation response into text using a speech recognition engine and analyzes the content. The input is voice data, and the output is the analyzed text data. For example, a final confirmation text such as "Yes, that's it, please" is generated.

[0135] Step 12:

[0136] The server registers the order details that have been finally confirmed in the database. The input is the data of the final confirmed order details, and the output is the order information saved as a record in the database.

[0137] Step 13:

[0138] The server generates a voice message notifying the terminal that the order has been processed successfully, and sends it to the terminal. Specifically, it sends a message such as "Your order has been completed. Thank you." The input is the order completion information, and the output is the generated voice message data.

[0139] Step 14:

[0140] The terminal plays audio data to notify the user that the order has been completed. The user hears this audio notification and confirms that the order has been accepted. The input is audio data from the server, and the output is audio playback.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] Conventional food delivery ordering systems are often complex and difficult to operate for elderly people, visually impaired people, and users who are not familiar with the system. Language barriers and misunderstandings of detailed order details can also be problematic. This can lead to ordering errors and user dissatisfaction. Therefore, the objective of this invention is to provide a system that uses voice recognition technology and generative AI to allow users to easily and accurately order food delivery.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for providing voice feedback on the generated order details, means for registering the final confirmed order details in a database, means for generating a confirmation message for the user using a smartphone and converting it into voice data, and means for analyzing the user's voice input and identifying the food delivery order details by combining voice recognition technology and generation AI. This enables even users who are particularly elderly, visually impaired, or unfamiliar with operation to easily place orders by voice and accurately confirm their orders.

[0146] "Voice input" is the process by which a user sends voice data to a system through a microphone.

[0147] "Voice data" refers to information that is a digital recording of a user's speech.

[0148] "Text conversion" is the process of analyzing audio data and converting it into corresponding text data.

[0149] "Text data" is voice data expressed as text information.

[0150] "Order Details" refers to specific product or service requests made by a user through voice.

[0151] "Generative AI" is an artificial intelligence technology that performs natural language processing based on large amounts of text data and generates appropriate text data according to demand.

[0152] "Feedback" is the process by which the system responds to the user with analysis results and confirmation items.

[0153] "Database" means a storage device for storing and managing information such as final confirmed order details.

[0154] A "smartphone" is a mobile device with advanced processing power and internet connectivity.

[0155] A "confirmation message" is a message that the system sends to a user after receiving the order details from the user to confirm the order.

[0156] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[0157] "Order identification" is the process of analyzing voice and text data to determine the products or services the user is trying to order.

[0158] "Re-analysis" is the process of receiving an acknowledgment from the user and re-analyzing it.

[0159] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for analysis and generation.

[0160] This invention is a system that combines voice recognition technology and generative AI, allowing users to order food delivery through voice input. An embodiment of this system is shown below.

[0161] Server-side processing

[0162] The server processes the audio data in the following steps:

[0163] 1. Voice input reception

[0164] The voice data that the user speaks into the smartphone is received. The smartphone is equipped with a microphone, and the voice data is collected from there. The API for receiving the voice data uses the standard HTTP protocol.

[0165] 2. Converting voice data to text

[0166] The received voice data is converted to text data using the Google Speech-to-Text API, and the order details are then ready to be analyzed based on the converted text.

[0167] 3. Order Analysis

[0168] A generative AI model is used to analyze the text data and identify specific order details, such as if you are ordering "one pizza and one Pepsi."

[0169] 4. Generate a confirmation message

[0170] Based on the analyzed order details, a confirmation message is generated to provide feedback to the user. The generated message is in a natural conversational style, such as "Is one pizza and one Pepsi okay?"

[0171] 5. Conversion to audio data

[0172] The confirmation message is converted into voice data and sent to the terminal. Voice synthesis technology is used to generate the confirmation message as natural-sounding voice data.

[0173] 6. Final confirmation and registration in the database

[0174] The confirmation response from the user is analyzed again, and the final order details are confirmed and registered in the database, again using the Google Speech-to-Text API and generative AI models.

[0175] Terminal side processing

[0176] The smartphone plays back the feedback from the server to the user for final confirmation, as follows:

[0177] 1. Start voice input

[0178] Voice input is initiated by the user tapping the microphone icon or speaking a voice input command.

[0179] 2. Feedback playback

[0180] The confirmation message sent from the server is received as audio data and played back to the user, who then confirms the feedback.

[0181] 3. Receiving an Acknowledgment

[0182] If the user responds with a "yes" or "no" confirmation, the voice data is sent to the server again, providing a final confirmation.

[0183] 4. Notification of order completion

[0184] Once the final confirmed order details are registered in the database, a voice message informing the user that the order has been completed is sent and played to the user.

[0185] User operations

[0186] The user can enter the specific details of their order by voice, and the system will analyze, confirm, and register them appropriately, allowing for a smooth food delivery order. For example, if an elderly person or a visually impaired person wants to order "one pizza and one Pepsi," they can do so simply and accurately.

[0187] Examples and prompts

[0188] Specific examples

[0189] If an elderly user is trying to order a pizza:

[0190] 1. User: "One pizza and one Pepsi, please."

[0191] 2. Server: "One pizza and one Pepsi, please?"

[0192] 3. User: "Yes"

[0193] 4. Server: "Thank you for your order. Your order of one pizza and one Pepsi has been completed."

[0194] Prompt Sentence Examples

[0195] If a user orders one curry and one salad:

[0196] User: "I'd like to order one curry and one salad."

[0197] Examples of how to respond after receiving a prompt:

[0198] Transliteration: "I'd like to order one curry and one salad."

[0199] Generated AI response: "Would you like one plate of curry and one salad?"

[0200] Final check:

[0201] Final text entry: "Yes"

[0202] The above is an embodiment of the present invention.

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] The user speaks their order into the smartphone. For example, they might say, "One pizza and one Pepsi, please." This voice data is recorded through the smartphone's microphone. Input data: User's voice data. Output data: Recorded voice data file.

[0206] Step 2:

[0207] The smartphone sends the recorded voice data to the server. Input data: Recorded voice data file. Output data: Voice data sent to the server.

[0208] Step 3:

[0209] The server receives the audio data and converts it to text using the Google Speech-to-Text API. Input data: Audio data. Output data: Text data.

[0210] Step 4:

[0211] The server uses the generative AI model to analyze the order details from the text data. Specifically, the speech-recognized text data is sent to the generative AI model as a prompt to identify the order details. Input data: Text data. Output data: Identified order details.

[0212] Step 5:

[0213] The server generates a confirmation message based on the parsed order details. For example, it generates a message like "One pizza and one Pepsi, okay?" Input data: Identified order details. Output data: Text of the confirmation message.

[0214] Step 6:

[0215] The server converts the text of the confirmation message into audio data, generates the audio data, and sends it to the smartphone. Input data: Text of the confirmation message. Output data: Audio data of the confirmation message.

[0216] Step 7:

[0217] The smartphone plays the received audio data to the user. The user listens to the played confirmation message and confirms its contents. Input data: Audio data of the confirmation message. Output data: User's understanding and judgment.

[0218] Step 8:

[0219] The user responds with "yes" or "no." For example, the user utters "yes." This voice data is recorded again. Input data: User's confirmation voice data. Output data: Recorded confirmation voice data file.

[0220] Step 9:

[0221] The smartphone sends the recorded confirmation response voice data to the server. Input data: Recorded confirmation response voice data file. Output data: Confirmation response voice data sent to the server.

[0222] Step 10:

[0223] The server receives the confirmation response audio data and converts it to text again using the Google Speech-to-Text API. Input data: confirmation response audio data. Output data: text data.

[0224] Step 11:

[0225] The server analyzes the text data of the confirmation response and performs a final confirmation. For example, if the response is "yes," the order is confirmed, and if the response is "no," the order details are revised and a confirmation message is generated. Input data: Text data of the confirmation response. Output data: Final confirmation result.

[0226] Step 12:

[0227] The final confirmed order details are registered in the database. Input data: Final confirmation results. Output data: Order details registered in the database.

[0228] Step 13:

[0229] The server generates a message indicating that the order has been completed, converts it into voice data, and sends it to the smartphone. For example, a voice message such as "Thank you for your order. Your order for one pizza and one Pepsi has been completed." Input data: Text of the order completion message. Output data: Voice data of the order completion message.

[0230] Step 14:

[0231] The smartphone plays the received voice data of the order completion message to the user. Input data: Voice data of the order completion message. Output data: User's understanding and confirmation of order acceptance.

[0232] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0233] The present invention is a system that combines speech recognition technology, generative AI, and an emotion engine, and when a user places an order through voice input, it recognizes the user's emotion and adjusts feedback based on that emotion, providing a more natural and comfortable experience. Below, the program processing of this system is explained in natural language, and an embodiment is described with specific examples.

[0234] Server-side processing

[0235] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Through this process, what the user wants to order becomes clear in text form.

[0236] The server then uses generative AI to analyze the text data and identify the order details, such as "one beer and one hamburger." The emotion engine then analyzes the user's voice data and recognizes the emotion expressed during the feedback.

[0237] The server generates a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion recognized by the emotion engine. For example, if the user is impatient, the server generates feedback in a gentle tone such as "Please wait a moment, we will respond immediately."

[0238] The server converts this feedback message into voice data and sends it to the device. It also receives additional voice data from the user (such as "yes" or "no"), converts it into text, and analyzes it. The server also considers the user's emotions when making a final confirmation.

[0239] Finally, the final confirmed order details are registered in the database. The server also generates a message to notify the completion of the order and sends it to the terminal.

[0240] Terminal side processing

[0241] The device detects that the user has started voice input and activates recording mode. The recorded voice data is sent to the server. When the device receives the feedback voice data from the server, it plays it back to the user. The user checks the feedback and responds by voice if they want to input the answer again. The device then sends the response voice back to the server.

[0242] User operations

[0243] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks the details and, if there are no problems, responds "Yes." If corrections are needed, they can say "No, not that," followed by the new order details. Once the user has completed the final confirmation and received a notification from the device that the order has been completed, they can feel reassured that their order has been successfully accepted.

[0244] Specific examples

[0245] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[0246] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[0250] Step 2:

[0251] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[0252] Step 3:

[0253] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[0254] Step 4:

[0255] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[0256] Step 5:

[0257] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, recognizing emotions such as impatience or anxiety from the tone and speed of the user's voice.

[0258] Step 6:

[0259] The server generates a feedback message based on the analyzed order details and the user's emotional data. Generative AI is used to create a feedback message that takes into account the appropriate tone and wording. For example, if the user is impatient, the server might say, "Please wait a moment, we will respond shortly."

[0260] Step 7:

[0261] The server converts the generated feedback message into voice data and transmits it to the terminal.

[0262] Step 8:

[0263] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[0264] Step 9:

[0265] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[0266] Step 10:

[0267] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[0268] Step 11:

[0269] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[0270] Step 12:

[0271] The server registers the final confirmed order details in the database, which officially accepts the order.

[0272] Step 13:

[0273] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal. At this time, feedback is provided taking into account emotion data.

[0274] Step 14:

[0275] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[0276] Specific examples

[0277] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[0278] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] The problem that the present invention aims to solve is the inability to provide a more natural and comfortable experience by recognizing a user's emotions and adjusting feedback based on them when placing an order via voice input. Conventional systems provide uniform feedback without considering the user's emotions, which can impair the user experience. Another problem is that they are unable to provide appropriate services to a variety of users, including the elderly, the visually impaired, and foreigners.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0283] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for analyzing the user's emotions from the voice data, means for generating a feedback message according to the user's emotions using a generative AI model, means for converting the generated feedback message into voice data, means for providing the generated feedback message by voice, and means for registering the final confirmed order details in a database. This allows appropriate feedback to be given based on the user's emotions, and makes it possible to provide a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with operation.

[0284] The "means for receiving voice input" refers to a device or method for receiving voice data spoken by a user and incorporating it into the system.

[0285] A "means for converting voice data to text" is a device or method for generating text data from voice data using voice recognition technology.

[0286] The "means for analyzing text data to identify order details" refers to a device or method for processing text data and extracting specific order information requested by the user.

[0287] The "means for analyzing user emotions from voice data" is a device or method for analyzing voice data to extract the user's emotional state.

[0288] "Means for generating feedback messages according to user emotions using a generative AI model" refers to a device or method for using generative AI to create appropriate feedback messages based on analyzed user emotions.

[0289] The "means for converting the generated feedback message into voice data" is a device or method for converting the textual feedback message into voice data.

[0290] The "means for providing a generated feedback message by voice" refers to a device or method for transmitting a feedback message converted into voice data to a user.

[0291] The "means for registering the final confirmed order details in the database" refers to a device or method for storing the order details that have been finally confirmed by the system in the database.

[0292] The present invention is a system that combines voice recognition technology, generative AI, and an emotion engine, and provides a more natural and comfortable experience by recognizing the user's emotions when placing an order through voice input and adjusting feedback accordingly. This system is configured as follows.

[0293] Server-side processing

[0294] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Specifically, Google Cloud Speech-to-Text is used to convert the voice data into text data. This makes it clear what the user wants to order in text format.

[0295] Next, the server uses a natural language processing model (e.g., a generative AI model) to analyze the text data and identify the order. For example, it extracts specific order information such as "one beer and one hamburger." In parallel, it uses emotion analysis technology (e.g., a voice emotion analysis engine) to analyze the user's emotions from the voice data.

[0296] The server uses a generative AI model to generate a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion as recognized by the emotion engine. For example, if the user is impatient, a gentle tone of feedback such as "Please wait a moment, we will respond immediately" is generated.

[0297] To convert the generated feedback message into audio data, a speech synthesis technology (e.g., a speech synthesis engine) is used. Specifically, Amazon Polly is used to convert the text message into audio data. This audio data is then sent to the device.

[0298] Furthermore, the system receives voice data for additional input from the user (such as "yes" or "no"), performs voice recognition and emotion analysis again, and performs final confirmation. The final confirmed order details are registered in a database system (e.g., a database management system). This ensures that the order record is saved.

[0299] Terminal side processing

[0300] The device detects that the user has started voice input and activates recording mode. In this mode, the device's microphone records the user's voice and transmits the voice data to the server. The device also receives feedback voice data transmitted from the server, plays it back, and provides it to the user.

[0301] The audio data recorded by the device is converted into an appropriate format (e.g., WAV or MP3) and then sent to the server. This process is performed through an HTTP request. Upon receiving the feedback audio data from the server, the device plays it back to the user.

[0302] User operations

[0303] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks it and, if satisfied, responds with "Yes." If a correction is needed, they can say, "No, not that," followed by the new order details. Once the final confirmation is complete, the user receives a notification from the device that the order has been successfully accepted, providing peace of mind.

[0304] Specific examples

[0305] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion analysis engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording that corresponds to the user's emotion, giving the user a sense of security.

[0306] Prompt Sentence Examples

[0307] Use the following prompt for the generative AI model:

[0308] prompt:

[0309] User utterance: "One cup of coffee, no milk"

[0310] User Emotion: "Impatience"

[0311] Produced message: "Please wait a moment, your coffee will be ready shortly."

[0312] By using such prompt sentences, it is possible to generate appropriate feedback messages according to the user's emotions.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1:

[0315] The server receives the voice data transmitted from the terminal.

[0316] Input: User spoken voice data (WAV or MP3 format)

[0317] Processing: The audio data is sent to the server via an HTTP request, and the server stores it in its storage.

[0318] Output: Audio data file saved in storage

[0319] Step 2:

[0320] The server converts the received voice data into text using a voice recognition engine.

[0321] Input: Audio data file saved in storage

[0322] Processing: The audio data is input into a speech recognition engine (for example, Google Cloud Speech-to-Text) and converted into text data.

[0323] Output: Converted text data

[0324] Step 3:

[0325] The server uses generative AI to analyze the text data and identify the order details.

[0326] Input: Converted text data

[0327] Processing: The text data is fed into a generative AI model (e.g., GPT-4®) to analyze and identify the order details. For example, specific order information such as "one cup of coffee, no milk" is extracted.

[0328] Output: Identified order details

[0329] Step 4:

[0330] The server uses an emotion engine to analyze the user's emotion from the voice data.

[0331] Input: Audio data file saved in storage

[0332] Processing: Input the voice data into an emotion engine (e.g., Emotion API) and analyze the user's emotions (e.g., impatience, relief, joy).

[0333] Output: Parsed emotion data

[0334] Step 5:

[0335] The server uses a generative AI model to generate a feedback message based on the analyzed order content and user sentiment data.

[0336] Input: Identified order details, analyzed sentiment data

[0337] Processing: The following prompt sentence is input into the generative AI model, and a feedback message corresponding to the user's emotion is output.

[0338] prompt:

[0339] User utterance: "One cup of coffee, no milk"

[0340] User Emotion: "Impatience"

[0341] Produced message: "Please wait a moment, your coffee will be ready shortly."

[0342] Output: The generated feedback message

[0343] Step 6:

[0344] The server converts the generated feedback message into audio data.

[0345] Input: The generated feedback message

[0346] Processing: A text message is input into speech synthesis technology (e.g., Amazon Polly) and converted into audio data.

[0347] Output: Feedback message converted to audio data

[0348] Step 7:

[0349] The server transmits the generated feedback message to the terminal as voice data.

[0350] Input: Feedback message converted to audio data

[0351] Processing: Sends feedback audio data to the device via an HTTP response.

[0352] Output: Feedback audio data sent to the device

[0353] Step 8:

[0354] The terminal plays back the feedback voice data received from the server to the user.

[0355] Input: Feedback audio data sent from the server

[0356] Processing: The audio data is played through the device speaker to provide feedback to the user.

[0357] Output: Feedback audio played to the user

[0358] Step 9:

[0359] The user reviews the feedback and speaks additional responses.

[0360] Input: Feedback audio played from the device

[0361] Processing: The user speaks the acknowledgment "yes" or "no," which is recorded by the terminal.

[0362] Output: Additional audio data

[0363] Step 10:

[0364] The terminal transmits the additional voice data from the user to the server again.

[0365] Input: Additional voice data from the user

[0366] Processing: The recorded audio data is sent to the server via an HTTP request.

[0367] Output: Additional audio data sent to the server

[0368] Step 11:

[0369] The server receives the additional voice data, performs voice recognition and emotion analysis again, and performs a final confirmation.

[0370] Input: Additional audio data

[0371] Processing: The voice data is converted to text and analyzed again to determine the final order and user sentiment.

[0372] Output: Final confirmed order details and sentiment data

[0373] Step 12:

[0374] The server registers the final confirmed order details in a database and generates a notification of order completion.

[0375] Input: Final confirmed order details and sentiment data

[0376] Processing: The order details are saved in the database system, an order completion message is generated, and then the message is converted into voice data and sent to the terminal.

[0377] Output: Order details registered in the database and voice data of the completed order sent to the terminal

[0378] Step 13:

[0379] The terminal reproduces order completion voice data to the user.

[0380] Input: Order completion voice data sent from the server

[0381] Processing: Plays audio data to notify the user that the order has been completed.

[0382] Output: Order complete audio played to the user

[0383] This series of processing steps provides appropriate feedback according to the user's emotions, resulting in a pleasant ordering experience.

[0384] (Application example 2)

[0385] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] Conventional voice input systems provide mechanical responses without considering the user's emotions, and therefore do not provide a user-friendly or reassuring interface. Furthermore, users who are unfamiliar with the system, such as the elderly, the visually impaired, and foreigners, often find it difficult to use and feel frustrated. There is a need for a system that provides natural feedback based on the user's emotions to resolve these issues and facilitate the ordering process.

[0387] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to specify the order details, means for providing voice feedback of the generated order details, means for analyzing user emotion data, means for adjusting the feedback message based on the emotion data, and means for registering the final confirmed order details in a database. This enables natural feedback according to the user's emotions, allowing even elderly people, visually impaired people, and users unfamiliar with operation to place orders without stress.

[0388] "Voice input" is the process of receiving user spoken sounds through a device's microphone.

[0389] "Voice data" is data that represents in digital form the voice spoken by the user.

[0390] The "means for converting to text" refers to a technology that performs a process of analyzing audio data and converting it into corresponding text data.

[0391] "Text data" is voice data converted into character information, and is a sentence describing the contents of an order or conversation.

[0392] The "means for identifying the order content" is a technology that analyzes text data and identifies the specific product or service content that the user wants to order.

[0393] "Feedback means" refers to a technology that generates and transmits a voice message of confirmation or guidance to the user based on the analyzed order details.

[0394] "User emotion data" is data that indicates the user's emotion or emotional state, analyzed from the user's speech and voice characteristics.

[0395] "Means for analyzing emotional data" refers to technology that analyzes a user's voice and text data to identify the user's emotional state.

[0396] The "means for adjusting a feedback message based on emotional data" is a technique for adjusting the content and tone of a feedback message based on analyzed emotional data of a user.

[0397] The "finally confirmed order details" are the final order details that have been confirmed after being confirmed by the user.

[0398] The "means for registering in a database" refers to a technique for storing the confirmed order details in a database in digital form.

[0399] This invention is a food and beverage delivery ordering system that uses voice recognition technology, generative AI, and an emotion engine to assist users in the ordering process. Specifically, it uses a smartphone application to receive orders using the following procedure and provides feedback based on the user's emotions.

[0400] Hardware used

[0401] Smartphone (ANDROID (registered trademark) / iOS)

[0402] microphone

[0403] server

[0404] Software used

[0405] Speech recognition engine (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[0406] Emotion engines (e.g., AWS® Rekognition, IBM Watson® Tone Analyzer)

[0407] Generative AI (e.g., GPT-3 by OpenAI (registered trademark))

[0408] Database (e.g., MySQL (registered trademark), Firebase)

[0409] Server-side processing

[0410] The server receives voice data sent from the smartphone. The voice data is first converted into text using a speech recognition engine. The text data is then analyzed to identify the order details. The analyzed order details are then processed together with the user's emotion data using an emotion engine.

[0411] The emotion engine analyzes the user's emotions from the tone and tempo of their voice, and the generation AI generates a feedback message based on this. The generated feedback message is composed of a tone and wording that corresponds to the user's emotions and is converted back into audio data. This audio data is sent to the smartphone and played back to the user.

[0412] After receiving the final confirmation response from the user, the order is analyzed again and the order details are finally confirmed. The final confirmation details are registered in the database and the order is completed.

[0413] Terminal side processing

[0414] When a user speaks their order details into their smartphone, the device detects this voice input and activates recording mode. The recorded voice data is sent to the server, and when feedback voice data is received, it is played back to the user. If the user checks the feedback and wants to re-enter the answer, they respond by voice, and this response voice is also sent back to the server.

[0415] User operations

[0416] The user speaks the specific details of their order into the device's microphone. For example, they might say, "One pizza, lots of cheese, tomato sauce." This voice data is sent to the server, where it undergoes a series of processes including analysis, feedback generation, and sentiment analysis. When the feedback is returned, the user confirms it and finally responds with "yes" or "no." As a result of this exchange, the order is confirmed and a notification of completion is sent.

[0417] Specific examples

[0418] For example, if a user says, "One pizza, lots of cheese, and tomato sauce," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order, and generates feedback. If the emotion engine analyzes the user's voice as relaxed, the generation AI will generate, "Thank you for your order. One pizza, lots of cheese, and tomato sauce, please. Is that okay?"

[0419] Prompt Sentence Examples

[0420] "When a user orders one pizza, lots of cheese, and tomato sauce, generate a feedback message in a relaxed tone."

[0421] This design allows users to complete their orders in a natural conversational manner, and is intended to be stress-free for users, especially the elderly, visually impaired, and those unfamiliar with the system.

[0422] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0423] Step 1:

[0424] Input: User voice input

[0425] How it works: The user speaks specific details of their order into the smartphone's microphone.

[0426] Output: Recorded audio data

[0427] Step 2:

[0428] Input: Recorded audio data

[0429] Operation: The device detects the user's voice input and activates recording mode. The recorded voice data is sent to the server.

[0430] Output: Audio data sent to the server

[0431] Step 3:

[0432] Input: Audio data sent to the server

[0433] How it works: The server uses a speech recognition engine to convert the voice data into text data.

[0434] Output: Text data

[0435] Step 4:

[0436] Input: Text data

[0437] Operation: The server analyzes the text data and identifies the order contents.

[0438] Output: Identifying the order

[0439] Step 5:

[0440] Input: User's voice data and identified order details

[0441] How it works: The emotion engine analyzes the user's voice data and generates the user's emotion data.

[0442] Output: User emotion data

[0443] Step 6:

[0444] Input: Identified order details and user sentiment data

[0445] How it works: The server uses a generative AI to generate feedback messages based on the user's emotions.

[0446] Output: Feedback message

[0447] Step 7:

[0448] Input: Feedback message

[0449] Operation: The server converts the generated feedback message into audio data.

[0450] Output: Feedback message as audio data

[0451] Step 8:

[0452] Input: Feedback message as audio data

[0453] Operation: The server sends a feedback message to the terminal, which then plays it back to the user.

[0454] Output: The feedback message played to the user

[0455] Step 9:

[0456] Input: The user's response to the played feedback message

[0457] How it works: The user checks the feedback and responds again verbally. The device sends this response to the server.

[0458] Output: Response audio data sent to the server

[0459] Step 10:

[0460] Input: Response voice data sent to the server

[0461] Operation: The server analyzes the response voice data again and performs a final confirmation of the order details.

[0462] Output: Final confirmed order details

[0463] Step 11:

[0464] Input: Final confirmed order details

[0465] Operation: The server registers the final confirmed order details in the database.

[0466] Output: Order details registered in the database

[0467] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0468] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0469] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0470] [Second embodiment]

[0471] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0472] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0473] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0474] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0475] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0476] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0477] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0478] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0479] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0480] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0481] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0482] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0483] The present invention is a system that combines voice recognition technology and generative AI, allowing users to place orders through voice input. Below, the program processing of this system is explained in natural language, and an embodiment is explained using specific examples.

[0484] Server-side processing

[0485] The server receives the voice data sent from the device, which is the first step in recognizing what the user wants to order. Once the voice data is received, the server uses a speech recognition engine to convert the received voice data into text. For example, the server can use the Google Speech-to-Text API or other speech recognition services.

[0486] The server then uses generative AI to analyze the order details from the converted text data. For example, it extracts specific order details such as "one cup of coffee, no milk." Based on this analysis, the server generates a confirmation message to provide feedback to the user. The generated message is in a natural conversational format, such as "One cup of coffee, no milk, okay?"

[0487] The server converts this confirmation message into voice data and sends the generated voice data to the terminal. The server then receives additional input from the user, such as a confirmation response of "yes" or "no," analyzes the content, and performs a final confirmation. The final confirmed order details are then registered in a database, ensuring that the order is processed accurately.

[0488] Finally, the server generates a voice message to notify the user that the order has been completed and sends the voice message to the terminal, allowing the user to confirm that the order has been successfully received.

[0489] Terminal side processing

[0490] The device detects when the user starts voice input and activates recording mode. Voice input begins when the user taps the microphone icon or speaks a voice input command. The device records the user's speech and sends the voice data to the server.

[0491] When the terminal receives the feedback voice data from the server, it plays this voice data to the user. The user checks the feedback content and responds again if necessary. The terminal also receives a notification voice indicating that the order has been completed and plays it to the user.

[0492] User operations

[0493] The user speaks into the device's microphone to input the specific details of their order. For example, they might say, "I'd like a beer and a hamburger, please." When the feedback is returned, the user checks the details and responds "Yes" if there are no problems. If corrections are needed, they can say "No, not that one," followed by the new order details.

[0494] Once the final confirmation is complete, the user receives a notification from the terminal that the order has been completed, knowing that the order has been successfully accepted and can wait for the product in peace.

[0495] Specific examples

[0496] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. It then converts the confirmation message "One cup of coffee, no milk, okay?" into voice data and sends it to the device. If the user responds "Yes," the server performs a final confirmation, registers the order details in the database, and generates a notification of order completion that is sent to the device.

[0497] This system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation, which improves store operational efficiency and overall user satisfaction.

[0498] The processing flow will be explained below.

[0499] Step 1:

[0500] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[0501] Step 2:

[0502] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[0503] Step 3:

[0504] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[0505] Step 4:

[0506] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[0507] Step 5:

[0508] The server generates a feedback message based on the parsed order details, using generative AI to create a natural conversational message, such as "Would you like a beer and a hamburger?"

[0509] Step 6:

[0510] The server converts the generated feedback message into voice data and transmits it to the terminal.

[0511] Step 7:

[0512] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[0513] Step 8:

[0514] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[0515] Step 9:

[0516] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[0517] Step 10:

[0518] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[0519] Step 11:

[0520] The server registers the final confirmed order details in the database, which officially accepts the order.

[0521] Step 12:

[0522] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal.

[0523] Step 13:

[0524] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[0525] This process flow provides a natural ordering experience using voice, making it easy for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation of the system to order, improving store operational efficiency and overall user satisfaction.

[0526] Example 1

[0527] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] In conventional voice input systems, the accuracy of voice recognition and the generation of natural conversational formats were issues in the process of accurately understanding a user's order and providing feedback. Furthermore, when a user speaks multiple languages, it was difficult to switch languages ​​and adapt feedback accordingly. This resulted in problems that reduced user satisfaction for users who were unfamiliar with the system, such as the elderly, the visually impaired, and foreigners.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0530] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order details, means for analyzing the order details using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order details, and means for registering the final confirmed order details in a database. This allows the user to accurately communicate the order details through voice input and receive feedback in a natural conversational style. Furthermore, appropriate feedback can be provided even when the user speaks multiple languages.

[0531] "Means for receiving voice input" refers to technology for capturing and recording voice data spoken by a user into the terminal.

[0532] "Means for converting voice data to text" refers to a technique that uses a voice recognition engine to convert received voice data into corresponding text data.

[0533] The "means for identifying the order details by analyzing the text data" refers to a technology for analyzing the converted text data and extracting the specific order details that the user is trying to convey.

[0534] "Means of analyzing order content using a generative AI model" refers to technology that utilizes generative AI to accurately read the user's intended order content from secured text data.

[0535] The "means for generating a feedback message in a natural conversational style based on the analysis results" refers to a technology for generating a confirmation message to the user in a natural conversational style based on the analyzed order details.

[0536] The "means for providing voice feedback on the generated order details" refers to a technology for converting the generated confirmation message into voice data and providing it to the user.

[0537] "Means for registering the final confirmed order details in a database" refers to a technology for saving the order details finally confirmed by the user in a database and managing them as order processing information.

[0538] "Means for playing back audio feedback to the user" refers to a technique for playing back the generated audio data at the terminal and providing feedback to the user.

[0539] The "means for receiving and re-analyzing a confirmation response from the user" refers to a technology for re-analyzing the user's response voice and determining the next action based on the content of the analysis.

[0540] "Means for identifying multiple languages ​​and providing feedback in a specified language" refers to technology that automatically determines the language used by a user and provides appropriate feedback in that language.

[0541] This invention is a system that allows users to place orders through voice input using a system that combines voice recognition technology and a generative AI model. This system is mainly composed of a server, a terminal, and a user.

[0542] The server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order contents, means for analyzing the order contents using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order contents, means for registering the final confirmed order contents in a database, and means for identifying multiple languages ​​and providing feedback in a specified language.

[0543] The terminal includes means for detecting a user's voice input, activating a recording mode, and playing feedback voice data from the server. The user provides voice input through a microphone of the terminal and transmits the voice data to the server.

[0544] The server converts the received voice data into text using the Google Speech-to-Text API or other speech recognition services. It then uses a generative AI model (e.g., GPT-3) to analyze the order details from the converted text data. Based on the analysis results, it generates a feedback message in a natural conversational style, such as "Would you like a cup of coffee, without milk?", converts it into voice data, and sends it to the device.

[0545] The terminal plays a feedback voice to the user, and the user responds with a confirmation. The user's response voice is also sent from the terminal to the server. The server again performs voice recognition and analysis, and registers the final confirmed order details in the database. This ensures that the order is processed accurately.

[0546] As a concrete example, consider the case where a domestic customer says, "One hamburger, one glass of coke." When the user speaks, the device records this voice and sends it to the server. The server converts the voice data into text and analyzes the order using a generative AI model. Based on the analysis results, it generates a feedback message saying, "One hamburger, one glass of coke, would you like?" and sends it as voice data to the device. If the user responds "Yes," the server analyzes again, registers the order in the database, and notifies the user that "the order has been completed."

[0547] For example, by inputting an example of a prompt sentence such as "Please explain the detailed steps of the process in which the user says 'one hamburger, one glass of coke'", the detailed process described above is generated.

[0548] The system provides an intuitive and easy-to-use interface, particularly for elderly people, visually impaired people, foreigners, and users unfamiliar with the operation, improving store operational efficiency and overall user satisfaction.

[0549] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0550] Step 1:

[0551] The user begins voice input. The user speaks into the device's microphone, saying, "A cup of hot coffee, no sugar." This becomes the base data for subsequent processing.

[0552] Step 2:

[0553] The device detects the user's speech and activates recording mode. The device's microphone captures and records the audio data. The input is audio data, and the output is a recorded audio file.

[0554] Step 3:

[0555] The device sends the recorded audio data to the server via Wi-Fi or mobile data network. The input is the recorded audio data, and the output is the audio data sent as an HTTP request to the server.

[0556] Step 4:

[0557] The server converts the received voice data into text. This is done using speech recognition software such as the Google Speech-to-Text API. The input is voice data, and the output is text data generated based on that voice data. Specifically, the generated text might be something like "A cup of hot coffee, no sugar."

[0558] Step 5:

[0559] The server uses a generative AI model to analyze the order. During analysis, it uses text data as input to identify each element of the order (drink, temperature, whether sugar is added, etc.). The input is text data, and the output is a data structure of the analyzed order. For example, the drink might be "coffee," the temperature "warm," and the sugar "no."

[0560] Step 6:

[0561] The server generates a feedback message in a natural conversational format based on the analysis results. It uses a generative AI model to generate a message such as "Would you like a cup of coffee, no sugar?" and converts it into voice data. The input is the analysis results, and the output is the generated voice message.

[0562] Step 7:

[0563] The server sends the generated feedback voice data to the device. The communication method is again Wi-Fi or mobile data network. The input is voice data, and the output is voice data sent as an HTTP response to the device.

[0564] Step 8:

[0565] The terminal plays back the feedback audio data received from the server. The user can hear the audio feedback, "Would you like a cup of coffee, no sugar?" The input is the audio data received from the server, and the output is the audio playback.

[0566] Step 9:

[0567] The user confirms the feedback voice and responds again with voice, for example, saying "Yes, that's fine." This provides the user with final confirmation.

[0568] Step 10:

[0569] The terminal re-records the user's confirmation response and sends it to the server. The input is the re-recorded voice data, and the output is the response voice data to be sent as an HTTP request to the server.

[0570] Step 11:

[0571] The server converts the user's confirmation response into text using a speech recognition engine and analyzes the content. The input is voice data, and the output is the analyzed text data. For example, a final confirmation text such as "Yes, that's it, please" is generated.

[0572] Step 12:

[0573] The server registers the order details that have been finally confirmed in the database. The input is the data of the final confirmed order details, and the output is the order information saved as a record in the database.

[0574] Step 13:

[0575] The server generates a voice message notifying the terminal that the order has been processed successfully, and sends it to the terminal. Specifically, it sends a message such as "Your order has been completed. Thank you." The input is the order completion information, and the output is the generated voice message data.

[0576] Step 14:

[0577] The terminal plays audio data to notify the user that the order has been completed. The user hears this audio notification and confirms that the order has been accepted. The input is audio data from the server, and the output is audio playback.

[0578] (Application example 1)

[0579] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0580] Conventional food delivery ordering systems are often complex and difficult to operate for elderly people, visually impaired people, and users who are not familiar with the system. Language barriers and misunderstandings of detailed order details can also be problematic. This can lead to ordering errors and user dissatisfaction. Therefore, the objective of this invention is to provide a system that uses voice recognition technology and generative AI to allow users to easily and accurately order food delivery.

[0581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0582] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for providing voice feedback on the generated order details, means for registering the final confirmed order details in a database, means for generating a confirmation message for the user using a smartphone and converting it into voice data, and means for analyzing the user's voice input and identifying the food delivery order details by combining voice recognition technology and generation AI. This enables even users who are particularly elderly, visually impaired, or unfamiliar with operation to easily place orders by voice and accurately confirm their orders.

[0583] "Voice input" is the process by which a user sends voice data to a system through a microphone.

[0584] "Voice data" refers to information that is a digital recording of a user's speech.

[0585] "Text conversion" is the process of analyzing audio data and converting it into corresponding text data.

[0586] "Text data" is voice data expressed as text information.

[0587] "Order Details" refers to specific product or service requests made by a user through voice.

[0588] "Generative AI" is an artificial intelligence technology that performs natural language processing based on large amounts of text data and generates appropriate text data according to demand.

[0589] "Feedback" is the process by which the system responds to the user with analysis results and confirmation items.

[0590] "Database" means a storage device for storing and managing information such as final confirmed order details.

[0591] A "smartphone" is a mobile device with advanced processing power and internet connectivity.

[0592] A "confirmation message" is a message that the system sends to a user after receiving the order details from the user to confirm the order.

[0593] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[0594] "Order identification" is the process of analyzing voice and text data to determine the products or services the user is trying to order.

[0595] "Re-analysis" is the process of receiving an acknowledgment from the user and re-analyzing it.

[0596] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for analysis and generation.

[0597] This invention is a system that combines voice recognition technology and generative AI, allowing users to order food delivery through voice input. An embodiment of this system is shown below.

[0598] Server-side processing

[0599] The server processes the audio data in the following steps:

[0600] 1. Voice input reception

[0601] The voice data that the user speaks into the smartphone is received. The smartphone is equipped with a microphone, and the voice data is collected from there. The API for receiving the voice data uses the standard HTTP protocol.

[0602] 2. Converting voice data to text

[0603] The received voice data is converted to text data using the Google Speech-to-Text API, and the order details are then ready to be analyzed based on the converted text.

[0604] 3. Order Analysis

[0605] A generative AI model is used to analyze the text data and identify specific order details, such as if you are ordering "one pizza and one Pepsi."

[0606] 4. Generate a confirmation message

[0607] Based on the analyzed order details, a confirmation message is generated to provide feedback to the user. The generated message is in a natural conversational style, such as "Is one pizza and one Pepsi okay?"

[0608] 5. Conversion to audio data

[0609] The confirmation message is converted into voice data and sent to the terminal. Voice synthesis technology is used to generate the confirmation message as natural-sounding voice data.

[0610] 6. Final confirmation and registration in the database

[0611] The confirmation response from the user is analyzed again, and the final order details are confirmed and registered in the database, again using the Google Speech-to-Text API and generative AI models.

[0612] Terminal side processing

[0613] The smartphone plays back the feedback from the server to the user for final confirmation, as follows:

[0614] 1. Start voice input

[0615] Voice input is initiated by the user tapping the microphone icon or speaking a voice input command.

[0616] 2. Feedback playback

[0617] The confirmation message sent from the server is received as audio data and played back to the user, who then confirms the feedback.

[0618] 3. Receiving an Acknowledgment

[0619] If the user responds with a "yes" or "no" confirmation, the voice data is sent to the server again, providing a final confirmation.

[0620] 4. Notification of order completion

[0621] Once the final confirmed order details are registered in the database, a voice message informing the user that the order has been completed is sent and played to the user.

[0622] User operations

[0623] The user can enter the specific details of their order by voice, and the system will analyze, confirm, and register them appropriately, allowing for a smooth food delivery order. For example, if an elderly person or a visually impaired person wants to order "one pizza and one Pepsi," they can do so simply and accurately.

[0624] Examples and prompts

[0625] Specific examples

[0626] If an elderly user is trying to order a pizza:

[0627] 1. User: "One pizza and one Pepsi, please."

[0628] 2. Server: "One pizza and one Pepsi, please?"

[0629] 3. User: "Yes"

[0630] 4. Server: "Thank you for your order. Your order of one pizza and one Pepsi has been completed."

[0631] Prompt Sentence Examples

[0632] If a user orders one curry and one salad:

[0633] User: "I'd like to order one curry and one salad."

[0634] Examples of how to respond after receiving a prompt:

[0635] Transliteration: "I'd like to order one curry and one salad."

[0636] Generated AI response: "Would you like one plate of curry and one salad?"

[0637] Final check:

[0638] Final text entry: "Yes"

[0639] The above is an embodiment of the present invention.

[0640] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0641] Step 1:

[0642] The user speaks their order into the smartphone. For example, they might say, "One pizza and one Pepsi, please." This voice data is recorded through the smartphone's microphone. Input data: User's voice data. Output data: Recorded voice data file.

[0643] Step 2:

[0644] The smartphone sends the recorded voice data to the server. Input data: Recorded voice data file. Output data: Voice data sent to the server.

[0645] Step 3:

[0646] The server receives the audio data and converts it to text using the Google Speech-to-Text API. Input data: Audio data. Output data: Text data.

[0647] Step 4:

[0648] The server uses the generative AI model to analyze the order details from the text data. Specifically, the speech-recognized text data is sent to the generative AI model as a prompt to identify the order details. Input data: Text data. Output data: Identified order details.

[0649] Step 5:

[0650] The server generates a confirmation message based on the parsed order details. For example, it generates a message like "One pizza and one Pepsi, okay?" Input data: Identified order details. Output data: Text of the confirmation message.

[0651] Step 6:

[0652] The server converts the text of the confirmation message into audio data, generates the audio data, and sends it to the smartphone. Input data: Text of the confirmation message. Output data: Audio data of the confirmation message.

[0653] Step 7:

[0654] The smartphone plays the received audio data to the user. The user listens to the played confirmation message and confirms its contents. Input data: Audio data of the confirmation message. Output data: User's understanding and judgment.

[0655] Step 8:

[0656] The user responds with "yes" or "no." For example, the user utters "yes." This voice data is recorded again. Input data: User's confirmation voice data. Output data: Recorded confirmation voice data file.

[0657] Step 9:

[0658] The smartphone sends the recorded confirmation response voice data to the server. Input data: Recorded confirmation response voice data file. Output data: Confirmation response voice data sent to the server.

[0659] Step 10:

[0660] The server receives the confirmation response audio data and converts it to text again using the Google Speech-to-Text API. Input data: confirmation response audio data. Output data: text data.

[0661] Step 11:

[0662] The server analyzes the text data of the confirmation response and performs a final confirmation. For example, if the response is "yes," the order is confirmed, and if the response is "no," the order details are revised and a confirmation message is generated. Input data: Text data of the confirmation response. Output data: Final confirmation result.

[0663] Step 12:

[0664] The final confirmed order details are registered in the database. Input data: Final confirmation results. Output data: Order details registered in the database.

[0665] Step 13:

[0666] The server generates a message indicating that the order has been completed, converts it into voice data, and sends it to the smartphone. For example, a voice message such as "Thank you for your order. Your order for one pizza and one Pepsi has been completed." Input data: Text of the order completion message. Output data: Voice data of the order completion message.

[0667] Step 14:

[0668] The smartphone plays the received voice data of the order completion message to the user. Input data: Voice data of the order completion message. Output data: User's understanding and confirmation of order acceptance.

[0669] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0670] The present invention is a system that combines speech recognition technology, generative AI, and an emotion engine, and when a user places an order through voice input, it recognizes the user's emotion and adjusts feedback based on that emotion, providing a more natural and comfortable experience. Below, the program processing of this system is explained in natural language, and an embodiment is described with specific examples.

[0671] Server-side processing

[0672] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Through this process, what the user wants to order becomes clear in text form.

[0673] The server then uses generative AI to analyze the text data and identify the order details, such as "one beer and one hamburger." The emotion engine then analyzes the user's voice data and recognizes the emotion expressed during the feedback.

[0674] The server generates a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion recognized by the emotion engine. For example, if the user is impatient, the server generates feedback in a gentle tone such as "Please wait a moment, we will respond immediately."

[0675] The server converts this feedback message into voice data and sends it to the device. It also receives additional voice data from the user (such as "yes" or "no"), converts it into text, and analyzes it. The server also considers the user's emotions when making a final confirmation.

[0676] Finally, the final confirmed order details are registered in the database. The server also generates a message to notify the completion of the order and sends it to the terminal.

[0677] Terminal side processing

[0678] The device detects that the user has started voice input and activates recording mode. The recorded voice data is sent to the server. When the device receives the feedback voice data from the server, it plays it back to the user. The user checks the feedback and responds by voice if they want to input the answer again. The device then sends the response voice back to the server.

[0679] User operations

[0680] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks the details and, if there are no problems, responds "Yes." If corrections are needed, they can say "No, not that," followed by the new order details. Once the user has completed the final confirmation and received a notification from the device that the order has been completed, they can feel reassured that their order has been successfully accepted.

[0681] Specific examples

[0682] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[0683] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[0684] The processing flow will be explained below.

[0685] Step 1:

[0686] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[0687] Step 2:

[0688] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[0689] Step 3:

[0690] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[0691] Step 4:

[0692] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[0693] Step 5:

[0694] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, recognizing emotions such as impatience or anxiety from the tone and speed of the user's voice.

[0695] Step 6:

[0696] The server generates a feedback message based on the analyzed order details and the user's emotional data. Generative AI is used to create a feedback message that takes into account the appropriate tone and wording. For example, if the user is impatient, the server might say, "Please wait a moment, we will respond shortly."

[0697] Step 7:

[0698] The server converts the generated feedback message into voice data and transmits it to the terminal.

[0699] Step 8:

[0700] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[0701] Step 9:

[0702] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[0703] Step 10:

[0704] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[0705] Step 11:

[0706] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[0707] Step 12:

[0708] The server registers the final confirmed order details in the database, which officially accepts the order.

[0709] Step 13:

[0710] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal. At this time, feedback is provided taking into account emotion data.

[0711] Step 14:

[0712] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[0713] Specific examples

[0714] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[0715] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[0716] Example 2

[0717] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0718] The problem that the present invention aims to solve is the inability to provide a more natural and comfortable experience by recognizing a user's emotions and adjusting feedback based on them when placing an order via voice input. Conventional systems provide uniform feedback without considering the user's emotions, which can impair the user experience. Another problem is that they are unable to provide appropriate services to a variety of users, including the elderly, the visually impaired, and foreigners.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0720] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for analyzing the user's emotions from the voice data, means for generating a feedback message according to the user's emotions using a generative AI model, means for converting the generated feedback message into voice data, means for providing the generated feedback message by voice, and means for registering the final confirmed order details in a database. This allows appropriate feedback to be given based on the user's emotions, and makes it possible to provide a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with operation.

[0721] The "means for receiving voice input" refers to a device or method for receiving voice data spoken by a user and incorporating it into the system.

[0722] A "means for converting voice data to text" is a device or method for generating text data from voice data using voice recognition technology.

[0723] The "means for analyzing text data to identify order details" refers to a device or method for processing text data and extracting specific order information requested by the user.

[0724] The "means for analyzing user emotions from voice data" is a device or method for analyzing voice data to extract the user's emotional state.

[0725] "Means for generating feedback messages according to user emotions using a generative AI model" refers to a device or method for using generative AI to create appropriate feedback messages based on analyzed user emotions.

[0726] The "means for converting the generated feedback message into voice data" is a device or method for converting the textual feedback message into voice data.

[0727] The "means for providing a generated feedback message by voice" refers to a device or method for transmitting a feedback message converted into voice data to a user.

[0728] The "means for registering the final confirmed order details in the database" refers to a device or method for storing the order details that have been finally confirmed by the system in the database.

[0729] The present invention is a system that combines voice recognition technology, generative AI, and an emotion engine, and provides a more natural and comfortable experience by recognizing the user's emotions when placing an order through voice input and adjusting feedback accordingly. This system is configured as follows.

[0730] Server-side processing

[0731] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Specifically, Google Cloud Speech-to-Text is used to convert the voice data into text data. This makes it clear what the user wants to order in text format.

[0732] Next, the server uses a natural language processing model (e.g., a generative AI model) to analyze the text data and identify the order. For example, it extracts specific order information such as "one beer and one hamburger." In parallel, it uses emotion analysis technology (e.g., a voice emotion analysis engine) to analyze the user's emotions from the voice data.

[0733] The server uses a generative AI model to generate a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion as recognized by the emotion engine. For example, if the user is impatient, a gentle tone of feedback such as "Please wait a moment, we will respond immediately" is generated.

[0734] To convert the generated feedback message into audio data, a speech synthesis technology (e.g., a speech synthesis engine) is used. Specifically, Amazon Polly is used to convert the text message into audio data. This audio data is then sent to the device.

[0735] Furthermore, the system receives voice data for additional input from the user (such as "yes" or "no"), performs voice recognition and emotion analysis again, and performs final confirmation. The final confirmed order details are registered in a database system (e.g., a database management system). This ensures that the order record is saved.

[0736] Terminal side processing

[0737] The device detects that the user has started voice input and activates recording mode. In this mode, the device's microphone records the user's voice and transmits the voice data to the server. The device also receives feedback voice data transmitted from the server, plays it back, and provides it to the user.

[0738] The audio data recorded by the device is converted into an appropriate format (e.g., WAV or MP3) and then sent to the server. This process is performed through an HTTP request. Upon receiving the feedback audio data from the server, the device plays it back to the user.

[0739] User operations

[0740] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks it and, if satisfied, responds with "Yes." If a correction is needed, they can say, "No, not that," followed by the new order details. Once the final confirmation is complete, the user receives a notification from the device that the order has been successfully accepted, providing peace of mind.

[0741] Specific examples

[0742] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion analysis engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording that corresponds to the user's emotion, giving the user a sense of security.

[0743] Prompt Sentence Examples

[0744] Use the following prompt for the generative AI model:

[0745] prompt:

[0746] User utterance: "One cup of coffee, no milk"

[0747] User Emotion: "Impatience"

[0748] Produced message: "Please wait a moment, your coffee will be ready shortly."

[0749] By using such prompt sentences, it is possible to generate appropriate feedback messages according to the user's emotions.

[0750] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] The server receives the voice data transmitted from the terminal.

[0753] Input: User spoken voice data (WAV or MP3 format)

[0754] Processing: The audio data is sent to the server via an HTTP request, and the server stores it in its storage.

[0755] Output: Audio data file saved in storage

[0756] Step 2:

[0757] The server converts the received voice data into text using a voice recognition engine.

[0758] Input: Audio data file saved in storage

[0759] Processing: The audio data is input into a speech recognition engine (for example, Google Cloud Speech-to-Text) and converted into text data.

[0760] Output: Converted text data

[0761] Step 3:

[0762] The server uses generative AI to analyze the text data and identify the order details.

[0763] Input: Converted text data

[0764] Processing: A generative AI model (e.g., GPT-4) is fed the text data and used to analyze and identify the order, e.g., extracting specific order information such as "one cup of coffee, no milk."

[0765] Output: Identified order details

[0766] Step 4:

[0767] The server uses an emotion engine to analyze the user's emotion from the voice data.

[0768] Input: Audio data file saved in storage

[0769] Processing: Input the voice data into an emotion engine (e.g., Emotion API) and analyze the user's emotions (e.g., impatience, relief, joy).

[0770] Output: Parsed emotion data

[0771] Step 5:

[0772] The server uses a generative AI model to generate a feedback message based on the analyzed order content and user sentiment data.

[0773] Input: Identified order details, analyzed sentiment data

[0774] Processing: The following prompt sentence is input into the generative AI model, and a feedback message corresponding to the user's emotion is output.

[0775] prompt:

[0776] User utterance: "One cup of coffee, no milk"

[0777] User Emotion: "Impatience"

[0778] Produced message: "Please wait a moment, your coffee will be ready shortly."

[0779] Output: The generated feedback message

[0780] Step 6:

[0781] The server converts the generated feedback message into audio data.

[0782] Input: The generated feedback message

[0783] Processing: A text message is input into speech synthesis technology (e.g., Amazon Polly) and converted into audio data.

[0784] Output: Feedback message converted to audio data

[0785] Step 7:

[0786] The server transmits the generated feedback message to the terminal as voice data.

[0787] Input: Feedback message converted to audio data

[0788] Processing: Sends feedback audio data to the device via an HTTP response.

[0789] Output: Feedback audio data sent to the device

[0790] Step 8:

[0791] The terminal plays back the feedback voice data received from the server to the user.

[0792] Input: Feedback audio data sent from the server

[0793] Processing: The audio data is played through the device speaker to provide feedback to the user.

[0794] Output: Feedback audio played to the user

[0795] Step 9:

[0796] The user reviews the feedback and speaks additional responses.

[0797] Input: Feedback audio played from the device

[0798] Processing: The user speaks the acknowledgment "yes" or "no," which is recorded by the terminal.

[0799] Output: Additional audio data

[0800] Step 10:

[0801] The terminal transmits the additional voice data from the user to the server again.

[0802] Input: Additional voice data from the user

[0803] Processing: The recorded audio data is sent to the server via an HTTP request.

[0804] Output: Additional audio data sent to the server

[0805] Step 11:

[0806] The server receives the additional voice data, performs voice recognition and emotion analysis again, and performs a final confirmation.

[0807] Input: Additional audio data

[0808] Processing: The voice data is converted to text and analyzed again to determine the final order and user sentiment.

[0809] Output: Final confirmed order details and sentiment data

[0810] Step 12:

[0811] The server registers the final confirmed order details in a database and generates a notification of order completion.

[0812] Input: Final confirmed order details and sentiment data

[0813] Processing: The order details are saved in the database system, an order completion message is generated, and then the message is converted into voice data and sent to the terminal.

[0814] Output: Order details registered in the database and voice data of the completed order sent to the terminal

[0815] Step 13:

[0816] The terminal reproduces order completion voice data to the user.

[0817] Input: Order completion voice data sent from the server

[0818] Processing: Plays audio data to notify the user that the order has been completed.

[0819] Output: Order complete audio played to the user

[0820] This series of processing steps provides appropriate feedback according to the user's emotions, resulting in a pleasant ordering experience.

[0821] (Application example 2)

[0822] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0823] Conventional voice input systems provide mechanical responses without considering the user's emotions, and therefore do not provide a user-friendly or reassuring interface. Furthermore, users who are unfamiliar with the system, such as the elderly, the visually impaired, and foreigners, often find it difficult to use and feel frustrated. There is a need for a system that provides natural feedback based on the user's emotions to resolve these issues and facilitate the ordering process.

[0824] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to specify the order details, means for providing voice feedback of the generated order details, means for analyzing user emotion data, means for adjusting the feedback message based on the emotion data, and means for registering the final confirmed order details in a database. This enables natural feedback according to the user's emotions, allowing even elderly people, visually impaired people, and users unfamiliar with operation to place orders without stress.

[0825] "Voice input" is the process of receiving user spoken sounds through a device's microphone.

[0826] "Voice data" is data that represents in digital form the voice spoken by the user.

[0827] The "means for converting to text" refers to a technology that performs a process of analyzing audio data and converting it into corresponding text data.

[0828] "Text data" is voice data converted into character information, and is a sentence describing the contents of an order or conversation.

[0829] The "means for identifying the order content" is a technology that analyzes text data and identifies the specific product or service content that the user wants to order.

[0830] "Feedback means" refers to a technology that generates and transmits a voice message of confirmation or guidance to the user based on the analyzed order details.

[0831] "User emotion data" is data that indicates the user's emotion or emotional state, analyzed from the user's speech and voice characteristics.

[0832] "Means for analyzing emotional data" refers to technology that analyzes a user's voice and text data to identify the user's emotional state.

[0833] The "means for adjusting a feedback message based on emotional data" is a technique for adjusting the content and tone of a feedback message based on analyzed emotional data of a user.

[0834] The "finally confirmed order details" are the final order details that have been confirmed after being confirmed by the user.

[0835] The "means for registering in a database" refers to a technique for storing the confirmed order details in a database in digital form.

[0836] This invention is a food and beverage delivery ordering system that uses voice recognition technology, generative AI, and an emotion engine to assist users in the ordering process. Specifically, it uses a smartphone application to receive orders using the following procedure and provides feedback based on the user's emotions.

[0837] Hardware used

[0838] Smartphone (Android / iOS)

[0839] microphone

[0840] server

[0841] Software used

[0842] Speech recognition engine (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[0843] Emotion engines (e.g., AWS Rekognition, IBM Watson Tone Analyzer)

[0844] Generative AI (e.g., OpenAI's GPT-3)

[0845] Database (e.g. MySQL, Firebase)

[0846] Server-side processing

[0847] The server receives voice data sent from the smartphone. The voice data is first converted into text using a speech recognition engine. The text data is then analyzed to identify the order details. The analyzed order details are then processed together with the user's emotion data using an emotion engine.

[0848] The emotion engine analyzes the user's emotions from the tone and tempo of their voice, and the generation AI generates a feedback message based on this. The generated feedback message is composed of a tone and wording that corresponds to the user's emotions and is converted back into audio data. This audio data is sent to the smartphone and played back to the user.

[0849] After receiving the final confirmation response from the user, the order is analyzed again and the order details are finally confirmed. The final confirmation details are registered in the database and the order is completed.

[0850] Terminal side processing

[0851] When a user speaks their order details into their smartphone, the device detects this voice input and activates recording mode. The recorded voice data is sent to the server, and when feedback voice data is received, it is played back to the user. If the user checks the feedback and wants to re-enter the answer, they respond by voice, and this response voice is also sent back to the server.

[0852] User operations

[0853] The user speaks the specific details of their order into the device's microphone. For example, they might say, "One pizza, lots of cheese, tomato sauce." This voice data is sent to the server, where it undergoes a series of processes including analysis, feedback generation, and sentiment analysis. When the feedback is returned, the user confirms it and finally responds with "yes" or "no." As a result of this exchange, the order is confirmed and a notification of completion is sent.

[0854] Specific examples

[0855] For example, if a user says, "One pizza, lots of cheese, and tomato sauce," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order, and generates feedback. If the emotion engine analyzes the user's voice as relaxed, the generation AI will generate, "Thank you for your order. One pizza, lots of cheese, and tomato sauce, please. Is that okay?"

[0856] Prompt Sentence Examples

[0857] "When a user orders one pizza, lots of cheese, and tomato sauce, generate a feedback message in a relaxed tone."

[0858] This design allows users to complete their orders in a natural conversational manner, and is intended to be stress-free for users, especially the elderly, visually impaired, and those unfamiliar with the system.

[0859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0860] Step 1:

[0861] Input: User voice input

[0862] How it works: The user speaks specific details of their order into the smartphone's microphone.

[0863] Output: Recorded audio data

[0864] Step 2:

[0865] Input: Recorded audio data

[0866] Operation: The device detects the user's voice input and activates recording mode. The recorded voice data is sent to the server.

[0867] Output: Audio data sent to the server

[0868] Step 3:

[0869] Input: Audio data sent to the server

[0870] How it works: The server uses a speech recognition engine to convert the voice data into text data.

[0871] Output: Text data

[0872] Step 4:

[0873] Input: Text data

[0874] Operation: The server analyzes the text data and identifies the order contents.

[0875] Output: Identifying the order

[0876] Step 5:

[0877] Input: User's voice data and identified order details

[0878] How it works: The emotion engine analyzes the user's voice data and generates the user's emotion data.

[0879] Output: User emotion data

[0880] Step 6:

[0881] Input: Identified order details and user sentiment data

[0882] How it works: The server uses a generative AI to generate feedback messages based on the user's emotions.

[0883] Output: Feedback message

[0884] Step 7:

[0885] Input: Feedback message

[0886] Operation: The server converts the generated feedback message into audio data.

[0887] Output: Feedback message as audio data

[0888] Step 8:

[0889] Input: Feedback message as audio data

[0890] Operation: The server sends a feedback message to the terminal, which then plays it back to the user.

[0891] Output: The feedback message played to the user

[0892] Step 9:

[0893] Input: The user's response to the played feedback message

[0894] How it works: The user checks the feedback and responds again verbally. The device sends this response to the server.

[0895] Output: Response audio data sent to the server

[0896] Step 10:

[0897] Input: Response voice data sent to the server

[0898] Operation: The server analyzes the response voice data again and performs a final confirmation of the order details.

[0899] Output: Final confirmed order details

[0900] Step 11:

[0901] Input: Final confirmed order details

[0902] Operation: The server registers the final confirmed order details in the database.

[0903] Output: Order details registered in the database

[0904] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0906] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0907] [Third embodiment]

[0908] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0909] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0910] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0911] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0912] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0913] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0914] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0915] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0916] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0917] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0918] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0919] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0920] The present invention is a system that combines voice recognition technology and generative AI, allowing users to place orders through voice input. Below, the program processing of this system is explained in natural language, and an embodiment is explained using specific examples.

[0921] Server-side processing

[0922] The server receives the voice data sent from the device, which is the first step in recognizing what the user wants to order. Once the voice data is received, the server uses a speech recognition engine to convert the received voice data into text. For example, the server can use the Google Speech-to-Text API or other speech recognition services.

[0923] The server then uses generative AI to analyze the order details from the converted text data. For example, it extracts specific order details such as "one cup of coffee, no milk." Based on this analysis, the server generates a confirmation message to provide feedback to the user. The generated message is in a natural conversational format, such as "One cup of coffee, no milk, okay?"

[0924] The server converts this confirmation message into voice data and sends the generated voice data to the terminal. The server then receives additional input from the user, such as a confirmation response of "yes" or "no," analyzes the content, and performs a final confirmation. The final confirmed order details are then registered in a database, ensuring that the order is processed accurately.

[0925] Finally, the server generates a voice message to notify the user that the order has been completed and sends the voice message to the terminal, allowing the user to confirm that the order has been successfully received.

[0926] Terminal side processing

[0927] The device detects when the user starts voice input and activates recording mode. Voice input begins when the user taps the microphone icon or speaks a voice input command. The device records the user's speech and sends the voice data to the server.

[0928] When the terminal receives the feedback voice data from the server, it plays this voice data to the user. The user checks the feedback content and responds again if necessary. The terminal also receives a notification voice indicating that the order has been completed and plays it to the user.

[0929] User operations

[0930] The user speaks into the device's microphone to input the specific details of their order. For example, they might say, "I'd like a beer and a hamburger, please." When the feedback is returned, the user checks the details and responds "Yes" if there are no problems. If corrections are needed, they can say "No, not that one," followed by the new order details.

[0931] Once the final confirmation is complete, the user receives a notification from the terminal that the order has been completed, knowing that the order has been successfully accepted and can wait for the product in peace.

[0932] Specific examples

[0933] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. It then converts the confirmation message "One cup of coffee, no milk, okay?" into voice data and sends it to the device. If the user responds "Yes," the server performs a final confirmation, registers the order details in the database, and generates a notification of order completion that is sent to the device.

[0934] This system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation, which improves store operational efficiency and overall user satisfaction.

[0935] The processing flow will be explained below.

[0936] Step 1:

[0937] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[0938] Step 2:

[0939] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[0940] Step 3:

[0941] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[0942] Step 4:

[0943] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[0944] Step 5:

[0945] The server generates a feedback message based on the parsed order details, using generative AI to create a natural conversational message, such as "Would you like a beer and a hamburger?"

[0946] Step 6:

[0947] The server converts the generated feedback message into voice data and transmits it to the terminal.

[0948] Step 7:

[0949] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[0950] Step 8:

[0951] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[0952] Step 9:

[0953] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[0954] Step 10:

[0955] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[0956] Step 11:

[0957] The server registers the final confirmed order details in the database, which officially accepts the order.

[0958] Step 12:

[0959] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal.

[0960] Step 13:

[0961] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[0962] This process flow provides a natural ordering experience using voice, making it easy for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation of the system to order, improving store operational efficiency and overall user satisfaction.

[0963] Example 1

[0964] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0965] In conventional voice input systems, the accuracy of voice recognition and the generation of natural conversational formats were issues in the process of accurately understanding a user's order and providing feedback. Furthermore, when a user speaks multiple languages, it was difficult to switch languages ​​and adapt feedback accordingly. This resulted in problems that reduced user satisfaction for users who were unfamiliar with the system, such as the elderly, the visually impaired, and foreigners.

[0966] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0967] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order details, means for analyzing the order details using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order details, and means for registering the final confirmed order details in a database. This allows the user to accurately communicate the order details through voice input and receive feedback in a natural conversational style. Furthermore, appropriate feedback can be provided even when the user speaks multiple languages.

[0968] "Means for receiving voice input" refers to technology for capturing and recording voice data spoken by a user into the terminal.

[0969] "Means for converting voice data to text" refers to a technique that uses a voice recognition engine to convert received voice data into corresponding text data.

[0970] The "means for identifying the order details by analyzing the text data" refers to a technology for analyzing the converted text data and extracting the specific order details that the user is trying to convey.

[0971] "Means of analyzing order content using a generative AI model" refers to technology that utilizes generative AI to accurately read the user's intended order content from secured text data.

[0972] The "means for generating a feedback message in a natural conversational style based on the analysis results" refers to a technology for generating a confirmation message to the user in a natural conversational style based on the analyzed order details.

[0973] The "means for providing voice feedback on the generated order details" refers to a technology for converting the generated confirmation message into voice data and providing it to the user.

[0974] "Means for registering the final confirmed order details in a database" refers to a technology for saving the order details finally confirmed by the user in a database and managing them as order processing information.

[0975] "Means for playing back audio feedback to the user" refers to a technique for playing back the generated audio data at the terminal and providing feedback to the user.

[0976] The "means for receiving and re-analyzing a confirmation response from the user" refers to a technology for re-analyzing the user's response voice and determining the next action based on the content of the analysis.

[0977] "Means for identifying multiple languages ​​and providing feedback in a specified language" refers to technology that automatically determines the language used by a user and provides appropriate feedback in that language.

[0978] This invention is a system that allows users to place orders through voice input using a system that combines voice recognition technology and a generative AI model. This system is mainly composed of a server, a terminal, and a user.

[0979] The server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order contents, means for analyzing the order contents using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order contents, means for registering the final confirmed order contents in a database, and means for identifying multiple languages ​​and providing feedback in a specified language.

[0980] The terminal includes means for detecting a user's voice input, activating a recording mode, and playing feedback voice data from the server. The user provides voice input through a microphone of the terminal and transmits the voice data to the server.

[0981] The server converts the received voice data into text using the Google Speech-to-Text API or other speech recognition services. It then uses a generative AI model (e.g., GPT-3) to analyze the order details from the converted text data. Based on the analysis results, it generates a feedback message in a natural conversational style, such as "Would you like a cup of coffee, without milk?", converts it into voice data, and sends it to the device.

[0982] The terminal plays a feedback voice to the user, and the user responds with a confirmation. The user's response voice is also sent from the terminal to the server. The server again performs voice recognition and analysis, and registers the final confirmed order details in the database. This ensures that the order is processed accurately.

[0983] As a concrete example, consider the case where a domestic customer says, "One hamburger, one glass of coke." When the user speaks, the device records this voice and sends it to the server. The server converts the voice data into text and analyzes the order using a generative AI model. Based on the analysis results, it generates a feedback message saying, "One hamburger, one glass of coke, would you like?" and sends it as voice data to the device. If the user responds "Yes," the server analyzes again, registers the order in the database, and notifies the user that "the order has been completed."

[0984] For example, by inputting an example of a prompt sentence such as "Please explain the detailed steps of the process in which the user says 'one hamburger, one glass of coke'", the detailed process described above is generated.

[0985] The system provides an intuitive and easy-to-use interface, particularly for elderly people, visually impaired people, foreigners, and users unfamiliar with the operation, improving store operational efficiency and overall user satisfaction.

[0986] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0987] Step 1:

[0988] The user begins voice input. The user speaks into the device's microphone, saying, "A cup of hot coffee, no sugar." This becomes the base data for subsequent processing.

[0989] Step 2:

[0990] The device detects the user's speech and activates recording mode. The device's microphone captures and records the audio data. The input is audio data, and the output is a recorded audio file.

[0991] Step 3:

[0992] The device sends the recorded audio data to the server via Wi-Fi or mobile data network. The input is the recorded audio data, and the output is the audio data sent as an HTTP request to the server.

[0993] Step 4:

[0994] The server converts the received voice data into text. This is done using speech recognition software such as the Google Speech-to-Text API. The input is voice data, and the output is text data generated based on that voice data. Specifically, the generated text might be something like "A cup of hot coffee, no sugar."

[0995] Step 5:

[0996] The server uses a generative AI model to analyze the order. During analysis, it uses text data as input to identify each element of the order (drink, temperature, whether sugar is added, etc.). The input is text data, and the output is a data structure of the analyzed order. For example, the drink might be "coffee," the temperature "warm," and the sugar "no."

[0997] Step 6:

[0998] The server generates a feedback message in a natural conversational format based on the analysis results. It uses a generative AI model to generate a message such as "Would you like a cup of coffee, no sugar?" and converts it into voice data. The input is the analysis results, and the output is the generated voice message.

[0999] Step 7:

[1000] The server sends the generated feedback voice data to the device. The communication method is again Wi-Fi or mobile data network. The input is voice data, and the output is voice data sent as an HTTP response to the device.

[1001] Step 8:

[1002] The terminal plays back the feedback audio data received from the server. The user can hear the audio feedback, "Would you like a cup of coffee, no sugar?" The input is the audio data received from the server, and the output is the audio playback.

[1003] Step 9:

[1004] The user confirms the feedback voice and responds again with voice, for example, saying "Yes, that's fine." This provides the user with final confirmation.

[1005] Step 10:

[1006] The terminal re-records the user's confirmation response and sends it to the server. The input is the re-recorded voice data, and the output is the response voice data to be sent as an HTTP request to the server.

[1007] Step 11:

[1008] The server converts the user's confirmation response into text using a speech recognition engine and analyzes the content. The input is voice data, and the output is the analyzed text data. For example, a final confirmation text such as "Yes, that's it, please" is generated.

[1009] Step 12:

[1010] The server registers the order details that have been finally confirmed in the database. The input is the data of the final confirmed order details, and the output is the order information saved as a record in the database.

[1011] Step 13:

[1012] The server generates a voice message notifying the terminal that the order has been processed successfully, and sends it to the terminal. Specifically, it sends a message such as "Your order has been completed. Thank you." The input is the order completion information, and the output is the generated voice message data.

[1013] Step 14:

[1014] The terminal plays audio data to notify the user that the order has been completed. The user hears this audio notification and confirms that the order has been accepted. The input is audio data from the server, and the output is audio playback.

[1015] (Application example 1)

[1016] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1017] Conventional food delivery ordering systems are often complex and difficult to operate for elderly people, visually impaired people, and users who are not familiar with the system. Language barriers and misunderstandings of detailed order details can also be problematic. This can lead to ordering errors and user dissatisfaction. Therefore, the objective of this invention is to provide a system that uses voice recognition technology and generative AI to allow users to easily and accurately order food delivery.

[1018] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1019] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for providing voice feedback on the generated order details, means for registering the final confirmed order details in a database, means for generating a confirmation message for the user using a smartphone and converting it into voice data, and means for analyzing the user's voice input and identifying the food delivery order details by combining voice recognition technology and generation AI. This enables even users who are particularly elderly, visually impaired, or unfamiliar with operation to easily place orders by voice and accurately confirm their orders.

[1020] "Voice input" is the process by which a user sends voice data to a system through a microphone.

[1021] "Voice data" refers to information that is a digital recording of a user's speech.

[1022] "Text conversion" is the process of analyzing audio data and converting it into corresponding text data.

[1023] "Text data" is voice data expressed as text information.

[1024] "Order Details" refers to specific product or service requests made by a user through voice.

[1025] "Generative AI" is an artificial intelligence technology that performs natural language processing based on large amounts of text data and generates appropriate text data according to demand.

[1026] "Feedback" is the process by which the system responds to the user with analysis results and confirmation items.

[1027] "Database" means a storage device for storing and managing information such as final confirmed order details.

[1028] A "smartphone" is a mobile device with advanced processing power and internet connectivity.

[1029] A "confirmation message" is a message that the system sends to a user after receiving the order details from the user to confirm the order.

[1030] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[1031] "Order identification" is the process of analyzing voice and text data to determine the products or services the user is trying to order.

[1032] "Re-analysis" is the process of receiving an acknowledgment from the user and re-analyzing it.

[1033] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for analysis and generation.

[1034] This invention is a system that combines voice recognition technology and generative AI, allowing users to order food delivery through voice input. An embodiment of this system is shown below.

[1035] Server-side processing

[1036] The server processes the audio data in the following steps:

[1037] 1. Voice input reception

[1038] The voice data that the user speaks into the smartphone is received. The smartphone is equipped with a microphone, and the voice data is collected from there. The API for receiving the voice data uses the standard HTTP protocol.

[1039] 2. Converting voice data to text

[1040] The received voice data is converted to text data using the Google Speech-to-Text API, and the order details are then ready to be analyzed based on the converted text.

[1041] 3. Order Analysis

[1042] A generative AI model is used to analyze the text data and identify specific order details, such as if you are ordering "one pizza and one Pepsi."

[1043] 4. Generate a confirmation message

[1044] Based on the analyzed order details, a confirmation message is generated to provide feedback to the user. The generated message is in a natural conversational style, such as "Is one pizza and one Pepsi okay?"

[1045] 5. Conversion to audio data

[1046] The confirmation message is converted into voice data and sent to the terminal. Voice synthesis technology is used to generate the confirmation message as natural-sounding voice data.

[1047] 6. Final confirmation and registration in the database

[1048] The confirmation response from the user is analyzed again, and the final order details are confirmed and registered in the database, again using the Google Speech-to-Text API and generative AI models.

[1049] Terminal side processing

[1050] The smartphone plays back the feedback from the server to the user for final confirmation, as follows:

[1051] 1. Start voice input

[1052] Voice input is initiated by the user tapping the microphone icon or speaking a voice input command.

[1053] 2. Feedback playback

[1054] The confirmation message sent from the server is received as audio data and played back to the user, who then confirms the feedback.

[1055] 3. Receiving an Acknowledgment

[1056] If the user responds with a "yes" or "no" confirmation, the voice data is sent to the server again, providing a final confirmation.

[1057] 4. Notification of order completion

[1058] Once the final confirmed order details are registered in the database, a voice message informing the user that the order has been completed is sent and played to the user.

[1059] User operations

[1060] The user can enter the specific details of their order by voice, and the system will analyze, confirm, and register them appropriately, allowing for a smooth food delivery order. For example, if an elderly person or a visually impaired person wants to order "one pizza and one Pepsi," they can do so simply and accurately.

[1061] Examples and prompts

[1062] Specific examples

[1063] If an elderly user is trying to order a pizza:

[1064] 1. User: "One pizza and one Pepsi, please."

[1065] 2. Server: "One pizza and one Pepsi, please?"

[1066] 3. User: "Yes"

[1067] 4. Server: "Thank you for your order. Your order of one pizza and one Pepsi has been completed."

[1068] Prompt Sentence Examples

[1069] If a user orders one curry and one salad:

[1070] User: "I'd like to order one curry and one salad."

[1071] Examples of how to respond after receiving a prompt:

[1072] Transliteration: "I'd like to order one curry and one salad."

[1073] Generated AI response: "Would you like one plate of curry and one salad?"

[1074] Final check:

[1075] Final text entry: "Yes"

[1076] The above is an embodiment of the present invention.

[1077] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1078] Step 1:

[1079] The user speaks their order into the smartphone. For example, they might say, "One pizza and one Pepsi, please." This voice data is recorded through the smartphone's microphone. Input data: User's voice data. Output data: Recorded voice data file.

[1080] Step 2:

[1081] The smartphone sends the recorded voice data to the server. Input data: Recorded voice data file. Output data: Voice data sent to the server.

[1082] Step 3:

[1083] The server receives the audio data and converts it to text using the Google Speech-to-Text API. Input data: Audio data. Output data: Text data.

[1084] Step 4:

[1085] The server uses the generative AI model to analyze the order details from the text data. Specifically, the speech-recognized text data is sent to the generative AI model as a prompt to identify the order details. Input data: Text data. Output data: Identified order details.

[1086] Step 5:

[1087] The server generates a confirmation message based on the parsed order details. For example, it generates a message like "One pizza and one Pepsi, okay?" Input data: Identified order details. Output data: Text of the confirmation message.

[1088] Step 6:

[1089] The server converts the text of the confirmation message into audio data, generates the audio data, and sends it to the smartphone. Input data: Text of the confirmation message. Output data: Audio data of the confirmation message.

[1090] Step 7:

[1091] The smartphone plays the received audio data to the user. The user listens to the played confirmation message and confirms its contents. Input data: Audio data of the confirmation message. Output data: User's understanding and judgment.

[1092] Step 8:

[1093] The user responds with "yes" or "no." For example, the user utters "yes." This voice data is recorded again. Input data: User's confirmation voice data. Output data: Recorded confirmation voice data file.

[1094] Step 9:

[1095] The smartphone sends the recorded confirmation response voice data to the server. Input data: Recorded confirmation response voice data file. Output data: Confirmation response voice data sent to the server.

[1096] Step 10:

[1097] The server receives the confirmation response audio data and converts it to text again using the Google Speech-to-Text API. Input data: confirmation response audio data. Output data: text data.

[1098] Step 11:

[1099] The server analyzes the text data of the confirmation response and performs a final confirmation. For example, if the response is "yes," the order is confirmed, and if the response is "no," the order details are revised and a confirmation message is generated. Input data: Text data of the confirmation response. Output data: Final confirmation result.

[1100] Step 12:

[1101] The final confirmed order details are registered in the database. Input data: Final confirmation results. Output data: Order details registered in the database.

[1102] Step 13:

[1103] The server generates a message indicating that the order has been completed, converts it into voice data, and sends it to the smartphone. For example, a voice message such as "Thank you for your order. Your order for one pizza and one Pepsi has been completed." Input data: Text of the order completion message. Output data: Voice data of the order completion message.

[1104] Step 14:

[1105] The smartphone plays the received voice data of the order completion message to the user. Input data: Voice data of the order completion message. Output data: User's understanding and confirmation of order acceptance.

[1106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1107] The present invention is a system that combines speech recognition technology, generative AI, and an emotion engine, and when a user places an order through voice input, it recognizes the user's emotion and adjusts feedback based on that emotion, providing a more natural and comfortable experience. Below, the program processing of this system is explained in natural language, and an embodiment is described with specific examples.

[1108] Server-side processing

[1109] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Through this process, what the user wants to order becomes clear in text form.

[1110] The server then uses generative AI to analyze the text data and identify the order details, such as "one beer and one hamburger." The emotion engine then analyzes the user's voice data and recognizes the emotion expressed during the feedback.

[1111] The server generates a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion recognized by the emotion engine. For example, if the user is impatient, the server generates feedback in a gentle tone such as "Please wait a moment, we will respond immediately."

[1112] The server converts this feedback message into voice data and sends it to the device. It also receives additional voice data from the user (such as "yes" or "no"), converts it into text, and analyzes it. The server also considers the user's emotions when making a final confirmation.

[1113] Finally, the final confirmed order details are registered in the database. The server also generates a message to notify the completion of the order and sends it to the terminal.

[1114] Terminal side processing

[1115] The device detects that the user has started voice input and activates recording mode. The recorded voice data is sent to the server. When the device receives the feedback voice data from the server, it plays it back to the user. The user checks the feedback and responds by voice if they want to input the answer again. The device then sends the response voice back to the server.

[1116] User operations

[1117] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks the details and, if there are no problems, responds "Yes." If corrections are needed, they can say "No, not that," followed by the new order details. Once the user has completed the final confirmation and received a notification from the device that the order has been completed, they can feel reassured that their order has been successfully accepted.

[1118] Specific examples

[1119] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[1120] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[1124] Step 2:

[1125] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[1126] Step 3:

[1127] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[1128] Step 4:

[1129] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[1130] Step 5:

[1131] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, recognizing emotions such as impatience or anxiety from the tone and speed of the user's voice.

[1132] Step 6:

[1133] The server generates a feedback message based on the analyzed order details and the user's emotional data. Generative AI is used to create a feedback message that takes into account the appropriate tone and wording. For example, if the user is impatient, the server might say, "Please wait a moment, we will respond shortly."

[1134] Step 7:

[1135] The server converts the generated feedback message into voice data and transmits it to the terminal.

[1136] Step 8:

[1137] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[1138] Step 9:

[1139] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[1140] Step 10:

[1141] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[1142] Step 11:

[1143] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[1144] Step 12:

[1145] The server registers the final confirmed order details in the database, which officially accepts the order.

[1146] Step 13:

[1147] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal. At this time, feedback is provided taking into account emotion data.

[1148] Step 14:

[1149] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[1150] Specific examples

[1151] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[1152] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[1153] Example 2

[1154] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1155] The problem that the present invention aims to solve is the inability to provide a more natural and comfortable experience by recognizing a user's emotions and adjusting feedback based on them when placing an order via voice input. Conventional systems provide uniform feedback without considering the user's emotions, which can impair the user experience. Another problem is that they are unable to provide appropriate services to a variety of users, including the elderly, the visually impaired, and foreigners.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1157] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for analyzing the user's emotions from the voice data, means for generating a feedback message according to the user's emotions using a generative AI model, means for converting the generated feedback message into voice data, means for providing the generated feedback message by voice, and means for registering the final confirmed order details in a database. This allows appropriate feedback to be given based on the user's emotions, and makes it possible to provide a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with operation.

[1158] The "means for receiving voice input" refers to a device or method for receiving voice data spoken by a user and incorporating it into the system.

[1159] A "means for converting voice data to text" is a device or method for generating text data from voice data using voice recognition technology.

[1160] The "means for analyzing text data to identify order details" refers to a device or method for processing text data and extracting specific order information requested by the user.

[1161] The "means for analyzing user emotions from voice data" is a device or method for analyzing voice data to extract the user's emotional state.

[1162] "Means for generating feedback messages according to user emotions using a generative AI model" refers to a device or method for using generative AI to create appropriate feedback messages based on analyzed user emotions.

[1163] The "means for converting the generated feedback message into voice data" is a device or method for converting the textual feedback message into voice data.

[1164] The "means for providing a generated feedback message by voice" refers to a device or method for transmitting a feedback message converted into voice data to a user.

[1165] The "means for registering the final confirmed order details in the database" refers to a device or method for storing the order details that have been finally confirmed by the system in the database.

[1166] The present invention is a system that combines voice recognition technology, generative AI, and an emotion engine, and provides a more natural and comfortable experience by recognizing the user's emotions when placing an order through voice input and adjusting feedback accordingly. This system is configured as follows.

[1167] Server-side processing

[1168] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Specifically, Google Cloud Speech-to-Text is used to convert the voice data into text data. This makes it clear what the user wants to order in text format.

[1169] Next, the server uses a natural language processing model (e.g., a generative AI model) to analyze the text data and identify the order. For example, it extracts specific order information such as "one beer and one hamburger." In parallel, it uses emotion analysis technology (e.g., a voice emotion analysis engine) to analyze the user's emotions from the voice data.

[1170] The server uses a generative AI model to generate a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion as recognized by the emotion engine. For example, if the user is impatient, a gentle tone of feedback such as "Please wait a moment, we will respond immediately" is generated.

[1171] To convert the generated feedback message into audio data, a speech synthesis technology (e.g., a speech synthesis engine) is used. Specifically, Amazon Polly is used to convert the text message into audio data. This audio data is then sent to the device.

[1172] Furthermore, the system receives voice data for additional input from the user (such as "yes" or "no"), performs voice recognition and emotion analysis again, and performs final confirmation. The final confirmed order details are registered in a database system (e.g., a database management system). This ensures that the order record is saved.

[1173] Terminal side processing

[1174] The device detects that the user has started voice input and activates recording mode. In this mode, the device's microphone records the user's voice and transmits the voice data to the server. The device also receives feedback voice data transmitted from the server, plays it back, and provides it to the user.

[1175] The audio data recorded by the device is converted into an appropriate format (e.g., WAV or MP3) and then sent to the server. This process is performed through an HTTP request. Upon receiving the feedback audio data from the server, the device plays it back to the user.

[1176] User operations

[1177] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks it and, if satisfied, responds with "Yes." If a correction is needed, they can say, "No, not that," followed by the new order details. Once the final confirmation is complete, the user receives a notification from the device that the order has been successfully accepted, providing peace of mind.

[1178] Specific examples

[1179] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion analysis engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording that corresponds to the user's emotion, giving the user a sense of security.

[1180] Prompt Sentence Examples

[1181] Use the following prompt for the generative AI model:

[1182] prompt:

[1183] User utterance: "One cup of coffee, no milk"

[1184] User Emotion: "Impatience"

[1185] Produced message: "Please wait a moment, your coffee will be ready shortly."

[1186] By using such prompt sentences, it is possible to generate appropriate feedback messages according to the user's emotions.

[1187] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1188] Step 1:

[1189] The server receives the voice data transmitted from the terminal.

[1190] Input: User spoken voice data (WAV or MP3 format)

[1191] Processing: The audio data is sent to the server via an HTTP request, and the server stores it in its storage.

[1192] Output: Audio data file saved in storage

[1193] Step 2:

[1194] The server converts the received voice data into text using a voice recognition engine.

[1195] Input: Audio data file saved in storage

[1196] Processing: The audio data is input into a speech recognition engine (for example, Google Cloud Speech-to-Text) and converted into text data.

[1197] Output: Converted text data

[1198] Step 3:

[1199] The server uses generative AI to analyze the text data and identify the order details.

[1200] Input: Converted text data

[1201] Processing: A generative AI model (e.g., GPT-4) is fed the text data and used to analyze and identify the order, e.g., extracting specific order information such as "one cup of coffee, no milk."

[1202] Output: Identified order details

[1203] Step 4:

[1204] The server uses an emotion engine to analyze the user's emotion from the voice data.

[1205] Input: Audio data file saved in storage

[1206] Processing: Input the voice data into an emotion engine (e.g., Emotion API) and analyze the user's emotions (e.g., impatience, relief, joy).

[1207] Output: Parsed emotion data

[1208] Step 5:

[1209] The server uses a generative AI model to generate a feedback message based on the analyzed order content and user sentiment data.

[1210] Input: Identified order details, analyzed sentiment data

[1211] Processing: The following prompt sentence is input into the generative AI model, and a feedback message corresponding to the user's emotion is output.

[1212] prompt:

[1213] User utterance: "One cup of coffee, no milk"

[1214] User Emotion: "Impatience"

[1215] Produced message: "Please wait a moment, your coffee will be ready shortly."

[1216] Output: The generated feedback message

[1217] Step 6:

[1218] The server converts the generated feedback message into audio data.

[1219] Input: The generated feedback message

[1220] Processing: A text message is input into speech synthesis technology (e.g., Amazon Polly) and converted into audio data.

[1221] Output: Feedback message converted to audio data

[1222] Step 7:

[1223] The server transmits the generated feedback message to the terminal as voice data.

[1224] Input: Feedback message converted to audio data

[1225] Processing: Sends feedback audio data to the device via an HTTP response.

[1226] Output: Feedback audio data sent to the device

[1227] Step 8:

[1228] The terminal plays back the feedback voice data received from the server to the user.

[1229] Input: Feedback audio data sent from the server

[1230] Processing: The audio data is played through the device speaker to provide feedback to the user.

[1231] Output: Feedback audio played to the user

[1232] Step 9:

[1233] The user reviews the feedback and speaks additional responses.

[1234] Input: Feedback audio played from the device

[1235] Processing: The user speaks the acknowledgment "yes" or "no," which is recorded by the terminal.

[1236] Output: Additional audio data

[1237] Step 10:

[1238] The terminal transmits the additional voice data from the user to the server again.

[1239] Input: Additional voice data from the user

[1240] Processing: The recorded audio data is sent to the server via an HTTP request.

[1241] Output: Additional audio data sent to the server

[1242] Step 11:

[1243] The server receives the additional voice data, performs voice recognition and emotion analysis again, and performs a final confirmation.

[1244] Input: Additional audio data

[1245] Processing: The voice data is converted to text and analyzed again to determine the final order and user sentiment.

[1246] Output: Final confirmed order details and sentiment data

[1247] Step 12:

[1248] The server registers the final confirmed order details in a database and generates a notification of order completion.

[1249] Input: Final confirmed order details and sentiment data

[1250] Processing: The order details are saved in the database system, an order completion message is generated, and then the message is converted into voice data and sent to the terminal.

[1251] Output: Order details registered in the database and voice data of the completed order sent to the terminal

[1252] Step 13:

[1253] The terminal reproduces order completion voice data to the user.

[1254] Input: Order completion voice data sent from the server

[1255] Processing: Plays audio data to notify the user that the order has been completed.

[1256] Output: Order complete audio played to the user

[1257] This series of processing steps provides appropriate feedback according to the user's emotions, resulting in a pleasant ordering experience.

[1258] (Application example 2)

[1259] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1260] Conventional voice input systems provide mechanical responses without considering the user's emotions, and therefore do not provide a user-friendly or reassuring interface. Furthermore, users who are unfamiliar with the system, such as the elderly, the visually impaired, and foreigners, often find it difficult to use and feel frustrated. There is a need for a system that provides natural feedback based on the user's emotions to resolve these issues and facilitate the ordering process.

[1261] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to specify the order details, means for providing voice feedback of the generated order details, means for analyzing user emotion data, means for adjusting the feedback message based on the emotion data, and means for registering the final confirmed order details in a database. This enables natural feedback according to the user's emotions, allowing even elderly people, visually impaired people, and users unfamiliar with operation to place orders without stress.

[1262] "Voice input" is the process of receiving user spoken sounds through a device's microphone.

[1263] "Voice data" is data that represents in digital form the voice spoken by the user.

[1264] The "means for converting to text" refers to a technology that performs a process of analyzing audio data and converting it into corresponding text data.

[1265] "Text data" is voice data converted into character information, and is a sentence describing the contents of an order or conversation.

[1266] The "means for identifying the order content" is a technology that analyzes text data and identifies the specific product or service content that the user wants to order.

[1267] "Feedback means" refers to a technology that generates and transmits a voice message of confirmation or guidance to the user based on the analyzed order details.

[1268] "User emotion data" is data that indicates the user's emotion or emotional state, analyzed from the user's speech and voice characteristics.

[1269] "Means for analyzing emotional data" refers to technology that analyzes a user's voice and text data to identify the user's emotional state.

[1270] The "means for adjusting a feedback message based on emotional data" is a technique for adjusting the content and tone of a feedback message based on analyzed emotional data of a user.

[1271] The "finally confirmed order details" are the final order details that have been confirmed after being confirmed by the user.

[1272] The "means for registering in a database" refers to a technique for storing the confirmed order details in a database in digital form.

[1273] This invention is a food and beverage delivery ordering system that uses voice recognition technology, generative AI, and an emotion engine to assist users in the ordering process. Specifically, it uses a smartphone application to receive orders using the following procedure and provides feedback based on the user's emotions.

[1274] Hardware used

[1275] Smartphone (Android / iOS)

[1276] microphone

[1277] server

[1278] Software used

[1279] Speech recognition engine (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[1280] Emotion engines (e.g., AWS Rekognition, IBM Watson Tone Analyzer)

[1281] Generative AI (e.g., OpenAI's GPT-3)

[1282] Database (e.g. MySQL, Firebase)

[1283] Server-side processing

[1284] The server receives voice data sent from the smartphone. The voice data is first converted into text using a speech recognition engine. The text data is then analyzed to identify the order details. The analyzed order details are then processed together with the user's emotion data using an emotion engine.

[1285] The emotion engine analyzes the user's emotions from the tone and tempo of their voice, and the generation AI generates a feedback message based on this. The generated feedback message is composed of a tone and wording that corresponds to the user's emotions and is converted back into audio data. This audio data is sent to the smartphone and played back to the user.

[1286] After receiving the final confirmation response from the user, the order is analyzed again and the order details are finally confirmed. The final confirmation details are registered in the database and the order is completed.

[1287] Terminal side processing

[1288] When a user speaks their order details into their smartphone, the device detects this voice input and activates recording mode. The recorded voice data is sent to the server, and when feedback voice data is received, it is played back to the user. If the user checks the feedback and wants to re-enter the answer, they respond by voice, and this response voice is also sent back to the server.

[1289] User operations

[1290] The user speaks the specific details of their order into the device's microphone. For example, they might say, "One pizza, lots of cheese, tomato sauce." This voice data is sent to the server, where it undergoes a series of processes including analysis, feedback generation, and sentiment analysis. When the feedback is returned, the user confirms it and finally responds with "yes" or "no." As a result of this exchange, the order is confirmed and a notification of completion is sent.

[1291] Specific examples

[1292] For example, if a user says, "One pizza, lots of cheese, and tomato sauce," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order, and generates feedback. If the emotion engine analyzes the user's voice as relaxed, the generation AI will generate, "Thank you for your order. One pizza, lots of cheese, and tomato sauce, please. Is that okay?"

[1293] Prompt Sentence Examples

[1294] "When a user orders one pizza, lots of cheese, and tomato sauce, generate a feedback message in a relaxed tone."

[1295] This design allows users to complete their orders in a natural conversational manner, and is intended to be stress-free for users, especially the elderly, visually impaired, and those unfamiliar with the system.

[1296] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1297] Step 1:

[1298] Input: User voice input

[1299] How it works: The user speaks specific details of their order into the smartphone's microphone.

[1300] Output: Recorded audio data

[1301] Step 2:

[1302] Input: Recorded audio data

[1303] Operation: The device detects the user's voice input and activates recording mode. The recorded voice data is sent to the server.

[1304] Output: Audio data sent to the server

[1305] Step 3:

[1306] Input: Audio data sent to the server

[1307] How it works: The server uses a speech recognition engine to convert the voice data into text data.

[1308] Output: Text data

[1309] Step 4:

[1310] Input: Text data

[1311] Operation: The server analyzes the text data and identifies the order contents.

[1312] Output: Identifying the order

[1313] Step 5:

[1314] Input: User's voice data and identified order details

[1315] How it works: The emotion engine analyzes the user's voice data and generates the user's emotion data.

[1316] Output: User emotion data

[1317] Step 6:

[1318] Input: Identified order details and user sentiment data

[1319] How it works: The server uses a generative AI to generate feedback messages based on the user's emotions.

[1320] Output: Feedback message

[1321] Step 7:

[1322] Input: Feedback message

[1323] Operation: The server converts the generated feedback message into audio data.

[1324] Output: Feedback message as audio data

[1325] Step 8:

[1326] Input: Feedback message as audio data

[1327] Operation: The server sends a feedback message to the terminal, which then plays it back to the user.

[1328] Output: The feedback message played to the user

[1329] Step 9:

[1330] Input: The user's response to the played feedback message

[1331] How it works: The user checks the feedback and responds again verbally. The device sends this response to the server.

[1332] Output: Response audio data sent to the server

[1333] Step 10:

[1334] Input: Response voice data sent to the server

[1335] Operation: The server analyzes the response voice data again and performs a final confirmation of the order details.

[1336] Output: Final confirmed order details

[1337] Step 11:

[1338] Input: Final confirmed order details

[1339] Operation: The server registers the final confirmed order details in the database.

[1340] Output: Order details registered in the database

[1341] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1342] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1343] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1344] [Fourth embodiment]

[1345] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1346] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1347] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1348] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1349] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1350] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1351] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1352] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1353] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1354] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1355] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1356] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1357] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] The present invention is a system that combines voice recognition technology and generative AI, allowing users to place orders through voice input. Below, the program processing of this system is explained in natural language, and an embodiment is explained using specific examples.

[1359] Server-side processing

[1360] The server receives the voice data sent from the device, which is the first step in recognizing what the user wants to order. Once the voice data is received, the server uses a speech recognition engine to convert the received voice data into text. For example, the server can use the Google Speech-to-Text API or other speech recognition services.

[1361] The server then uses generative AI to analyze the order details from the converted text data. For example, it extracts specific order details such as "one cup of coffee, no milk." Based on this analysis, the server generates a confirmation message to provide feedback to the user. The generated message is in a natural conversational format, such as "One cup of coffee, no milk, okay?"

[1362] The server converts this confirmation message into voice data and sends the generated voice data to the terminal. The server then receives additional input from the user, such as a confirmation response of "yes" or "no," analyzes the content, and performs a final confirmation. The final confirmed order details are then registered in a database, ensuring that the order is processed accurately.

[1363] Finally, the server generates a voice message to notify the user that the order has been completed and sends the voice message to the terminal, allowing the user to confirm that the order has been successfully received.

[1364] Terminal side processing

[1365] The device detects when the user starts voice input and activates recording mode. Voice input begins when the user taps the microphone icon or speaks a voice input command. The device records the user's speech and sends the voice data to the server.

[1366] When the terminal receives the feedback voice data from the server, it plays this voice data to the user. The user checks the feedback content and responds again if necessary. The terminal also receives a notification voice indicating that the order has been completed and plays it to the user.

[1367] User operations

[1368] The user speaks into the device's microphone to input the specific details of their order. For example, they might say, "I'd like a beer and a hamburger, please." When the feedback is returned, the user checks the details and responds "Yes" if there are no problems. If corrections are needed, they can say "No, not that one," followed by the new order details.

[1369] Once the final confirmation is complete, the user receives a notification from the terminal that the order has been completed, knowing that the order has been successfully accepted and can wait for the product in peace.

[1370] Specific examples

[1371] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. It then converts the confirmation message "One cup of coffee, no milk, okay?" into voice data and sends it to the device. If the user responds "Yes," the server performs a final confirmation, registers the order details in the database, and generates a notification of order completion that is sent to the device.

[1372] This system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation, which improves store operational efficiency and overall user satisfaction.

[1373] The processing flow will be explained below.

[1374] Step 1:

[1375] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[1376] Step 2:

[1377] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[1378] Step 3:

[1379] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[1380] Step 4:

[1381] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[1382] Step 5:

[1383] The server generates a feedback message based on the parsed order details, using generative AI to create a natural conversational message, such as "Would you like a beer and a hamburger?"

[1384] Step 6:

[1385] The server converts the generated feedback message into voice data and transmits it to the terminal.

[1386] Step 7:

[1387] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[1388] Step 8:

[1389] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[1390] Step 9:

[1391] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[1392] Step 10:

[1393] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[1394] Step 11:

[1395] The server registers the final confirmed order details in the database, which officially accepts the order.

[1396] Step 12:

[1397] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal.

[1398] Step 13:

[1399] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[1400] This process flow provides a natural ordering experience using voice, making it easy for elderly people, visually impaired people, foreigners, and users who are not familiar with the operation of the system to order, improving store operational efficiency and overall user satisfaction.

[1401] Example 1

[1402] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] In conventional voice input systems, the accuracy of voice recognition and the generation of natural conversational formats were issues in the process of accurately understanding a user's order and providing feedback. Furthermore, when a user speaks multiple languages, it was difficult to switch languages ​​and adapt feedback accordingly. This resulted in problems that reduced user satisfaction for users who were unfamiliar with the system, such as the elderly, the visually impaired, and foreigners.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1405] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order details, means for analyzing the order details using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order details, and means for registering the final confirmed order details in a database. This allows the user to accurately communicate the order details through voice input and receive feedback in a natural conversational style. Furthermore, appropriate feedback can be provided even when the user speaks multiple languages.

[1406] "Means for receiving voice input" refers to technology for capturing and recording voice data spoken by a user into the terminal.

[1407] "Means for converting voice data to text" refers to a technique that uses a voice recognition engine to convert received voice data into corresponding text data.

[1408] The "means for identifying the order details by analyzing the text data" refers to a technology for analyzing the converted text data and extracting the specific order details that the user is trying to convey.

[1409] "Means of analyzing order content using a generative AI model" refers to technology that utilizes generative AI to accurately read the user's intended order content from secured text data.

[1410] The "means for generating a feedback message in a natural conversational style based on the analysis results" refers to a technology for generating a confirmation message to the user in a natural conversational style based on the analyzed order details.

[1411] The "means for providing voice feedback on the generated order details" refers to a technology for converting the generated confirmation message into voice data and providing it to the user.

[1412] "Means for registering the final confirmed order details in a database" refers to a technology for saving the order details finally confirmed by the user in a database and managing them as order processing information.

[1413] "Means for playing back audio feedback to the user" refers to a technique for playing back the generated audio data at the terminal and providing feedback to the user.

[1414] The "means for receiving and re-analyzing a confirmation response from the user" refers to a technology for re-analyzing the user's response voice and determining the next action based on the content of the analysis.

[1415] "Means for identifying multiple languages ​​and providing feedback in a specified language" refers to technology that automatically determines the language used by a user and provides appropriate feedback in that language.

[1416] This invention is a system that allows users to place orders through voice input using a system that combines voice recognition technology and a generative AI model. This system is mainly composed of a server, a terminal, and a user.

[1417] The server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify order contents, means for analyzing the order contents using a generative AI model, means for generating a feedback message in a natural conversational style based on the analysis results, means for providing voice feedback of the generated order contents, means for registering the final confirmed order contents in a database, and means for identifying multiple languages ​​and providing feedback in a specified language.

[1418] The terminal includes means for detecting a user's voice input, activating a recording mode, and playing feedback voice data from the server. The user provides voice input through a microphone of the terminal and transmits the voice data to the server.

[1419] The server converts the received voice data into text using the Google Speech-to-Text API or other speech recognition services. It then uses a generative AI model (e.g., GPT-3) to analyze the order details from the converted text data. Based on the analysis results, it generates a feedback message in a natural conversational style, such as "Would you like a cup of coffee, without milk?", converts it into voice data, and sends it to the device.

[1420] The terminal plays a feedback voice to the user, and the user responds with a confirmation. The user's response voice is also sent from the terminal to the server. The server again performs voice recognition and analysis, and registers the final confirmed order details in the database. This ensures that the order is processed accurately.

[1421] As a concrete example, consider the case where a domestic customer says, "One hamburger, one glass of coke." When the user speaks, the device records this voice and sends it to the server. The server converts the voice data into text and analyzes the order using a generative AI model. Based on the analysis results, it generates a feedback message saying, "One hamburger, one glass of coke, would you like?" and sends it as voice data to the device. If the user responds "Yes," the server analyzes again, registers the order in the database, and notifies the user that "the order has been completed."

[1422] For example, by inputting an example of a prompt sentence such as "Please explain the detailed steps of the process in which the user says 'one hamburger, one glass of coke'", the detailed process described above is generated.

[1423] The system provides an intuitive and easy-to-use interface, particularly for elderly people, visually impaired people, foreigners, and users unfamiliar with the operation, improving store operational efficiency and overall user satisfaction.

[1424] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1425] Step 1:

[1426] The user begins voice input. The user speaks into the device's microphone, saying, "A cup of hot coffee, no sugar." This becomes the base data for subsequent processing.

[1427] Step 2:

[1428] The device detects the user's speech and activates recording mode. The device's microphone captures and records the audio data. The input is audio data, and the output is a recorded audio file.

[1429] Step 3:

[1430] The device sends the recorded audio data to the server via Wi-Fi or mobile data network. The input is the recorded audio data, and the output is the audio data sent as an HTTP request to the server.

[1431] Step 4:

[1432] The server converts the received voice data into text. This is done using speech recognition software such as the Google Speech-to-Text API. The input is voice data, and the output is text data generated based on that voice data. Specifically, the generated text might be something like "A cup of hot coffee, no sugar."

[1433] Step 5:

[1434] The server uses a generative AI model to analyze the order. During analysis, it uses text data as input to identify each element of the order (drink, temperature, whether sugar is added, etc.). The input is text data, and the output is a data structure of the analyzed order. For example, the drink might be "coffee," the temperature "warm," and the sugar "no."

[1435] Step 6:

[1436] The server generates a feedback message in a natural conversational format based on the analysis results. It uses a generative AI model to generate a message such as "Would you like a cup of coffee, no sugar?" and converts it into voice data. The input is the analysis results, and the output is the generated voice message.

[1437] Step 7:

[1438] The server sends the generated feedback voice data to the device. The communication method is again Wi-Fi or mobile data network. The input is voice data, and the output is voice data sent as an HTTP response to the device.

[1439] Step 8:

[1440] The terminal plays back the feedback audio data received from the server. The user can hear the audio feedback, "Would you like a cup of coffee, no sugar?" The input is the audio data received from the server, and the output is the audio playback.

[1441] Step 9:

[1442] The user confirms the feedback voice and responds again with voice, for example, saying "Yes, that's fine." This provides the user with final confirmation.

[1443] Step 10:

[1444] The terminal re-records the user's confirmation response and sends it to the server. The input is the re-recorded voice data, and the output is the response voice data to be sent as an HTTP request to the server.

[1445] Step 11:

[1446] The server converts the user's confirmation response into text using a speech recognition engine and analyzes the content. The input is voice data, and the output is the analyzed text data. For example, a final confirmation text such as "Yes, that's it, please" is generated.

[1447] Step 12:

[1448] The server registers the order details that have been finally confirmed in the database. The input is the data of the final confirmed order details, and the output is the order information saved as a record in the database.

[1449] Step 13:

[1450] The server generates a voice message notifying the terminal that the order has been processed successfully, and sends it to the terminal. Specifically, it sends a message such as "Your order has been completed. Thank you." The input is the order completion information, and the output is the generated voice message data.

[1451] Step 14:

[1452] The terminal plays audio data to notify the user that the order has been completed. The user hears this audio notification and confirms that the order has been accepted. The input is audio data from the server, and the output is audio playback.

[1453] (Application example 1)

[1454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1455] Conventional food delivery ordering systems are often complex and difficult to operate for elderly people, visually impaired people, and users who are not familiar with the system. Language barriers and misunderstandings of detailed order details can also be problematic. This can lead to ordering errors and user dissatisfaction. Therefore, the objective of this invention is to provide a system that uses voice recognition technology and generative AI to allow users to easily and accurately order food delivery.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1457] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for providing voice feedback on the generated order details, means for registering the final confirmed order details in a database, means for generating a confirmation message for the user using a smartphone and converting it into voice data, and means for analyzing the user's voice input and identifying the food delivery order details by combining voice recognition technology and generation AI. This enables even users who are particularly elderly, visually impaired, or unfamiliar with operation to easily place orders by voice and accurately confirm their orders.

[1458] "Voice input" is the process by which a user sends voice data to a system through a microphone.

[1459] "Voice data" refers to information that is a digital recording of a user's speech.

[1460] "Text conversion" is the process of analyzing audio data and converting it into corresponding text data.

[1461] "Text data" is voice data expressed as text information.

[1462] "Order Details" refers to specific product or service requests made by a user through voice.

[1463] "Generative AI" is an artificial intelligence technology that performs natural language processing based on large amounts of text data and generates appropriate text data according to demand.

[1464] "Feedback" is the process by which the system responds to the user with analysis results and confirmation items.

[1465] "Database" means a storage device for storing and managing information such as final confirmed order details.

[1466] A "smartphone" is a mobile device with advanced processing power and internet connectivity.

[1467] A "confirmation message" is a message that the system sends to a user after receiving the order details from the user to confirm the order.

[1468] "Speech recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[1469] "Order identification" is the process of analyzing voice and text data to determine the products or services the user is trying to order.

[1470] "Re-analysis" is the process of receiving an acknowledgment from the user and re-analyzing it.

[1471] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for analysis and generation.

[1472] This invention is a system that combines voice recognition technology and generative AI, allowing users to order food delivery through voice input. An embodiment of this system is shown below.

[1473] Server-side processing

[1474] The server processes the audio data in the following steps:

[1475] 1. Voice input reception

[1476] The voice data that the user speaks into the smartphone is received. The smartphone is equipped with a microphone, and the voice data is collected from there. The API for receiving the voice data uses the standard HTTP protocol.

[1477] 2. Converting voice data to text

[1478] The received voice data is converted to text data using the Google Speech-to-Text API, and the order details are then ready to be analyzed based on the converted text.

[1479] 3. Order Analysis

[1480] A generative AI model is used to analyze the text data and identify specific order details, such as if you are ordering "one pizza and one Pepsi."

[1481] 4. Generate a confirmation message

[1482] Based on the analyzed order details, a confirmation message is generated to provide feedback to the user. The generated message is in a natural conversational style, such as "Is one pizza and one Pepsi okay?"

[1483] 5. Conversion to audio data

[1484] The confirmation message is converted into voice data and sent to the terminal. Voice synthesis technology is used to generate the confirmation message as natural-sounding voice data.

[1485] 6. Final confirmation and registration in the database

[1486] The confirmation response from the user is analyzed again, and the final order details are confirmed and registered in the database, again using the Google Speech-to-Text API and generative AI models.

[1487] Terminal side processing

[1488] The smartphone plays back the feedback from the server to the user for final confirmation, as follows:

[1489] 1. Start voice input

[1490] Voice input is initiated by the user tapping the microphone icon or speaking a voice input command.

[1491] 2. Feedback playback

[1492] The confirmation message sent from the server is received as audio data and played back to the user, who then confirms the feedback.

[1493] 3. Receiving an Acknowledgment

[1494] If the user responds with a "yes" or "no" confirmation, the voice data is sent to the server again, providing a final confirmation.

[1495] 4. Notification of order completion

[1496] Once the final confirmed order details are registered in the database, a voice message informing the user that the order has been completed is sent and played to the user.

[1497] User operations

[1498] The user can enter the specific details of their order by voice, and the system will analyze, confirm, and register them appropriately, allowing for a smooth food delivery order. For example, if an elderly person or a visually impaired person wants to order "one pizza and one Pepsi," they can do so simply and accurately.

[1499] Examples and prompts

[1500] Specific examples

[1501] If an elderly user is trying to order a pizza:

[1502] 1. User: "One pizza and one Pepsi, please."

[1503] 2. Server: "One pizza and one Pepsi, please?"

[1504] 3. User: "Yes"

[1505] 4. Server: "Thank you for your order. Your order of one pizza and one Pepsi has been completed."

[1506] Prompt Sentence Examples

[1507] If a user orders one curry and one salad:

[1508] User: "I'd like to order one curry and one salad."

[1509] Examples of how to respond after receiving a prompt:

[1510] Transliteration: "I'd like to order one curry and one salad."

[1511] Generated AI response: "Would you like one plate of curry and one salad?"

[1512] Final check:

[1513] Final text entry: "Yes"

[1514] The above is an embodiment of the present invention.

[1515] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1516] Step 1:

[1517] The user speaks their order into the smartphone. For example, they might say, "One pizza and one Pepsi, please." This voice data is recorded through the smartphone's microphone. Input data: User's voice data. Output data: Recorded voice data file.

[1518] Step 2:

[1519] The smartphone sends the recorded voice data to the server. Input data: Recorded voice data file. Output data: Voice data sent to the server.

[1520] Step 3:

[1521] The server receives the audio data and converts it to text using the Google Speech-to-Text API. Input data: Audio data. Output data: Text data.

[1522] Step 4:

[1523] The server uses the generative AI model to analyze the order details from the text data. Specifically, the speech-recognized text data is sent to the generative AI model as a prompt to identify the order details. Input data: Text data. Output data: Identified order details.

[1524] Step 5:

[1525] The server generates a confirmation message based on the parsed order details. For example, it generates a message like "One pizza and one Pepsi, okay?" Input data: Identified order details. Output data: Text of the confirmation message.

[1526] Step 6:

[1527] The server converts the text of the confirmation message into audio data, generates the audio data, and sends it to the smartphone. Input data: Text of the confirmation message. Output data: Audio data of the confirmation message.

[1528] Step 7:

[1529] The smartphone plays the received audio data to the user. The user listens to the played confirmation message and confirms its contents. Input data: Audio data of the confirmation message. Output data: User's understanding and judgment.

[1530] Step 8:

[1531] The user responds with "yes" or "no." For example, the user utters "yes." This voice data is recorded again. Input data: User's confirmation voice data. Output data: Recorded confirmation voice data file.

[1532] Step 9:

[1533] The smartphone sends the recorded confirmation response voice data to the server. Input data: Recorded confirmation response voice data file. Output data: Confirmation response voice data sent to the server.

[1534] Step 10:

[1535] The server receives the confirmation response audio data and converts it to text again using the Google Speech-to-Text API. Input data: confirmation response audio data. Output data: text data.

[1536] Step 11:

[1537] The server analyzes the text data of the confirmation response and performs a final confirmation. For example, if the response is "yes," the order is confirmed, and if the response is "no," the order details are revised and a confirmation message is generated. Input data: Text data of the confirmation response. Output data: Final confirmation result.

[1538] Step 12:

[1539] The final confirmed order details are registered in the database. Input data: Final confirmation results. Output data: Order details registered in the database.

[1540] Step 13:

[1541] The server generates a message indicating that the order has been completed, converts it into voice data, and sends it to the smartphone. For example, a voice message such as "Thank you for your order. Your order for one pizza and one Pepsi has been completed." Input data: Text of the order completion message. Output data: Voice data of the order completion message.

[1542] Step 14:

[1543] The smartphone plays the received voice data of the order completion message to the user. Input data: Voice data of the order completion message. Output data: User's understanding and confirmation of order acceptance.

[1544] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1545] The present invention is a system that combines speech recognition technology, generative AI, and an emotion engine, and when a user places an order through voice input, it recognizes the user's emotion and adjusts feedback based on that emotion, providing a more natural and comfortable experience. Below, the program processing of this system is explained in natural language, and an embodiment is described with specific examples.

[1546] Server-side processing

[1547] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Through this process, what the user wants to order becomes clear in text form.

[1548] The server then uses generative AI to analyze the text data and identify the order details, such as "one beer and one hamburger." The emotion engine then analyzes the user's voice data and recognizes the emotion expressed during the feedback.

[1549] The server generates a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion recognized by the emotion engine. For example, if the user is impatient, the server generates feedback in a gentle tone such as "Please wait a moment, we will respond immediately."

[1550] The server converts this feedback message into voice data and sends it to the device. It also receives additional voice data from the user (such as "yes" or "no"), converts it into text, and analyzes it. The server also considers the user's emotions when making a final confirmation.

[1551] Finally, the final confirmed order details are registered in the database. The server also generates a message to notify the completion of the order and sends it to the terminal.

[1552] Terminal side processing

[1553] The device detects that the user has started voice input and activates recording mode. The recorded voice data is sent to the server. When the device receives the feedback voice data from the server, it plays it back to the user. The user checks the feedback and responds by voice if they want to input the answer again. The device then sends the response voice back to the server.

[1554] User operations

[1555] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks the details and, if there are no problems, responds "Yes." If corrections are needed, they can say "No, not that," followed by the new order details. Once the user has completed the final confirmation and received a notification from the device that the order has been completed, they can feel reassured that their order has been successfully accepted.

[1556] Specific examples

[1557] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[1558] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[1559] The processing flow will be explained below.

[1560] Step 1:

[1561] The user speaks into the microphone of the terminal to input the details of their order, for example, saying, "I'd like a beer and a hamburger, please."

[1562] Step 2:

[1563] The device detects voice input from the user and activates recording mode. The recorded voice data is sent to the server in an appropriate format (e.g., PCM format).

[1564] Step 3:

[1565] The server receives the voice data sent from the device, which is then passed to a speech recognition engine (e.g., Google Speech-to-Text API) and converted into text.

[1566] Step 4:

[1567] The server analyzes the text data returned by the speech recognition engine and identifies the order details, such as "one beer and one hamburger."

[1568] Step 5:

[1569] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, recognizing emotions such as impatience or anxiety from the tone and speed of the user's voice.

[1570] Step 6:

[1571] The server generates a feedback message based on the analyzed order details and the user's emotional data. Generative AI is used to create a feedback message that takes into account the appropriate tone and wording. For example, if the user is impatient, the server might say, "Please wait a moment, we will respond shortly."

[1572] Step 7:

[1573] The server converts the generated feedback message into voice data and transmits it to the terminal.

[1574] Step 8:

[1575] The terminal receives the feedback voice data transmitted from the server and plays it back to the user, who then listens to the feedback voice.

[1576] Step 9:

[1577] The user checks the feedback and responds with a confirmation such as "Yes" or "No," and makes corrections as necessary.

[1578] Step 10:

[1579] The device re-records the user's confirmation response and sends it to the server, thereby clearly conveying the user's intention.

[1580] Step 11:

[1581] The server converts the received voice data into text and analyzes the confirmation response. For example, if the response is "yes," the order details are confirmed.

[1582] Step 12:

[1583] The server registers the final confirmed order details in the database, which officially accepts the order.

[1584] Step 13:

[1585] The server generates an order completion message to notify the user that the order has been completed. The generated message is converted into voice data and sent to the terminal. At this time, feedback is provided taking into account emotion data.

[1586] Step 14:

[1587] The terminal receives the voice data notifying the user that the order has been completed and plays it back to the user, who then confirms that the order has been processed correctly.

[1588] Specific examples

[1589] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording appropriate to the user's emotion, giving the user a sense of security.

[1590] The system provides a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with the system, thereby improving store operational efficiency and increasing overall user satisfaction.

[1591] Example 2

[1592] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1593] The problem that the present invention aims to solve is the inability to provide a more natural and comfortable experience by recognizing a user's emotions and adjusting feedback based on them when placing an order via voice input. Conventional systems provide uniform feedback without considering the user's emotions, which can impair the user experience. Another problem is that they are unable to provide appropriate services to a variety of users, including the elderly, the visually impaired, and foreigners.

[1594] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1595] In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to identify the order details, means for analyzing the user's emotions from the voice data, means for generating a feedback message according to the user's emotions using a generative AI model, means for converting the generated feedback message into voice data, means for providing the generated feedback message by voice, and means for registering the final confirmed order details in a database. This allows appropriate feedback to be given based on the user's emotions, and makes it possible to provide a simple and easy-to-understand interface, especially for elderly people, visually impaired people, foreigners, and users who are not familiar with operation.

[1596] The "means for receiving voice input" refers to a device or method for receiving voice data spoken by a user and incorporating it into the system.

[1597] A "means for converting voice data to text" is a device or method for generating text data from voice data using voice recognition technology.

[1598] The "means for analyzing text data to identify order details" refers to a device or method for processing text data and extracting specific order information requested by the user.

[1599] The "means for analyzing user emotions from voice data" is a device or method for analyzing voice data to extract the user's emotional state.

[1600] "Means for generating feedback messages according to user emotions using a generative AI model" refers to a device or method for using generative AI to create appropriate feedback messages based on analyzed user emotions.

[1601] The "means for converting the generated feedback message into voice data" is a device or method for converting the textual feedback message into voice data.

[1602] The "means for providing a generated feedback message by voice" refers to a device or method for transmitting a feedback message converted into voice data to a user.

[1603] The "means for registering the final confirmed order details in the database" refers to a device or method for storing the order details that have been finally confirmed by the system in the database.

[1604] The present invention is a system that combines voice recognition technology, generative AI, and an emotion engine, and provides a more natural and comfortable experience by recognizing the user's emotions when placing an order through voice input and adjusting feedback accordingly. This system is configured as follows.

[1605] Server-side processing

[1606] The server receives the voice data sent from the terminal. The received voice data is first converted into text using a speech recognition engine. Specifically, Google Cloud Speech-to-Text is used to convert the voice data into text data. This makes it clear what the user wants to order in text format.

[1607] Next, the server uses a natural language processing model (e.g., a generative AI model) to analyze the text data and identify the order. For example, it extracts specific order information such as "one beer and one hamburger." In parallel, it uses emotion analysis technology (e.g., a voice emotion analysis engine) to analyze the user's emotions from the voice data.

[1608] The server uses a generative AI model to generate a feedback message based on the analyzed order details and the user's emotional data. The generated message is composed of a tone and wording that corresponds to the user's emotion as recognized by the emotion engine. For example, if the user is impatient, a gentle tone of feedback such as "Please wait a moment, we will respond immediately" is generated.

[1609] To convert the generated feedback message into audio data, a speech synthesis technology (e.g., a speech synthesis engine) is used. Specifically, Amazon Polly is used to convert the text message into audio data. This audio data is then sent to the device.

[1610] Furthermore, the system receives voice data for additional input from the user (such as "yes" or "no"), performs voice recognition and emotion analysis again, and performs final confirmation. The final confirmed order details are registered in a database system (e.g., a database management system). This ensures that the order record is saved.

[1611] Terminal side processing

[1612] The device detects that the user has started voice input and activates recording mode. In this mode, the device's microphone records the user's voice and transmits the voice data to the server. The device also receives feedback voice data transmitted from the server, plays it back, and provides it to the user.

[1613] The audio data recorded by the device is converted into an appropriate format (e.g., WAV or MP3) and then sent to the server. This process is performed through an HTTP request. Upon receiving the feedback audio data from the server, the device plays it back to the user.

[1614] User operations

[1615] The user speaks into the device's microphone to enter the specific details of their order. For example, they might say, "One cup of coffee, no milk." When the feedback is returned, the user checks it and, if satisfied, responds with "Yes." If a correction is needed, they can say, "No, not that," followed by the new order details. Once the final confirmation is complete, the user receives a notification from the device that the order has been successfully accepted, providing peace of mind.

[1616] Specific examples

[1617] As a concrete example, consider the case where a domestic customer places an order. When the user says, "One cup of coffee, no milk," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order details, and generates feedback. If the emotion analysis engine determines that the user is impatient, it uses a tone such as "Please wait a moment, your coffee will be ready shortly." If the user responds "Yes," the server confirms the order details and registers them in the database. The notification of order completion is also generated using wording that corresponds to the user's emotion, giving the user a sense of security.

[1618] Prompt Sentence Examples

[1619] Use the following prompt for the generative AI model:

[1620] prompt:

[1621] User utterance: "One cup of coffee, no milk"

[1622] User Emotion: "Impatience"

[1623] Produced message: "Please wait a moment, your coffee will be ready shortly."

[1624] By using such prompt sentences, it is possible to generate appropriate feedback messages according to the user's emotions.

[1625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1626] Step 1:

[1627] The server receives the voice data transmitted from the terminal.

[1628] Input: User spoken voice data (WAV or MP3 format)

[1629] Processing: The audio data is sent to the server via an HTTP request, and the server stores it in its storage.

[1630] Output: Audio data file saved in storage

[1631] Step 2:

[1632] The server converts the received voice data into text using a voice recognition engine.

[1633] Input: Audio data file saved in storage

[1634] Processing: The audio data is input into a speech recognition engine (for example, Google Cloud Speech-to-Text) and converted into text data.

[1635] Output: Converted text data

[1636] Step 3:

[1637] The server uses generative AI to analyze the text data and identify the order details.

[1638] Input: Converted text data

[1639] Processing: A generative AI model (e.g., GPT-4) is fed the text data and used to analyze and identify the order, e.g., extracting specific order information such as "one cup of coffee, no milk."

[1640] Output: Identified order details

[1641] Step 4:

[1642] The server uses an emotion engine to analyze the user's emotion from the voice data.

[1643] Input: Audio data file saved in storage

[1644] Processing: Input the voice data into an emotion engine (e.g., Emotion API) and analyze the user's emotions (e.g., impatience, relief, joy).

[1645] Output: Parsed emotion data

[1646] Step 5:

[1647] The server uses a generative AI model to generate a feedback message based on the analyzed order content and user sentiment data.

[1648] Input: Identified order details, analyzed sentiment data

[1649] Processing: The following prompt sentence is input into the generative AI model, and a feedback message corresponding to the user's emotion is output.

[1650] prompt:

[1651] User utterance: "One cup of coffee, no milk"

[1652] User Emotion: "Impatience"

[1653] Produced message: "Please wait a moment, your coffee will be ready shortly."

[1654] Output: The generated feedback message

[1655] Step 6:

[1656] The server converts the generated feedback message into audio data.

[1657] Input: The generated feedback message

[1658] Processing: A text message is input into speech synthesis technology (e.g., Amazon Polly) and converted into audio data.

[1659] Output: Feedback message converted to audio data

[1660] Step 7:

[1661] The server transmits the generated feedback message to the terminal as voice data.

[1662] Input: Feedback message converted to audio data

[1663] Processing: Sends feedback audio data to the device via an HTTP response.

[1664] Output: Feedback audio data sent to the device

[1665] Step 8:

[1666] The terminal plays back the feedback voice data received from the server to the user.

[1667] Input: Feedback audio data sent from the server

[1668] Processing: The audio data is played through the device speaker to provide feedback to the user.

[1669] Output: Feedback audio played to the user

[1670] Step 9:

[1671] The user reviews the feedback and speaks additional responses.

[1672] Input: Feedback audio played from the device

[1673] Processing: The user speaks the acknowledgment "yes" or "no," which is recorded by the terminal.

[1674] Output: Additional audio data

[1675] Step 10:

[1676] The terminal transmits the additional voice data from the user to the server again.

[1677] Input: Additional voice data from the user

[1678] Processing: The recorded audio data is sent to the server via an HTTP request.

[1679] Output: Additional audio data sent to the server

[1680] Step 11:

[1681] The server receives the additional voice data, performs voice recognition and emotion analysis again, and performs a final confirmation.

[1682] Input: Additional audio data

[1683] Processing: The voice data is converted to text and analyzed again to determine the final order and user sentiment.

[1684] Output: Final confirmed order details and sentiment data

[1685] Step 12:

[1686] The server registers the final confirmed order details in a database and generates a notification of order completion.

[1687] Input: Final confirmed order details and sentiment data

[1688] Processing: The order details are saved in the database system, an order completion message is generated, and then the message is converted into voice data and sent to the terminal.

[1689] Output: Order details registered in the database and voice data of the completed order sent to the terminal

[1690] Step 13:

[1691] The terminal reproduces order completion voice data to the user.

[1692] Input: Order completion voice data sent from the server

[1693] Processing: Plays audio data to notify the user that the order has been completed.

[1694] Output: Order complete audio played to the user

[1695] This series of processing steps provides appropriate feedback according to the user's emotions, resulting in a pleasant ordering experience.

[1696] (Application example 2)

[1697] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1698] Conventional voice input systems provide mechanical responses without considering the user's emotions, and therefore do not provide a user-friendly or reassuring interface. Furthermore, users who are unfamiliar with the system, such as the elderly, the visually impaired, and foreigners, often find it difficult to use and feel frustrated. There is a need for a system that provides natural feedback based on the user's emotions to resolve these issues and facilitate the ordering process.

[1699] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input, means for converting voice data into text, means for analyzing the text data to specify the order details, means for providing voice feedback of the generated order details, means for analyzing user emotion data, means for adjusting the feedback message based on the emotion data, and means for registering the final confirmed order details in a database. This enables natural feedback according to the user's emotions, allowing even elderly people, visually impaired people, and users unfamiliar with operation to place orders without stress.

[1700] "Voice input" is the process of receiving user spoken sounds through a device's microphone.

[1701] "Voice data" is data that represents in digital form the voice spoken by the user.

[1702] The "means for converting to text" refers to a technology that performs a process of analyzing audio data and converting it into corresponding text data.

[1703] "Text data" is voice data converted into character information, and is a sentence describing the contents of an order or conversation.

[1704] The "means for identifying the order content" is a technology that analyzes text data and identifies the specific product or service content that the user wants to order.

[1705] "Feedback means" refers to a technology that generates and transmits a voice message of confirmation or guidance to the user based on the analyzed order details.

[1706] "User emotion data" is data that indicates the user's emotion or emotional state, analyzed from the user's speech and voice characteristics.

[1707] "Means for analyzing emotional data" refers to technology that analyzes a user's voice and text data to identify the user's emotional state.

[1708] The "means for adjusting a feedback message based on emotional data" is a technique for adjusting the content and tone of a feedback message based on analyzed emotional data of a user.

[1709] The "finally confirmed order details" are the final order details that have been confirmed after being confirmed by the user.

[1710] The "means for registering in a database" refers to a technique for storing the confirmed order details in a database in digital form.

[1711] This invention is a food and beverage delivery ordering system that uses voice recognition technology, generative AI, and an emotion engine to assist users in the ordering process. Specifically, it uses a smartphone application to receive orders using the following procedure and provides feedback based on the user's emotions.

[1712] Hardware used

[1713] Smartphone (Android / iOS)

[1714] microphone

[1715] server

[1716] Software used

[1717] Speech recognition engine (e.g., Google Cloud Speech-to-Text, Amazon Transcribe)

[1718] Emotion engines (e.g., AWS Rekognition, IBM Watson Tone Analyzer)

[1719] Generative AI (e.g., OpenAI's GPT-3)

[1720] Database (e.g. MySQL, Firebase)

[1721] Server-side processing

[1722] The server receives voice data sent from the smartphone. The voice data is first converted into text using a speech recognition engine. The text data is then analyzed to identify the order details. The analyzed order details are then processed together with the user's emotion data using an emotion engine.

[1723] The emotion engine analyzes the user's emotions from the tone and tempo of their voice, and the generation AI generates a feedback message based on this. The generated feedback message is composed of a tone and wording that corresponds to the user's emotions and is converted back into audio data. This audio data is sent to the smartphone and played back to the user.

[1724] After receiving the final confirmation response from the user, the order is analyzed again and the order details are finally confirmed. The final confirmation details are registered in the database and the order is completed.

[1725] Terminal side processing

[1726] When a user speaks their order details into their smartphone, the device detects this voice input and activates recording mode. The recorded voice data is sent to the server, and when feedback voice data is received, it is played back to the user. If the user checks the feedback and wants to re-enter the answer, they respond by voice, and this response voice is also sent back to the server.

[1727] User operations

[1728] The user speaks the specific details of their order into the device's microphone. For example, they might say, "One pizza, lots of cheese, tomato sauce." This voice data is sent to the server, where it undergoes a series of processes including analysis, feedback generation, and sentiment analysis. When the feedback is returned, the user confirms it and finally responds with "yes" or "no." As a result of this exchange, the order is confirmed and a notification of completion is sent.

[1729] Specific examples

[1730] For example, if a user says, "One pizza, lots of cheese, and tomato sauce," the device records this voice and sends it to the server. The server converts the voice data into text, analyzes the order, and generates feedback. If the emotion engine analyzes the user's voice as relaxed, the generation AI will generate, "Thank you for your order. One pizza, lots of cheese, and tomato sauce, please. Is that okay?"

[1731] Prompt Sentence Examples

[1732] "When a user orders one pizza, lots of cheese, and tomato sauce, generate a feedback message in a relaxed tone."

[1733] This design allows users to complete their orders in a natural conversational manner, and is intended to be stress-free for users, especially the elderly, visually impaired, and those unfamiliar with the system.

[1734] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1735] Step 1:

[1736] Input: User voice input

[1737] How it works: The user speaks specific details of their order into the smartphone's microphone.

[1738] Output: Recorded audio data

[1739] Step 2:

[1740] Input: Recorded audio data

[1741] Operation: The device detects the user's voice input and activates recording mode. The recorded voice data is sent to the server.

[1742] Output: Audio data sent to the server

[1743] Step 3:

[1744] Input: Audio data sent to the server

[1745] How it works: The server uses a speech recognition engine to convert the voice data into text data.

[1746] Output: Text data

[1747] Step 4:

[1748] Input: Text data

[1749] Operation: The server analyzes the text data and identifies the order contents.

[1750] Output: Identifying the order

[1751] Step 5:

[1752] Input: User's voice data and identified order details

[1753] How it works: The emotion engine analyzes the user's voice data and generates the user's emotion data.

[1754] Output: User emotion data

[1755] Step 6:

[1756] Input: Identified order details and user sentiment data

[1757] How it works: The server uses a generative AI to generate feedback messages based on the user's emotions.

[1758] Output: Feedback message

[1759] Step 7:

[1760] Input: Feedback message

[1761] Operation: The server converts the generated feedback message into audio data.

[1762] Output: Feedback message as audio data

[1763] Step 8:

[1764] Input: Feedback message as audio data

[1765] Operation: The server sends a feedback message to the terminal, which then plays it back to the user.

[1766] Output: The feedback message played to the user

[1767] Step 9:

[1768] Input: The user's response to the played feedback message

[1769] How it works: The user checks the feedback and responds again verbally. The device sends this response to the server.

[1770] Output: Response audio data sent to the server

[1771] Step 10:

[1772] Input: Response voice data sent to the server

[1773] Operation: The server analyzes the response voice data again and performs a final confirmation of the order details.

[1774] Output: Final confirmed order details

[1775] Step 11:

[1776] Input: Final confirmed order details

[1777] Operation: The server registers the final confirmed order details in the database.

[1778] Output: Order details registered in the database

[1779] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1781] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1782] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1783] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1784] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1785] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1786] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1787] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1788] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1789] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1790] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1791] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1792] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1793] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1794] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1795] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1796] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1797] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1798] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1799] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1800] The following is further disclosed regarding the above embodiment.

[1801] (Claim 1)

[1802] means for receiving audio input;

[1803] a means for converting the audio data into text;

[1804] A means for analyzing the text data to identify the order contents;

[1805] A means for providing voice feedback on the generated order details;

[1806] a means for registering the final confirmed order details in a database;

[1807] A system including:

[1808] (Claim 2)

[1809] means for playing audio feedback to the user;

[1810] means for receiving and re-analyzing an acknowledgment from the user;

[1811] 10. The system of claim 1, further comprising:

[1812] (Claim 3)

[1813] 10. The system of claim 1, further comprising means for identifying a plurality of languages ​​and providing feedback in a specified language.

[1814] (Claim 4)

[1815] 10. The system of claim 1, further comprising means for analyzing the voice data and the text data to collect and store preference data.

[1816] (Claim 5)

[1817] A terminal includes a means for detecting the start of voice input and recording voice data, and a means for transmitting the recorded voice data to a server;

[1818] 10. The system of claim 1, further comprising:

[1819] "Example 1"

[1820] (Claim 1)

[1821] means for receiving audio input;

[1822] a means for converting the audio data into text;

[1823] A means for analyzing the text data to identify the order contents;

[1824] A means for analyzing order content using a generative AI model; and

[1825] a means for generating feedback messages in a natural conversational style based on the analysis results;

[1826] A means for providing voice feedback on the generated order details;

[1827] a means for registering the final confirmed order details in a database;

[1828] A system including:

[1829] (Claim 2)

[1830] means for playing audio feedback to the user;

[1831] means for receiving and re-analyzing an acknowledgment from the user;

[1832] means for generating and sending to the user a notification of order completion after final confirmation;

[1833] 10. The system of claim 1.

[1834] (Claim 3)

[1835] including means for identifying multiple languages ​​and providing feedback in a specified language;

[1836] 10. The system of claim 1.

[1837] "Application Example 1"

[1838] (Claim 1)

[1839] means for receiving audio input;

[1840] a means for converting the audio data into text;

[1841] A means for analyzing the text data to identify the order contents;

[1842] A means for providing voice feedback on the generated order details;

[1843] a means for registering the final confirmed order details in a database;

[1844] means for generating a confirmation message for a user using a smartphone and converting the message into voice data;

[1845] A means to combine voice recognition technology and generative AI to analyze user voice input and identify food delivery order details;

[1846] A system including:

[1847] (Claim 2)

[1848] means for playing audio feedback to the user;

[1849] means for receiving and re-analyzing an acknowledgment from the user;

[1850] A means for the generative AI model to generate a response regarding the order content using the prompt sentence;

[1851] 10. The system of claim 1, further comprising:

[1852] (Claim 3)

[1853] 10. The system of claim 1, further comprising means for identifying a plurality of languages ​​and providing feedback in a specified language.

[1854] "Example 2: Combining Emotion Engines"

[1855] (Claim 1)

[1856] means for receiving audio input;

[1857] a means for converting the audio data into text;

[1858] A means for analyzing the text data to identify the order contents;

[1859] A means for analyzing user emotions from voice data;

[1860] A means for generating a feedback message according to the user's emotions using a generative AI model;

[1861] means for converting the generated feedback message into audio data;

[1862] a means for providing audio feedback of the generated feedback message;

[1863] a means for registering the final confirmed order details in a database;

[1864] A system including:

[1865] (Claim 2)

[1866] means for playing audio feedback to the user;

[1867] means for receiving and re-analyzing an acknowledgment from the user;

[1868] 10. The system of claim 1, further comprising:

[1869] (Claim 3)

[1870] 10. The system of claim 1, further comprising means for identifying a plurality of languages ​​and providing feedback in a specified language.

[1871] "Application example 2 when combining emotion engines"

[1872] (Claim 1)

[1873] means for receiving audio input;

[1874] a means for converting the audio data into text;

[1875] A means for analyzing the text data to identify the order contents;

[1876] A means for providing voice feedback on the generated order details;

[1877] means for analyzing user emotion data;

[1878] means for adjusting the feedback message based on the emotion data;

[1879] a means for registering the final confirmed order details in a database;

[1880] A system including:

[1881] (Claim 2)

[1882] means for playing audio feedback to the user;

[1883] means for receiving and re-analyzing an acknowledgment from the user;

[1884] 10. The system of claim 1.

[1885] (Claim 3)

[1886] further including means for identifying multiple languages ​​and providing feedback in a specified language;

[1887] 10. The system of claim 1. [Explanation of symbols]

[1888] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving audio input; a means for converting the audio data into text; A means for analyzing the text data to identify the order contents; A means for providing voice feedback on the generated order details; a means for registering the final confirmed order details in a database; A system including:

2. means for playing audio feedback to the user; means for receiving and re-analyzing an acknowledgment from the user; The system of claim 1 further comprising:

3. The system of claim 1 further comprising means for identifying a plurality of languages ​​and providing feedback in a specified language.

4. The system of claim 1 , further comprising means for analyzing the voice data and the text data to collect and store preference data.

5. A terminal includes a means for detecting the start of voice input and recording voice data, and a means for transmitting the recorded voice data to a server; The system of claim 1 further comprising:

Images