System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting speech by a user; means for converting the speech into text data; means for generating an answer based on the text data using a generative model; means for converting the answer into speech data; means for referring to past conversation data using user authentication information; means for generating an answer based on the past conversation data and a current context; means for returning the speech data to the user; means for detecting and correcting pronunciation and grammar errors; and means for generating a summary report of the conversation after the end of the conversation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While online English conversation is becoming mainstream these days, face-to-face English conversation can be stressful and difficult to ask for clarification. This makes it particularly difficult for beginners to practice English conversation casually. It is also difficult to receive appropriate feedback when making pronunciation or grammar mistakes. This makes it difficult for beginners to learn English effectively, resulting in a time-consuming learning process. This invention aims to solve these problems and lower the barriers to learning English conversation. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including the following means: a means for a user to input speech, a means for converting the speech into text data, a means for generating a response based on the text data using a generative model, a means for converting the response into speech data, a means for referencing past conversation data using user authentication information, a means for generating a response based on the past conversation data and the current context, a means for returning the speech data to the user, a means for detecting and correcting pronunciation and grammar errors, and a means for generating a summary report of the conversation after the conversation ends. This system allows users to practice English conversation without stress, appropriately correct pronunciation and grammar errors, and receive effective feedback. It also enables users to understand context based on past conversation data and promote consistent learning.

[0006] "User Credentials" means unique identification information used by a user to access a system, including authentication data such as a username and password.

[0007] "Voice input means" refers to a device such as a microphone that allows a user to input voice data into the system, and its processing function.

[0008] A "voice recognition engine" refers to software or algorithms used to convert voice data into text data.

[0009] A "generative model" refers to an artificial intelligence or machine learning model that understands context and generates appropriate responses based on received text data and past conversation data.

[0010] "Text data" refers to character string data that represents what the user has said, converted by a voice recognition engine.

[0011] A "speech synthesis engine" refers to software or algorithms used to convert text data into speech data.

[0012] "Past conversation data" refers to data that records the content of conversations that a user has had in the past, and is maintained based on a user profile.

[0013] "Context" refers to situational or background information based on a user's past conversation data and current conversation content.

[0014] "Speech Data" refers to data in speech form that is generated by a speech synthesis engine and returned to the user.

[0015] "Means for providing feedback" refers to a function for detecting pronunciation and grammar errors and providing appropriate corrections and advice to the user in real time.

[0016] "Summary Report" refers to a report generated after a conversation that contains summary information that allows a user to identify mistakes and areas for improvement in their conversation. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This system allows users to input speech and uses a generative AI to generate and provide appropriate responses. The system includes a user terminal, a server, a generative AI, and a series of databases and algorithms that support the AI.

[0039] User Authentication and Login

[0040] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0041] Voice input and speech generation

[0042] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[0043] Audio Output

[0044] The generated response text is sent to a speech synthesis engine by the server and converted into voice data. The voice data is then sent back to the user's device, and the user receives the response by voice, providing a natural conversational experience.

[0045] Feedback and Error Correction

[0046] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[0047] Generate a summary report

[0048] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0049] Specific examples

[0050] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[0051] In this way, users are provided with an environment in which they can practice English conversation without feeling nervous and while receiving appropriate feedback.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] User terminal: The user enters the username and password and clicks the login button.

[0055] Step 2:

[0056] Terminal: Sends the entered authentication information to the server.

[0057] Step 3:

[0058] Server: Compares the received authentication information with a database and authenticates the user.

[0059] Step 4:

[0060] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[0061] Step 5:

[0062] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[0063] Step 6:

[0064] User device: The user uses a microphone to input voice spoken in English.

[0065] Step 7:

[0066] Terminal: Sends audio data captured by the microphone to the server.

[0067] Step 8:

[0068] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[0069] Step 9:

[0070] Server: Sends the converted text data to the generative AI model and generates a response.

[0071] Step 10:

[0072] Generative AI: Uses text data and past conversation data to understand the user's context and generate appropriate responses.

[0073] Step 11:

[0074] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[0075] Step 12:

[0076] Server: Sends voice data to the user terminal.

[0077] Step 13:

[0078] User terminal: Receives the voice data and plays back the response from the speaker.

[0079] Step 14:

[0080] Generative AI: Detects user pronunciation and grammar errors in real time.

[0081] Step 15:

[0082] Generative AI: Generates appropriate corrections and feedback when errors are detected.

[0083] Step 16:

[0084] Server: Sends corrective feedback to the user terminal.

[0085] Step 17:

[0086] User terminal: Corrective feedback is played back and provided to the user.

[0087] Step 18:

[0088] Server: After the conversation ends, generate a summary report based on the entire history.

[0089] Step 19:

[0090] Server: Sends the generated summary report to the user terminal.

[0091] Step 20:

[0092] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Conventional voice dialogue systems have difficulty accurately converting user input speech into text data and generating appropriate responses. They also struggle to generate responses that fully understand past conversation data and context, making it impossible to provide users with a natural, personalized conversation experience. Furthermore, they lack the ability to correct pronunciation and grammar errors in real time and provide feedback to users. This reduces the user's learning effectiveness and the quality of their conversation experience.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for encrypting authentication information entered by the user and transmitting it to the server, means for loading the user's profile information if authentication is successful, and means for asynchronously processing voice data received from the user via the terminal. This allows for the generation of appropriate responses based on the user's voice input, enabling personalized responses that take into account past conversation data and context. Furthermore, pronunciation and grammar errors can be corrected in real time and feedback can be provided to improve the user's learning effect. Furthermore, the use of a cloud-based speech recognition and speech synthesis engine enables highly accurate voice processing, providing the user with a natural, high-quality conversation experience.

[0098] "User authentication information" is an identifier used by a user when accessing a system, and is information that mainly consists of a user name and password.

[0099] "Voice input" refers to the act of a user speaking to the system using a microphone, or the voice data acquired thereby.

[0100] A "voice recognition engine" is software or algorithm that converts captured voice data into text data.

[0101] A "generative model" is an algorithm or system that uses natural language processing to generate appropriate responses based on input text data.

[0102] "Voice data" is data input by voice that is expressed in digital format and is used for voice recognition and voice synthesis.

[0103] "Past conversation data" is historical information about conversations the user has had in the past, and is used as basic data for analysis and response generation.

[0104] "Context" is information used to understand the content, flow, and context of a conversation, and is an important element for a generative model to provide an appropriate response.

[0105] "Real-time feedback" refers to the process of providing instant feedback to users immediately after they speak, immediately pointing out and correcting pronunciation and grammar errors.

[0106] "Profile information" refers to information such as a user's individual settings, past usage history, conversation history, etc., and is used to provide personalized services.

[0107] A "cloud-based engine" is a processing engine for speech recognition, speech synthesis, etc. that is provided via the Internet on a cloud service.

[0108] This invention is a system that allows a user to input speech and generates and provides an appropriate response using a generative AI model. The system includes a user terminal, a server, a generative AI model, and a series of databases and algorithms that support the model.

[0109] User Authentication and Login

[0110] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal encrypts this authentication information and sends it to the server. The server authenticates the user by checking it against a database. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0111] Voice input

[0112] The user inputs voice using a microphone installed on the user device. The device temporarily stores the acquired voice data and then transmits it to the server.

[0113] Voice Recognition

[0114] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into text data, which is temporarily stored on the server.

[0115] Conversation Generation

[0116] The server inputs this text data into a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response based on the context and past conversation data. The generated response text is temporarily stored on the server.

[0117] Audio Output

[0118] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. The converted voice data is sent to the user's device, and the user receives the response by voice.

[0119] Feedback and Error Correction

[0120] During the conversation, the generative AI model detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error in "goed" and generates text correcting it to "went." The server generates corrective feedback, converts it into audio data, and sends it to the user's device.

[0121] Generate a summary report

[0122] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0123] Specific examples

[0124] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation. In this way, users are provided with an environment where they can practice English conversation without feeling stressed and while receiving appropriate feedback.

[0125] Prompt Sentence Examples

[0126] "Hello, I'd like to practice my English."

[0127] This allows users to continue the conversation in real time and receive real-time feedback on pronunciation and grammar errors, which is expected to improve users' English conversation skills.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1: User authentication and login

[0130] The user uses a dedicated app or web browser on the user device (PC or smartphone) to enter their username and password and click the login button.

[0131] Input: Username, Password

[0132] Specific behavior:

[0133] The terminal encrypts the entered authentication information and sends it to the server using the HTTPS protocol.

[0134] The server authenticates the user by checking the received authentication information against a database (e.g., MySQL).

[0135] Output: Authentication result (success or failure)

[0136] If authentication is successful, the server creates a session and loads the user's profile information into memory.

[0137] Step 2: Voice Input

[0138] The user speaks into a microphone installed in the user terminal.

[0139] Input: Audio data

[0140] Specific behavior:

[0141] The terminal temporarily stores the audio data acquired from the microphone.

[0142] The device asynchronously transmits the stored audio data to the server.

[0143] Output: Audio data (binary format)

[0144] Step 3: Voice Recognition

[0145] The server passes the received voice data to a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data.

[0146] Input: Audio data (binary format)

[0147] Specific behavior:

[0148] The server sends the audio data as an API request.

[0149] The voice recognition engine analyzes the voice data and converts it into text data.

[0150] Output: Text data (natural language text)

[0151] The converted text data is temporarily stored on the server.

[0152] Step 4: Conversation generation

[0153] The server inputs the text data generated by speech recognition into a generative AI model (e.g., OpenAI GPT-3).

[0154] Input: Text data (natural language text)

[0155] Specific behavior:

[0156] The server sends the text data to the generative AI model.

[0157] The generative AI model takes into account past conversation data and context to generate appropriate responses.

[0158] Output: Response text data (natural language text)

[0159] The generated response text is temporarily stored on the server.

[0160] Step 5: Audio Output

[0161] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data.

[0162] Input: Response text data (natural language text)

[0163] Specific behavior:

[0164] The server passes the response text to the speech synthesis engine.

[0165] The speech synthesis engine analyzes the text data and converts it into speech data.

[0166] Output: Audio data (audio file)

[0167] The generated voice data is transmitted to the user terminal and returned to the user.

[0168] Step 6: Feedback and Error Correction

[0169] During the conversation, the generative AI model detects the user's pronunciation and grammar errors and corrects them in real time.

[0170] Input: User utterance text data

[0171] Specific behavior:

[0172] The server analyzes the speech using a generative AI model and detects pronunciation and grammatical errors.

[0173] If an error is detected, the AI ​​generates corrected text data.

[0174] Output: Corrective feedback text (natural language text)

[0175] The server passes the corrective feedback to a speech synthesis engine, converts it into speech data, and provides the feedback to the user.

[0176] Step 7: Generate a summary report

[0177] After the conversation ends, the server generates a summary report based on the entire conversation history.

[0178] Input: Text data of conversation history

[0179] Specific behavior:

[0180] The server analyzes the conversation history and extracts pronunciation and grammatical errors and areas for improvement.

[0181] Generate summary reports and collate details.

[0182] Output: Summary report (text or PDF)

[0183] The summary report is sent to the user terminal so that the user can view it.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] In conventional food delivery apps, voice recognition is often inaccurate when users input their order details, and menu recommendations are text-based, making them less convenient. Furthermore, it is difficult to provide a personalized user experience. Therefore, there was a need for a system that allows users to easily order by voice and can use a generative AI model to provide appropriate menu recommendations via voice.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammar errors, means for generating a conversation summary report after the conversation ends, means for causing a generative AI model to generate prompts based on the user's order and recommending appropriate menu items, and means for providing the generated recommended menu items to the user as speech data. This allows a user to place an order by voice, and the generative AI to suggest appropriate recommended menu items by voice.

[0189] "Means for inputting voice" refers to a device or interface that takes in what the user has said and recognizes it as voice data.

[0190] "Means for converting into text data" refers to technology that analyzes voice data and converts its contents into text information.

[0191] "Means for generating a response using a generative model" refers to a method for creating an appropriate response based on input text data using a pre-trained artificial intelligence model.

[0192] "Means for converting into voice data" refers to a technology that analyzes text data and generates its contents as voice.

[0193] "Means for referencing past conversation data using user authentication information" refers to a system that searches and refers to the history of past conversations based on user authentication information.

[0194] "Means for generating a response based on the current context" refers to a technology that understands the current conversation content and situation and creates a response that matches that context.

[0195] The "means for returning voice data to the user" is a mechanism for transmitting the generated voice data to the user in an appropriate manner and for the user to receive it.

[0196] The "means for detecting and correcting pronunciation and grammar errors" refers to a technology that analyzes the user's speech and detects and corrects incorrect pronunciation and grammar.

[0197] The "means for generating a summary report of a conversation" is a technique for summarizing the contents of a conversation after it has ended and creating a report that includes errors and areas for improvement.

[0198] "Means for generating prompts" refers to technology that creates appropriate inputs for a generative AI model based on the user's order, enabling the AI ​​to generate appropriate answers.

[0199] The "means for providing recommended menus" is a system that presents via voice the dishes and products suggested by the generative AI model based on the user's requests.

[0200] MODE FOR CARRYING OUT THE INVENTION

[0201] The present invention is a system for food delivery applications that allows users to place orders using voice commands and provides appropriate menu recommendations using a generative AI model. The system includes a user terminal, a server, a generative AI, and a series of supporting databases and algorithms.

[0202] User voice input

[0203] The user inputs the order details by voice using a microphone installed on the user terminal such as a smartphone, and the user terminal transmits this voice data to the server.

[0204] Speech recognition and text conversion

[0205] The server receives the voice data and converts it into text data using a speech recognition engine (for example, the SpeechRecognition library). This text data is used for further processing.

[0206] Response generation by generative AI

[0207] The server sends the text data to a generative AI model, which generates an appropriate response based on the user's order. The generative AI model (e.g., OpenAI API) generates a response based on past conversation data and the current context.

[0208] Menu recommendations

[0209] Based on the user's order, the generative AI model generates a prompt and provides appropriate menu recommendations. For example, this prompt might look like this:

[0210] The user specifies "pizza and cola" as the menu item they would like to order, and asks for recommended side dishes.

[0211] An example of a prompt that the AI ​​generates is as follows:

[0212] Recommended side dishes include garlic bread and potato salad.

[0213] Audio Output

[0214] The generated response text is sent by the server to a speech synthesis engine (e.g., pyttsx3) and converted into voice data, which is then sent back to the user's terminal, where the user receives the response by voice.

[0215] Error Detection and Feedback

[0216] The server detects and corrects pronunciation and grammar errors in real time during conversations, providing audible feedback to the user, such as "You should say 'I went to the park'."

[0217] Conversation Summary Report

[0218] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0219] In this way, the present invention is a system that allows users to easily place orders by voice, and the generation AI suggests appropriate recommended menu items by voice.

[0220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0221] Step 1:

[0222] The user inputs the order details by voice into a user device such as a smartphone. Specifically, the voice data is acquired using the microphone of the user device. The input is the user's voice data, and the output is that voice data.

[0223] Step 2:

[0224] The user terminal transmits the acquired voice data to the server. The input is voice data, and the output is the transmission of voice data to the server.

[0225] Step 3:

[0226] The server uses a speech recognition engine to convert the voice data into text data. Specifically, the server analyzes the voice data using the SpeechRecognition library. The input is voice data and the output is text data.

[0227] Step 4:

[0228] The server sends the text data to the generative AI model and generates an appropriate response based on the user's order. Specifically, the server calls the OpenAI API and uses the generative AI model to create a response. The input is text data, and the output is the generated response text.

[0229] Step 5:

[0230] The server generates a prompt from the generative AI model and provides appropriate menu recommendations. The server generates a prompt based on the user's order and obtains suggestions from the generative AI model. The input is the user's order text and prompt text, and the output is a response text of recommended menu items.

[0231] Step 6:

[0232] The server sends the generated response text to the speech synthesis engine and converts it into speech data. Specifically, the server converts text to speech using the pyttsx3 library. The input is the response text data, and the output is speech data.

[0233] Step 7:

[0234] The server returns the converted voice data to the user terminal. The input is the voice data, and the output is the transmission of the voice data to the user terminal.

[0235] Step 8:

[0236] The server detects pronunciation and grammar errors in real time and corrects them. Specifically, the server analyzes the user's speech and provides feedback as needed. The input is the user's voice data, and the output is text or voice data of corrective feedback.

[0237] Step 9:

[0238] When the conversation ends, the server generates a summary report based on the entire conversation history. Specifically, the server extracts errors and areas for improvement from the conversation history and creates a report. The input is the conversation history data, and the output is the text data of the summary report.

[0239] Step 10:

[0240] The server sends the generated summary report to the user terminal. The input is the summary report text data, and the output is the report data sent to the user terminal.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] This invention is a system in which a user inputs speech, a generative AI generates and provides an appropriate response, and an emotion engine recognizes the user's emotional state and responds accordingly. This system includes a user terminal, a server, a generative AI, an emotion engine, and a series of supporting databases and algorithms.

[0243] User Authentication and Login

[0244] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server generates a session ID and user profile information and sends them back to the user terminal. This allows the user's past conversation data and individual settings to be inherited.

[0245] Voice input and speech generation

[0246] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[0247] Emotion Recognition and Voice Output

[0248] The server sends the text data to an emotion engine to analyze the user's emotional state. The generated response text is adjusted by the emotion engine, and the tone and content are changed according to the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The generated response text is sent by the server to a speech synthesis engine and converted into voice data. The voice data is sent back to the user's device, allowing the user to receive the response by voice.

[0249] Feedback and Error Correction

[0250] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[0251] Generate a summary report

[0252] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0253] Specific examples

[0254] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The emotion engine analyzes the user's emotional state and adjusts the tone and content of the response as needed. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[0255] In this way, users are provided with an environment where they can practice English conversation without feeling nervous and receive appropriate feedback. Furthermore, by utilizing an emotion engine, responses are provided that take into account the user's emotional state.

[0256] The processing flow will be explained below.

[0257] Step 1:

[0258] User terminal: The user enters the username and password and clicks the login button.

[0259] Step 2:

[0260] Terminal: Sends the entered authentication information to the server.

[0261] Step 3:

[0262] Server: Compares the received authentication information with a database and authenticates the user.

[0263] Step 4:

[0264] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[0265] Step 5:

[0266] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[0267] Step 6:

[0268] User device: The user uses a microphone to input voice spoken in English.

[0269] Step 7:

[0270] Terminal: Sends audio data captured by the microphone to the server.

[0271] Step 8:

[0272] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[0273] Step 9:

[0274] Server: The converted text data is sent to the emotion engine to analyze the user's emotional state.

[0275] Step 10:

[0276] Emotion engine: Based on the analysis results, it recognizes the user's emotional state. For example, it analyzes the tone and pace of the user's voice to determine whether they are happy or stressed.

[0277] Step 11:

[0278] Server: Receives the analysis results from the emotion engine and sends them to the generative AI model.

[0279] Step 12:

[0280] Generative AI: It uses text data, sentiment analysis results, and past conversation data to understand the context and generate responses that are tailored to the user's emotional state.

[0281] Step 13:

[0282] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[0283] Step 14:

[0284] Server: Sends voice data to the user terminal.

[0285] Step 15:

[0286] User terminal: Receives the voice data and plays back the response from the speaker.

[0287] Step 16:

[0288] Generative AI: Detects user pronunciation and grammar errors in real time during conversation.

[0289] Step 17:

[0290] Generative AI: If an error is detected, it generates appropriate corrections and feedback, for example, correcting the statement "I went to the park" to "I went to the park."

[0291] Step 18:

[0292] Server: Sends corrective feedback to the user terminal.

[0293] Step 19:

[0294] User terminal: Corrective feedback is played back and provided to the user.

[0295] Step 20:

[0296] Server: After the conversation ends, a summary report is generated based on the entire history, including details about pronunciation and grammar errors made during the conversation, areas for improvement, and the user's emotional state.

[0297] Step 21:

[0298] Server: Sends the generated summary report to the user terminal.

[0299] Step 22:

[0300] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[0301] Example 2

[0302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] Conventional speech recognition systems and response generation systems using generative models have difficulty generating appropriate responses that take the user's emotional state into account. Furthermore, they lack the functionality to correct users' pronunciation and grammar errors in real time and provide feedback, which reduces learning efficiency. Furthermore, there is no way to visualize the user's progress after the conversation and suggest areas for improvement, making it difficult to continuously improve the user's learning experience.

[0304] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammatical errors, means for generating a summary report of the conversation after the conversation ends, and means for recognizing the user's emotional state and adjusting the content of the response accordingly. This makes it possible to generate appropriate responses according to the user's emotional state, correct pronunciation and grammatical errors in real time, and visualize the user's learning progress.

[0305] "Means for user voice input" refers to a device or function that allows a user to transmit voice information to the system.

[0306] The "means for converting the voice into text data" refers to a device or function that analyzes input voice information and converts it into corresponding text data.

[0307] "Means for generating a response based on the text data using a generative model" refers to a device or function that utilizes a generative AI model to generate an appropriate response based on the text data.

[0308] "Means for converting said response into voice data" refers to a device or function that converts the generated text-based response into voice data.

[0309] The "means for referencing past conversation data using user authentication information" refers to a device or function that uses user authentication information to access and refer to conversation data that has taken place with a user in the past.

[0310] The "means for generating a response based on the past conversation data and the current context" refers to a device or function that generates an appropriate response by taking into account the past conversation data and the current conversation context.

[0311] The "means for returning the voice data to the user" refers to a device or function for transmitting the generated voice data to the user's terminal and letting the user hear it.

[0312] The "means for detecting and correcting pronunciation and grammar errors" refers to a device or function that identifies pronunciation and grammar errors contained in the user's speech and corrects them to the correct form.

[0313] "Means for generating a summary report of a conversation after the conversation has ended" refers to a device or function that, after the conversation has ended, summarizes the content of the conversation and generates a report that includes information such as pronunciation and grammatical errors, areas for improvement, and emotional state.

[0314] The "means for recognizing the user's emotional state and adjusting the content of the reply accordingly" refers to a device or function that analyzes the user's emotional state and adjusts the tone and content of the reply based on the results of the analysis.

[0315] This invention is a system for generating appropriate responses using user voice data, utilizing a generative AI model and an emotion recognition engine, and improving the user's learning experience. This system consists of a user terminal, a server, and various related engines and databases.

[0316] User Authentication and Login

[0317] A user logs in using their own device (e.g., PC or smartphone) by entering their username and password. The device sends the authentication information to the server, which checks it against a database. If authentication is successful, the server generates a session ID and user profile information and sends them back to the device. This allows the user to retain their settings and past conversation data.

[0318] Voice input and speech generation

[0319] The user inputs voice using the microphone on the device. The device then sends the captured voice data to the server. The server then sends the voice data to a speech recognition engine, which converts it into text data. This text data is then sent to a generative AI model, which generates an appropriate response based on the context and past conversation data. This generative AI model can be an open-source model or a commercial model provided by a company.

[0320] Emotion Recognition and Voice Output

[0321] The server sends the generated text data to an emotion engine to analyze the user's emotional state. The emotion engine identifies the user's emotion and adjusts the tone and content of the response accordingly. For example, if it determines that the user is feeling stressed, the response will be adjusted to a calmer, more encouraging tone. This adjusted response text is sent to a speech synthesis engine and converted into voice data. This voice data is sent back to the user's device via the server, allowing the user to receive the response via voice.

[0322] Feedback and Error Correction

[0323] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI detects the error and generates text correcting it to "went." The server then sends the corrective feedback to the user's device, providing audio feedback such as, "You should say 'I went to the park'."

[0324] Generate a summary report

[0325] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can view it to understand their learning progress and areas for improvement.

[0326] Specific examples

[0327] For example, if a user says, "Hello, I would like to practice English," the voice data is sent from the device to the server, where it is converted by the speech recognition engine into text data: "Hello, I would like to practice English." The generative AI model uses this text data to generate a response: "Sure, let's get started!" The emotion engine analyzes the user's emotional state and adjusts the tone and content as necessary. The speech synthesis engine then converts this response into voice data and sends it back to the user's device. Hearing the voice, "Sure, let's get started!", the user can continue practicing English conversation with peace of mind.

[0328] The system allows users to receive real-time feedback as they learn, providing a better learning experience through appropriate responses based on their emotional state.

[0329] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0330] Step 1: User authentication and login

[0331] The user enters their username and password using their own device (PC or smartphone) and clicks the login button. The device sends this authentication information to the server. The server compares the received authentication information with the database, and if the user is authenticated, generates a session ID and user profile information and sends these back to the device. This allows the user to retain their settings and past conversation data.

[0332] Input: Username, Password

[0333] Output: Session ID, user profile

[0334] Step 2: Receiving Audio Input

[0335] The user inputs speech using the device's microphone, which converts the speech data into a digital format and sends it to the server, which then sends it to a speech recognition engine for analysis.

[0336] Input: Audio data

[0337] Output: Digital audio data

[0338] Step 3: Convert audio data to text

[0339] The server converts the received voice data into text data using a speech recognition engine, which analyzes the voice waveform and generates corresponding text, which is then sent to the generative AI model.

[0340] Input: Digital audio data

[0341] Output: Text data

[0342] Step 4: Generative AI model generates a response

[0343] The server sends the generated text data to the generative AI model, which generates an appropriate response based on the text data, past conversation data, and the current context. The generated response text is then sent to the emotion engine.

[0344] Input: Text data, past conversation data, contextual information

[0345] Output: Response text

[0346] Step 5: Response tailoring with the emotion engine

[0347] The server sends the generated response text to the emotion engine, which analyzes the user's emotional state. The emotion engine adjusts the tone and content of the response based on the user's emotional state. For example, if the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The adjusted response text is then sent to the speech synthesis engine.

[0348] Input: Reply text, emotional state

[0349] Output: Adjusted response text

[0350] Step 6: Speech conversion using a speech synthesis engine

[0351] The server sends the adjusted response text to a speech synthesis engine, which converts it into voice data. The speech synthesis engine generates natural-sounding voice based on the text data. This voice data is then sent back to the user's device via the server.

[0352] Input: Adjusted response text

[0353] Output: Reply audio data

[0354] Step 7: Provide a response

[0355] The user terminal receives the returned voice data and provides it to the user through a speaker, so that the user receives a voice response.

[0356] Input: Response audio data

[0357] Output: Audio output

[0358] Step 8: Error detection and feedback

[0359] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI model detects the error in "goed" and generates a corrected text to change it to "went." The server then sends the corrected text feedback to the user's device and provides audio feedback.

[0360] Input: Raw audio data, speech content

[0361] Output: Corrective feedback text, audio feedback

[0362] Step 9: Generate a summary report

[0363] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors and areas for improvement during the conversation, as well as the user's emotional state. The server then sends the summary report to the user's device, allowing the user to review it and understand their learning progress.

[0364] Input: conversation history

[0365] Output: Summary report

[0366] (Application example 2)

[0367] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] During the food delivery ordering process, users face challenges in effectively and quickly selecting meals that suit their needs and emotions. Current systems are unable to provide personalized recommendations that take into account the user's emotional state, potentially resulting in lower customer satisfaction. Furthermore, the lack of real-time correction and feedback for pronunciation and grammar errors hinders smooth user experience.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input speech, a means for converting the speech into text data, a means for generating a response based on the text data using a generative model, a means for converting the response into speech data, a means for referencing past conversation data using user authentication information, a means for generating a response based on the past conversation data and the current context, a means for returning the speech data to the user, a means for detecting and correcting pronunciation and grammar errors, a means for generating a summary report of the conversation after the conversation ends, a means for recognizing the user's emotional state and responding according to the emotion, and a means for suggesting food delivery-related services and products. This allows the user to receive personalized food delivery suggestions based on their emotional state, enabling the ordering process to be carried out quickly and efficiently. Furthermore, real-time correction of pronunciation and grammar errors is provided, allowing for a smooth user experience.

[0370] "Means for user voice input" refers to a function for acquiring voice data through an input device (e.g., a microphone).

[0371] The "means for converting voice into text data" is a function that converts voice data acquired using voice recognition technology into a corresponding text format.

[0372] "Means for generating responses based on text data using a generative model" refers to a function that uses generative AI to analyze input text data and generate appropriate responses.

[0373] The "means for converting a response into voice data" is a function that converts the generated text response into voice data using synthesized voice technology.

[0374] The "means for referencing past conversation data using user authentication information" is a function for retrieving and referencing past conversation history from a database using authentication information such as a user name and password.

[0375] The "means for generating a response based on past conversation data and current context" is a function for generating a personalized response based on past conversation history and real-time context information.

[0376] The "means for returning voice data to the user" is a function that transmits the generated voice data to the user device, allowing the user to receive a voice response.

[0377] The "means for detecting and correcting pronunciation and grammatical errors" is a function that analyzes the content of the user's speech, detects pronunciation and grammatical errors, corrects them, and provides feedback.

[0378] The "means for generating a summary report of a conversation after the conversation has ended" is a function that compiles the conversation history with the user and generates a report that includes pronunciation and grammatical errors, emotional state, and so on.

[0379] "Means for recognizing the user's emotional state and responding accordingly" refers to a function that uses emotion recognition technology to analyze the user's emotional state and adjusts the content and tone of the response based on this.

[0380] "Means for suggesting food delivery-related services and products" refers to a function that recommends appropriate food delivery services and products based on the user's emotions and needs.

[0381] To implement this invention, multiple technical components are required: a user terminal, a server, a speech recognition engine, a generative AI model, an emotion engine, and a speech synthesis engine. A user places a food delivery order by voice. The server processes this voice data and provides a system that generates personalized responses based on the user's emotional state.

[0382] 1. User authentication and login

[0383] Users log in by entering their username and password on a device such as a smartphone. This authentication information is sent to the server and checked against a database. If authentication is successful, the server retrieves past conversation data and user profile information and returns it to the user's device, starting the session.

[0384] 2. Voice to Text

[0385] Users input their voice using the microphone on their smartphone. This voice data is sent to the server and converted into text data through a speech recognition engine. Speech recognition engines such as Google Speech-to-Text and Microsoft Azure Speech can be used.

[0386] 3. Response generation using generative AI models

[0387] The server sends the acquired text data to a generative AI model, which can use models such as GPT-3 or ChatGPT, to generate an appropriate response.

[0388] 4. Emotion recognition and response regulation

[0389] The generated text response is sent by the server to an emotion engine, which analyzes the user's emotional state (using a model like Hugging Face's emotion analysis model), and adjusts the content and tone of the response based on the user's emotional state.

[0390] 5. Speech synthesis and response return

[0391] The adjusted response text is sent to a speech synthesis engine, which converts it into voice data. The speech synthesis engine can be Amazon Polly or Google Text-to-Speech. This voice data is sent back to the user's device, where the user can receive a voice response.

[0392] 6. Food delivery proposal

[0393] The server will suggest suitable food delivery options to the user based on their emotional state and past ordering history. For example, if a user says, "I'm tired today," the system will suggest, "Would you like hot soup or pizza?"

[0394] 7. Summary report generation

[0395] After the conversation ends, the server generates a summary report based on the entire conversation history. This summary report includes any pronunciation or grammar errors detected during the conversation, information about the user's emotional state, and a history of food delivery suggestions. This report is sent to the user's device, where the user can review their conversation history.

[0396] Specific examples

[0397] For example, if a user asks, "I'm tired today, what would you recommend for dinner?", the server processes the voice data and uses an emotion engine to identify the emotion "tired." It then generates a response, "Would you like some hot soup or pizza?", and delivers it to the user as speech.

[0398] Prompt Sentence Examples

[0399] When a user asks, "I'm tired today, what food would you recommend?", your smart food assistant should use its emotion engine to identify the emotion "tired" and respond with, "You seem tired today. Would you like some hot soup or pizza to help you relax?"

[0400] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0401] Step 1:

[0402] The user inputs voice through the smartphone microphone. This input is the user's action of making a request or asking a question for food delivery by voice. For example, the user might say, "I'm tired today, so what meal would you recommend?"

[0403] Step 2:

[0404] The device collects voice data and sends it to the server. The data sent is raw voice data. In this step, the device does not perform any preprocessing for voice recognition.

[0405] Step 3:

[0406] The server sends the received voice data to a speech recognition engine. This engine converts the voice data into text data. For example, Google Speech-to-Text is used. In this step, the voice data is converted into text data such as "I'm tired today, so what meal would you recommend?"

[0407] Step 4:

[0408] The server sends the converted text data to a generative AI model, which then generates an appropriate response based on the text data. The input at this stage is the converted text data, and the output is the generated response text: "Would you like some warm soup or pizza to relax?"

[0409] Step 5:

[0410] The server sends the generated text response to the emotion engine, which analyzes the user's emotional state. In this case, based on the text data "I'm tired today...", the server analyzes that the user's state is "tired." Based on this input, the emotion engine adjusts the tone and content. The output is a response text with adjusted tone.

[0411] Step 6:

[0412] The server sends the tone-adjusted response text to a speech synthesis engine, which converts it into speech data. The speech synthesis engine (e.g., Amazon Polly) converts the text data into speech data. The output is the speech data, "Would you like some hot soup or pizza to relax?"

[0413] Step 7:

[0414] The server sends the generated voice data to the user terminal, and the user receives a voice response. The input is the adjusted voice data, and the output is the voice information that the user hears.

[0415] Step 8:

[0416] When the conversation ends, the server generates a summary report based on the conversation history and emotional state, including the user's comments, responses, and emotional analysis results. The server then sends this report to the user's device so that the user can review it.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0418] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0420] [Second embodiment]

[0421] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0422] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0423] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0425] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0427] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0428] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0429] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0432] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0433] This system allows users to input speech and uses a generative AI to generate and provide appropriate responses. The system includes a user terminal, a server, a generative AI, and a series of databases and algorithms that support the AI.

[0434] User Authentication and Login

[0435] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0436] Voice input and speech generation

[0437] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[0438] Audio Output

[0439] The generated response text is sent to a speech synthesis engine by the server and converted into voice data. The voice data is then sent back to the user's device, and the user receives the response by voice, providing a natural conversational experience.

[0440] Feedback and Error Correction

[0441] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[0442] Generate a summary report

[0443] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0444] Specific examples

[0445] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[0446] In this way, users are provided with an environment in which they can practice English conversation without feeling nervous and while receiving appropriate feedback.

[0447] The processing flow will be explained below.

[0448] Step 1:

[0449] User terminal: The user enters the username and password and clicks the login button.

[0450] Step 2:

[0451] Terminal: Sends the entered authentication information to the server.

[0452] Step 3:

[0453] Server: Compares the received authentication information with a database and authenticates the user.

[0454] Step 4:

[0455] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[0456] Step 5:

[0457] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[0458] Step 6:

[0459] User device: The user uses a microphone to input voice spoken in English.

[0460] Step 7:

[0461] Terminal: Sends audio data captured by the microphone to the server.

[0462] Step 8:

[0463] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[0464] Step 9:

[0465] Server: Sends the converted text data to the generative AI model and generates a response.

[0466] Step 10:

[0467] Generative AI: Uses text data and past conversation data to understand the user's context and generate appropriate responses.

[0468] Step 11:

[0469] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[0470] Step 12:

[0471] Server: Sends voice data to the user terminal.

[0472] Step 13:

[0473] User terminal: Receives the voice data and plays back the response from the speaker.

[0474] Step 14:

[0475] Generative AI: Detects user pronunciation and grammar errors in real time.

[0476] Step 15:

[0477] Generative AI: Generates appropriate corrections and feedback when errors are detected.

[0478] Step 16:

[0479] Server: Sends corrective feedback to the user terminal.

[0480] Step 17:

[0481] User terminal: Corrective feedback is played back and provided to the user.

[0482] Step 18:

[0483] Server: After the conversation ends, generate a summary report based on the entire history.

[0484] Step 19:

[0485] Server: Sends the generated summary report to the user terminal.

[0486] Step 20:

[0487] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] Conventional voice dialogue systems have difficulty accurately converting user input speech into text data and generating appropriate responses. They also struggle to generate responses that fully understand past conversation data and context, making it impossible to provide users with a natural, personalized conversation experience. Furthermore, they lack the ability to correct pronunciation and grammar errors in real time and provide feedback to users. This reduces the user's learning effectiveness and the quality of their conversation experience.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes means for encrypting authentication information entered by the user and transmitting it to the server, means for loading the user's profile information if authentication is successful, and means for asynchronously processing voice data received from the user via the terminal. This allows for the generation of appropriate responses based on the user's voice input, enabling personalized responses that take into account past conversation data and context. Furthermore, pronunciation and grammar errors can be corrected in real time and feedback can be provided to improve the user's learning effect. Furthermore, the use of a cloud-based speech recognition and speech synthesis engine enables highly accurate voice processing, providing the user with a natural, high-quality conversation experience.

[0493] "User authentication information" is an identifier used by a user when accessing a system, and is information that mainly consists of a user name and password.

[0494] "Voice input" refers to the act of a user speaking to the system using a microphone, or the voice data acquired thereby.

[0495] A "voice recognition engine" is software or algorithm that converts captured voice data into text data.

[0496] A "generative model" is an algorithm or system that uses natural language processing to generate appropriate responses based on input text data.

[0497] "Voice data" is data input by voice that is expressed in digital format and is used for voice recognition and voice synthesis.

[0498] "Past conversation data" is historical information about conversations the user has had in the past, and is used as basic data for analysis and response generation.

[0499] "Context" is information used to understand the content, flow, and context of a conversation, and is an important element for a generative model to provide an appropriate response.

[0500] "Real-time feedback" refers to the process of providing instant feedback to users immediately after they speak, immediately pointing out and correcting pronunciation and grammar errors.

[0501] "Profile information" refers to information such as a user's individual settings, past usage history, conversation history, etc., and is used to provide personalized services.

[0502] A "cloud-based engine" is a processing engine for speech recognition, speech synthesis, etc. that is provided via the Internet on a cloud service.

[0503] This invention is a system that allows a user to input speech and generates and provides an appropriate response using a generative AI model. The system includes a user terminal, a server, a generative AI model, and a series of databases and algorithms that support the model.

[0504] User Authentication and Login

[0505] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal encrypts this authentication information and sends it to the server. The server authenticates the user by checking it against a database. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0506] Voice input

[0507] The user inputs voice using a microphone installed on the user device. The device temporarily stores the acquired voice data and then transmits it to the server.

[0508] Voice Recognition

[0509] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into text data, which is temporarily stored on the server.

[0510] Conversation Generation

[0511] The server inputs this text data into a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response based on the context and past conversation data. The generated response text is temporarily stored on the server.

[0512] Audio Output

[0513] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. The converted voice data is sent to the user's device, and the user receives the response by voice.

[0514] Feedback and Error Correction

[0515] During the conversation, the generative AI model detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error in "goed" and generates text correcting it to "went." The server generates corrective feedback, converts it into audio data, and sends it to the user's device.

[0516] Generate a summary report

[0517] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0518] Specific examples

[0519] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation. In this way, users are provided with an environment where they can practice English conversation without feeling stressed and while receiving appropriate feedback.

[0520] Prompt Sentence Examples

[0521] "Hello, I'd like to practice my English."

[0522] This allows users to continue the conversation in real time and receive real-time feedback on pronunciation and grammar errors, which is expected to improve users' English conversation skills.

[0523] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0524] Step 1: User authentication and login

[0525] The user uses a dedicated app or web browser on the user device (PC or smartphone) to enter their username and password and click the login button.

[0526] Input: Username, Password

[0527] Specific behavior:

[0528] The terminal encrypts the entered authentication information and sends it to the server using the HTTPS protocol.

[0529] The server authenticates the user by checking the received authentication information against a database (e.g., MySQL).

[0530] Output: Authentication result (success or failure)

[0531] If authentication is successful, the server creates a session and loads the user's profile information into memory.

[0532] Step 2: Voice Input

[0533] The user speaks into a microphone installed in the user terminal.

[0534] Input: Audio data

[0535] Specific behavior:

[0536] The terminal temporarily stores the audio data acquired from the microphone.

[0537] The device asynchronously transmits the stored audio data to the server.

[0538] Output: Audio data (binary format)

[0539] Step 3: Voice Recognition

[0540] The server passes the received voice data to a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data.

[0541] Input: Audio data (binary format)

[0542] Specific behavior:

[0543] The server sends the audio data as an API request.

[0544] The voice recognition engine analyzes the voice data and converts it into text data.

[0545] Output: Text data (natural language text)

[0546] The converted text data is temporarily stored on the server.

[0547] Step 4: Conversation generation

[0548] The server inputs the text data generated by speech recognition into a generative AI model (e.g., OpenAI GPT-3).

[0549] Input: Text data (natural language text)

[0550] Specific behavior:

[0551] The server sends the text data to the generative AI model.

[0552] The generative AI model takes into account past conversation data and context to generate appropriate responses.

[0553] Output: Response text data (natural language text)

[0554] The generated response text is temporarily stored on the server.

[0555] Step 5: Audio Output

[0556] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data.

[0557] Input: Response text data (natural language text)

[0558] Specific behavior:

[0559] The server passes the response text to the speech synthesis engine.

[0560] The speech synthesis engine analyzes the text data and converts it into speech data.

[0561] Output: Audio data (audio file)

[0562] The generated voice data is transmitted to the user terminal and returned to the user.

[0563] Step 6: Feedback and Error Correction

[0564] During the conversation, the generative AI model detects the user's pronunciation and grammar errors and corrects them in real time.

[0565] Input: User utterance text data

[0566] Specific behavior:

[0567] The server analyzes the speech using a generative AI model and detects pronunciation and grammatical errors.

[0568] If an error is detected, the AI ​​generates corrected text data.

[0569] Output: Corrective feedback text (natural language text)

[0570] The server passes the corrective feedback to a speech synthesis engine, converts it into speech data, and provides the feedback to the user.

[0571] Step 7: Generate a summary report

[0572] After the conversation ends, the server generates a summary report based on the entire conversation history.

[0573] Input: Text data of conversation history

[0574] Specific behavior:

[0575] The server analyzes the conversation history and extracts pronunciation and grammatical errors and areas for improvement.

[0576] Generate summary reports and collate details.

[0577] Output: Summary report (text or PDF)

[0578] The summary report is sent to the user terminal so that the user can view it.

[0579] (Application example 1)

[0580] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0581] In conventional food delivery apps, voice recognition is often inaccurate when users input their order details, and menu recommendations are text-based, making them less convenient. Furthermore, it is difficult to provide a personalized user experience. Therefore, there was a need for a system that allows users to easily order by voice and can use a generative AI model to provide appropriate menu recommendations via voice.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0583] In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammar errors, means for generating a conversation summary report after the conversation ends, means for causing a generative AI model to generate prompts based on the user's order and recommending appropriate menu items, and means for providing the generated recommended menu items to the user as speech data. This allows a user to place an order by voice, and the generative AI to suggest appropriate recommended menu items by voice.

[0584] "Means for inputting voice" refers to a device or interface that takes in what the user has said and recognizes it as voice data.

[0585] "Means for converting into text data" refers to technology that analyzes voice data and converts its contents into text information.

[0586] "Means for generating a response using a generative model" refers to a method for creating an appropriate response based on input text data using a pre-trained artificial intelligence model.

[0587] "Means for converting into voice data" refers to a technology that analyzes text data and generates its contents as voice.

[0588] "Means for referencing past conversation data using user authentication information" refers to a system that searches and refers to the history of past conversations based on user authentication information.

[0589] "Means for generating a response based on the current context" refers to a technology that understands the current conversation content and situation and creates a response that matches that context.

[0590] The "means for returning voice data to the user" is a mechanism for transmitting the generated voice data to the user in an appropriate manner and for the user to receive it.

[0591] The "means for detecting and correcting pronunciation and grammar errors" refers to a technology that analyzes the user's speech and detects and corrects incorrect pronunciation and grammar.

[0592] The "means for generating a summary report of a conversation" is a technique for summarizing the contents of a conversation after it has ended and creating a report that includes errors and areas for improvement.

[0593] "Means for generating prompts" refers to technology that creates appropriate inputs for a generative AI model based on the user's order, enabling the AI ​​to generate appropriate answers.

[0594] The "means for providing recommended menus" is a system that presents via voice the dishes and products suggested by the generative AI model based on the user's requests.

[0595] MODE FOR CARRYING OUT THE INVENTION

[0596] The present invention is a system for food delivery applications that allows users to place orders using voice commands and provides appropriate menu recommendations using a generative AI model. The system includes a user terminal, a server, a generative AI, and a series of supporting databases and algorithms.

[0597] User voice input

[0598] The user inputs the order details by voice using a microphone installed on the user terminal such as a smartphone, and the user terminal transmits this voice data to the server.

[0599] Speech recognition and text conversion

[0600] The server receives the voice data and converts it into text data using a speech recognition engine (for example, the SpeechRecognition library). This text data is used for further processing.

[0601] Response generation by generative AI

[0602] The server sends the text data to a generative AI model, which generates an appropriate response based on the user's order. The generative AI model (e.g., OpenAI API) generates a response based on past conversation data and the current context.

[0603] Menu recommendations

[0604] Based on the user's order, the generative AI model generates a prompt and provides appropriate menu recommendations. For example, this prompt might look like this:

[0605] The user specifies "pizza and cola" as the menu item they would like to order, and asks for recommended side dishes.

[0606] An example of a prompt that the AI ​​generates is as follows:

[0607] Recommended side dishes include garlic bread and potato salad.

[0608] Audio Output

[0609] The generated response text is sent by the server to a speech synthesis engine (e.g., pyttsx3) and converted into voice data, which is then sent back to the user's terminal, where the user receives the response by voice.

[0610] Error Detection and Feedback

[0611] The server detects and corrects pronunciation and grammar errors in real time during conversations, providing audible feedback to the user, such as "You should say 'I went to the park'."

[0612] Conversation Summary Report

[0613] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0614] In this way, the present invention is a system that allows users to easily place orders by voice, and the generation AI suggests appropriate recommended menu items by voice.

[0615] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0616] Step 1:

[0617] The user inputs the order details by voice into a user device such as a smartphone. Specifically, the voice data is acquired using the microphone of the user device. The input is the user's voice data, and the output is that voice data.

[0618] Step 2:

[0619] The user terminal transmits the acquired voice data to the server. The input is voice data, and the output is the transmission of voice data to the server.

[0620] Step 3:

[0621] The server uses a speech recognition engine to convert the voice data into text data. Specifically, the server analyzes the voice data using the SpeechRecognition library. The input is voice data and the output is text data.

[0622] Step 4:

[0623] The server sends the text data to the generative AI model and generates an appropriate response based on the user's order. Specifically, the server calls the OpenAI API and uses the generative AI model to create a response. The input is text data, and the output is the generated response text.

[0624] Step 5:

[0625] The server generates a prompt from the generative AI model and provides appropriate menu recommendations. The server generates a prompt based on the user's order and obtains suggestions from the generative AI model. The input is the user's order text and prompt text, and the output is a response text of recommended menu items.

[0626] Step 6:

[0627] The server sends the generated response text to the speech synthesis engine and converts it into speech data. Specifically, the server converts text to speech using the pyttsx3 library. The input is the response text data, and the output is speech data.

[0628] Step 7:

[0629] The server returns the converted voice data to the user terminal. The input is the voice data, and the output is the transmission of the voice data to the user terminal.

[0630] Step 8:

[0631] The server detects pronunciation and grammar errors in real time and corrects them. Specifically, the server analyzes the user's speech and provides feedback as needed. The input is the user's voice data, and the output is text or voice data of corrective feedback.

[0632] Step 9:

[0633] When the conversation ends, the server generates a summary report based on the entire conversation history. Specifically, the server extracts errors and areas for improvement from the conversation history and creates a report. The input is the conversation history data, and the output is the text data of the summary report.

[0634] Step 10:

[0635] The server sends the generated summary report to the user terminal. The input is the summary report text data, and the output is the report data sent to the user terminal.

[0636] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0637] This invention is a system in which a user inputs speech, a generative AI generates and provides an appropriate response, and an emotion engine recognizes the user's emotional state and responds accordingly. This system includes a user terminal, a server, a generative AI, an emotion engine, and a series of supporting databases and algorithms.

[0638] User Authentication and Login

[0639] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server generates a session ID and user profile information and sends them back to the user terminal. This allows the user's past conversation data and individual settings to be inherited.

[0640] Voice input and speech generation

[0641] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[0642] Emotion Recognition and Voice Output

[0643] The server sends the text data to an emotion engine to analyze the user's emotional state. The generated response text is adjusted by the emotion engine, and the tone and content are changed according to the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The generated response text is sent by the server to a speech synthesis engine and converted into voice data. The voice data is sent back to the user's device, allowing the user to receive the response by voice.

[0644] Feedback and Error Correction

[0645] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[0646] Generate a summary report

[0647] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0648] Specific examples

[0649] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The emotion engine analyzes the user's emotional state and adjusts the tone and content of the response as needed. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[0650] In this way, users are provided with an environment where they can practice English conversation without feeling nervous and receive appropriate feedback. Furthermore, by utilizing an emotion engine, responses are provided that take into account the user's emotional state.

[0651] The processing flow will be explained below.

[0652] Step 1:

[0653] User terminal: The user enters the username and password and clicks the login button.

[0654] Step 2:

[0655] Terminal: Sends the entered authentication information to the server.

[0656] Step 3:

[0657] Server: Compares the received authentication information with a database and authenticates the user.

[0658] Step 4:

[0659] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[0660] Step 5:

[0661] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[0662] Step 6:

[0663] User device: The user uses a microphone to input voice spoken in English.

[0664] Step 7:

[0665] Terminal: Sends audio data captured by the microphone to the server.

[0666] Step 8:

[0667] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[0668] Step 9:

[0669] Server: The converted text data is sent to the emotion engine to analyze the user's emotional state.

[0670] Step 10:

[0671] Emotion engine: Based on the analysis results, it recognizes the user's emotional state. For example, it analyzes the tone and pace of the user's voice to determine whether they are happy or stressed.

[0672] Step 11:

[0673] Server: Receives the analysis results from the emotion engine and sends them to the generative AI model.

[0674] Step 12:

[0675] Generative AI: It uses text data, sentiment analysis results, and past conversation data to understand the context and generate responses that are tailored to the user's emotional state.

[0676] Step 13:

[0677] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[0678] Step 14:

[0679] Server: Sends voice data to the user terminal.

[0680] Step 15:

[0681] User terminal: Receives the voice data and plays back the response from the speaker.

[0682] Step 16:

[0683] Generative AI: Detects user pronunciation and grammar errors in real time during conversation.

[0684] Step 17:

[0685] Generative AI: If an error is detected, it generates appropriate corrections and feedback, for example, correcting the statement "I went to the park" to "I went to the park."

[0686] Step 18:

[0687] Server: Sends corrective feedback to the user terminal.

[0688] Step 19:

[0689] User terminal: Corrective feedback is played back and provided to the user.

[0690] Step 20:

[0691] Server: After the conversation ends, a summary report is generated based on the entire history, including details about pronunciation and grammar errors made during the conversation, areas for improvement, and the user's emotional state.

[0692] Step 21:

[0693] Server: Sends the generated summary report to the user terminal.

[0694] Step 22:

[0695] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[0696] Example 2

[0697] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0698] Conventional speech recognition systems and response generation systems using generative models have difficulty generating appropriate responses that take the user's emotional state into account. Furthermore, they lack the functionality to correct users' pronunciation and grammar errors in real time and provide feedback, which reduces learning efficiency. Furthermore, there is no way to visualize the user's progress after the conversation and suggest areas for improvement, making it difficult to continuously improve the user's learning experience.

[0699] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammatical errors, means for generating a summary report of the conversation after the conversation ends, and means for recognizing the user's emotional state and adjusting the content of the response accordingly. This makes it possible to generate appropriate responses according to the user's emotional state, correct pronunciation and grammatical errors in real time, and visualize the user's learning progress.

[0700] "Means for user voice input" refers to a device or function that allows a user to transmit voice information to the system.

[0701] The "means for converting the voice into text data" refers to a device or function that analyzes input voice information and converts it into corresponding text data.

[0702] "Means for generating a response based on the text data using a generative model" refers to a device or function that utilizes a generative AI model to generate an appropriate response based on the text data.

[0703] "Means for converting said response into voice data" refers to a device or function that converts the generated text-based response into voice data.

[0704] The "means for referencing past conversation data using user authentication information" refers to a device or function that uses user authentication information to access and refer to conversation data that has taken place with a user in the past.

[0705] The "means for generating a response based on the past conversation data and the current context" refers to a device or function that generates an appropriate response by taking into account the past conversation data and the current conversation context.

[0706] The "means for returning the voice data to the user" refers to a device or function for transmitting the generated voice data to the user's terminal and letting the user hear it.

[0707] The "means for detecting and correcting pronunciation and grammar errors" refers to a device or function that identifies pronunciation and grammar errors contained in the user's speech and corrects them to the correct form.

[0708] "Means for generating a summary report of a conversation after the conversation has ended" refers to a device or function that, after the conversation has ended, summarizes the content of the conversation and generates a report that includes information such as pronunciation and grammatical errors, areas for improvement, and emotional state.

[0709] The "means for recognizing the user's emotional state and adjusting the content of the reply accordingly" refers to a device or function that analyzes the user's emotional state and adjusts the tone and content of the reply based on the results of the analysis.

[0710] This invention is a system for generating appropriate responses using user voice data, utilizing a generative AI model and an emotion recognition engine, and improving the user's learning experience. This system consists of a user terminal, a server, and various related engines and databases.

[0711] User Authentication and Login

[0712] A user logs in using their own device (e.g., PC or smartphone) by entering their username and password. The device sends the authentication information to the server, which checks it against a database. If authentication is successful, the server generates a session ID and user profile information and sends them back to the device. This allows the user to retain their settings and past conversation data.

[0713] Voice input and speech generation

[0714] The user inputs voice using the microphone on the device. The device then sends the captured voice data to the server. The server then sends the voice data to a speech recognition engine, which converts it into text data. This text data is then sent to a generative AI model, which generates an appropriate response based on the context and past conversation data. This generative AI model can be an open-source model or a commercial model provided by a company.

[0715] Emotion Recognition and Voice Output

[0716] The server sends the generated text data to an emotion engine to analyze the user's emotional state. The emotion engine identifies the user's emotion and adjusts the tone and content of the response accordingly. For example, if it determines that the user is feeling stressed, the response will be adjusted to a calmer, more encouraging tone. This adjusted response text is sent to a speech synthesis engine and converted into voice data. This voice data is sent back to the user's device via the server, allowing the user to receive the response via voice.

[0717] Feedback and Error Correction

[0718] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI detects the error and generates text correcting it to "went." The server then sends the corrective feedback to the user's device, providing audio feedback such as, "You should say 'I went to the park'."

[0719] Generate a summary report

[0720] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can view it to understand their learning progress and areas for improvement.

[0721] Specific examples

[0722] For example, if a user says, "Hello, I would like to practice English," the voice data is sent from the device to the server, where it is converted by the speech recognition engine into text data: "Hello, I would like to practice English." The generative AI model uses this text data to generate a response: "Sure, let's get started!" The emotion engine analyzes the user's emotional state and adjusts the tone and content as necessary. The speech synthesis engine then converts this response into voice data and sends it back to the user's device. Hearing the voice, "Sure, let's get started!", the user can continue practicing English conversation with peace of mind.

[0723] The system allows users to receive real-time feedback as they learn, providing a better learning experience through appropriate responses based on their emotional state.

[0724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0725] Step 1: User authentication and login

[0726] The user enters their username and password using their own device (PC or smartphone) and clicks the login button. The device sends this authentication information to the server. The server compares the received authentication information with the database, and if the user is authenticated, generates a session ID and user profile information and sends these back to the device. This allows the user to retain their settings and past conversation data.

[0727] Input: Username, Password

[0728] Output: Session ID, user profile

[0729] Step 2: Receiving Audio Input

[0730] The user inputs speech using the device's microphone, which converts the speech data into a digital format and sends it to the server, which then sends it to a speech recognition engine for analysis.

[0731] Input: Audio data

[0732] Output: Digital audio data

[0733] Step 3: Convert audio data to text

[0734] The server converts the received voice data into text data using a speech recognition engine, which analyzes the voice waveform and generates corresponding text, which is then sent to the generative AI model.

[0735] Input: Digital audio data

[0736] Output: Text data

[0737] Step 4: Generative AI model generates a response

[0738] The server sends the generated text data to the generative AI model, which generates an appropriate response based on the text data, past conversation data, and the current context. The generated response text is then sent to the emotion engine.

[0739] Input: Text data, past conversation data, contextual information

[0740] Output: Response text

[0741] Step 5: Response tailoring with the emotion engine

[0742] The server sends the generated response text to the emotion engine, which analyzes the user's emotional state. The emotion engine adjusts the tone and content of the response based on the user's emotional state. For example, if the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The adjusted response text is then sent to the speech synthesis engine.

[0743] Input: Reply text, emotional state

[0744] Output: Adjusted response text

[0745] Step 6: Speech conversion using a speech synthesis engine

[0746] The server sends the adjusted response text to a speech synthesis engine, which converts it into voice data. The speech synthesis engine generates natural-sounding voice based on the text data. This voice data is then sent back to the user's device via the server.

[0747] Input: Adjusted response text

[0748] Output: Reply audio data

[0749] Step 7: Provide a response

[0750] The user terminal receives the returned voice data and provides it to the user through a speaker, so that the user receives a voice response.

[0751] Input: Response audio data

[0752] Output: Audio output

[0753] Step 8: Error detection and feedback

[0754] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI model detects the error in "goed" and generates a corrected text to change it to "went." The server then sends the corrected text feedback to the user's device and provides audio feedback.

[0755] Input: Raw audio data, speech content

[0756] Output: Corrective feedback text, audio feedback

[0757] Step 9: Generate a summary report

[0758] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors and areas for improvement during the conversation, as well as the user's emotional state. The server then sends the summary report to the user's device, allowing the user to review it and understand their learning progress.

[0759] Input: conversation history

[0760] Output: Summary report

[0761] (Application example 2)

[0762] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0763] During the food delivery ordering process, users face challenges in effectively and quickly selecting meals that suit their needs and emotions. Current systems are unable to provide personalized recommendations that take into account the user's emotional state, potentially resulting in lower customer satisfaction. Furthermore, the lack of real-time correction and feedback for pronunciation and grammar errors hinders smooth user experience.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input speech, a means for converting the speech into text data, a means for generating a response based on the text data using a generative model, a means for converting the response into speech data, a means for referencing past conversation data using user authentication information, a means for generating a response based on the past conversation data and the current context, a means for returning the speech data to the user, a means for detecting and correcting pronunciation and grammar errors, a means for generating a summary report of the conversation after the conversation ends, a means for recognizing the user's emotional state and responding according to the emotion, and a means for suggesting food delivery-related services and products. This allows the user to receive personalized food delivery suggestions based on their emotional state, enabling the ordering process to be carried out quickly and efficiently. Furthermore, real-time correction of pronunciation and grammar errors is provided, allowing for a smooth user experience.

[0765] "Means for user voice input" refers to a function for acquiring voice data through an input device (e.g., a microphone).

[0766] The "means for converting voice into text data" is a function that converts voice data acquired using voice recognition technology into a corresponding text format.

[0767] "Means for generating responses based on text data using a generative model" refers to a function that uses generative AI to analyze input text data and generate appropriate responses.

[0768] The "means for converting a response into voice data" is a function that converts the generated text response into voice data using synthesized voice technology.

[0769] The "means for referencing past conversation data using user authentication information" is a function for retrieving and referencing past conversation history from a database using authentication information such as a user name and password.

[0770] The "means for generating a response based on past conversation data and current context" is a function for generating a personalized response based on past conversation history and real-time context information.

[0771] The "means for returning voice data to the user" is a function that transmits the generated voice data to the user device, allowing the user to receive a voice response.

[0772] The "means for detecting and correcting pronunciation and grammatical errors" is a function that analyzes the content of the user's speech, detects pronunciation and grammatical errors, corrects them, and provides feedback.

[0773] The "means for generating a summary report of a conversation after the conversation has ended" is a function that compiles the conversation history with the user and generates a report that includes pronunciation and grammatical errors, emotional state, and so on.

[0774] "Means for recognizing the user's emotional state and responding accordingly" refers to a function that uses emotion recognition technology to analyze the user's emotional state and adjusts the content and tone of the response based on this.

[0775] "Means for suggesting food delivery-related services and products" refers to a function that recommends appropriate food delivery services and products based on the user's emotions and needs.

[0776] To implement this invention, multiple technical components are required: a user terminal, a server, a speech recognition engine, a generative AI model, an emotion engine, and a speech synthesis engine. A user places a food delivery order by voice. The server processes this voice data and provides a system that generates personalized responses based on the user's emotional state.

[0777] 1. User authentication and login

[0778] Users log in by entering their username and password on a device such as a smartphone. This authentication information is sent to the server and checked against a database. If authentication is successful, the server retrieves past conversation data and user profile information and returns it to the user's device, starting the session.

[0779] 2. Voice to Text

[0780] Users input their voice using the microphone on their smartphone. This voice data is sent to the server and converted into text data through a speech recognition engine. Speech recognition engines such as Google Speech-to-Text and Microsoft Azure Speech can be used.

[0781] 3. Response generation using generative AI models

[0782] The server sends the acquired text data to a generative AI model, which can use models such as GPT-3 or ChatGPT, to generate an appropriate response.

[0783] 4. Emotion recognition and response regulation

[0784] The generated text response is sent by the server to an emotion engine, which analyzes the user's emotional state (using a model like Hugging Face's emotion analysis model), and adjusts the content and tone of the response based on the user's emotional state.

[0785] 5. Speech synthesis and response return

[0786] The adjusted response text is sent to a speech synthesis engine, which converts it into voice data. The speech synthesis engine can be Amazon Polly or Google Text-to-Speech. This voice data is sent back to the user's device, where the user can receive a voice response.

[0787] 6. Food delivery proposal

[0788] The server will suggest suitable food delivery options to the user based on their emotional state and past ordering history. For example, if a user says, "I'm tired today," the system will suggest, "Would you like hot soup or pizza?"

[0789] 7. Summary report generation

[0790] After the conversation ends, the server generates a summary report based on the entire conversation history. This summary report includes any pronunciation or grammar errors detected during the conversation, information about the user's emotional state, and a history of food delivery suggestions. This report is sent to the user's device, where the user can review their conversation history.

[0791] Specific examples

[0792] For example, if a user asks, "I'm tired today, what would you recommend for dinner?", the server processes the voice data and uses an emotion engine to identify the emotion "tired." It then generates a response, "Would you like some hot soup or pizza?", and delivers it to the user as speech.

[0793] Prompt Sentence Examples

[0794] When a user asks, "I'm tired today, what food would you recommend?", your smart food assistant should use its emotion engine to identify the emotion "tired" and respond with, "You seem tired today. Would you like some hot soup or pizza to help you relax?"

[0795] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0796] Step 1:

[0797] The user inputs voice through the smartphone microphone. This input is the user's action of making a request or asking a question for food delivery by voice. For example, the user might say, "I'm tired today, so what meal would you recommend?"

[0798] Step 2:

[0799] The device collects voice data and sends it to the server. The data sent is raw voice data. In this step, the device does not perform any preprocessing for voice recognition.

[0800] Step 3:

[0801] The server sends the received voice data to a speech recognition engine. This engine converts the voice data into text data. For example, Google Speech-to-Text is used. In this step, the voice data is converted into text data such as "I'm tired today, so what meal would you recommend?"

[0802] Step 4:

[0803] The server sends the converted text data to a generative AI model, which then generates an appropriate response based on the text data. The input at this stage is the converted text data, and the output is the generated response text: "Would you like some warm soup or pizza to relax?"

[0804] Step 5:

[0805] The server sends the generated text response to the emotion engine, which analyzes the user's emotional state. In this case, based on the text data "I'm tired today...", the server analyzes that the user's state is "tired." Based on this input, the emotion engine adjusts the tone and content. The output is a response text with adjusted tone.

[0806] Step 6:

[0807] The server sends the tone-adjusted response text to a speech synthesis engine, which converts it into speech data. The speech synthesis engine (e.g., Amazon Polly) converts the text data into speech data. The output is the speech data, "Would you like some hot soup or pizza to relax?"

[0808] Step 7:

[0809] The server sends the generated voice data to the user terminal, and the user receives a voice response. The input is the adjusted voice data, and the output is the voice information that the user hears.

[0810] Step 8:

[0811] When the conversation ends, the server generates a summary report based on the conversation history and emotional state, including the user's comments, responses, and emotional analysis results. The server then sends this report to the user's device so that the user can review it.

[0812] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0813] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0814] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0815] [Third embodiment]

[0816] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0817] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0818] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0819] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0820] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0821] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0822] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0823] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0824] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0825] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0826] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0827] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0828] This system allows users to input speech and uses a generative AI to generate and provide appropriate responses. The system includes a user terminal, a server, a generative AI, and a series of databases and algorithms that support the AI.

[0829] User Authentication and Login

[0830] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0831] Voice input and speech generation

[0832] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[0833] Audio Output

[0834] The generated response text is sent to a speech synthesis engine by the server and converted into voice data. The voice data is then sent back to the user's device, and the user receives the response by voice, providing a natural conversational experience.

[0835] Feedback and Error Correction

[0836] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[0837] Generate a summary report

[0838] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0839] Specific examples

[0840] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[0841] In this way, users are provided with an environment in which they can practice English conversation without feeling nervous and while receiving appropriate feedback.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] User terminal: The user enters the username and password and clicks the login button.

[0845] Step 2:

[0846] Terminal: Sends the entered authentication information to the server.

[0847] Step 3:

[0848] Server: Compares the received authentication information with a database and authenticates the user.

[0849] Step 4:

[0850] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[0851] Step 5:

[0852] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[0853] Step 6:

[0854] User device: The user uses a microphone to input voice spoken in English.

[0855] Step 7:

[0856] Terminal: Sends audio data captured by the microphone to the server.

[0857] Step 8:

[0858] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[0859] Step 9:

[0860] Server: Sends the converted text data to the generative AI model and generates a response.

[0861] Step 10:

[0862] Generative AI: Uses text data and past conversation data to understand the user's context and generate appropriate responses.

[0863] Step 11:

[0864] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[0865] Step 12:

[0866] Server: Sends voice data to the user terminal.

[0867] Step 13:

[0868] User terminal: Receives the voice data and plays back the response from the speaker.

[0869] Step 14:

[0870] Generative AI: Detects user pronunciation and grammar errors in real time.

[0871] Step 15:

[0872] Generative AI: Generates appropriate corrections and feedback when errors are detected.

[0873] Step 16:

[0874] Server: Sends corrective feedback to the user terminal.

[0875] Step 17:

[0876] User terminal: Corrective feedback is played back and provided to the user.

[0877] Step 18:

[0878] Server: After the conversation ends, generate a summary report based on the entire history.

[0879] Step 19:

[0880] Server: Sends the generated summary report to the user terminal.

[0881] Step 20:

[0882] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[0883] Example 1

[0884] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0885] Conventional voice dialogue systems have difficulty accurately converting user input speech into text data and generating appropriate responses. They also struggle to generate responses that fully understand past conversation data and context, making it impossible to provide users with a natural, personalized conversation experience. Furthermore, they lack the ability to correct pronunciation and grammar errors in real time and provide feedback to users. This reduces the user's learning effectiveness and the quality of their conversation experience.

[0886] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0887] In this invention, the server includes means for encrypting authentication information entered by the user and transmitting it to the server, means for loading the user's profile information if authentication is successful, and means for asynchronously processing voice data received from the user via the terminal. This allows for the generation of appropriate responses based on the user's voice input, enabling personalized responses that take into account past conversation data and context. Furthermore, pronunciation and grammar errors can be corrected in real time and feedback can be provided to improve the user's learning effect. Furthermore, the use of a cloud-based speech recognition and speech synthesis engine enables highly accurate voice processing, providing the user with a natural, high-quality conversation experience.

[0888] "User authentication information" is an identifier used by a user when accessing a system, and is information that mainly consists of a user name and password.

[0889] "Voice input" refers to the act of a user speaking to the system using a microphone, or the voice data acquired thereby.

[0890] A "voice recognition engine" is software or algorithm that converts captured voice data into text data.

[0891] A "generative model" is an algorithm or system that uses natural language processing to generate appropriate responses based on input text data.

[0892] "Voice data" is data input by voice that is expressed in digital format and is used for voice recognition and voice synthesis.

[0893] "Past conversation data" is historical information about conversations the user has had in the past, and is used as basic data for analysis and response generation.

[0894] "Context" is information used to understand the content, flow, and context of a conversation, and is an important element for a generative model to provide an appropriate response.

[0895] "Real-time feedback" refers to the process of providing instant feedback to users immediately after they speak, immediately pointing out and correcting pronunciation and grammar errors.

[0896] "Profile information" refers to information such as a user's individual settings, past usage history, conversation history, etc., and is used to provide personalized services.

[0897] A "cloud-based engine" is a processing engine for speech recognition, speech synthesis, etc. that is provided via the Internet on a cloud service.

[0898] This invention is a system that allows a user to input speech and generates and provides an appropriate response using a generative AI model. The system includes a user terminal, a server, a generative AI model, and a series of databases and algorithms that support the model.

[0899] User Authentication and Login

[0900] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal encrypts this authentication information and sends it to the server. The server authenticates the user by checking it against a database. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[0901] Voice input

[0902] The user inputs voice using a microphone installed on the user device. The device temporarily stores the acquired voice data and then transmits it to the server.

[0903] Voice Recognition

[0904] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into text data, which is temporarily stored on the server.

[0905] Conversation Generation

[0906] The server inputs this text data into a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response based on the context and past conversation data. The generated response text is temporarily stored on the server.

[0907] Audio Output

[0908] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. The converted voice data is sent to the user's device, and the user receives the response by voice.

[0909] Feedback and Error Correction

[0910] During the conversation, the generative AI model detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error in "goed" and generates text correcting it to "went." The server generates corrective feedback, converts it into audio data, and sends it to the user's device.

[0911] Generate a summary report

[0912] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[0913] Specific examples

[0914] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation. In this way, users are provided with an environment where they can practice English conversation without feeling stressed and while receiving appropriate feedback.

[0915] Prompt Sentence Examples

[0916] "Hello, I'd like to practice my English."

[0917] This allows users to continue the conversation in real time and receive real-time feedback on pronunciation and grammar errors, which is expected to improve users' English conversation skills.

[0918] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0919] Step 1: User authentication and login

[0920] The user uses a dedicated app or web browser on the user device (PC or smartphone) to enter their username and password and click the login button.

[0921] Input: Username, Password

[0922] Specific behavior:

[0923] The terminal encrypts the entered authentication information and sends it to the server using the HTTPS protocol.

[0924] The server authenticates the user by checking the received authentication information against a database (e.g., MySQL).

[0925] Output: Authentication result (success or failure)

[0926] If authentication is successful, the server creates a session and loads the user's profile information into memory.

[0927] Step 2: Voice Input

[0928] The user speaks into a microphone installed in the user terminal.

[0929] Input: Audio data

[0930] Specific behavior:

[0931] The terminal temporarily stores the audio data acquired from the microphone.

[0932] The device asynchronously transmits the stored audio data to the server.

[0933] Output: Audio data (binary format)

[0934] Step 3: Voice Recognition

[0935] The server passes the received voice data to a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data.

[0936] Input: Audio data (binary format)

[0937] Specific behavior:

[0938] The server sends the audio data as an API request.

[0939] The voice recognition engine analyzes the voice data and converts it into text data.

[0940] Output: Text data (natural language text)

[0941] The converted text data is temporarily stored on the server.

[0942] Step 4: Conversation generation

[0943] The server inputs the text data generated by speech recognition into a generative AI model (e.g., OpenAI GPT-3).

[0944] Input: Text data (natural language text)

[0945] Specific behavior:

[0946] The server sends the text data to the generative AI model.

[0947] The generative AI model takes into account past conversation data and context to generate appropriate responses.

[0948] Output: Response text data (natural language text)

[0949] The generated response text is temporarily stored on the server.

[0950] Step 5: Audio Output

[0951] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data.

[0952] Input: Response text data (natural language text)

[0953] Specific behavior:

[0954] The server passes the response text to the speech synthesis engine.

[0955] The speech synthesis engine analyzes the text data and converts it into speech data.

[0956] Output: Audio data (audio file)

[0957] The generated voice data is transmitted to the user terminal and returned to the user.

[0958] Step 6: Feedback and Error Correction

[0959] During the conversation, the generative AI model detects the user's pronunciation and grammar errors and corrects them in real time.

[0960] Input: User utterance text data

[0961] Specific behavior:

[0962] The server analyzes the speech using a generative AI model and detects pronunciation and grammatical errors.

[0963] If an error is detected, the AI ​​generates corrected text data.

[0964] Output: Corrective feedback text (natural language text)

[0965] The server passes the corrective feedback to a speech synthesis engine, converts it into speech data, and provides the feedback to the user.

[0966] Step 7: Generate a summary report

[0967] After the conversation ends, the server generates a summary report based on the entire conversation history.

[0968] Input: Text data of conversation history

[0969] Specific behavior:

[0970] The server analyzes the conversation history and extracts pronunciation and grammatical errors and areas for improvement.

[0971] Generate summary reports and collate details.

[0972] Output: Summary report (text or PDF)

[0973] The summary report is sent to the user terminal so that the user can view it.

[0974] (Application example 1)

[0975] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0976] In conventional food delivery apps, voice recognition is often inaccurate when users input their order details, and menu recommendations are text-based, making them less convenient. Furthermore, it is difficult to provide a personalized user experience. Therefore, there was a need for a system that allows users to easily order by voice and can use a generative AI model to provide appropriate menu recommendations via voice.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0978] In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammar errors, means for generating a conversation summary report after the conversation ends, means for causing a generative AI model to generate prompts based on the user's order and recommending appropriate menu items, and means for providing the generated recommended menu items to the user as speech data. This allows a user to place an order by voice, and the generative AI to suggest appropriate recommended menu items by voice.

[0979] "Means for inputting voice" refers to a device or interface that takes in what the user has said and recognizes it as voice data.

[0980] "Means for converting into text data" refers to technology that analyzes voice data and converts its contents into text information.

[0981] "Means for generating a response using a generative model" refers to a method for creating an appropriate response based on input text data using a pre-trained artificial intelligence model.

[0982] "Means for converting into voice data" refers to a technology that analyzes text data and generates its contents as voice.

[0983] "Means for referencing past conversation data using user authentication information" refers to a system that searches and refers to the history of past conversations based on user authentication information.

[0984] "Means for generating a response based on the current context" refers to a technology that understands the current conversation content and situation and creates a response that matches that context.

[0985] The "means for returning voice data to the user" is a mechanism for transmitting the generated voice data to the user in an appropriate manner and for the user to receive it.

[0986] The "means for detecting and correcting pronunciation and grammar errors" refers to a technology that analyzes the user's speech and detects and corrects incorrect pronunciation and grammar.

[0987] The "means for generating a summary report of a conversation" is a technique for summarizing the contents of a conversation after it has ended and creating a report that includes errors and areas for improvement.

[0988] "Means for generating prompts" refers to technology that creates appropriate inputs for a generative AI model based on the user's order, enabling the AI ​​to generate appropriate answers.

[0989] The "means for providing recommended menus" is a system that presents via voice the dishes and products suggested by the generative AI model based on the user's requests.

[0990] MODE FOR CARRYING OUT THE INVENTION

[0991] The present invention is a system for food delivery applications that allows users to place orders using voice commands and provides appropriate menu recommendations using a generative AI model. The system includes a user terminal, a server, a generative AI, and a series of supporting databases and algorithms.

[0992] User voice input

[0993] The user inputs the order details by voice using a microphone installed on the user terminal such as a smartphone, and the user terminal transmits this voice data to the server.

[0994] Speech recognition and text conversion

[0995] The server receives the voice data and converts it into text data using a speech recognition engine (for example, the SpeechRecognition library). This text data is used for further processing.

[0996] Response generation by generative AI

[0997] The server sends the text data to a generative AI model, which generates an appropriate response based on the user's order. The generative AI model (e.g., OpenAI API) generates a response based on past conversation data and the current context.

[0998] Menu recommendations

[0999] Based on the user's order, the generative AI model generates a prompt and provides appropriate menu recommendations. For example, this prompt might look like this:

[1000] The user specifies "pizza and cola" as the menu item they would like to order, and asks for recommended side dishes.

[1001] An example of a prompt that the AI ​​generates is as follows:

[1002] Recommended side dishes include garlic bread and potato salad.

[1003] Audio Output

[1004] The generated response text is sent by the server to a speech synthesis engine (e.g., pyttsx3) and converted into voice data, which is then sent back to the user's terminal, where the user receives the response by voice.

[1005] Error Detection and Feedback

[1006] The server detects and corrects pronunciation and grammar errors in real time during conversations, providing audible feedback to the user, such as "You should say 'I went to the park'."

[1007] Conversation Summary Report

[1008] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1009] In this way, the present invention is a system that allows users to easily place orders by voice, and the generation AI suggests appropriate recommended menu items by voice.

[1010] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1011] Step 1:

[1012] The user inputs the order details by voice into a user device such as a smartphone. Specifically, the voice data is acquired using the microphone of the user device. The input is the user's voice data, and the output is that voice data.

[1013] Step 2:

[1014] The user terminal transmits the acquired voice data to the server. The input is voice data, and the output is the transmission of voice data to the server.

[1015] Step 3:

[1016] The server uses a speech recognition engine to convert the voice data into text data. Specifically, the server analyzes the voice data using the SpeechRecognition library. The input is voice data and the output is text data.

[1017] Step 4:

[1018] The server sends the text data to the generative AI model and generates an appropriate response based on the user's order. Specifically, the server calls the OpenAI API and uses the generative AI model to create a response. The input is text data, and the output is the generated response text.

[1019] Step 5:

[1020] The server generates a prompt from the generative AI model and provides appropriate menu recommendations. The server generates a prompt based on the user's order and obtains suggestions from the generative AI model. The input is the user's order text and prompt text, and the output is a response text of recommended menu items.

[1021] Step 6:

[1022] The server sends the generated response text to the speech synthesis engine and converts it into speech data. Specifically, the server converts text to speech using the pyttsx3 library. The input is the response text data, and the output is speech data.

[1023] Step 7:

[1024] The server returns the converted voice data to the user terminal. The input is the voice data, and the output is the transmission of the voice data to the user terminal.

[1025] Step 8:

[1026] The server detects pronunciation and grammar errors in real time and corrects them. Specifically, the server analyzes the user's speech and provides feedback as needed. The input is the user's voice data, and the output is text or voice data of corrective feedback.

[1027] Step 9:

[1028] When the conversation ends, the server generates a summary report based on the entire conversation history. Specifically, the server extracts errors and areas for improvement from the conversation history and creates a report. The input is the conversation history data, and the output is the text data of the summary report.

[1029] Step 10:

[1030] The server sends the generated summary report to the user terminal. The input is the summary report text data, and the output is the report data sent to the user terminal.

[1031] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1032] This invention is a system in which a user inputs speech, a generative AI generates and provides an appropriate response, and an emotion engine recognizes the user's emotional state and responds accordingly. This system includes a user terminal, a server, a generative AI, an emotion engine, and a series of supporting databases and algorithms.

[1033] User Authentication and Login

[1034] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server generates a session ID and user profile information and sends them back to the user terminal. This allows the user's past conversation data and individual settings to be inherited.

[1035] Voice input and speech generation

[1036] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[1037] Emotion Recognition and Voice Output

[1038] The server sends the text data to an emotion engine to analyze the user's emotional state. The generated response text is adjusted by the emotion engine, and the tone and content are changed according to the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The generated response text is sent by the server to a speech synthesis engine and converted into voice data. The voice data is sent back to the user's device, allowing the user to receive the response by voice.

[1039] Feedback and Error Correction

[1040] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[1041] Generate a summary report

[1042] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1043] Specific examples

[1044] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The emotion engine analyzes the user's emotional state and adjusts the tone and content of the response as needed. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[1045] In this way, users are provided with an environment where they can practice English conversation without feeling nervous and receive appropriate feedback. Furthermore, by utilizing an emotion engine, responses are provided that take into account the user's emotional state.

[1046] The processing flow will be explained below.

[1047] Step 1:

[1048] User terminal: The user enters the username and password and clicks the login button.

[1049] Step 2:

[1050] Terminal: Sends the entered authentication information to the server.

[1051] Step 3:

[1052] Server: Compares the received authentication information with a database and authenticates the user.

[1053] Step 4:

[1054] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[1055] Step 5:

[1056] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[1057] Step 6:

[1058] User device: The user uses a microphone to input voice spoken in English.

[1059] Step 7:

[1060] Terminal: Sends audio data captured by the microphone to the server.

[1061] Step 8:

[1062] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[1063] Step 9:

[1064] Server: The converted text data is sent to the emotion engine to analyze the user's emotional state.

[1065] Step 10:

[1066] Emotion engine: Based on the analysis results, it recognizes the user's emotional state. For example, it analyzes the tone and pace of the user's voice to determine whether they are happy or stressed.

[1067] Step 11:

[1068] Server: Receives the analysis results from the emotion engine and sends them to the generative AI model.

[1069] Step 12:

[1070] Generative AI: It uses text data, sentiment analysis results, and past conversation data to understand the context and generate responses that are tailored to the user's emotional state.

[1071] Step 13:

[1072] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[1073] Step 14:

[1074] Server: Sends voice data to the user terminal.

[1075] Step 15:

[1076] User terminal: Receives the voice data and plays back the response from the speaker.

[1077] Step 16:

[1078] Generative AI: Detects user pronunciation and grammar errors in real time during conversation.

[1079] Step 17:

[1080] Generative AI: If an error is detected, it generates appropriate corrections and feedback, for example, correcting the statement "I went to the park" to "I went to the park."

[1081] Step 18:

[1082] Server: Sends corrective feedback to the user terminal.

[1083] Step 19:

[1084] User terminal: Corrective feedback is played back and provided to the user.

[1085] Step 20:

[1086] Server: After the conversation ends, a summary report is generated based on the entire history, including details about pronunciation and grammar errors made during the conversation, areas for improvement, and the user's emotional state.

[1087] Step 21:

[1088] Server: Sends the generated summary report to the user terminal.

[1089] Step 22:

[1090] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[1091] Example 2

[1092] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1093] Conventional speech recognition systems and response generation systems using generative models have difficulty generating appropriate responses that take the user's emotional state into account. Furthermore, they lack the functionality to correct users' pronunciation and grammar errors in real time and provide feedback, which reduces learning efficiency. Furthermore, there is no way to visualize the user's progress after the conversation and suggest areas for improvement, making it difficult to continuously improve the user's learning experience.

[1094] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammatical errors, means for generating a summary report of the conversation after the conversation ends, and means for recognizing the user's emotional state and adjusting the content of the response accordingly. This makes it possible to generate appropriate responses according to the user's emotional state, correct pronunciation and grammatical errors in real time, and visualize the user's learning progress.

[1095] "Means for user voice input" refers to a device or function that allows a user to transmit voice information to the system.

[1096] The "means for converting the voice into text data" refers to a device or function that analyzes input voice information and converts it into corresponding text data.

[1097] "Means for generating a response based on the text data using a generative model" refers to a device or function that utilizes a generative AI model to generate an appropriate response based on the text data.

[1098] "Means for converting said response into voice data" refers to a device or function that converts the generated text-based response into voice data.

[1099] The "means for referencing past conversation data using user authentication information" refers to a device or function that uses user authentication information to access and refer to conversation data that has taken place with a user in the past.

[1100] The "means for generating a response based on the past conversation data and the current context" refers to a device or function that generates an appropriate response by taking into account the past conversation data and the current conversation context.

[1101] The "means for returning the voice data to the user" refers to a device or function for transmitting the generated voice data to the user's terminal and letting the user hear it.

[1102] The "means for detecting and correcting pronunciation and grammar errors" refers to a device or function that identifies pronunciation and grammar errors contained in the user's speech and corrects them to the correct form.

[1103] "Means for generating a summary report of a conversation after the conversation has ended" refers to a device or function that, after the conversation has ended, summarizes the content of the conversation and generates a report that includes information such as pronunciation and grammatical errors, areas for improvement, and emotional state.

[1104] The "means for recognizing the user's emotional state and adjusting the content of the reply accordingly" refers to a device or function that analyzes the user's emotional state and adjusts the tone and content of the reply based on the results of the analysis.

[1105] This invention is a system for generating appropriate responses using user voice data, utilizing a generative AI model and an emotion recognition engine, and improving the user's learning experience. This system consists of a user terminal, a server, and various related engines and databases.

[1106] User Authentication and Login

[1107] A user logs in using their own device (e.g., PC or smartphone) by entering their username and password. The device sends the authentication information to the server, which checks it against a database. If authentication is successful, the server generates a session ID and user profile information and sends them back to the device. This allows the user to retain their settings and past conversation data.

[1108] Voice input and speech generation

[1109] The user inputs voice using the microphone on the device. The device then sends the captured voice data to the server. The server then sends the voice data to a speech recognition engine, which converts it into text data. This text data is then sent to a generative AI model, which generates an appropriate response based on the context and past conversation data. This generative AI model can be an open-source model or a commercial model provided by a company.

[1110] Emotion Recognition and Voice Output

[1111] The server sends the generated text data to an emotion engine to analyze the user's emotional state. The emotion engine identifies the user's emotion and adjusts the tone and content of the response accordingly. For example, if it determines that the user is feeling stressed, the response will be adjusted to a calmer, more encouraging tone. This adjusted response text is sent to a speech synthesis engine and converted into voice data. This voice data is sent back to the user's device via the server, allowing the user to receive the response via voice.

[1112] Feedback and Error Correction

[1113] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI detects the error and generates text correcting it to "went." The server then sends the corrective feedback to the user's device, providing audio feedback such as, "You should say 'I went to the park'."

[1114] Generate a summary report

[1115] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can view it to understand their learning progress and areas for improvement.

[1116] Specific examples

[1117] For example, if a user says, "Hello, I would like to practice English," the voice data is sent from the device to the server, where it is converted by the speech recognition engine into text data: "Hello, I would like to practice English." The generative AI model uses this text data to generate a response: "Sure, let's get started!" The emotion engine analyzes the user's emotional state and adjusts the tone and content as necessary. The speech synthesis engine then converts this response into voice data and sends it back to the user's device. Hearing the voice, "Sure, let's get started!", the user can continue practicing English conversation with peace of mind.

[1118] The system allows users to receive real-time feedback as they learn, providing a better learning experience through appropriate responses based on their emotional state.

[1119] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1120] Step 1: User authentication and login

[1121] The user enters their username and password using their own device (PC or smartphone) and clicks the login button. The device sends this authentication information to the server. The server compares the received authentication information with the database, and if the user is authenticated, generates a session ID and user profile information and sends these back to the device. This allows the user to retain their settings and past conversation data.

[1122] Input: Username, Password

[1123] Output: Session ID, user profile

[1124] Step 2: Receiving Audio Input

[1125] The user inputs speech using the device's microphone, which converts the speech data into a digital format and sends it to the server, which then sends it to a speech recognition engine for analysis.

[1126] Input: Audio data

[1127] Output: Digital audio data

[1128] Step 3: Convert audio data to text

[1129] The server converts the received voice data into text data using a speech recognition engine, which analyzes the voice waveform and generates corresponding text, which is then sent to the generative AI model.

[1130] Input: Digital audio data

[1131] Output: Text data

[1132] Step 4: Generative AI model generates a response

[1133] The server sends the generated text data to the generative AI model, which generates an appropriate response based on the text data, past conversation data, and the current context. The generated response text is then sent to the emotion engine.

[1134] Input: Text data, past conversation data, contextual information

[1135] Output: Response text

[1136] Step 5: Response tailoring with the emotion engine

[1137] The server sends the generated response text to the emotion engine, which analyzes the user's emotional state. The emotion engine adjusts the tone and content of the response based on the user's emotional state. For example, if the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The adjusted response text is then sent to the speech synthesis engine.

[1138] Input: Reply text, emotional state

[1139] Output: Adjusted response text

[1140] Step 6: Speech conversion using a speech synthesis engine

[1141] The server sends the adjusted response text to a speech synthesis engine, which converts it into voice data. The speech synthesis engine generates natural-sounding voice based on the text data. This voice data is then sent back to the user's device via the server.

[1142] Input: Adjusted response text

[1143] Output: Reply audio data

[1144] Step 7: Provide a response

[1145] The user terminal receives the returned voice data and provides it to the user through a speaker, so that the user receives a voice response.

[1146] Input: Response audio data

[1147] Output: Audio output

[1148] Step 8: Error detection and feedback

[1149] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI model detects the error in "goed" and generates a corrected text to change it to "went." The server then sends the corrected text feedback to the user's device and provides audio feedback.

[1150] Input: Raw audio data, speech content

[1151] Output: Corrective feedback text, audio feedback

[1152] Step 9: Generate a summary report

[1153] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors and areas for improvement during the conversation, as well as the user's emotional state. The server then sends the summary report to the user's device, allowing the user to review it and understand their learning progress.

[1154] Input: conversation history

[1155] Output: Summary report

[1156] (Application example 2)

[1157] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] During the food delivery ordering process, users face challenges in effectively and quickly selecting meals that suit their needs and emotions. Current systems are unable to provide personalized recommendations that take into account the user's emotional state, potentially resulting in lower customer satisfaction. Furthermore, the lack of real-time correction and feedback for pronunciation and grammar errors hinders smooth user experience.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input speech, a means for converting the speech into text data, a means for generating a response based on the text data using a generative model, a means for converting the response into speech data, a means for referencing past conversation data using user authentication information, a means for generating a response based on the past conversation data and the current context, a means for returning the speech data to the user, a means for detecting and correcting pronunciation and grammar errors, a means for generating a summary report of the conversation after the conversation ends, a means for recognizing the user's emotional state and responding according to the emotion, and a means for suggesting food delivery-related services and products. This allows the user to receive personalized food delivery suggestions based on their emotional state, enabling the ordering process to be carried out quickly and efficiently. Furthermore, real-time correction of pronunciation and grammar errors is provided, allowing for a smooth user experience.

[1160] "Means for user voice input" refers to a function for acquiring voice data through an input device (e.g., a microphone).

[1161] The "means for converting voice into text data" is a function that converts voice data acquired using voice recognition technology into a corresponding text format.

[1162] "Means for generating responses based on text data using a generative model" refers to a function that uses generative AI to analyze input text data and generate appropriate responses.

[1163] The "means for converting a response into voice data" is a function that converts the generated text response into voice data using synthesized voice technology.

[1164] The "means for referencing past conversation data using user authentication information" is a function for retrieving and referencing past conversation history from a database using authentication information such as a user name and password.

[1165] The "means for generating a response based on past conversation data and current context" is a function for generating a personalized response based on past conversation history and real-time context information.

[1166] The "means for returning voice data to the user" is a function that transmits the generated voice data to the user device, allowing the user to receive a voice response.

[1167] The "means for detecting and correcting pronunciation and grammatical errors" is a function that analyzes the content of the user's speech, detects pronunciation and grammatical errors, corrects them, and provides feedback.

[1168] The "means for generating a summary report of a conversation after the conversation has ended" is a function that compiles the conversation history with the user and generates a report that includes pronunciation and grammatical errors, emotional state, and so on.

[1169] "Means for recognizing the user's emotional state and responding accordingly" refers to a function that uses emotion recognition technology to analyze the user's emotional state and adjusts the content and tone of the response based on this.

[1170] "Means for suggesting food delivery-related services and products" refers to a function that recommends appropriate food delivery services and products based on the user's emotions and needs.

[1171] To implement this invention, multiple technical components are required: a user terminal, a server, a speech recognition engine, a generative AI model, an emotion engine, and a speech synthesis engine. A user places a food delivery order by voice. The server processes this voice data and provides a system that generates personalized responses based on the user's emotional state.

[1172] 1. User authentication and login

[1173] Users log in by entering their username and password on a device such as a smartphone. This authentication information is sent to the server and checked against a database. If authentication is successful, the server retrieves past conversation data and user profile information and returns it to the user's device, starting the session.

[1174] 2. Voice to Text

[1175] Users input their voice using the microphone on their smartphone. This voice data is sent to the server and converted into text data through a speech recognition engine. Speech recognition engines such as Google Speech-to-Text and Microsoft Azure Speech can be used.

[1176] 3. Response generation using generative AI models

[1177] The server sends the acquired text data to a generative AI model, which can use models such as GPT-3 or ChatGPT, to generate an appropriate response.

[1178] 4. Emotion recognition and response regulation

[1179] The generated text response is sent by the server to an emotion engine, which analyzes the user's emotional state (using a model like Hugging Face's emotion analysis model), and adjusts the content and tone of the response based on the user's emotional state.

[1180] 5. Speech synthesis and response return

[1181] The adjusted response text is sent to a speech synthesis engine, which converts it into voice data. The speech synthesis engine can be Amazon Polly or Google Text-to-Speech. This voice data is sent back to the user's device, where the user can receive a voice response.

[1182] 6. Food delivery proposal

[1183] The server will suggest suitable food delivery options to the user based on their emotional state and past ordering history. For example, if a user says, "I'm tired today," the system will suggest, "Would you like hot soup or pizza?"

[1184] 7. Summary report generation

[1185] After the conversation ends, the server generates a summary report based on the entire conversation history. This summary report includes any pronunciation or grammar errors detected during the conversation, information about the user's emotional state, and a history of food delivery suggestions. This report is sent to the user's device, where the user can review their conversation history.

[1186] Specific examples

[1187] For example, if a user asks, "I'm tired today, what would you recommend for dinner?", the server processes the voice data and uses an emotion engine to identify the emotion "tired." It then generates a response, "Would you like some hot soup or pizza?", and delivers it to the user as speech.

[1188] Prompt Sentence Examples

[1189] When a user asks, "I'm tired today, what food would you recommend?", your smart food assistant should use its emotion engine to identify the emotion "tired" and respond with, "You seem tired today. Would you like some hot soup or pizza to help you relax?"

[1190] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1191] Step 1:

[1192] The user inputs voice through the smartphone microphone. This input is the user's action of making a request or asking a question for food delivery by voice. For example, the user might say, "I'm tired today, so what meal would you recommend?"

[1193] Step 2:

[1194] The device collects voice data and sends it to the server. The data sent is raw voice data. In this step, the device does not perform any preprocessing for voice recognition.

[1195] Step 3:

[1196] The server sends the received voice data to a speech recognition engine. This engine converts the voice data into text data. For example, Google Speech-to-Text is used. In this step, the voice data is converted into text data such as "I'm tired today, so what meal would you recommend?"

[1197] Step 4:

[1198] The server sends the converted text data to a generative AI model, which then generates an appropriate response based on the text data. The input at this stage is the converted text data, and the output is the generated response text: "Would you like some warm soup or pizza to relax?"

[1199] Step 5:

[1200] The server sends the generated text response to the emotion engine, which analyzes the user's emotional state. In this case, based on the text data "I'm tired today...", the server analyzes that the user's state is "tired." Based on this input, the emotion engine adjusts the tone and content. The output is a response text with adjusted tone.

[1201] Step 6:

[1202] The server sends the tone-adjusted response text to a speech synthesis engine, which converts it into speech data. The speech synthesis engine (e.g., Amazon Polly) converts the text data into speech data. The output is the speech data, "Would you like some hot soup or pizza to relax?"

[1203] Step 7:

[1204] The server sends the generated voice data to the user terminal, and the user receives a voice response. The input is the adjusted voice data, and the output is the voice information that the user hears.

[1205] Step 8:

[1206] When the conversation ends, the server generates a summary report based on the conversation history and emotional state, including the user's comments, responses, and emotional analysis results. The server then sends this report to the user's device so that the user can review it.

[1207] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1208] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1209] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1210] [Fourth embodiment]

[1211] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1212] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1213] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1214] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1215] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1216] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1217] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1218] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1219] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1220] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1221] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1222] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1223] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] This system allows users to input speech and uses a generative AI to generate and provide appropriate responses. The system includes a user terminal, a server, a generative AI, and a series of databases and algorithms that support the AI.

[1225] User Authentication and Login

[1226] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[1227] Voice input and speech generation

[1228] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[1229] Audio Output

[1230] The generated response text is sent to a speech synthesis engine by the server and converted into voice data. The voice data is then sent back to the user's device, and the user receives the response by voice, providing a natural conversational experience.

[1231] Feedback and Error Correction

[1232] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[1233] Generate a summary report

[1234] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1235] Specific examples

[1236] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[1237] In this way, users are provided with an environment in which they can practice English conversation without feeling nervous and while receiving appropriate feedback.

[1238] The processing flow will be explained below.

[1239] Step 1:

[1240] User terminal: The user enters the username and password and clicks the login button.

[1241] Step 2:

[1242] Terminal: Sends the entered authentication information to the server.

[1243] Step 3:

[1244] Server: Compares the received authentication information with a database and authenticates the user.

[1245] Step 4:

[1246] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[1247] Step 5:

[1248] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[1249] Step 6:

[1250] User device: The user uses a microphone to input voice spoken in English.

[1251] Step 7:

[1252] Terminal: Sends audio data captured by the microphone to the server.

[1253] Step 8:

[1254] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[1255] Step 9:

[1256] Server: Sends the converted text data to the generative AI model and generates a response.

[1257] Step 10:

[1258] Generative AI: Uses text data and past conversation data to understand the user's context and generate appropriate responses.

[1259] Step 11:

[1260] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[1261] Step 12:

[1262] Server: Sends voice data to the user terminal.

[1263] Step 13:

[1264] User terminal: Receives the voice data and plays back the response from the speaker.

[1265] Step 14:

[1266] Generative AI: Detects user pronunciation and grammar errors in real time.

[1267] Step 15:

[1268] Generative AI: Generates appropriate corrections and feedback when errors are detected.

[1269] Step 16:

[1270] Server: Sends corrective feedback to the user terminal.

[1271] Step 17:

[1272] User terminal: Corrective feedback is played back and provided to the user.

[1273] Step 18:

[1274] Server: After the conversation ends, generate a summary report based on the entire history.

[1275] Step 19:

[1276] Server: Sends the generated summary report to the user terminal.

[1277] Step 20:

[1278] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[1279] Example 1

[1280] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1281] Conventional voice dialogue systems have difficulty accurately converting user input speech into text data and generating appropriate responses. They also struggle to generate responses that fully understand past conversation data and context, making it impossible to provide users with a natural, personalized conversation experience. Furthermore, they lack the ability to correct pronunciation and grammar errors in real time and provide feedback to users. This reduces the user's learning effectiveness and the quality of their conversation experience.

[1282] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1283] In this invention, the server includes means for encrypting authentication information entered by the user and transmitting it to the server, means for loading the user's profile information if authentication is successful, and means for asynchronously processing voice data received from the user via the terminal. This allows for the generation of appropriate responses based on the user's voice input, enabling personalized responses that take into account past conversation data and context. Furthermore, pronunciation and grammar errors can be corrected in real time and feedback can be provided to improve the user's learning effect. Furthermore, the use of a cloud-based speech recognition and speech synthesis engine enables highly accurate voice processing, providing the user with a natural, high-quality conversation experience.

[1284] "User authentication information" is an identifier used by a user when accessing a system, and is information that mainly consists of a user name and password.

[1285] "Voice input" refers to the act of a user speaking to the system using a microphone, or the voice data acquired thereby.

[1286] A "voice recognition engine" is software or algorithm that converts captured voice data into text data.

[1287] A "generative model" is an algorithm or system that uses natural language processing to generate appropriate responses based on input text data.

[1288] "Voice data" is data input by voice that is expressed in digital format and is used for voice recognition and voice synthesis.

[1289] "Past conversation data" is historical information about conversations the user has had in the past, and is used as basic data for analysis and response generation.

[1290] "Context" is information used to understand the content, flow, and context of a conversation, and is an important element for a generative model to provide an appropriate response.

[1291] "Real-time feedback" refers to the process of providing instant feedback to users immediately after they speak, immediately pointing out and correcting pronunciation and grammar errors.

[1292] "Profile information" refers to information such as a user's individual settings, past usage history, conversation history, etc., and is used to provide personalized services.

[1293] A "cloud-based engine" is a processing engine for speech recognition, speech synthesis, etc. that is provided via the Internet on a cloud service.

[1294] This invention is a system that allows a user to input speech and generates and provides an appropriate response using a generative AI model. The system includes a user terminal, a server, a generative AI model, and a series of databases and algorithms that support the model.

[1295] User Authentication and Login

[1296] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal encrypts this authentication information and sends it to the server. The server authenticates the user by checking it against a database. If authentication is successful, the server creates a session and loads the user's profile information. This allows the user's past conversation data and individual settings to be carried over.

[1297] Voice input

[1298] The user inputs voice using a microphone installed on the user device. The device temporarily stores the acquired voice data and then transmits it to the server.

[1299] Voice Recognition

[1300] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts it into text data, which is temporarily stored on the server.

[1301] Conversation Generation

[1302] The server inputs this text data into a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response based on the context and past conversation data. The generated response text is temporarily stored on the server.

[1303] Audio Output

[1304] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. The converted voice data is sent to the user's device, and the user receives the response by voice.

[1305] Feedback and Error Correction

[1306] During the conversation, the generative AI model detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error in "goed" and generates text correcting it to "went." The server generates corrective feedback, converts it into audio data, and sends it to the user's device.

[1307] Generate a summary report

[1308] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1309] Specific examples

[1310] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation. In this way, users are provided with an environment where they can practice English conversation without feeling stressed and while receiving appropriate feedback.

[1311] Prompt Sentence Examples

[1312] "Hello, I'd like to practice my English."

[1313] This allows users to continue the conversation in real time and receive real-time feedback on pronunciation and grammar errors, which is expected to improve users' English conversation skills.

[1314] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1315] Step 1: User authentication and login

[1316] The user uses a dedicated app or web browser on the user device (PC or smartphone) to enter their username and password and click the login button.

[1317] Input: Username, Password

[1318] Specific behavior:

[1319] The terminal encrypts the entered authentication information and sends it to the server using the HTTPS protocol.

[1320] The server authenticates the user by checking the received authentication information against a database (e.g., MySQL).

[1321] Output: Authentication result (success or failure)

[1322] If authentication is successful, the server creates a session and loads the user's profile information into memory.

[1323] Step 2: Voice Input

[1324] The user speaks into a microphone installed in the user terminal.

[1325] Input: Audio data

[1326] Specific behavior:

[1327] The terminal temporarily stores the audio data acquired from the microphone.

[1328] The device asynchronously transmits the stored audio data to the server.

[1329] Output: Audio data (binary format)

[1330] Step 3: Voice Recognition

[1331] The server passes the received voice data to a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data.

[1332] Input: Audio data (binary format)

[1333] Specific behavior:

[1334] The server sends the audio data as an API request.

[1335] The voice recognition engine analyzes the voice data and converts it into text data.

[1336] Output: Text data (natural language text)

[1337] The converted text data is temporarily stored on the server.

[1338] Step 4: Conversation generation

[1339] The server inputs the text data generated by speech recognition into a generative AI model (e.g., OpenAI GPT-3).

[1340] Input: Text data (natural language text)

[1341] Specific behavior:

[1342] The server sends the text data to the generative AI model.

[1343] The generative AI model takes into account past conversation data and context to generate appropriate responses.

[1344] Output: Response text data (natural language text)

[1345] The generated response text is temporarily stored on the server.

[1346] Step 5: Audio Output

[1347] The server sends the generated response text to a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data.

[1348] Input: Response text data (natural language text)

[1349] Specific behavior:

[1350] The server passes the response text to the speech synthesis engine.

[1351] The speech synthesis engine analyzes the text data and converts it into speech data.

[1352] Output: Audio data (audio file)

[1353] The generated voice data is transmitted to the user terminal and returned to the user.

[1354] Step 6: Feedback and Error Correction

[1355] During the conversation, the generative AI model detects the user's pronunciation and grammar errors and corrects them in real time.

[1356] Input: User utterance text data

[1357] Specific behavior:

[1358] The server analyzes the speech using a generative AI model and detects pronunciation and grammatical errors.

[1359] If an error is detected, the AI ​​generates corrected text data.

[1360] Output: Corrective feedback text (natural language text)

[1361] The server passes the corrective feedback to a speech synthesis engine, converts it into speech data, and provides the feedback to the user.

[1362] Step 7: Generate a summary report

[1363] After the conversation ends, the server generates a summary report based on the entire conversation history.

[1364] Input: Text data of conversation history

[1365] Specific behavior:

[1366] The server analyzes the conversation history and extracts pronunciation and grammatical errors and areas for improvement.

[1367] Generate summary reports and collate details.

[1368] Output: Summary report (text or PDF)

[1369] The summary report is sent to the user terminal so that the user can view it.

[1370] (Application example 1)

[1371] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1372] In conventional food delivery apps, voice recognition is often inaccurate when users input their order details, and menu recommendations are text-based, making them less convenient. Furthermore, it is difficult to provide a personalized user experience. Therefore, there was a need for a system that allows users to easily order by voice and can use a generative AI model to provide appropriate menu recommendations via voice.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1374] In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammar errors, means for generating a conversation summary report after the conversation ends, means for causing a generative AI model to generate prompts based on the user's order and recommending appropriate menu items, and means for providing the generated recommended menu items to the user as speech data. This allows a user to place an order by voice, and the generative AI to suggest appropriate recommended menu items by voice.

[1375] "Means for inputting voice" refers to a device or interface that takes in what the user has said and recognizes it as voice data.

[1376] "Means for converting into text data" refers to technology that analyzes voice data and converts its contents into text information.

[1377] "Means for generating a response using a generative model" refers to a method for creating an appropriate response based on input text data using a pre-trained artificial intelligence model.

[1378] "Means for converting into voice data" refers to a technology that analyzes text data and generates its contents as voice.

[1379] "Means for referencing past conversation data using user authentication information" refers to a system that searches and refers to the history of past conversations based on user authentication information.

[1380] "Means for generating a response based on the current context" refers to a technology that understands the current conversation content and situation and creates a response that matches that context.

[1381] The "means for returning voice data to the user" is a mechanism for transmitting the generated voice data to the user in an appropriate manner and for the user to receive it.

[1382] The "means for detecting and correcting pronunciation and grammar errors" refers to a technology that analyzes the user's speech and detects and corrects incorrect pronunciation and grammar.

[1383] The "means for generating a summary report of a conversation" is a technique for summarizing the contents of a conversation after it has ended and creating a report that includes errors and areas for improvement.

[1384] "Means for generating prompts" refers to technology that creates appropriate inputs for a generative AI model based on the user's order, enabling the AI ​​to generate appropriate answers.

[1385] The "means for providing recommended menus" is a system that presents via voice the dishes and products suggested by the generative AI model based on the user's requests.

[1386] MODE FOR CARRYING OUT THE INVENTION

[1387] The present invention is a system for food delivery applications that allows users to place orders using voice commands and provides appropriate menu recommendations using a generative AI model. The system includes a user terminal, a server, a generative AI, and a series of supporting databases and algorithms.

[1388] User voice input

[1389] The user inputs the order details by voice using a microphone installed on the user terminal such as a smartphone, and the user terminal transmits this voice data to the server.

[1390] Speech recognition and text conversion

[1391] The server receives the voice data and converts it into text data using a speech recognition engine (for example, the SpeechRecognition library). This text data is used for further processing.

[1392] Response generation by generative AI

[1393] The server sends the text data to a generative AI model, which generates an appropriate response based on the user's order. The generative AI model (e.g., OpenAI API) generates a response based on past conversation data and the current context.

[1394] Menu recommendations

[1395] Based on the user's order, the generative AI model generates a prompt and provides appropriate menu recommendations. For example, this prompt might look like this:

[1396] The user specifies "pizza and cola" as the menu item they would like to order, and asks for recommended side dishes.

[1397] An example of a prompt that the AI ​​generates is as follows:

[1398] Recommended side dishes include garlic bread and potato salad.

[1399] Audio Output

[1400] The generated response text is sent by the server to a speech synthesis engine (e.g., pyttsx3) and converted into voice data, which is then sent back to the user's terminal, where the user receives the response by voice.

[1401] Error Detection and Feedback

[1402] The server detects and corrects pronunciation and grammar errors in real time during conversations, providing audible feedback to the user, such as "You should say 'I went to the park'."

[1403] Conversation Summary Report

[1404] Once the conversation is over, the server generates a summary report based on the entire conversation history. This summary report includes details of pronunciation and grammar errors made during the conversation, as well as areas for improvement. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1405] In this way, the present invention is a system that allows users to easily place orders by voice, and the generation AI suggests appropriate recommended menu items by voice.

[1406] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1407] Step 1:

[1408] The user inputs the order details by voice into a user device such as a smartphone. Specifically, the voice data is acquired using the microphone of the user device. The input is the user's voice data, and the output is that voice data.

[1409] Step 2:

[1410] The user terminal transmits the acquired voice data to the server. The input is voice data, and the output is the transmission of voice data to the server.

[1411] Step 3:

[1412] The server uses a speech recognition engine to convert the voice data into text data. Specifically, the server analyzes the voice data using the SpeechRecognition library. The input is voice data and the output is text data.

[1413] Step 4:

[1414] The server sends the text data to the generative AI model and generates an appropriate response based on the user's order. Specifically, the server calls the OpenAI API and uses the generative AI model to create a response. The input is text data, and the output is the generated response text.

[1415] Step 5:

[1416] The server generates a prompt from the generative AI model and provides appropriate menu recommendations. The server generates a prompt based on the user's order and obtains suggestions from the generative AI model. The input is the user's order text and prompt text, and the output is a response text of recommended menu items.

[1417] Step 6:

[1418] The server sends the generated response text to the speech synthesis engine and converts it into speech data. Specifically, the server converts text to speech using the pyttsx3 library. The input is the response text data, and the output is speech data.

[1419] Step 7:

[1420] The server returns the converted voice data to the user terminal. The input is the voice data, and the output is the transmission of the voice data to the user terminal.

[1421] Step 8:

[1422] The server detects pronunciation and grammar errors in real time and corrects them. Specifically, the server analyzes the user's speech and provides feedback as needed. The input is the user's voice data, and the output is text or voice data of corrective feedback.

[1423] Step 9:

[1424] When the conversation ends, the server generates a summary report based on the entire conversation history. Specifically, the server extracts errors and areas for improvement from the conversation history and creates a report. The input is the conversation history data, and the output is the text data of the summary report.

[1425] Step 10:

[1426] The server sends the generated summary report to the user terminal. The input is the summary report text data, and the output is the report data sent to the user terminal.

[1427] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1428] This invention is a system in which a user inputs speech, a generative AI generates and provides an appropriate response, and an emotion engine recognizes the user's emotional state and responds accordingly. This system includes a user terminal, a server, a generative AI, an emotion engine, and a series of supporting databases and algorithms.

[1429] User Authentication and Login

[1430] A user terminal is a device (e.g., PC, smartphone) that the user uses to input information. The user enters their username and password on the terminal and clicks the login button. The terminal sends this authentication information to the server. The server collates the username and password in a database and performs authentication. If authentication is successful, the server generates a session ID and user profile information and sends them back to the user terminal. This allows the user's past conversation data and individual settings to be inherited.

[1431] Voice input and speech generation

[1432] The user inputs speech using a microphone on the user's device. The device then transmits the captured speech data to a server. The server then sends the speech data to a speech recognition engine, which converts the speech into text. This text is then sent to a generative AI model, which generates an appropriate response based on the context and previous conversation data.

[1433] Emotion Recognition and Voice Output

[1434] The server sends the text data to an emotion engine to analyze the user's emotional state. The generated response text is adjusted by the emotion engine, and the tone and content are changed according to the user's emotional state. For example, if the emotion engine determines that the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The generated response text is sent by the server to a speech synthesis engine and converted into voice data. The voice data is sent back to the user's device, allowing the user to receive the response by voice.

[1435] Feedback and Error Correction

[1436] During the conversation, the AI ​​detects and corrects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the AI ​​detects the error and generates text correcting it to "went." The server sends the correction feedback to the user's device, and the user receives audible feedback such as, "You should say 'I went to the park'."

[1437] Generate a summary report

[1438] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can review it to understand their learning progress and areas for improvement.

[1439] Specific examples

[1440] For example, if a user types "Hello, I would like to practice English," the voice data is sent to the server and converted into text data by a speech recognition engine. The generative AI model generates the response "Sure, let's get started!" based on past conversation data and context. The emotion engine analyzes the user's emotional state and adjusts the tone and content of the response as needed. The speech synthesis engine converts this response into voice data and sends it back to the user's device. The user hears "Sure, let's get started!" and continues the conversation.

[1441] In this way, users are provided with an environment where they can practice English conversation without feeling nervous and receive appropriate feedback. Furthermore, by utilizing an emotion engine, responses are provided that take into account the user's emotional state.

[1442] The processing flow will be explained below.

[1443] Step 1:

[1444] User terminal: The user enters the username and password and clicks the login button.

[1445] Step 2:

[1446] Terminal: Sends the entered authentication information to the server.

[1447] Step 3:

[1448] Server: Compares the received authentication information with a database and authenticates the user.

[1449] Step 4:

[1450] Server: If authentication is successful, generates a session ID and user profile information and returns them to the user terminal.

[1451] Step 5:

[1452] User terminal: Receives the session ID and profile information and displays a successful login notification to the user.

[1453] Step 6:

[1454] User device: The user uses a microphone to input voice spoken in English.

[1455] Step 7:

[1456] Terminal: Sends audio data captured by the microphone to the server.

[1457] Step 8:

[1458] Server: Receives voice data, sends it to a speech recognition engine, and converts it into text data.

[1459] Step 9:

[1460] Server: The converted text data is sent to the emotion engine to analyze the user's emotional state.

[1461] Step 10:

[1462] Emotion engine: Based on the analysis results, it recognizes the user's emotional state. For example, it analyzes the tone and pace of the user's voice to determine whether they are happy or stressed.

[1463] Step 11:

[1464] Server: Receives the analysis results from the emotion engine and sends them to the generative AI model.

[1465] Step 12:

[1466] Generative AI: It uses text data, sentiment analysis results, and past conversation data to understand the context and generate responses that are tailored to the user's emotional state.

[1467] Step 13:

[1468] Server: The generated response text is sent to the speech synthesis engine and converted into voice data.

[1469] Step 14:

[1470] Server: Sends voice data to the user terminal.

[1471] Step 15:

[1472] User terminal: Receives the voice data and plays back the response from the speaker.

[1473] Step 16:

[1474] Generative AI: Detects user pronunciation and grammar errors in real time during conversation.

[1475] Step 17:

[1476] Generative AI: If an error is detected, it generates appropriate corrections and feedback, for example, correcting the statement "I went to the park" to "I went to the park."

[1477] Step 18:

[1478] Server: Sends corrective feedback to the user terminal.

[1479] Step 19:

[1480] User terminal: Corrective feedback is played back and provided to the user.

[1481] Step 20:

[1482] Server: After the conversation ends, a summary report is generated based on the entire history, including details about pronunciation and grammar errors made during the conversation, areas for improvement, and the user's emotional state.

[1483] Step 21:

[1484] Server: Sends the generated summary report to the user terminal.

[1485] Step 22:

[1486] User device: A summary report is displayed, allowing users to identify areas for improvement and mistakes in the conversation.

[1487] Example 2

[1488] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1489] Conventional speech recognition systems and response generation systems using generative models have difficulty generating appropriate responses that take the user's emotional state into account. Furthermore, they lack the functionality to correct users' pronunciation and grammar errors in real time and provide feedback, which reduces learning efficiency. Furthermore, there is no way to visualize the user's progress after the conversation and suggest areas for improvement, making it difficult to continuously improve the user's learning experience.

[1490] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input speech, means for converting the speech into text data, means for generating a response based on the text data using a generative model, means for converting the response into speech data, means for referencing past conversation data using user authentication information, means for generating a response based on the past conversation data and the current context, means for returning the speech data to the user, means for detecting and correcting pronunciation and grammatical errors, means for generating a summary report of the conversation after the conversation ends, and means for recognizing the user's emotional state and adjusting the content of the response accordingly. This makes it possible to generate appropriate responses according to the user's emotional state, correct pronunciation and grammatical errors in real time, and visualize the user's learning progress.

[1491] "Means for user voice input" refers to a device or function that allows a user to transmit voice information to the system.

[1492] The "means for converting the voice into text data" refers to a device or function that analyzes input voice information and converts it into corresponding text data.

[1493] "Means for generating a response based on the text data using a generative model" refers to a device or function that utilizes a generative AI model to generate an appropriate response based on the text data.

[1494] "Means for converting said response into voice data" refers to a device or function that converts the generated text-based response into voice data.

[1495] The "means for referencing past conversation data using user authentication information" refers to a device or function that uses user authentication information to access and refer to conversation data that has taken place with a user in the past.

[1496] The "means for generating a response based on the past conversation data and the current context" refers to a device or function that generates an appropriate response by taking into account the past conversation data and the current conversation context.

[1497] The "means for returning the voice data to the user" refers to a device or function for transmitting the generated voice data to the user's terminal and letting the user hear it.

[1498] The "means for detecting and correcting pronunciation and grammar errors" refers to a device or function that identifies pronunciation and grammar errors contained in the user's speech and corrects them to the correct form.

[1499] "Means for generating a summary report of a conversation after the conversation has ended" refers to a device or function that, after the conversation has ended, summarizes the content of the conversation and generates a report that includes information such as pronunciation and grammatical errors, areas for improvement, and emotional state.

[1500] The "means for recognizing the user's emotional state and adjusting the content of the reply accordingly" refers to a device or function that analyzes the user's emotional state and adjusts the tone and content of the reply based on the results of the analysis.

[1501] This invention is a system for generating appropriate responses using user voice data, utilizing a generative AI model and an emotion recognition engine, and improving the user's learning experience. This system consists of a user terminal, a server, and various related engines and databases.

[1502] User Authentication and Login

[1503] A user logs in using their own device (e.g., PC or smartphone) by entering their username and password. The device sends the authentication information to the server, which checks it against a database. If authentication is successful, the server generates a session ID and user profile information and sends them back to the device. This allows the user to retain their settings and past conversation data.

[1504] Voice input and speech generation

[1505] The user inputs voice using the microphone on the device. The device then sends the captured voice data to the server. The server then sends the voice data to a speech recognition engine, which converts it into text data. This text data is then sent to a generative AI model, which generates an appropriate response based on the context and past conversation data. This generative AI model can be an open-source model or a commercial model provided by a company.

[1506] Emotion Recognition and Voice Output

[1507] The server sends the generated text data to an emotion engine to analyze the user's emotional state. The emotion engine identifies the user's emotion and adjusts the tone and content of the response accordingly. For example, if it determines that the user is feeling stressed, the response will be adjusted to a calmer, more encouraging tone. This adjusted response text is sent to a speech synthesis engine and converted into voice data. This voice data is sent back to the user's device via the server, allowing the user to receive the response via voice.

[1508] Feedback and Error Correction

[1509] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI detects the error and generates text correcting it to "went." The server then sends the corrective feedback to the user's device, providing audio feedback such as, "You should say 'I went to the park'."

[1510] Generate a summary report

[1511] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors during the conversation, areas for improvement, and the user's emotional state. The summary report is sent to the user's device, where the user can view it to understand their learning progress and areas for improvement.

[1512] Specific examples

[1513] For example, if a user says, "Hello, I would like to practice English," the voice data is sent from the device to the server, where it is converted by the speech recognition engine into text data: "Hello, I would like to practice English." The generative AI model uses this text data to generate a response: "Sure, let's get started!" The emotion engine analyzes the user's emotional state and adjusts the tone and content as necessary. The speech synthesis engine then converts this response into voice data and sends it back to the user's device. Hearing the voice, "Sure, let's get started!", the user can continue practicing English conversation with peace of mind.

[1514] The system allows users to receive real-time feedback as they learn, providing a better learning experience through appropriate responses based on their emotional state.

[1515] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1516] Step 1: User authentication and login

[1517] The user enters their username and password using their own device (PC or smartphone) and clicks the login button. The device sends this authentication information to the server. The server compares the received authentication information with the database, and if the user is authenticated, generates a session ID and user profile information and sends these back to the device. This allows the user to retain their settings and past conversation data.

[1518] Input: Username, Password

[1519] Output: Session ID, user profile

[1520] Step 2: Receiving Audio Input

[1521] The user inputs speech using the device's microphone, which converts the speech data into a digital format and sends it to the server, which then sends it to a speech recognition engine for analysis.

[1522] Input: Audio data

[1523] Output: Digital audio data

[1524] Step 3: Convert audio data to text

[1525] The server converts the received voice data into text data using a speech recognition engine, which analyzes the voice waveform and generates corresponding text, which is then sent to the generative AI model.

[1526] Input: Digital audio data

[1527] Output: Text data

[1528] Step 4: Generative AI model generates a response

[1529] The server sends the generated text data to the generative AI model, which generates an appropriate response based on the text data, past conversation data, and the current context. The generated response text is then sent to the emotion engine.

[1530] Input: Text data, past conversation data, contextual information

[1531] Output: Response text

[1532] Step 5: Response tailoring with the emotion engine

[1533] The server sends the generated response text to the emotion engine, which analyzes the user's emotional state. The emotion engine adjusts the tone and content of the response based on the user's emotional state. For example, if the user is feeling stressed, the response will be adjusted to a more gentle and encouraging tone. The adjusted response text is then sent to the speech synthesis engine.

[1534] Input: Reply text, emotional state

[1535] Output: Adjusted response text

[1536] Step 6: Speech conversion using a speech synthesis engine

[1537] The server sends the adjusted response text to a speech synthesis engine, which converts it into voice data. The speech synthesis engine generates natural-sounding voice based on the text data. This voice data is then sent back to the user's device via the server.

[1538] Input: Adjusted response text

[1539] Output: Reply audio data

[1540] Step 7: Provide a response

[1541] The user terminal receives the returned voice data and provides it to the user through a speaker, so that the user receives a voice response.

[1542] Input: Response audio data

[1543] Output: Audio output

[1544] Step 8: Error detection and feedback

[1545] During the conversation, the generative AI model detects the user's pronunciation and grammar errors in real time. For example, if the user says, "I went to the park," the generative AI model detects the error in "goed" and generates a corrected text to change it to "went." The server then sends the corrected text feedback to the user's device and provides audio feedback.

[1546] Input: Raw audio data, speech content

[1547] Output: Corrective feedback text, audio feedback

[1548] Step 9: Generate a summary report

[1549] At the end of the conversation, the server generates a summary report based on the entire conversation history. This summary report includes details about pronunciation and grammar errors and areas for improvement during the conversation, as well as the user's emotional state. The server then sends the summary report to the user's device, allowing the user to review it and understand their learning progress.

[1550] Input: conversation history

[1551] Output: Summary report

[1552] (Application example 2)

[1553] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] During the food delivery ordering process, users face challenges in effectively and quickly selecting meals that suit their needs and emotions. Current systems are unable to provide personalized recommendations that take into account the user's emotional state, potentially resulting in lower customer satisfaction. Furthermore, the lack of real-time correction and feedback for pronunciation and grammar errors hinders smooth user experience.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input speech, a means for converting the speech into text data, a means for generating a response based on the text data using a generative model, a means for converting the response into speech data, a means for referencing past conversation data using user authentication information, a means for generating a response based on the past conversation data and the current context, a means for returning the speech data to the user, a means for detecting and correcting pronunciation and grammar errors, a means for generating a summary report of the conversation after the conversation ends, a means for recognizing the user's emotional state and responding according to the emotion, and a means for suggesting food delivery-related services and products. This allows the user to receive personalized food delivery suggestions based on their emotional state, enabling the ordering process to be carried out quickly and efficiently. Furthermore, real-time correction of pronunciation and grammar errors is provided, allowing for a smooth user experience.

[1556] "Means for user voice input" refers to a function for acquiring voice data through an input device (e.g., a microphone).

[1557] The "means for converting voice into text data" is a function that converts voice data acquired using voice recognition technology into a corresponding text format.

[1558] "Means for generating responses based on text data using a generative model" refers to a function that uses generative AI to analyze input text data and generate appropriate responses.

[1559] The "means for converting a response into voice data" is a function that converts the generated text response into voice data using synthesized voice technology.

[1560] The "means for referencing past conversation data using user authentication information" is a function for retrieving and referencing past conversation history from a database using authentication information such as a user name and password.

[1561] The "means for generating a response based on past conversation data and current context" is a function for generating a personalized response based on past conversation history and real-time context information.

[1562] The "means for returning voice data to the user" is a function that transmits the generated voice data to the user device, allowing the user to receive a voice response.

[1563] The "means for detecting and correcting pronunciation and grammatical errors" is a function that analyzes the content of the user's speech, detects pronunciation and grammatical errors, corrects them, and provides feedback.

[1564] The "means for generating a summary report of a conversation after the conversation has ended" is a function that compiles the conversation history with the user and generates a report that includes pronunciation and grammatical errors, emotional state, and so on.

[1565] "Means for recognizing the user's emotional state and responding accordingly" refers to a function that uses emotion recognition technology to analyze the user's emotional state and adjusts the content and tone of the response based on this.

[1566] "Means for suggesting food delivery-related services and products" refers to a function that recommends appropriate food delivery services and products based on the user's emotions and needs.

[1567] To implement this invention, multiple technical components are required: a user terminal, a server, a speech recognition engine, a generative AI model, an emotion engine, and a speech synthesis engine. A user places a food delivery order by voice. The server processes this voice data and provides a system that generates personalized responses based on the user's emotional state.

[1568] 1. User authentication and login

[1569] Users log in by entering their username and password on a device such as a smartphone. This authentication information is sent to the server and checked against a database. If authentication is successful, the server retrieves past conversation data and user profile information and returns it to the user's device, starting the session.

[1570] 2. Voice to Text

[1571] Users input their voice using the microphone on their smartphone. This voice data is sent to the server and converted into text data through a speech recognition engine. Speech recognition engines such as Google Speech-to-Text and Microsoft Azure Speech can be used.

[1572] 3. Response generation using generative AI models

[1573] The server sends the acquired text data to a generative AI model, which can use models such as GPT-3 or ChatGPT, to generate an appropriate response.

[1574] 4. Emotion recognition and response regulation

[1575] The generated text response is sent by the server to an emotion engine, which analyzes the user's emotional state (using a model like Hugging Face's emotion analysis model), and adjusts the content and tone of the response based on the user's emotional state.

[1576] 5. Speech synthesis and response return

[1577] The adjusted response text is sent to a speech synthesis engine, which converts it into voice data. The speech synthesis engine can be Amazon Polly or Google Text-to-Speech. This voice data is sent back to the user's device, where the user can receive a voice response.

[1578] 6. Food delivery proposal

[1579] The server will suggest suitable food delivery options to the user based on their emotional state and past ordering history. For example, if a user says, "I'm tired today," the system will suggest, "Would you like hot soup or pizza?"

[1580] 7. Summary report generation

[1581] After the conversation ends, the server generates a summary report based on the entire conversation history. This summary report includes any pronunciation or grammar errors detected during the conversation, information about the user's emotional state, and a history of food delivery suggestions. This report is sent to the user's device, where the user can review their conversation history.

[1582] Specific examples

[1583] For example, if a user asks, "I'm tired today, what would you recommend for dinner?", the server processes the voice data and uses an emotion engine to identify the emotion "tired." It then generates a response, "Would you like some hot soup or pizza?", and delivers it to the user as speech.

[1584] Prompt Sentence Examples

[1585] When a user asks, "I'm tired today, what food would you recommend?", your smart food assistant should use its emotion engine to identify the emotion "tired" and respond with, "You seem tired today. Would you like some hot soup or pizza to help you relax?"

[1586] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1587] Step 1:

[1588] The user inputs voice through the smartphone microphone. This input is the user's action of making a request or asking a question for food delivery by voice. For example, the user might say, "I'm tired today, so what meal would you recommend?"

[1589] Step 2:

[1590] The device collects voice data and sends it to the server. The data sent is raw voice data. In this step, the device does not perform any preprocessing for voice recognition.

[1591] Step 3:

[1592] The server sends the received voice data to a speech recognition engine. This engine converts the voice data into text data. For example, Google Speech-to-Text is used. In this step, the voice data is converted into text data such as "I'm tired today, so what meal would you recommend?"

[1593] Step 4:

[1594] The server sends the converted text data to a generative AI model, which then generates an appropriate response based on the text data. The input at this stage is the converted text data, and the output is the generated response text: "Would you like some warm soup or pizza to relax?"

[1595] Step 5:

[1596] The server sends the generated text response to the emotion engine, which analyzes the user's emotional state. In this case, based on the text data "I'm tired today...", the server analyzes that the user's state is "tired." Based on this input, the emotion engine adjusts the tone and content. The output is a response text with adjusted tone.

[1597] Step 6:

[1598] The server sends the tone-adjusted response text to a speech synthesis engine, which converts it into speech data. The speech synthesis engine (e.g., Amazon Polly) converts the text data into speech data. The output is the speech data, "Would you like some hot soup or pizza to relax?"

[1599] Step 7:

[1600] The server sends the generated voice data to the user terminal, and the user receives a voice response. The input is the adjusted voice data, and the output is the voice information that the user hears.

[1601] Step 8:

[1602] When the conversation ends, the server generates a summary report based on the conversation history and emotional state, including the user's comments, responses, and emotional analysis results. The server then sends this report to the user's device so that the user can review it.

[1603] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1604] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1605] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1606] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1607] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1608] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1609] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1610] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1611] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1612] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1613] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1614] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1615] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1616] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1617] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1618] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1619] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1620] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1621] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1622] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1623] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1624] The following is further disclosed regarding the above embodiment.

[1625] (Claim 1)

[1626] a means for a user to input speech;

[1627] means for converting the voice into text data;

[1628] means for generating a response based on the text data using a generative model;

[1629] means for converting the response into voice data;

[1630] A means for referencing past conversation data using user authentication information;

[1631] means for generating a response based on the past conversation data and a current context;

[1632] means for returning said audio data to a user;

[1633] means for detecting and correcting pronunciation and grammar errors;

[1634] a means for generating a summary report of the conversation after the conversation has concluded;

[1635] A system including:

[1636] (Claim 2)

[1637] 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback.

[1638] (Claim 3)

[1639] 2. The system of claim 1, wherein the generative model understands context based on the user's past conversation data and generates personalized responses.

[1640] "Example 1"

[1641] (Claim 1)

[1642] a means for a user to input speech;

[1643] means for converting the voice into text data;

[1644] means for generating a response based on the text data using a generative model;

[1645] means for converting the response into voice data;

[1646] A means for referencing past conversation data using user authentication information;

[1647] means for generating a response based on the past conversation data and a current context;

[1648] means for returning said audio data to a user;

[1649] means for detecting and correcting pronunciation and grammar errors;

[1650] a means for generating a summary report of the conversation after the conversation has concluded;

[1651] means for encrypting authentication information entered by a user and transmitting the information to a server;

[1652] means for loading the user's profile information if authentication is successful;

[1653] means for asynchronously processing voice data received from a user via a terminal;

[1654] means for executing the speech recognition means and the speech synthesis means on a cloud-based engine;

[1655] a means for managing the process of playing back the generated response aloud on the user terminal;

[1656] A system including:

[1657] (Claim 2)

[1658] 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback.

[1659] (Claim 3)

[1660] 2. The system of claim 1, wherein the generative model understands context based on the user's past conversation data and generates personalized responses.

[1661] "Application Example 1"

[1662] (Claim 1)

[1663] a means for a user to input speech;

[1664] means for converting the voice into text data;

[1665] means for generating a response based on the text data using a generative model;

[1666] means for converting the response into voice data;

[1667] A means for referencing past conversation data using user authentication information;

[1668] means for generating a response based on the past conversation data and a current context;

[1669] means for returning said audio data to a user;

[1670] means for detecting and correcting pronunciation and grammar errors;

[1671] a means for generating a summary report of the conversation after the conversation has concluded;

[1672] A means for having a generative AI model generate prompts based on the user's order to provide appropriate menu recommendations; and

[1673] a means for providing the generated recommended menu to the user as voice data;

[1674] A system including:

[1675] (Claim 2)

[1676] 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback.

[1677] (Claim 3)

[1678] 2. The system of claim 1, wherein the generative model understands context based on the user's past conversation data and generates personalized responses.

[1679] "Example 2: Combining Emotion Engines"

[1680] (Claim 1)

[1681] a means for a user to input speech;

[1682] means for converting the voice into text data;

[1683] means for generating a response based on the text data using a generative model;

[1684] means for converting the response into voice data;

[1685] A means for referencing past conversation data using user authentication information;

[1686] means for generating a response based on the past conversation data and a current context;

[1687] means for returning said audio data to a user;

[1688] means for detecting and correcting pronunciation and grammar errors;

[1689] a means for generating a summary report of the conversation after the conversation has concluded;

[1690] a means of recognizing emotional states and adjusting responses accordingly;

[1691] A system including:

[1692] (Claim 2)

[1693] 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback, recognizing the emotional state, and adjusting the tone of responses accordingly.

[1694] (Claim 3)

[1695] 10. The system of claim 1, wherein the generative model understands context based on the user's past conversation data, generates personalized responses, and adjusts replies based on the user's emotional state.

[1696] "Application example 2 when combining emotion engines"

[1697] (Claim 1)

[1698] a means for a user to input speech;

[1699] means for converting the voice into text data;

[1700] means for generating a response based on the text data using a generative model;

[1701] means for converting the response into voice data;

[1702] A means for referencing past conversation data using user authentication information;

[1703] means for generating a response based on the past conversation data and a current context;

[1704] means for returning said audio data to a user;

[1705] means for detecting and correcting pronunciation and grammar errors;

[1706] a means for generating a summary report of the conversation after the conversation has concluded;

[1707] means for recognizing the emotional state of a user and responding accordingly;

[1708] A means of proposing services and products related to food delivery,

[1709] A system including:

[1710] (Claim 2)

[1711] 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback and further comprising tone adjustments in response to the emotional state.

[1712] (Claim 3)

[1713] The system of claim 1, wherein the generative model understands the context based on the user's past conversation data, generates individualized responses, and provides personalized food delivery suggestions that take user emotions into account. [Explanation of symbols]

[1714] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to input speech; means for converting the voice into text data; means for generating a response based on the text data using a generative model; means for converting the response into voice data; A means for referencing past conversation data using user authentication information; means for generating a response based on the past conversation data and a current context; means for returning said audio data to a user; means for detecting and correcting pronunciation and grammar errors; a means for generating a summary report of the conversation after the conversation has concluded; A system including:

2. 10. The system of claim 1, wherein the means for detecting and correcting pronunciation and grammar errors is capable of providing real-time feedback.

3. The system of claim 1 , wherein the generative model understands context based on past conversational data of the user and generates personalized responses.

Images