system

Abstract

We provide the system. [Solution] A means for collecting voice data input from the user, Means for converting the aforementioned audio data into text data, A means for analyzing the aforementioned text data and performing emotion recognition, A means for generating an appropriate response based on the results of the aforementioned emotion recognition, Means for displaying or playing the generated response to the user, A system that includes this.

Description

Technical Field

[0006] , ,

[0005] , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional dialogue system, it has been difficult to recognize the emotional state of a user and generate an appropriate response accordingly. For this reason, there has been a problem that a sympathetic response to the user's emotion and intention cannot be obtained, and the improvement of the user experience cannot be sufficiently achieved. In addition, appropriate processing has not been performed on voice input, and the convenience when the user interacts with the system based on voice has been low.

Means for Solving the Problems

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The present invention provides a system that includes means for collecting voice data input by a user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, means for generating an appropriate response based on the results of the emotion recognition, and means for displaying or playing the generated response to the user.

[0007] This makes it possible to accurately recognize the emotional state of a user who inputs voice or text data, and to generate and provide an appropriate response based on that emotion. Furthermore, by using speech recognition and synthesized speech, natural voice dialogue can be achieved, which is expected to significantly improve the user experience.

[0008] A "user" refers to a person who uses a system or an end-user.

[0009] "Voice data" refers to voice information entered by the user, and includes audio files recorded in a digital format.

[0010] "Text data" refers to data that has been converted from audio data into written information and is recognizable as sentences or words.

[0011] "Means of collection" refers to the technology or equipment used to obtain voice data or text data from users and incorporate it into the system.

[0012] "Means of conversion" refers to technologies or devices for converting audio data into text data, and includes speech recognition engines and natural language processing engines.

[0013] "Means of analysis" refers to technologies or devices used to analyze collected text data and identify specific meanings, intentions, emotions, etc.

[0014] "Emotion recognition" is the process of identifying an emotional state from the content of text data or the user's voice information.

[0015] "The result of emotion recognition" refers to information indicating the emotional state of the user extracted from the analyzed text data.

[0016] "Means for generating an appropriate response" refers to a technology or device that automatically generates appropriate response content for the user based on the result of emotion recognition.

[0017] "Means for displaying or playing" refers to a technology or device that conveys the generated response to the user visually or audibly, such as a display or a speaker.

Brief Description of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The system's operation and specific usage examples are shown below.

[0040] System Configuration

[0041] This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0042] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0043] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0044] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, a synthesized speech engine, and so on.

[0045] Program processing

[0046] 1. Collection of audio and text data

[0047] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[0048] The device collects voice data and sends it to the server.

[0049] 2. Converting audio data

[0050] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[0051] 3. Emotion recognition

[0052] The server sends text data to the NLP engine, which then executes an emotion recognition algorithm. As a result of the emotion recognition, "anxiety" is extracted.

[0053] 4. Response generation

[0054] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0055] 5. Sending and displaying responses

[0056] The server sends the generated response to the terminal as text data.

[0057] The terminal displays the received text to the user. If necessary, it plays a voice response using a text-to-speech engine.

[0058] Specific example

[0059] Examples of use in customer service

[0060] 1. The user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0061] 2. The terminal sends this input to the server.

[0062] 3. The server analyzes the input and recognizes the emotion of "anxiety."

[0063] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[0064] 5. The terminal displays a response to the user.

[0065] Examples of use in the field of education

[0066] 1. The user (student) inputs "I don't understand this problem" by voice.

[0067] 2. The device collects audio and sends it to the server.

[0068] 3. The server converts the speech into text and recognizes the emotion of "confusion."

[0069] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0070] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0071] This system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0072] The following describes the processing flow.

[0073] Step 1:

[0074] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[0075] Step 2:

[0076] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[0077] Step 3:

[0078] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[0079] Step 4:

[0080] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[0081] Step 5:

[0082] The server sends the converted text data to a natural language processing (NLP) engine, which then executes an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0083] Step 6:

[0084] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0085] Step 7:

[0086] The server sends the generated response to the terminal as text data.

[0087] Step 8:

[0088] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[0089] (Example 1)

[0090] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] Traditional dialogue systems have faced the challenge of difficulty in considering emotions when generating appropriate responses from user voice or text input. In particular, when users were experiencing emotions such as anxiety, confusion, or anger, the dialogue system often failed to provide appropriate responses that reflected those emotions. This resulted in a degraded user experience.

[0092] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0093] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, means for generating an appropriate response based on the emotion recognition result using a generative AI model, and means for displaying or playing the generated response to the user. This enables appropriate dialogue that responds to the user's emotions, thereby improving the user experience.

[0094] "Audio data" refers to data that records the voice spoken by a user in digital format.

[0095] "Text data" refers to data that represents audio data or input character information as a string of characters.

[0096] "Emotion recognition" is the process of analyzing emotions contained in text data and identifying the type of emotion.

[0097] A "generative AI model" is an artificial intelligence model trained using large-scale datasets, and is used for text generation and response generation.

[0098] An "appropriate response" is a response that is relevant to the situation and generated based on the user's input and emotions.

[0099] "Display" refers to providing the generated text response to the user visually.

[0100] "Playback" refers to converting the generated text response into audio and providing it to the user audibly.

[0101] "Collection means" refers to methods or devices for acquiring voice data or text data from users.

[0102] "Conversion means" refers to methods or devices for converting audio data into text data.

[0103] "Analysis means" refers to methods or devices for analyzing received text data and identifying its syntax and sentiment.

[0104] "Generative means" refers to a method or device that generates an appropriate response using a generative AI model based on the results of emotion recognition.

[0105] A "user" refers to a person who uses a dialogue system to input text or voice.

[0106] A "server" refers to a computer system used for data processing and analysis.

[0107] A "terminal" refers to a device used to collect user voice and text input and transmit it to a server.

[0108] Modes for carrying out the invention

[0109] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0110] System Configuration

[0111] 1. Terminal

[0112] A terminal is a device that collects voice or text input from a user and sends it to a server. Examples include smartphones, tablets, and personal computers. Terminals are equipped with microphones, keyboards, touchscreens, and other features that allow them to collect user input data.

[0113] 2. Server

[0114] A server is a computer system that receives, analyzes, and processes data sent from terminals. This computer system is equipped with a high-speed CPU or GPU, sufficient memory and storage, and processes data in real time.

[0115] 3. Software Components

[0116] The system's software components include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a speech synthesis engine. Specifically, these are as follows:

[0117] Speech recognition engine: Converts speech data into text data. As an example, Google® Cloud Speech-to-Text is used.

[0118] Natural Language Processing (NLP) engine: Analyzes text data and performs syntactic and semantic analysis. A concrete example is the Google Cloud Natural Language API.

[0119] Emotion recognition algorithm: Extracts emotions from text data analyzed by an NLP engine.

[0120] Response generation engine: Uses a generative AI model (e.g., GPT-3®) to generate appropriate responses based on emotion recognition results.

[0121] Text-to-speech engine: Converts generated responses into speech. A concrete example is Google Cloud Text-to-Speech.

[0122] Specific usage examples

[0123] Examples of use in customer service

[0124] A user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0125] The terminal sends this input to the server.

[0126] The server analyzes the input and recognizes the emotion of "anxiety."

[0127] The server generates an empathetic response saying, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0128] The device displays a response to the user. If necessary, it plays the response using synthesized speech.

[0129] Examples of use in the field of education

[0130] The user (student) inputs "I don't understand this problem" by voice.

[0131] The device collects audio and sends it to the server.

[0132] The server converts the speech into text and recognizes the emotion of "confusion."

[0133] The server generates a response saying, "Please tell me specifically what part you don't understand. Let's solve it together."

[0134] The device reads the text aloud using synthesized speech and plays it back to the user.

[0135] Through the above, this system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0136] Example of a prompt

[0137] "The user is feeling anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution."

[0138] "A student is confused by a math problem. Generate a response that suggests you try to solve the problem together."

[0139] These prompts generate appropriate responses that reflect the user's emotions.

[0140] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0141] Step 1: Collect audio and text data

[0142] The user provides input to the system via voice or text. For example, they might voice-input, "My recent order hasn't arrived yet, what's going on?" This input is collected by the terminal.

[0143] Input: [User's voice or text data]

[0144] Output: [Audio data or text data]

[0145] Specific operation: The device uses its built-in microphone and text input function to collect user voice and text, and sends them to the server as raw data.

[0146] Step 2: Convert audio data

[0147] The server receives the audio data sent from the terminal. The received audio data is passed to a speech recognition engine, which converts the audio into text data. Using the speech recognition engine (for example, Google Cloud Speech-to-Text), the server outputs the text data "My recent order hasn't arrived yet, what's going on?"

[0148] Input: [Audio data]

[0149] Output: [Text data]

[0150] Specific operation: The server converts the audio data into a string via a speech recognition engine and saves it as text data.

[0151] Step 3: Emotion Recognition

[0152] The server sends the converted text data to an NLP engine (e.g., Google Cloud Natural Language API) to run an emotion recognition algorithm. It analyzes the text "My recent order hasn't arrived yet, what's going on?" and extracts the emotion "anxiety" contained within it.

[0153] Input: [Text data]

[0154] Output: [emotion data]

[0155] Specific operation: The server uses an NLP engine to parse the text data and extracts emotional information from the text using an emotion recognition algorithm.

[0156] Step 4: Response Generation

[0157] The server uses an AI model (e.g., GPT-3) to generate an appropriate response based on the emotion recognition result "anxiety." The prompt used is "The user is anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution." The response generated is "We apologize for the inconvenience. We will check immediately and provide you with tracking information."

[0158] Input: [emotion data]

[0159] Output: [Response text data]

[0160] Specific operation: The response generation engine inputs a prompt sentence into the generation AI model and generates an appropriate response sentence based on emotion.

[0161] Step 5: Sending and displaying the response

[0162] The server sends the generated response text data to the terminal. The terminal receives this data and displays it to the user. If necessary, it converts it to speech using a text-to-speech engine (e.g., Google Cloud Text-to-Speech) and plays it back to the user.

[0163] Input: [Response text data]

[0164] Output: [Display or audio data]

[0165] Specific operation: The device displays text data using a display UI and plays it back as audio using a text-to-speech engine.

[0166] (Application Example 1)

[0167] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0168] Conventional dialogue systems lacked the ability to provide appropriate responses based on user emotions, making it difficult to fully improve the user experience. Furthermore, in specific applications such as food delivery services, there is a need to generate rapid and emotionally resonant responses to user inquiries. Existing technologies have not adequately addressed these challenges.

[0169] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0170] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, and means for generating a response using a generative AI model based on the emotion recognition results. This makes it possible to accurately recognize the user's emotions and quickly generate an appropriate response that is in line with those emotions.

[0171] "Audio data" refers to the audio signal that a user produces using an audio input device such as a microphone.

[0172] "Text data" refers to audio data that has been converted into a string by a speech recognition engine, or string information directly entered by the user.

[0173] "Emotion recognition" refers to the process of analyzing collected text data to identify the user's emotional state (e.g., anxiety, joy, anger, etc.).

[0174] An "appropriate response" refers to a reply message that is generated based on the results of emotion recognition and is tailored to the user's emotions and the content of their inquiry.

[0175] A "generative AI model" refers to an artificial intelligence algorithm that uses natural language processing to generate appropriate responses to user input.

[0176] "Means of generating responses" refers to the process of creating appropriate responses in text or audio format using emotion recognition results and generative AI models.

[0177] "Synthesized speech" refers to speech synthesis technology that converts text data into audio data and plays it back for the user.

[0178] "Means for displaying or reproducing" means devices and methods for providing the generated response to the user visually or audibly.

[0179] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses. The system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0180] System Configuration

[0181] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0182] 2. Server: A computer system that receives, analyzes, and processes data transmitted from terminals. The server includes software components such as a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, and a generative AI model.

[0183] 3. Software Components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a synthesized speech engine. These are necessary to analyze user input data and generate appropriate responses.

[0184] Program processing

[0185] When a user uses the application and says, "My pizza hasn't arrived yet," the device collects the audio data and sends it to the server. The server converts the audio data into text data through a speech recognition engine. This converted text data is then sent to an NLP engine, which applies an emotion recognition algorithm to recognize the emotion of "anxiety."

[0186] The server then uses a generative AI model to generate an appropriate response based on the emotion recognition results. For example, it might say, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." The generated response is sent to the device in text or synthesized speech format and displayed or played back to the user.

[0187] Hardware and software used

[0188] Speech recognition engine: This is an engine that converts speech data into text data, and commonly used are "Google Speech Recognition API" and "Microsoft Azure Speech Services".

[0189] Natural Language Processing (NLP): Used to analyze text data and recognize user intent and emotions. Models such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa are used for this purpose.

[0190] Emotion recognition algorithms: These are algorithms for identifying emotions from text data. For example, emotion recognition models such as "j-hartmann / emotion-english-distilroberta-base" are applied.

[0191] Generative AI model: This model generates appropriate responses in response to user input, and examples include OpenAI's "GPT-3".

[0192] Specific example

[0193] Suppose a user voice-inquiries to the app saying, "My pizza order hasn't arrived yet, what's going on?" This voice input is converted to text, and the emotion of "anxiety" is recognized. The server then generates a response saying, "We apologize for the inconvenience. We will check immediately and provide you with tracking information," which is then displayed or played back to the user.

[0194] Example of a prompt:

[0195] A user has inquired, "My pizza hasn't arrived yet..." This inquiry has caused the user to feel anxious. Please generate an appropriate response.

[0196] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0197] Step 1:

[0198] A user uses a food delivery application and says, "My pizza hasn't arrived yet..." The device uses its built-in microphone to collect this voice input data. The input data is the user's voice signal and is sent to the server after collection.

[0199] Step 2:

[0200] The server receives the audio data sent from the terminal. The server uses a speech recognition engine to convert the audio data into text data. In this process, the speech recognition engine (e.g., "Google Speech Recognition API") converts the audio signal into string information, and the output is the text data "My pizza hasn't arrived yet..."

[0201] Step 3:

[0202] The server sends the converted text data to a natural language processing (NLP) engine. The NLP engine (e.g., "BERT" or "RoBERTa") analyzes this text data and processes it to identify the user's intent and emotions. This process includes using a language model to perform contextual analysis and generate the data necessary for the emotion recognition algorithm. As output, the text data is processed into data for emotion analysis.

[0203] Step 4:

[0204] The analyzed data is sent to an emotion recognition algorithm on the server. The emotion recognition algorithm (e.g., "j-hartmann / emotion-english-distilroberta-base") extracts emotions from the text data. This specific operation involves assigning specific emotion labels (e.g., "anxiety," "joy," "anger," etc.). The output is an emotion recognition result of "anxiety."

[0205] Step 5:

[0206] The server generates a response using a generative AI model based on the obtained emotion recognition results. The generative AI model (e.g., "GPT-3") creates an appropriate response based on emotion and text data. This specific operation involves a process of requesting the model to generate a response using a prompt statement. The output is the generated text response, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0207] Step 6:

[0208] The generated response is sent from the server to the terminal. The terminal then displays or plays this text response to the user. This specific action includes displaying the text on the screen and converting it to speech using a speech synthesis engine (e.g., "Microsoft Azure Speech Services") and playing it through the speaker.

[0209] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0210] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. In particular, it features a "emotion engine" for emotion recognition, enabling highly accurate emotion recognition and empathetic responses. The system's operation and specific usage examples are described below.

[0211] System Configuration

[0212] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[0213] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0214] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0215] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, an emotion engine, a response generation engine, a synthesized speech engine, and so on.

[0216] Program processing

[0217] 1. Collection of audio and text data

[0218] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[0219] The device collects voice data and sends it to the server.

[0220] 2. Converting audio data

[0221] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[0222] 3. Emotion recognition

[0223] The server sends text data to the NLP engine, which then uses the emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0224] 4. Response generation

[0225] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0226] 5. Sending and displaying responses

[0227] The server sends the generated response to the terminal as text data.

[0228] The terminal displays the received text to the user. If necessary, the terminal uses a text-to-speech engine to convert the text into speech and play the response.

[0229] Specific example

[0230] Examples of use in customer service

[0231] 1. The user enters a message into customer service saying, "My ordered item hasn't arrived yet, what's going on?"

[0232] 2. The terminal sends this input to the server.

[0233] 3. The server analyzes the input data and uses an emotion engine to recognize the emotion of "anxiety."

[0234] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[0235] 5. The terminal displays a response to the user.

[0236] Examples of use in the field of education

[0237] 1. The user (student) inputs "I don't understand this problem" by voice.

[0238] 2. The device collects audio and sends it to the server.

[0239] 3. The server converts the audio into text and uses an emotion engine to recognize the emotion "confusion."

[0240] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0241] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0242] This system enables appropriate dialogue based on the user's emotions, and the introduction of an emotion engine, in particular, improves the accuracy of emotion recognition. This greatly enhances the user experience.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[0246] Step 2:

[0247] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[0248] Step 3:

[0249] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[0250] Step 4:

[0251] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[0252] Step 5:

[0253] The server sends the converted text data to a natural language processing (NLP) engine, which then uses an emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0254] Step 6:

[0255] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0256] Step 7:

[0257] The server sends the generated response to the terminal as text data.

[0258] Step 8:

[0259] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[0260] (Example 2)

[0261] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0262] In modern dialogue systems, accurately recognizing user emotions and providing appropriate, empathetic responses is challenging. Conventional systems suffer from low accuracy in emotion recognition, resulting in a poor user experience.

[0263] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0264] In this invention, the server includes means for collecting voice data input from a user, means for converting the voice data into text data, means for analyzing the text data and performing natural language processing, means for performing emotion recognition using the data analyzed by the natural language processing, means for generating an appropriate response based on the result of the emotion recognition, and means for displaying or playing the generated response to the user. This makes it possible to recognize the user's emotions with higher accuracy and provide an empathetic and appropriate response accordingly.

[0265] A "user" refers to a person who uses the system.

[0266] "Voice data" refers to digital data that electronically records the voice input by the user.

[0267] "Text data" refers to data obtained by converting audio data into written text.

[0268] "Natural language processing" refers to the technology that enables computers to understand and analyze human language.

[0269] "Emotion recognition" refers to the technology that identifies a user's emotions from text data.

[0270] "Response" refers to the reply or answer that the system generates to the user.

[0271] "Means of generation" refers to a function that generates a response using a specific process or algorithm.

[0272] "Means for displaying or reproducing" refers to technologies or devices for showing or letting the user hear the generated response.

[0273] Modes for carrying out the invention

[0274] This invention is a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The actual operation and usage examples of the system are shown below.

[0275] System Configuration

[0276] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[0277] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0278] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0279] 3. Software Components: These include speech recognition engines, natural language processing (NLP) engines, emotion recognition algorithms, emotion engines, response generation engines, and synthesized speech engines. Specifically, this includes software such as Google Cloud Speech-to-Text, spaCy, BERT, IBM Watson® Tone Analyzer, OpenAI GPT-3, and Amazon Polly.

[0280] Overview of program processing

[0281] 1. Collection of audio and text data

[0282] Users can use the application to make inquiries via voice or text. For example, they might say, "My recent order hasn't arrived yet, what's going on?"

[0283] The device collects user input and sends it to a server via the internet.

[0284] 2. Conversion of Voice Data

[0285] The server converts the received voice data into text data using a speech recognition engine (Google Cloud Speech-to-Text).

[0286] The converted text data is "I haven't received my recent order. What's going on?"

[0287] 3. Sentiment Recognition

[0288] The server sends the text data to an NLP engine (spaCy, BERT) and analyzes the sentiment using a sentiment recognition algorithm (IBM Watson Tone Analyzer).

[0289] For example, it identifies the sentiment of "uneasy" from the text "I haven't received my recent order. What's going on?"

[0290] 4. Response Generation

[0291] Based on the sentiment recognition result, the server uses a response generation engine (OpenAI GPT-3) to generate an appropriate response.

[0292] An example of the generated response is "We apologize for the inconvenience. We will check immediately and inform you of the tracking information."

[0293] 5. Sending and Displaying the Response

[0294] The server sends the generated response to the terminal as text data.

[0295] The terminal displays the received text to the user, and if necessary, uses a text-to-speech engine (Amazon Polly) to convert the text into voice and play it for the user.

[0296] Specific Example

[0297] Usage Example in Customer Service

[0298] 1. The user inputs "The ordered product hasn't arrived yet. What's going on?"

[0299] 2. The terminal sends this input to the server.

[0300] 3. The server receives the input data, converts it to text using a speech recognition engine, and recognizes the emotion of "uneasiness" using an emotion engine.

[0301] 4. The server generates a response saying "We will check the situation immediately and provide tracking information."

[0302] 5. The terminal displays the response to the user and plays it as voice if necessary.

[0303] Usage Example in the Educational Field

[0304] 1. The user (student) inputs "I don't understand this problem." by voice.

[0305] 2. The terminal collects the voice and sends it to the server.

[0306] 3. The server converts the voice to text and recognizes the emotion of "confusion" using an emotion engine.

[0307] 4. The server generates a response saying "Please tell me specifically which part you don't understand. Let's solve it together."

[0308] 5. The terminal reads the text aloud as synthesized voice and plays it for the user.

[0309] According to this invention, appropriate conversations based on the user's emotions become possible, and particularly, the accuracy of emotion recognition is improved by introducing an emotion engine. As a result, the user experience is greatly improved.

[0310] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0311] Step 1:

[0312] Users use the application to make inquiries via voice or text. For example, they might say to customer service, "My recent order hasn't arrived yet, what's going on?"

[0313] Input: User voice or text input

[0314] Output: Collected audio or text data

[0315] Specific operation: The device collects the user's voice via the microphone, or text input via the keyboard. This data is temporarily stored on the device.

[0316] Step 2:

[0317] The device sends the voice or text data entered by the user to the server.

[0318] Input: Collected audio or text data

[0319] Output: Audio or text data sent to the server

[0320] Specific operation: The device sends voice or text data to the server via the internet. Encrypted communication (e.g., SSL / TLS) is often used in this process.

[0321] Step 3:

[0322] The server converts the received audio data into text data using a speech recognition engine (e.g., speech recognition engine).

[0323] Input: Audio data

[0324] Output: Text data

[0325] Specific operation: The server invokes the speech recognition engine and provides voice data as input. The engine analyzes the voice waveform and converts it into text. For example, the voice saying "My recent order hasn't arrived, what's going on?" becomes the text "My recent order hasn't arrived, what's going on?".

[0326] Step 4:

[0327] The server sends the converted text data to a natural language processing (NLP) engine (e.g., a natural language processing engine) for sentiment recognition.

[0328] Input: Text data

[0329] Output: Emotional data (e.g., anxiety)

[0330] Specific operation: The server sends text data to the NLP engine, which performs contextual analysis. The emotion engine is used to recognize the user's emotions from the analyzed data. For example, the emotion "anxiety" is identified from the text "My recent order hasn't arrived, what's going on?".

[0331] Step 5:

[0332] The server generates an appropriate response using a response generation engine (e.g., a generative AI model) based on the emotion recognition results.

[0333] Input: Sentiment data, text data

[0334] Output: Response text

[0335] Specific operation: The server takes sentiment data and text data as input and invokes the response generation engine. The engine generates a response based on the provided data. For example, a response such as "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." might be generated.

[0336] Step 6:

[0337] The server sends the generated response text to the terminal.

[0338] Input: Response text

[0339] Output: Sent response text

[0340] Specific operation: The server sends the generated response text to the terminal. This also uses encrypted communication.

[0341] Step 7:

[0342] The terminal displays the received response text to the user. If necessary, it uses a text-to-speech engine (e.g., a speech synthesis engine) to convert the text to speech and play it back to the user.

[0343] Input: Response text

[0344] Output: Text displayed to the user or audio played back.

[0345] Specific operation: The device displays the received response text to the user. If necessary, it inputs the text into a text-to-speech engine and plays the generated audio through the speaker. For example, the user can see the text "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information" on the device screen, or hear the same content in audio.

[0346] In this way, a series of processes from user input to response generation are executed, creating a mechanism that improves the user experience.

[0347] (Application Example 2)

[0348] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0349] Traditional dialogue systems struggle to accurately recognize user emotions and generate appropriate and empathetic responses based on them. Furthermore, low accuracy in emotion recognition is likely to impair the user experience. In particular, customer service requires the rapid resolution of user anxiety and confusion, a task often failing to be adequately achieved by conventional systems.

[0350] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data or text data input from the user, means for converting voice data into text data, means for analyzing text data and performing emotion recognition, means for generating an appropriate response using natural language generation technology, means for converting the generated response into synthesized speech and displaying or playing it back to the user, means for inputting prompt sentences using a generation AI model, and means for adjusting based on the user's emotions and providing an empathetic response. This enables highly accurate emotion recognition and empathetic response generation, greatly improving the user experience.

[0351] "Voice data input by the user" refers to voice information provided by the user through a voice input device.

[0352] "Methods for converting to text data" refers to the process of converting speech information into text data using speech recognition technology.

[0353] "Means for performing emotion recognition" refers to methods that use natural language processing technology to analyze and identify a user's emotions from text data.

[0354] "Means for generating appropriate responses using natural language generation technology" refers to a function that generates appropriate responses based on recognized emotions and input information using a generative AI model.

[0355] "Means for converting to synthesized speech and displaying or playing it for the user" refers to the process of converting generated text responses into speech and providing it to the user.

[0356] "A method for inputting prompt sentences using a generative AI model" refers to a technique that inputs prompt sentences suitable for a generative AI model and generates an appropriate response.

[0357] "Means of adjusting based on emotions and providing empathetic responses" refers to a function that modifies the response content based on recognized emotions and provides empathetic responses that take the user's feelings into consideration.

[0358] One embodiment of this invention is a dialogue system that recognizes emotion from user input data, generates an appropriate response using natural language generation technology, and provides it as synthesized speech. This system uses the following hardware and software.

[0359] 1. Hardware to be used

[0360] User devices: Devices such as smartphones, tablets, and personal computers. These devices collect voice or text input from users and send it to the server.

[0361] Server: A computer system that performs data analysis and processing.

[0362] 2. Software to use

[0363] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert speech data into text data.

[0364] Emotion Recognition Engine: Using HUME AI, the system analyzes user emotions from text data.

[0365] The system uses a natural language generation engine, OpenAI GPT-4 (registered trademark), to generate appropriate responses based on emotion recognition results.

[0366] Text-to-speech engine: Uses the Google Text-to-Speech API to convert generated text responses into speech.

[0367] 3. Data flow and operating procedures

[0368] The server collects and processes voice or text data entered by the user. Specifically, the process proceeds as follows:

[0369] 1. Collection and conversion of audio data

[0370] The user speaks into the device and says, "The item I ordered hasn't arrived yet."

[0371] The device collects voice data and sends it to the server.

[0372] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[0373] 2. Emotion recognition

[0374] The server sends text data to HUME AI for emotion recognition.

[0375] The text "The item I ordered hasn't arrived yet" conveys the emotion of "anxiety."

[0376] 3. Response generation

[0377] The server sends a prompt message to OpenAI GPT-4 based on the emotion recognition results and text data, and generates a response.

[0378] The generated response is, "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0379] 4. Provision of synthesized speech and responses

[0380] The server converts the generated text response into speech using the Google Text-to-Speech API.

[0381] The device plays synthesized speech or displays text to the user.

[0382] Specific examples and prompt statements

[0383] Specific Scenario

[0384] 1. The user opens the app and voice-inputs, "My ordered item hasn't arrived yet."

[0385] 2. The app converts the voice to text and sends it to the server as "My ordered items have not yet arrived."

[0386] 3. The server uses HUME AI to recognize emotions in the text and identifies the emotion "anxiety."

[0387] 4. The server sends a prompt to OpenAI GPT-4 stating, "The user is feeling anxious. Please check the order status immediately," and generates a response.

[0388] 5. Provide the user with the generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0389] Examples of prompts for generative AI models

[0390] prompt:

[0391] User text input: "The item I ordered has not yet arrived."

[0392] Recognized emotion: “Anxiety”

[0393] Please generate an appropriate response:

[0394] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0395] Step 1:

[0396] The user makes a voice input into the device. For example, the user might say, "The item I ordered hasn't arrived yet." This voice data is collected via the device's microphone.

[0397] Input: Voice data "The item I ordered has not yet arrived."

[0398] Output: Audio data stored on the device

[0399] Step 2:

[0400] The terminal sends the collected audio data to the server. This transfer of audio data to the server in real time uses a network connection.

[0401] Input: Audio data stored on the device

[0402] Output: Audio data sent to the server

[0403] Step 3:

[0404] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The audio data is passed to the API, and the converted text data is received as the result.

[0405] Input: Audio data sent to the server

[0406] Data processing: Converting audio data to text data.

[0407] Output: Text data "The item I ordered has not yet arrived."

[0408] Step 4:

[0409] The server sends text data to HUME AI for emotion recognition. HUME AI analyzes the text data and recognizes the user's emotion as "anxiety."

[0410] Input: Text data "The item I ordered has not yet arrived."

[0411] Data processing: Apply emotion recognition algorithms.

[0412] Output: Emotion tag "anxiety"

[0413] Step 5:

[0414] The server generates an appropriate response to the OpenAI GPT-4 model using prompts. These prompts include user text input and recognized emotions.

[0415] Example of a prompt:

[0416] User text input: "The item I ordered has not yet arrived."

[0417] Recognized emotion: “Anxiety”

[0418] Please generate an appropriate response:

[0419] Input: Text data "The item I ordered has not yet arrived", emotion tag "anxiety", prompt text

[0420] Data processing: Uses a natural language generation AI model.

[0421] Output: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0422] Step 6:

[0423] The server uses the Google Text-to-Speech API to convert the generated response text into audio data. The generated response text is passed to the API and the audio data is received.

[0424] Input: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0425] Data processing: Converting text data to audio data.

[0426] Output: Synthesized speech data

[0427] Step 7:

[0428] The server sends synthesized speech data and text data to the terminal. Data is transferred in real time over the network.

[0429] Input: synthesized speech data, generated response text

[0430] Output: Synthesized speech data and text data sent to the terminal

[0431] Step 8:

[0432] The device plays the received synthesized speech data to the user and displays the generated response text. This allows the user to receive a response both audibly and visually.

[0433] Input: Synthesized speech data and text data sent to the terminal.

[0434] Output: Played audio response, displayed text response

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0436] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0438] [Second Embodiment]

[0439] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0440] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0441] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0443] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0445] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0446] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0447] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0450] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0451] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The system's operation and specific usage examples are shown below.

[0452] System Configuration

[0453] This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0454] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0455] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0456] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, a synthesized speech engine, and so on.

[0457] Program processing

[0458] 1. Collection of audio and text data

[0459] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[0460] The device collects voice data and sends it to the server.

[0461] 2. Converting audio data

[0462] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[0463] 3. Emotion recognition

[0464] The server sends text data to the NLP engine, which then executes an emotion recognition algorithm. As a result of the emotion recognition, "anxiety" is extracted.

[0465] 4. Response generation

[0466] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0467] 5. Sending and displaying responses

[0468] The server sends the generated response to the terminal as text data.

[0469] The terminal displays the received text to the user. If necessary, it plays a voice response using a text-to-speech engine.

[0470] Specific example

[0471] Examples of use in customer service

[0472] 1. The user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0473] 2. The terminal sends this input to the server.

[0474] 3. The server analyzes the input and recognizes the emotion of "anxiety."

[0475] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[0476] 5. The terminal displays a response to the user.

[0477] Examples of use in the field of education

[0478] 1. The user (student) inputs "I don't understand this problem" by voice.

[0479] 2. The device collects audio and sends it to the server.

[0480] 3. The server converts the speech into text and recognizes the emotion of "confusion."

[0481] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0482] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0483] This system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0484] The following describes the processing flow.

[0485] Step 1:

[0486] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[0487] Step 2:

[0488] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[0489] Step 3:

[0490] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[0491] Step 4:

[0492] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[0493] Step 5:

[0494] The server sends the converted text data to a natural language processing (NLP) engine, which then executes an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0495] Step 6:

[0496] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0497] Step 7:

[0498] The server sends the generated response to the terminal as text data.

[0499] Step 8:

[0500] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[0501] (Example 1)

[0502] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0503] Traditional dialogue systems have faced the challenge of difficulty in considering emotions when generating appropriate responses from user voice or text input. In particular, when users were experiencing emotions such as anxiety, confusion, or anger, the dialogue system often failed to provide appropriate responses that reflected those emotions. This resulted in a degraded user experience.

[0504] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0505] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, means for generating an appropriate response based on the emotion recognition result using a generative AI model, and means for displaying or playing the generated response to the user. This enables appropriate dialogue that responds to the user's emotions, thereby improving the user experience.

[0506] "Audio data" refers to data that records the voice spoken by a user in digital format.

[0507] "Text data" refers to data that represents audio data or input character information as a string of characters.

[0508] "Emotion recognition" is the process of analyzing emotions contained in text data and identifying the type of emotion.

[0509] A "generative AI model" is an artificial intelligence model trained using large-scale datasets, and is used for text generation and response generation.

[0510] An "appropriate response" is a response that is relevant to the situation and generated based on the user's input and emotions.

[0511] "Display" refers to providing the generated text response to the user visually.

[0512] "Playback" refers to converting the generated text response into audio and providing it to the user audibly.

[0513] "Collection means" refers to methods or devices for acquiring voice data or text data from users.

[0514] "Conversion means" refers to methods or devices for converting audio data into text data.

[0515] "Analysis means" refers to methods or devices for analyzing received text data and identifying its syntax and sentiment.

[0516] "Generative means" refers to a method or device that generates an appropriate response using a generative AI model based on the results of emotion recognition.

[0517] A "user" refers to a person who uses a dialogue system to input text or voice.

[0518] A "server" refers to a computer system used for data processing and analysis.

[0519] A "terminal" refers to a device used to collect user voice and text input and transmit it to a server.

[0520] Modes for carrying out the invention

[0521] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0522] System Configuration

[0523] 1. Terminal

[0524] A terminal is a device that collects voice or text input from a user and sends it to a server. Examples include smartphones, tablets, and personal computers. Terminals are equipped with microphones, keyboards, touchscreens, and other features that allow them to collect user input data.

[0525] 2. Server

[0526] A server is a computer system that receives, analyzes, and processes data sent from terminals. This computer system is equipped with a high-speed CPU or GPU, sufficient memory and storage, and processes data in real time.

[0527] 3. Software Components

[0528] The system's software components include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a speech synthesis engine. Specifically, these are as follows:

[0529] Speech recognition engine: Converts speech data into text data. Google Cloud Speech-to-Text is used as an example.

[0530] Natural Language Processing (NLP) engine: Analyzes text data and performs syntactic and semantic analysis. A concrete example is the Google Cloud Natural Language API.

[0531] Emotion recognition algorithm: Extracts emotions from text data analyzed by an NLP engine.

[0532] Response generation engine: Uses a generative AI model (e.g., GPT-3) to generate appropriate responses based on emotion recognition results.

[0533] Text-to-speech engine: Converts generated responses into speech. A concrete example is Google Cloud Text-to-Speech.

[0534] Specific usage examples

[0535] Examples of use in customer service

[0536] A user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0537] The terminal sends this input to the server.

[0538] The server analyzes the input and recognizes the emotion of "anxiety."

[0539] The server generates an empathetic response saying, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0540] The device displays a response to the user. If necessary, it plays the response using synthesized speech.

[0541] Examples of use in the field of education

[0542] The user (student) inputs "I don't understand this problem" by voice.

[0543] The device collects audio and sends it to the server.

[0544] The server converts the speech into text and recognizes the emotion of "confusion."

[0545] The server generates a response saying, "Please tell me specifically what part you don't understand. Let's solve it together."

[0546] The device reads the text aloud using synthesized speech and plays it back to the user.

[0547] Through the above, this system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0548] Example of a prompt

[0549] "The user is feeling anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution."

[0550] "A student is confused by a math problem. Generate a response that suggests you try to solve the problem together."

[0551] These prompts generate appropriate responses that reflect the user's emotions.

[0552] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0553] Step 1: Collect audio and text data

[0554] The user provides input to the system via voice or text. For example, they might voice-input, "My recent order hasn't arrived yet, what's going on?" This input is collected by the terminal.

[0555] Input: [User's voice or text data]

[0556] Output: [Audio data or text data]

[0557] Specific operation: The device uses its built-in microphone and text input function to collect user voice and text, and sends them to the server as raw data.

[0558] Step 2: Convert audio data

[0559] The server receives the audio data sent from the terminal. The received audio data is passed to a speech recognition engine, which converts the audio into text data. Using the speech recognition engine (for example, Google Cloud Speech-to-Text), the server outputs the text data "My recent order hasn't arrived yet, what's going on?"

[0560] Input: [Audio data]

[0561] Output: [Text data]

[0562] Specific operation: The server converts the audio data into a string via a speech recognition engine and saves it as text data.

[0563] Step 3: Emotion Recognition

[0564] The server sends the converted text data to an NLP engine (e.g., Google Cloud Natural Language API) to run an emotion recognition algorithm. It analyzes the text "My recent order hasn't arrived yet, what's going on?" and extracts the emotion "anxiety" contained within it.

[0565] Input: [Text data]

[0566] Output: [emotion data]

[0567] Specific operation: The server uses an NLP engine to parse the text data and extracts emotional information from the text using an emotion recognition algorithm.

[0568] Step 4: Response Generation

[0569] The server uses an AI model (e.g., GPT-3) to generate an appropriate response based on the emotion recognition result "anxiety." The prompt used is "The user is anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution." The response generated is "We apologize for the inconvenience. We will check immediately and provide you with tracking information."

[0570] Input: [emotion data]

[0571] Output: [Response text data]

[0572] Specific operation: The response generation engine inputs a prompt sentence into the generation AI model and generates an appropriate response sentence based on emotion.

[0573] Step 5: Sending and displaying the response

[0574] The server sends the generated response text data to the terminal. The terminal receives this data and displays it to the user. If necessary, it converts it to speech using a text-to-speech engine (e.g., Google Cloud Text-to-Speech) and plays it back to the user.

[0575] Input: [Response text data]

[0576] Output: [Display or audio data]

[0577] Specific operation: The device displays text data using a display UI and plays it back as audio using a text-to-speech engine.

[0578] (Application Example 1)

[0579] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0580] Conventional dialogue systems lacked the ability to provide appropriate responses based on user emotions, making it difficult to fully improve the user experience. Furthermore, in specific applications such as food delivery services, there is a need to generate rapid and emotionally resonant responses to user inquiries. Existing technologies have not adequately addressed these challenges.

[0581] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0582] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, and means for generating a response using a generative AI model based on the emotion recognition results. This makes it possible to accurately recognize the user's emotions and quickly generate an appropriate response that is in line with those emotions.

[0583] "Audio data" refers to the audio signal that a user produces using an audio input device such as a microphone.

[0584] "Text data" refers to audio data that has been converted into a string by a speech recognition engine, or string information directly entered by the user.

[0585] "Emotion recognition" refers to the process of analyzing collected text data to identify the user's emotional state (e.g., anxiety, joy, anger, etc.).

[0586] An "appropriate response" refers to a reply message that is generated based on the results of emotion recognition and is tailored to the user's emotions and the content of their inquiry.

[0587] A "generative AI model" refers to an artificial intelligence algorithm that uses natural language processing to generate appropriate responses to user input.

[0588] "Means of generating responses" refers to the process of creating appropriate responses in text or audio format using emotion recognition results and generative AI models.

[0589] "Synthesized speech" refers to speech synthesis technology that converts text data into audio data and plays it back for the user.

[0590] "Means for displaying or reproducing" means devices and methods for providing the generated response to the user visually or audibly.

[0591] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses. The system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0592] System Configuration

[0593] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0594] 2. Server: A computer system that receives, analyzes, and processes data transmitted from terminals. The server includes software components such as a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, and a generative AI model.

[0595] 3. Software Components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a synthesized speech engine. These are necessary to analyze user input data and generate appropriate responses.

[0596] Program processing

[0597] When a user uses the application and says, "My pizza hasn't arrived yet," the device collects the audio data and sends it to the server. The server converts the audio data into text data through a speech recognition engine. This converted text data is then sent to an NLP engine, which applies an emotion recognition algorithm to recognize the emotion of "anxiety."

[0598] The server then uses a generative AI model to generate an appropriate response based on the emotion recognition results. For example, it might say, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." The generated response is sent to the device in text or synthesized speech format and displayed or played back to the user.

[0599] Hardware and software used

[0600] Speech recognition engine: This is an engine that converts speech data into text data, and commonly used examples include "Google Speech Recognition API" and "Microsoft Azure Speech Services".

[0601] Natural Language Processing (NLP): Used to analyze text data and recognize user intent and emotions. Models such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa are used for this purpose.

[0602] Emotion recognition algorithms: These are algorithms for identifying emotions from text data. For example, emotion recognition models such as "j-hartmann / emotion-english-distilroberta-base" are applied.

[0603] Generative AI models: These are models that generate appropriate responses in response to user input, and OpenAI's "GPT-3" is one example.

[0604] Specific example

[0605] Suppose a user voice-inquiries to the app saying, "My pizza order hasn't arrived yet, what's going on?" This voice input is converted to text, and the emotion of "anxiety" is recognized. The server then generates a response saying, "We apologize for the inconvenience. We will check immediately and provide you with tracking information," which is then displayed or played back to the user.

[0606] Example of a prompt:

[0607] A user has inquired, "My pizza hasn't arrived yet..." This inquiry has caused the user to feel anxious. Please generate an appropriate response.

[0608] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0609] Step 1:

[0610] A user uses a food delivery application and says, "My pizza hasn't arrived yet..." The device uses its built-in microphone to collect this voice input data. The input data is the user's voice signal and is sent to the server after collection.

[0611] Step 2:

[0612] The server receives the audio data sent from the terminal. The server uses a speech recognition engine to convert the audio data into text data. In this process, the speech recognition engine (e.g., "Google Speech Recognition API") converts the audio signal into string information, and the output is the text data "My pizza hasn't arrived yet..."

[0613] Step 3:

[0614] The server sends the converted text data to a natural language processing (NLP) engine. The NLP engine (e.g., "BERT" or "RoBERTa") analyzes this text data and processes it to identify the user's intent and emotions. This process includes using a language model to perform contextual analysis and generate the data necessary for the emotion recognition algorithm. As output, the text data is processed into data for emotion analysis.

[0615] Step 4:

[0616] The analyzed data is sent to an emotion recognition algorithm on the server. The emotion recognition algorithm (e.g., "j-hartmann / emotion-english-distilroberta-base") extracts emotions from the text data. This specific operation involves assigning specific emotion labels (e.g., "anxiety," "joy," "anger," etc.). The output is an emotion recognition result of "anxiety."

[0617] Step 5:

[0618] The server generates a response using a generative AI model based on the obtained emotion recognition results. The generative AI model (e.g., "GPT-3") creates an appropriate response based on emotion and text data. This specific operation involves a process of requesting the model to generate a response using a prompt statement. The output is the generated text response, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0619] Step 6:

[0620] The generated response is sent from the server to the terminal. The terminal then displays or plays this text response to the user. This specific action includes displaying the text on the screen and converting it to speech using a speech synthesis engine (e.g., "Microsoft Azure Speech Services") and playing it through the speaker.

[0621] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0622] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. In particular, it features a "emotion engine" for emotion recognition, enabling highly accurate emotion recognition and empathetic responses. The system's operation and specific usage examples are described below.

[0623] System Configuration

[0624] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[0625] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0626] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0627] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, an emotion engine, a response generation engine, a synthesized speech engine, and so on.

[0628] Program processing

[0629] 1. Collection of audio and text data

[0630] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[0631] The device collects voice data and sends it to the server.

[0632] 2. Converting audio data

[0633] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[0634] 3. Emotion recognition

[0635] The server sends text data to the NLP engine, which then uses the emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0636] 4. Response generation

[0637] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0638] 5. Sending and displaying responses

[0639] The server sends the generated response to the terminal as text data.

[0640] The terminal displays the received text to the user. If necessary, the terminal uses a text-to-speech engine to convert the text into speech and play the response.

[0641] Specific example

[0642] Examples of use in customer service

[0643] 1. The user enters a message into customer service saying, "My ordered item hasn't arrived yet, what's going on?"

[0644] 2. The terminal sends this input to the server.

[0645] 3. The server analyzes the input data and uses an emotion engine to recognize the emotion of "anxiety."

[0646] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[0647] 5. The terminal displays a response to the user.

[0648] Examples of use in the field of education

[0649] 1. The user (student) inputs "I don't understand this problem" by voice.

[0650] 2. The device collects audio and sends it to the server.

[0651] 3. The server converts the audio into text and uses an emotion engine to recognize the emotion "confusion."

[0652] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0653] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0654] This system enables appropriate dialogue based on the user's emotions, and the introduction of an emotion engine, in particular, improves the accuracy of emotion recognition. This greatly enhances the user experience.

[0655] The following describes the processing flow.

[0656] Step 1:

[0657] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[0658] Step 2:

[0659] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[0660] Step 3:

[0661] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[0662] Step 4:

[0663] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[0664] Step 5:

[0665] The server sends the converted text data to a natural language processing (NLP) engine, which then uses an emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0666] Step 6:

[0667] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0668] Step 7:

[0669] The server sends the generated response to the terminal as text data.

[0670] Step 8:

[0671] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0674] In modern dialogue systems, accurately recognizing user emotions and providing appropriate, empathetic responses is challenging. Conventional systems suffer from low accuracy in emotion recognition, resulting in a poor user experience.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0676] In this invention, the server includes means for collecting voice data input from a user, means for converting the voice data into text data, means for analyzing the text data and performing natural language processing, means for performing emotion recognition using the data analyzed by the natural language processing, means for generating an appropriate response based on the result of the emotion recognition, and means for displaying or playing the generated response to the user. This makes it possible to recognize the user's emotions with higher accuracy and provide an empathetic and appropriate response accordingly.

[0677] A "user" refers to a person who uses the system.

[0678] "Voice data" refers to digital data that electronically records the voice input by the user.

[0679] "Text data" refers to data obtained by converting audio data into written text.

[0680] "Natural language processing" refers to the technology that enables computers to understand and analyze human language.

[0681] "Emotion recognition" refers to the technology that identifies a user's emotions from text data.

[0682] "Response" refers to the reply or answer that the system generates to the user.

[0683] "Means of generation" refers to a function that generates a response using a specific process or algorithm.

[0684] "Means for displaying or reproducing" refers to technologies or devices for showing or letting the user hear the generated response.

[0685] Modes for carrying out the invention

[0686] This invention is a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The actual operation and usage examples of the system are shown below.

[0687] System Configuration

[0688] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[0689] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0690] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0691] 3. Software Components: These include speech recognition engines, natural language processing (NLP) engines, emotion recognition algorithms, emotion engines, response generation engines, and synthesized speech engines. Specifically, this includes software such as Google Cloud Speech-to-Text, spaCy, BERT, IBM Watson Tone Analyzer, OpenAI GPT-3, and Amazon Polly.

[0692] Overview of program processing

[0693] 1. Collection of audio and text data

[0694] Users can use the application to make inquiries via voice or text. For example, they might say, "My recent order hasn't arrived yet, what's going on?"

[0695] The device collects user input and sends it to a server via the internet.

[0696] 2. Converting audio data

[0697] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text).

[0698] The converted text data is, "My recent order hasn't arrived yet, what's going on?"

[0699] 3. Emotion recognition

[0700] The server sends text data to an NLP engine (spaCy, BERT), which then analyzes the emotions using an emotion recognition algorithm (IBM Watson Tone Analyzer).

[0701] For example, the text "My recent order hasn't arrived yet, what's going on?" can be used to identify the emotion of "anxiety."

[0702] 4. Response generation

[0703] The server uses a response generation engine (OpenAI GPT-3) to generate an appropriate response based on the emotion recognition results.

[0704] One example of a response that might be generated is, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0705] 5. Sending and displaying responses

[0706] The server sends the generated response to the terminal as text data.

[0707] The device displays the received text to the user, and, if necessary, uses a text-to-speech engine (Amazon Polly) to convert the text into speech and play it back to the user.

[0708] Specific example

[0709] Examples of use in customer service

[0710] 1. The user enters, "My ordered item hasn't arrived yet. What's going on?"

[0711] 2. The terminal sends this input to the server.

[0712] 3. The server receives the input data, converts it to text using a speech recognition engine, and recognizes the emotion "anxiety" using an emotion engine.

[0713] 4. The server generates a response saying, "We will check the situation immediately and provide tracking information."

[0714] 5. The device displays the response to the user and plays it back as audio if necessary.

[0715] Examples of use in the field of education

[0716] 1. The user (student) inputs "I don't understand this problem" by voice.

[0717] 2. The device collects audio and sends it to the server.

[0718] 3. The server converts the audio to text, and the emotion engine recognizes the emotion as "confused."

[0719] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0720] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0721] This invention enables appropriate dialogue based on the user's emotions, and the accuracy of emotion recognition is particularly improved by the introduction of an emotion engine. This greatly enhances the user experience.

[0722] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0723] Step 1:

[0724] Users use the application to make inquiries via voice or text. For example, they might say to customer service, "My recent order hasn't arrived yet, what's going on?"

[0725] Input: User voice or text input

[0726] Output: Collected audio or text data

[0727] Specific operation: The device collects the user's voice via the microphone, or text input via the keyboard. This data is temporarily stored on the device.

[0728] Step 2:

[0729] The device sends the voice or text data entered by the user to the server.

[0730] Input: Collected audio or text data

[0731] Output: Audio or text data sent to the server

[0732] Specific operation: The device sends voice or text data to the server via the internet. Encrypted communication (e.g., SSL / TLS) is often used in this process.

[0733] Step 3:

[0734] The server converts the received audio data into text data using a speech recognition engine (e.g., speech recognition engine).

[0735] Input: Audio data

[0736] Output: Text data

[0737] Specific operation: The server invokes the speech recognition engine and provides voice data as input. The engine analyzes the voice waveform and converts it into text. For example, the voice saying "My recent order hasn't arrived, what's going on?" becomes the text "My recent order hasn't arrived, what's going on?".

[0738] Step 4:

[0739] The server sends the converted text data to a natural language processing (NLP) engine (e.g., a natural language processing engine) for sentiment recognition.

[0740] Input: Text data

[0741] Output: Emotional data (e.g., anxiety)

[0742] Specific operation: The server sends text data to the NLP engine, which performs contextual analysis. The emotion engine is used to recognize the user's emotions from the analyzed data. For example, the emotion "anxiety" is identified from the text "My recent order hasn't arrived, what's going on?".

[0743] Step 5:

[0744] The server generates an appropriate response using a response generation engine (e.g., a generative AI model) based on the emotion recognition results.

[0745] Input: Sentiment data, text data

[0746] Output: Response text

[0747] Specific operation: The server takes sentiment data and text data as input and invokes the response generation engine. The engine generates a response based on the provided data. For example, a response such as "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." might be generated.

[0748] Step 6:

[0749] The server sends the generated response text to the terminal.

[0750] Input: Response text

[0751] Output: Sent response text

[0752] Specific operation: The server sends the generated response text to the terminal. This also uses encrypted communication.

[0753] Step 7:

[0754] The terminal displays the received response text to the user. If necessary, it uses a text-to-speech engine (e.g., a speech synthesis engine) to convert the text to speech and play it back to the user.

[0755] Input: Response text

[0756] Output: Text displayed to the user or audio played back.

[0757] Specific operation: The device displays the received response text to the user. If necessary, it inputs the text into a text-to-speech engine and plays the generated audio through the speaker. For example, the user can see the text "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information" on the device screen, or hear the same content in audio.

[0758] In this way, a series of processes from user input to response generation are executed, creating a mechanism that improves the user experience.

[0759] (Application Example 2)

[0760] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0761] Traditional dialogue systems struggle to accurately recognize user emotions and generate appropriate and empathetic responses based on them. Furthermore, low accuracy in emotion recognition is likely to impair the user experience. In particular, customer service requires the rapid resolution of user anxiety and confusion, a task often failing to be adequately achieved by conventional systems.

[0762] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data or text data input from the user, means for converting voice data into text data, means for analyzing text data and performing emotion recognition, means for generating an appropriate response using natural language generation technology, means for converting the generated response into synthesized speech and displaying or playing it back to the user, means for inputting prompt sentences using a generation AI model, and means for adjusting based on the user's emotions and providing an empathetic response. This enables highly accurate emotion recognition and empathetic response generation, greatly improving the user experience.

[0763] "Voice data input by the user" refers to voice information provided by the user through a voice input device.

[0764] "Methods for converting to text data" refers to the process of converting speech information into text data using speech recognition technology.

[0765] "Means for performing emotion recognition" refers to methods that use natural language processing technology to analyze and identify a user's emotions from text data.

[0766] "Means for generating appropriate responses using natural language generation technology" refers to a function that generates appropriate responses based on recognized emotions and input information using a generative AI model.

[0767] "Means for converting to synthesized speech and displaying or playing it for the user" refers to the process of converting generated text responses into speech and providing it to the user.

[0768] "A method for inputting prompt sentences using a generative AI model" refers to a technique that inputs prompt sentences suitable for a generative AI model and generates an appropriate response.

[0769] "Means of adjusting based on emotions and providing empathetic responses" refers to a function that modifies the response content based on recognized emotions and provides empathetic responses that take the user's feelings into consideration.

[0770] One embodiment of this invention is a dialogue system that recognizes emotion from user input data, generates an appropriate response using natural language generation technology, and provides it as synthesized speech. This system uses the following hardware and software.

[0771] 1. Hardware to be used

[0772] User devices: Devices such as smartphones, tablets, and personal computers. These devices collect voice or text input from users and send it to the server.

[0773] Server: A computer system that performs data analysis and processing.

[0774] 2. Software to use

[0775] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert speech data into text data.

[0776] Emotion Recognition Engine: Using HUME AI, the system analyzes user emotions from text data.

[0777] Natural language generation engine: Uses OpenAI GPT-4 to generate appropriate responses based on emotion recognition results.

[0778] Text-to-speech engine: Uses the Google Text-to-Speech API to convert generated text responses into speech.

[0779] 3. Data flow and operating procedures

[0780] The server collects and processes voice or text data entered by the user. Specifically, the process proceeds as follows:

[0781] 1. Collection and conversion of audio data

[0782] The user speaks into the device and says, "The item I ordered hasn't arrived yet."

[0783] The device collects voice data and sends it to the server.

[0784] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[0785] 2. Emotion recognition

[0786] The server sends text data to HUME AI for emotion recognition.

[0787] The text "The item I ordered hasn't arrived yet" conveys the emotion of "anxiety."

[0788] 3. Response generation

[0789] The server sends a prompt message to OpenAI GPT-4 based on the emotion recognition results and text data, and generates a response.

[0790] The generated response is, "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0791] 4. Provision of synthesized speech and responses

[0792] The server converts the generated text response into speech using the Google Text-to-Speech API.

[0793] The device plays synthesized speech or displays text to the user.

[0794] Specific examples and prompt statements

[0795] Specific Scenario

[0796] 1. The user opens the app and voice-inputs, "My ordered item hasn't arrived yet."

[0797] 2. The app converts the voice to text and sends it to the server as "My ordered items have not yet arrived."

[0798] 3. The server uses HUME AI to recognize emotions in the text and identifies the emotion "anxiety."

[0799] 4. The server sends a prompt to OpenAI GPT-4 stating, "The user is feeling anxious. Please check the order status immediately," and generates a response.

[0800] 5. Provide the user with the generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0801] Examples of prompts for generative AI models

[0802] prompt:

[0803] User text input: "The item I ordered has not yet arrived."

[0804] Recognized emotion: "anxiety"

[0805] Please generate an appropriate response:

[0806] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0807] Step 1:

[0808] The user makes a voice input into the device. For example, the user might say, "The item I ordered hasn't arrived yet." This voice data is collected via the device's microphone.

[0809] Input: Voice data "The item I ordered has not yet arrived."

[0810] Output: Audio data stored on the device

[0811] Step 2:

[0812] The terminal sends the collected audio data to the server. This transfer of audio data to the server in real time uses a network connection.

[0813] Input: Audio data stored on the device

[0814] Output: Audio data sent to the server

[0815] Step 3:

[0816] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The audio data is passed to the API, and the converted text data is received as the result.

[0817] Input: Audio data sent to the server

[0818] Data processing: Converting audio data to text data.

[0819] Output: Text data "The item I ordered has not yet arrived."

[0820] Step 4:

[0821] The server sends text data to HUME AI for emotion recognition. HUME AI analyzes the text data and recognizes the user's emotion as "anxiety."

[0822] Input: Text data "The item I ordered has not yet arrived."

[0823] Data processing: Apply emotion recognition algorithms.

[0824] Output: Emotion tag "anxiety"

[0825] Step 5:

[0826] The server generates an appropriate response to the OpenAI GPT-4 model using prompts. These prompts include user text input and recognized emotions.

[0827] Example of a prompt:

[0828] User text input: "The item I ordered has not yet arrived."

[0829] Recognized emotion: "anxiety"

[0830] Please generate an appropriate response:

[0831] Input: Text data "The item I ordered has not yet arrived", emotion tag "anxiety", prompt text

[0832] Data processing: Uses a natural language generation AI model.

[0833] Output: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0834] Step 6:

[0835] The server uses the Google Text-to-Speech API to convert the generated response text into audio data. The generated response text is passed to the API and the audio data is received.

[0836] Input: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[0837] Data processing: Converting text data to audio data.

[0838] Output: Synthesized speech data

[0839] Step 7:

[0840] The server sends synthesized speech data and text data to the terminal. Data is transferred in real time over the network.

[0841] Input: synthesized speech data, generated response text

[0842] Output: Synthesized speech data and text data sent to the terminal

[0843] Step 8:

[0844] The device plays the received synthesized speech data to the user and displays the generated response text. This allows the user to receive a response both audibly and visually.

[0845] Input: Synthesized speech data and text data sent to the terminal.

[0846] Output: Played audio response, displayed text response

[0847] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0848] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0849] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0850] [Third Embodiment]

[0851] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0852] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0853] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0854] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0855] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0856] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0857] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0858] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0859] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0860] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0861] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0862] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0863] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The system's operation and specific usage examples are shown below.

[0864] System Configuration

[0865] This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0866] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[0867] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[0868] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, a synthesized speech engine, and so on.

[0869] Program processing

[0870] 1. Collection of audio and text data

[0871] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[0872] The device collects voice data and sends it to the server.

[0873] 2. Converting audio data

[0874] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[0875] 3. Emotion recognition

[0876] The server sends text data to the NLP engine, which then executes an emotion recognition algorithm. As a result of the emotion recognition, "anxiety" is extracted.

[0877] 4. Response generation

[0878] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0879] 5. Sending and displaying responses

[0880] The server sends the generated response to the terminal as text data.

[0881] The terminal displays the received text to the user. If necessary, it plays a voice response using a text-to-speech engine.

[0882] Specific example

[0883] Examples of use in customer service

[0884] 1. The user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0885] 2. The terminal sends this input to the server.

[0886] 3. The server analyzes the input and recognizes the emotion of "anxiety."

[0887] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[0888] 5. The terminal displays a response to the user.

[0889] Examples of use in the field of education

[0890] 1. The user (student) inputs "I don't understand this problem" by voice.

[0891] 2. The device collects audio and sends it to the server.

[0892] 3. The server converts the speech into text and recognizes the emotion of "confusion."

[0893] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[0894] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[0895] This system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0896] The following describes the processing flow.

[0897] Step 1:

[0898] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[0899] Step 2:

[0900] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[0901] Step 3:

[0902] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[0903] Step 4:

[0904] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[0905] Step 5:

[0906] The server sends the converted text data to a natural language processing (NLP) engine, which then executes an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[0907] Step 6:

[0908] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0909] Step 7:

[0910] The server sends the generated response to the terminal as text data.

[0911] Step 8:

[0912] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[0913] (Example 1)

[0914] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0915] Traditional dialogue systems have faced the challenge of difficulty in considering emotions when generating appropriate responses from user voice or text input. In particular, when users were experiencing emotions such as anxiety, confusion, or anger, the dialogue system often failed to provide appropriate responses that reflected those emotions. This resulted in a degraded user experience.

[0916] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0917] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, means for generating an appropriate response based on the emotion recognition result using a generative AI model, and means for displaying or playing the generated response to the user. This enables appropriate dialogue that responds to the user's emotions, thereby improving the user experience.

[0918] "Audio data" refers to data that records the voice spoken by a user in digital format.

[0919] "Text data" refers to data that represents audio data or input character information as a string of characters.

[0920] "Emotion recognition" is the process of analyzing emotions contained in text data and identifying the type of emotion.

[0921] A "generative AI model" is an artificial intelligence model trained using large-scale datasets, and is used for text generation and response generation.

[0922] An "appropriate response" is a response that is relevant to the situation and generated based on the user's input and emotions.

[0923] "Display" refers to providing the generated text response to the user visually.

[0924] "Playback" refers to converting the generated text response into audio and providing it to the user audibly.

[0925] "Collection means" refers to methods or devices for acquiring voice data or text data from users.

[0926] "Conversion means" refers to methods or devices for converting audio data into text data.

[0927] "Analysis means" refers to methods or devices for analyzing received text data and identifying its syntax and sentiment.

[0928] "Generative means" refers to a method or device that generates an appropriate response using a generative AI model based on the results of emotion recognition.

[0929] A "user" refers to a person who uses a dialogue system to input text or voice.

[0930] A "server" refers to a computer system used for data processing and analysis.

[0931] A "terminal" refers to a device used to collect user voice and text input and transmit it to a server.

[0932] Modes for carrying out the invention

[0933] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[0934] System Configuration

[0935] 1. Terminal

[0936] A terminal is a device that collects voice or text input from a user and sends it to a server. Examples include smartphones, tablets, and personal computers. Terminals are equipped with microphones, keyboards, touchscreens, and other features that allow them to collect user input data.

[0937] 2. Server

[0938] A server is a computer system that receives, analyzes, and processes data sent from terminals. This computer system is equipped with a high-speed CPU or GPU, sufficient memory and storage, and processes data in real time.

[0939] 3. Software Components

[0940] The system's software components include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a speech synthesis engine. Specifically, these are as follows:

[0941] Speech recognition engine: Converts speech data into text data. Google Cloud Speech-to-Text is used as an example.

[0942] Natural Language Processing (NLP) engine: Analyzes text data and performs syntactic and semantic analysis. A concrete example is the Google Cloud Natural Language API.

[0943] Emotion recognition algorithm: Extracts emotions from text data analyzed by an NLP engine.

[0944] Response generation engine: Uses a generative AI model (e.g., GPT-3) to generate appropriate responses based on emotion recognition results.

[0945] Text-to-speech engine: Converts generated responses into speech. A concrete example is Google Cloud Text-to-Speech.

[0946] Specific usage examples

[0947] Examples of use in customer service

[0948] A user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[0949] The terminal sends this input to the server.

[0950] The server analyzes the input and recognizes the emotion of "anxiety."

[0951] The server generates an empathetic response saying, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[0952] The device displays a response to the user. If necessary, it plays the response using synthesized speech.

[0953] Examples of use in the field of education

[0954] The user (student) inputs "I don't understand this problem" by voice.

[0955] The device collects audio and sends it to the server.

[0956] The server converts the speech into text and recognizes the emotion of "confusion."

[0957] The server generates a response saying, "Please tell me specifically what part you don't understand. Let's solve it together."

[0958] The device reads the text aloud using synthesized speech and plays it back to the user.

[0959] Through the above, this system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[0960] Example of a prompt

[0961] "The user is feeling anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution."

[0962] "A student is confused by a math problem. Generate a response that suggests you try to solve the problem together."

[0963] These prompts generate appropriate responses that reflect the user's emotions.

[0964] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0965] Step 1: Collect audio and text data

[0966] The user provides input to the system via voice or text. For example, they might voice-input, "My recent order hasn't arrived yet, what's going on?" This input is collected by the terminal.

[0967] Input: [User's voice or text data]

[0968] Output: [Audio data or text data]

[0969] Specific operation: The device uses its built-in microphone and text input function to collect user voice and text, and sends them to the server as raw data.

[0970] Step 2: Convert audio data

[0971] The server receives the audio data sent from the terminal. The received audio data is passed to a speech recognition engine, which converts the audio into text data. Using the speech recognition engine (for example, Google Cloud Speech-to-Text), the server outputs the text data "My recent order hasn't arrived yet, what's going on?"

[0972] Input: [Audio data]

[0973] Output: [Text data]

[0974] Specific operation: The server converts the audio data into a string via a speech recognition engine and saves it as text data.

[0975] Step 3: Emotion Recognition

[0976] The server sends the converted text data to an NLP engine (e.g., Google Cloud Natural Language API) to run an emotion recognition algorithm. It analyzes the text "My recent order hasn't arrived yet, what's going on?" and extracts the emotion "anxiety" contained within it.

[0977] Input: [Text data]

[0978] Output: [emotion data]

[0979] Specific operation: The server uses an NLP engine to parse the text data and extracts emotional information from the text using an emotion recognition algorithm.

[0980] Step 4: Response Generation

[0981] The server uses an AI model (e.g., GPT-3) to generate an appropriate response based on the emotion recognition result "anxiety." The prompt used is "The user is anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution." The response generated is "We apologize for the inconvenience. We will check immediately and provide you with tracking information."

[0982] Input: [emotion data]

[0983] Output: [Response text data]

[0984] Specific operation: The response generation engine inputs a prompt sentence into the generation AI model and generates an appropriate response sentence based on emotion.

[0985] Step 5: Sending and displaying the response

[0986] The server sends the generated response text data to the terminal. The terminal receives this data and displays it to the user. If necessary, it converts it to speech using a text-to-speech engine (e.g., Google Cloud Text-to-Speech) and plays it back to the user.

[0987] Input: [Response text data]

[0988] Output: [Display or audio data]

[0989] Specific operation: The device displays text data using a display UI and plays it back as audio using a text-to-speech engine.

[0990] (Application Example 1)

[0991] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0992] Conventional dialogue systems lacked the ability to provide appropriate responses based on user emotions, making it difficult to fully improve the user experience. Furthermore, in specific applications such as food delivery services, there is a need to generate rapid and emotionally resonant responses to user inquiries. Existing technologies have not adequately addressed these challenges.

[0993] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0994] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, and means for generating a response using a generative AI model based on the emotion recognition results. This makes it possible to accurately recognize the user's emotions and quickly generate an appropriate response that is in line with those emotions.

[0995] "Audio data" refers to the audio signal that a user produces using an audio input device such as a microphone.

[0996] "Text data" refers to audio data that has been converted into a string by a speech recognition engine, or string information directly entered by the user.

[0997] "Emotion recognition" refers to the process of analyzing collected text data to identify the user's emotional state (e.g., anxiety, joy, anger, etc.).

[0998] An "appropriate response" refers to a reply message that is generated based on the results of emotion recognition and is tailored to the user's emotions and the content of their inquiry.

[0999] A "generative AI model" refers to an artificial intelligence algorithm that uses natural language processing to generate appropriate responses to user input.

[1000] "Means of generating responses" refers to the process of creating appropriate responses in text or audio format using emotion recognition results and generative AI models.

[1001] "Synthesized speech" refers to speech synthesis technology that converts text data into audio data and plays it back for the user.

[1002] "Means for displaying or reproducing" means devices and methods for providing the generated response to the user visually or audibly.

[1003] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses. The system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[1004] System Configuration

[1005] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1006] 2. Server: A computer system that receives, analyzes, and processes data transmitted from terminals. The server includes software components such as a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, and a generative AI model.

[1007] 3. Software Components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a synthesized speech engine. These are necessary to analyze user input data and generate appropriate responses.

[1008] Program processing

[1009] When a user uses the application and says, "My pizza hasn't arrived yet," the device collects the audio data and sends it to the server. The server converts the audio data into text data through a speech recognition engine. This converted text data is then sent to an NLP engine, which applies an emotion recognition algorithm to recognize the emotion of "anxiety."

[1010] The server then uses a generative AI model to generate an appropriate response based on the emotion recognition results. For example, it might say, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." The generated response is sent to the device in text or synthesized speech format and displayed or played back to the user.

[1011] Hardware and software used

[1012] Speech recognition engine: This is an engine that converts speech data into text data, and commonly used examples include "Google Speech Recognition API" and "Microsoft Azure Speech Services".

[1013] Natural Language Processing (NLP): Used to analyze text data and recognize user intent and emotions. Models such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa are used for this purpose.

[1014] Emotion recognition algorithms: These are algorithms for identifying emotions from text data. For example, emotion recognition models such as "j-hartmann / emotion-english-distilroberta-base" are applied.

[1015] Generative AI models: These are models that generate appropriate responses in response to user input, and OpenAI's "GPT-3" is one example.

[1016] Specific example

[1017] Suppose a user voice-inquiries to the app saying, "My pizza order hasn't arrived yet, what's going on?" This voice input is converted to text, and the emotion of "anxiety" is recognized. The server then generates a response saying, "We apologize for the inconvenience. We will check immediately and provide you with tracking information," which is then displayed or played back to the user.

[1018] Example of a prompt:

[1019] A user has inquired, "My pizza hasn't arrived yet..." This inquiry has caused the user to feel anxious. Please generate an appropriate response.

[1020] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1021] Step 1:

[1022] A user uses a food delivery application and says, "My pizza hasn't arrived yet..." The device uses its built-in microphone to collect this voice input data. The input data is the user's voice signal and is sent to the server after collection.

[1023] Step 2:

[1024] The server receives the audio data sent from the terminal. The server uses a speech recognition engine to convert the audio data into text data. In this process, the speech recognition engine (e.g., "Google Speech Recognition API") converts the audio signal into string information, and the output is the text data "My pizza hasn't arrived yet..."

[1025] Step 3:

[1026] The server sends the converted text data to a natural language processing (NLP) engine. The NLP engine (e.g., "BERT" or "RoBERTa") analyzes this text data and processes it to identify the user's intent and emotions. This process includes using a language model to perform contextual analysis and generate the data necessary for the emotion recognition algorithm. As output, the text data is processed into data for emotion analysis.

[1027] Step 4:

[1028] The analyzed data is sent to an emotion recognition algorithm on the server. The emotion recognition algorithm (e.g., "j-hartmann / emotion-english-distilroberta-base") extracts emotions from the text data. This specific operation involves assigning specific emotion labels (e.g., "anxiety," "joy," "anger," etc.). The output is an emotion recognition result of "anxiety."

[1029] Step 5:

[1030] The server generates a response using a generative AI model based on the obtained emotion recognition results. The generative AI model (e.g., "GPT-3") creates an appropriate response based on emotion and text data. This specific operation involves a process of requesting the model to generate a response using a prompt statement. The output is the generated text response, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1031] Step 6:

[1032] The generated response is sent from the server to the terminal. The terminal then displays or plays this text response to the user. This specific action includes displaying the text on the screen and converting it to speech using a speech synthesis engine (e.g., "Microsoft Azure Speech Services") and playing it through the speaker.

[1033] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1034] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. In particular, it features a "emotion engine" for emotion recognition, enabling highly accurate emotion recognition and empathetic responses. The system's operation and specific usage examples are described below.

[1035] System Configuration

[1036] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[1037] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1038] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[1039] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, an emotion engine, a response generation engine, a synthesized speech engine, and so on.

[1040] Program processing

[1041] 1. Collection of audio and text data

[1042] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[1043] The device collects voice data and sends it to the server.

[1044] 2. Converting audio data

[1045] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[1046] 3. Emotion recognition

[1047] The server sends text data to the NLP engine, which then uses the emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[1048] 4. Response generation

[1049] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1050] 5. Sending and displaying responses

[1051] The server sends the generated response to the terminal as text data.

[1052] The terminal displays the received text to the user. If necessary, the terminal uses a text-to-speech engine to convert the text into speech and play the response.

[1053] Specific example

[1054] Examples of use in customer service

[1055] 1. The user enters a message into customer service saying, "My ordered item hasn't arrived yet, what's going on?"

[1056] 2. The terminal sends this input to the server.

[1057] 3. The server analyzes the input data and uses an emotion engine to recognize the emotion of "anxiety."

[1058] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[1059] 5. The terminal displays a response to the user.

[1060] Examples of use in the field of education

[1061] 1. The user (student) inputs "I don't understand this problem" by voice.

[1062] 2. The device collects audio and sends it to the server.

[1063] 3. The server converts the audio into text and uses an emotion engine to recognize the emotion "confusion."

[1064] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[1065] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[1066] This system enables appropriate dialogue based on the user's emotions, and the introduction of an emotion engine, in particular, improves the accuracy of emotion recognition. This greatly enhances the user experience.

[1067] The following describes the processing flow.

[1068] Step 1:

[1069] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[1070] Step 2:

[1071] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[1072] Step 3:

[1073] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[1074] Step 4:

[1075] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[1076] Step 5:

[1077] The server sends the converted text data to a natural language processing (NLP) engine, which then uses an emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[1078] Step 6:

[1079] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1080] Step 7:

[1081] The server sends the generated response to the terminal as text data.

[1082] Step 8:

[1083] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[1084] (Example 2)

[1085] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1086] In modern dialogue systems, accurately recognizing user emotions and providing appropriate, empathetic responses is challenging. Conventional systems suffer from low accuracy in emotion recognition, resulting in a poor user experience.

[1087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1088] In this invention, the server includes means for collecting voice data input from a user, means for converting the voice data into text data, means for analyzing the text data and performing natural language processing, means for performing emotion recognition using the data analyzed by the natural language processing, means for generating an appropriate response based on the result of the emotion recognition, and means for displaying or playing the generated response to the user. This makes it possible to recognize the user's emotions with higher accuracy and provide an empathetic and appropriate response accordingly.

[1089] A "user" refers to a person who uses the system.

[1090] "Voice data" refers to digital data that electronically records the voice input by the user.

[1091] "Text data" refers to data obtained by converting audio data into written text.

[1092] "Natural language processing" refers to the technology that enables computers to understand and analyze human language.

[1093] "Emotion recognition" refers to the technology that identifies a user's emotions from text data.

[1094] "Response" refers to the reply or answer that the system generates to the user.

[1095] "Means of generation" refers to a function that generates a response using a specific process or algorithm.

[1096] "Means for displaying or reproducing" refers to technologies or devices for showing or letting the user hear the generated response.

[1097] Modes for carrying out the invention

[1098] This invention is a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The actual operation and usage examples of the system are shown below.

[1099] System Configuration

[1100] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[1101] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1102] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[1103] 3. Software Components: These include speech recognition engines, natural language processing (NLP) engines, emotion recognition algorithms, emotion engines, response generation engines, and synthesized speech engines. Specifically, this includes software such as Google Cloud Speech-to-Text, spaCy, BERT, IBM Watson Tone Analyzer, OpenAI GPT-3, and Amazon Polly.

[1104] Overview of program processing

[1105] 1. Collection of audio and text data

[1106] Users can use the application to make inquiries via voice or text. For example, they might say, "My recent order hasn't arrived yet, what's going on?"

[1107] The device collects user input and sends it to a server via the internet.

[1108] 2. Converting audio data

[1109] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text).

[1110] The converted text data is, "My recent order hasn't arrived yet, what's going on?"

[1111] 3. Emotion recognition

[1112] The server sends text data to an NLP engine (spaCy, BERT), which then analyzes the emotions using an emotion recognition algorithm (IBM Watson Tone Analyzer).

[1113] For example, the text "My recent order hasn't arrived yet, what's going on?" can be used to identify the emotion of "anxiety."

[1114] 4. Response generation

[1115] The server uses a response generation engine (OpenAI GPT-3) to generate an appropriate response based on the emotion recognition results.

[1116] One example of a response that might be generated is, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1117] 5. Sending and displaying responses

[1118] The server sends the generated response to the terminal as text data.

[1119] The device displays the received text to the user, and, if necessary, uses a text-to-speech engine (Amazon Polly) to convert the text into speech and play it back to the user.

[1120] Specific example

[1121] Examples of use in customer service

[1122] 1. The user enters, "My ordered item hasn't arrived yet. What's going on?"

[1123] 2. The terminal sends this input to the server.

[1124] 3. The server receives the input data, converts it to text using a speech recognition engine, and recognizes the emotion "anxiety" using an emotion engine.

[1125] 4. The server generates a response saying, "We will check the situation immediately and provide tracking information."

[1126] 5. The device displays the response to the user and plays it back as audio if necessary.

[1127] Examples of use in the field of education

[1128] 1. The user (student) inputs "I don't understand this problem" by voice.

[1129] 2. The device collects audio and sends it to the server.

[1130] 3. The server converts the audio to text, and the emotion engine recognizes the emotion as "confused."

[1131] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[1132] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[1133] This invention enables appropriate dialogue based on the user's emotions, and the accuracy of emotion recognition is particularly improved by the introduction of an emotion engine. This greatly enhances the user experience.

[1134] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1135] Step 1:

[1136] Users use the application to make inquiries via voice or text. For example, they might say to customer service, "My recent order hasn't arrived yet, what's going on?"

[1137] Input: User voice or text input

[1138] Output: Collected audio or text data

[1139] Specific operation: The device collects the user's voice via the microphone, or text input via the keyboard. This data is temporarily stored on the device.

[1140] Step 2:

[1141] The device sends the voice or text data entered by the user to the server.

[1142] Input: Collected audio or text data

[1143] Output: Audio or text data sent to the server

[1144] Specific operation: The device sends voice or text data to the server via the internet. Encrypted communication (e.g., SSL / TLS) is often used in this process.

[1145] Step 3:

[1146] The server converts the received audio data into text data using a speech recognition engine (e.g., speech recognition engine).

[1147] Input: Audio data

[1148] Output: Text data

[1149] Specific operation: The server invokes the speech recognition engine and provides voice data as input. The engine analyzes the voice waveform and converts it into text. For example, the voice saying "My recent order hasn't arrived, what's going on?" becomes the text "My recent order hasn't arrived, what's going on?".

[1150] Step 4:

[1151] The server sends the converted text data to a natural language processing (NLP) engine (e.g., a natural language processing engine) for sentiment recognition.

[1152] Input: Text data

[1153] Output: Emotional data (e.g., anxiety)

[1154] Specific operation: The server sends text data to the NLP engine, which performs contextual analysis. The emotion engine is used to recognize the user's emotions from the analyzed data. For example, the emotion "anxiety" is identified from the text "My recent order hasn't arrived, what's going on?".

[1155] Step 5:

[1156] The server generates an appropriate response using a response generation engine (e.g., a generative AI model) based on the emotion recognition results.

[1157] Input: Sentiment data, text data

[1158] Output: Response text

[1159] Specific operation: The server takes sentiment data and text data as input and invokes the response generation engine. The engine generates a response based on the provided data. For example, a response such as "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." might be generated.

[1160] Step 6:

[1161] The server sends the generated response text to the terminal.

[1162] Input: Response text

[1163] Output: Sent response text

[1164] Specific operation: The server sends the generated response text to the terminal. This also uses encrypted communication.

[1165] Step 7:

[1166] The terminal displays the received response text to the user. If necessary, it uses a text-to-speech engine (e.g., a speech synthesis engine) to convert the text to speech and play it back to the user.

[1167] Input: Response text

[1168] Output: Text displayed to the user or audio played back.

[1169] Specific operation: The device displays the received response text to the user. If necessary, it inputs the text into a text-to-speech engine and plays the generated audio through the speaker. For example, the user can see the text "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information" on the device screen, or hear the same content in audio.

[1170] In this way, a series of processes from user input to response generation are executed, creating a mechanism that improves the user experience.

[1171] (Application Example 2)

[1172] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1173] Traditional dialogue systems struggle to accurately recognize user emotions and generate appropriate and empathetic responses based on them. Furthermore, low accuracy in emotion recognition is likely to impair the user experience. In particular, customer service requires the rapid resolution of user anxiety and confusion, a task often failing to be adequately achieved by conventional systems.

[1174] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data or text data input from the user, means for converting voice data into text data, means for analyzing text data and performing emotion recognition, means for generating an appropriate response using natural language generation technology, means for converting the generated response into synthesized speech and displaying or playing it back to the user, means for inputting prompt sentences using a generation AI model, and means for adjusting based on the user's emotions and providing an empathetic response. This enables highly accurate emotion recognition and empathetic response generation, greatly improving the user experience.

[1175] "Voice data input by the user" refers to voice information provided by the user through a voice input device.

[1176] "Methods for converting to text data" refers to the process of converting speech information into text data using speech recognition technology.

[1177] "Means for performing emotion recognition" refers to methods that use natural language processing technology to analyze and identify a user's emotions from text data.

[1178] "Means for generating appropriate responses using natural language generation technology" refers to a function that generates appropriate responses based on recognized emotions and input information using a generative AI model.

[1179] "Means for converting to synthesized speech and displaying or playing it for the user" refers to the process of converting generated text responses into speech and providing it to the user.

[1180] "A method for inputting prompt sentences using a generative AI model" refers to a technique that inputs prompt sentences suitable for a generative AI model and generates an appropriate response.

[1181] "Means of adjusting based on emotions and providing empathetic responses" refers to a function that modifies the response content based on recognized emotions and provides empathetic responses that take the user's feelings into consideration.

[1182] One embodiment of this invention is a dialogue system that recognizes emotion from user input data, generates an appropriate response using natural language generation technology, and provides it as synthesized speech. This system uses the following hardware and software.

[1183] 1. Hardware to be used

[1184] User devices: Devices such as smartphones, tablets, and personal computers. These devices collect voice or text input from users and send it to the server.

[1185] Server: A computer system that performs data analysis and processing.

[1186] 2. Software to use

[1187] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert speech data into text data.

[1188] Emotion Recognition Engine: Using HUME AI, the system analyzes user emotions from text data.

[1189] Natural language generation engine: Uses OpenAI GPT-4 to generate appropriate responses based on emotion recognition results.

[1190] Text-to-speech engine: Uses the Google Text-to-Speech API to convert generated text responses into speech.

[1191] 3. Data flow and operating procedures

[1192] The server collects and processes voice or text data entered by the user. Specifically, the process proceeds as follows:

[1193] 1. Collection and conversion of audio data

[1194] The user speaks into the device and says, "The item I ordered hasn't arrived yet."

[1195] The device collects voice data and sends it to the server.

[1196] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1197] 2. Emotion recognition

[1198] The server sends text data to HUME AI for emotion recognition.

[1199] The text "The item I ordered hasn't arrived yet" conveys the emotion of "anxiety."

[1200] 3. Response generation

[1201] The server sends a prompt message to OpenAI GPT-4 based on the emotion recognition results and text data, and generates a response.

[1202] The generated response is, "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1203] 4. Provision of synthesized speech and responses

[1204] The server converts the generated text response into speech using the Google Text-to-Speech API.

[1205] The device plays synthesized speech or displays text to the user.

[1206] Specific examples and prompt statements

[1207] Specific Scenario

[1208] 1. The user opens the app and voice-inputs, "My ordered item hasn't arrived yet."

[1209] 2. The app converts the voice to text and sends it to the server as "My ordered items have not yet arrived."

[1210] 3. The server uses HUME AI to recognize emotions in the text and identifies the emotion "anxiety."

[1211] 4. The server sends a prompt to OpenAI GPT-4 stating, "The user is feeling anxious. Please check the order status immediately," and generates a response.

[1212] 5. Provide the user with the generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1213] Examples of prompts for generative AI models

[1214] prompt:

[1215] User text input: "The item I ordered has not yet arrived."

[1216] Recognized emotion: "anxiety"

[1217] Please generate an appropriate response:

[1218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1219] Step 1:

[1220] The user makes a voice input into the device. For example, the user might say, "The item I ordered hasn't arrived yet." This voice data is collected via the device's microphone.

[1221] Input: Voice data "The item I ordered has not yet arrived."

[1222] Output: Audio data stored on the device

[1223] Step 2:

[1224] The terminal sends the collected audio data to the server. This transfer of audio data to the server in real time uses a network connection.

[1225] Input: Audio data stored on the device

[1226] Output: Audio data sent to the server

[1227] Step 3:

[1228] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The audio data is passed to the API, and the converted text data is received as the result.

[1229] Input: Audio data sent to the server

[1230] Data processing: Converting audio data to text data.

[1231] Output: Text data "The item I ordered has not yet arrived."

[1232] Step 4:

[1233] The server sends text data to HUME AI for emotion recognition. HUME AI analyzes the text data and recognizes the user's emotion as "anxiety."

[1234] Input: Text data "The item I ordered has not yet arrived."

[1235] Data processing: Apply emotion recognition algorithms.

[1236] Output: Emotion tag "anxiety"

[1237] Step 5:

[1238] The server generates an appropriate response to the OpenAI GPT-4 model using prompts. These prompts include user text input and recognized emotions.

[1239] Example of a prompt:

[1240] User text input: "The item I ordered has not yet arrived."

[1241] Recognized emotion: "anxiety"

[1242] Please generate an appropriate response:

[1243] Input: Text data "The item I ordered has not yet arrived", emotion tag "anxiety", prompt text

[1244] Data processing: Uses a natural language generation AI model.

[1245] Output: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1246] Step 6:

[1247] The server uses the Google Text-to-Speech API to convert the generated response text into audio data. The generated response text is passed to the API and the audio data is received.

[1248] Input: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1249] Data processing: Converting text data to audio data.

[1250] Output: Synthesized speech data

[1251] Step 7:

[1252] The server sends synthesized speech data and text data to the terminal. Data is transferred in real time over the network.

[1253] Input: synthesized speech data, generated response text

[1254] Output: Synthesized speech data and text data sent to the terminal

[1255] Step 8:

[1256] The device plays the received synthesized speech data to the user and displays the generated response text. This allows the user to receive a response both audibly and visually.

[1257] Input: Synthesized speech data and text data sent to the terminal.

[1258] Output: Played audio response, displayed text response

[1259] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1260] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1261] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1262] [Fourth Embodiment]

[1263] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1264] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1265] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1266] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1267] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1268] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1269] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1270] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1271] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1272] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1273] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1274] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1275] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1276] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The system's operation and specific usage examples are shown below.

[1277] System Configuration

[1278] This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[1279] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1280] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[1281] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, a synthesized speech engine, and so on.

[1282] Program processing

[1283] 1. Collection of audio and text data

[1284] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[1285] The device collects voice data and sends it to the server.

[1286] 2. Converting audio data

[1287] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[1288] 3. Emotion recognition

[1289] The server sends text data to the NLP engine, which then executes an emotion recognition algorithm. As a result of the emotion recognition, "anxiety" is extracted.

[1290] 4. Response generation

[1291] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1292] 5. Sending and displaying responses

[1293] The server sends the generated response to the terminal as text data.

[1294] The terminal displays the received text to the user. If necessary, it plays a voice response using a text-to-speech engine.

[1295] Specific example

[1296] Examples of use in customer service

[1297] 1. The user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[1298] 2. The terminal sends this input to the server.

[1299] 3. The server analyzes the input and recognizes the emotion of "anxiety."

[1300] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[1301] 5. The terminal displays a response to the user.

[1302] Examples of use in the field of education

[1303] 1. The user (student) inputs "I don't understand this problem" by voice.

[1304] 2. The device collects audio and sends it to the server.

[1305] 3. The server converts the speech into text and recognizes the emotion of "confusion."

[1306] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[1307] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[1308] This system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[1309] The following describes the processing flow.

[1310] Step 1:

[1311] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[1312] Step 2:

[1313] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[1314] Step 3:

[1315] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[1316] Step 4:

[1317] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[1318] Step 5:

[1319] The server sends the converted text data to a natural language processing (NLP) engine, which then executes an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[1320] Step 6:

[1321] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1322] Step 7:

[1323] The server sends the generated response to the terminal as text data.

[1324] Step 8:

[1325] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[1326] (Example 1)

[1327] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1328] Traditional dialogue systems have faced the challenge of difficulty in considering emotions when generating appropriate responses from user voice or text input. In particular, when users were experiencing emotions such as anxiety, confusion, or anger, the dialogue system often failed to provide appropriate responses that reflected those emotions. This resulted in a degraded user experience.

[1329] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1330] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, means for generating an appropriate response based on the emotion recognition result using a generative AI model, and means for displaying or playing the generated response to the user. This enables appropriate dialogue that responds to the user's emotions, thereby improving the user experience.

[1331] "Audio data" refers to data that records the voice spoken by a user in digital format.

[1332] "Text data" refers to data that represents audio data or input character information as a string of characters.

[1333] "Emotion recognition" is the process of analyzing emotions contained in text data and identifying the type of emotion.

[1334] A "generative AI model" is an artificial intelligence model trained using large-scale datasets, and is used for text generation and response generation.

[1335] An "appropriate response" is a response that is relevant to the situation and generated based on the user's input and emotions.

[1336] "Display" refers to providing the generated text response to the user visually.

[1337] "Playback" refers to converting the generated text response into audio and providing it to the user audibly.

[1338] "Collection means" refers to methods or devices for acquiring voice data or text data from users.

[1339] "Conversion means" refers to methods or devices for converting audio data into text data.

[1340] "Analysis means" refers to methods or devices for analyzing received text data and identifying its syntax and sentiment.

[1341] "Generative means" refers to a method or device that generates an appropriate response using a generative AI model based on the results of emotion recognition.

[1342] A "user" refers to a person who uses a dialogue system to input text or voice.

[1343] A "server" refers to a computer system used for data processing and analysis.

[1344] A "terminal" refers to a device used to collect user voice and text input and transmit it to a server.

[1345] Modes for carrying out the invention

[1346] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. This system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[1347] System Configuration

[1348] 1. Terminal

[1349] A terminal is a device that collects voice or text input from a user and sends it to a server. Examples include smartphones, tablets, and personal computers. Terminals are equipped with microphones, keyboards, touchscreens, and other features that allow them to collect user input data.

[1350] 2. Server

[1351] A server is a computer system that receives, analyzes, and processes data sent from terminals. This computer system is equipped with a high-speed CPU or GPU, sufficient memory and storage, and processes data in real time.

[1352] 3. Software Components

[1353] The system's software components include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a speech synthesis engine. Specifically, these are as follows:

[1354] Speech recognition engine: Converts speech data into text data. Google Cloud Speech-to-Text is used as an example.

[1355] Natural Language Processing (NLP) engine: Analyzes text data and performs syntactic and semantic analysis. A concrete example is the Google Cloud Natural Language API.

[1356] Emotion recognition algorithm: Extracts emotions from text data analyzed by an NLP engine.

[1357] Response generation engine: Uses a generative AI model (e.g., GPT-3) to generate appropriate responses based on emotion recognition results.

[1358] Text-to-speech engine: Converts generated responses into speech. A concrete example is Google Cloud Text-to-Speech.

[1359] Specific usage examples

[1360] Examples of use in customer service

[1361] A user enters an inquiry saying, "The item I ordered hasn't arrived yet. What's going on?"

[1362] The terminal sends this input to the server.

[1363] The server analyzes the input and recognizes the emotion of "anxiety."

[1364] The server generates an empathetic response saying, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1365] The device displays a response to the user. If necessary, it plays the response using synthesized speech.

[1366] Examples of use in the field of education

[1367] The user (student) inputs "I don't understand this problem" by voice.

[1368] The device collects audio and sends it to the server.

[1369] The server converts the speech into text and recognizes the emotion of "confusion."

[1370] The server generates a response saying, "Please tell me specifically what part you don't understand. Let's solve it together."

[1371] The device reads the text aloud using synthesized speech and plays it back to the user.

[1372] Through the above, this system enables appropriate dialogue based on the user's emotions, greatly improving the user experience.

[1373] Example of a prompt

[1374] "The user is feeling anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution."

[1375] "A student is confused by a math problem. Generate a response that suggests you try to solve the problem together."

[1376] These prompts generate appropriate responses that reflect the user's emotions.

[1377] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1378] Step 1: Collect audio and text data

[1379] The user provides input to the system via voice or text. For example, they might voice-input, "My recent order hasn't arrived yet, what's going on?" This input is collected by the terminal.

[1380] Input: [User's voice or text data]

[1381] Output: [Audio data or text data]

[1382] Specific operation: The device uses its built-in microphone and text input function to collect user voice and text, and sends them to the server as raw data.

[1383] Step 2: Convert audio data

[1384] The server receives the audio data sent from the terminal. The received audio data is passed to a speech recognition engine, which converts the audio into text data. Using the speech recognition engine (for example, Google Cloud Speech-to-Text), the server outputs the text data "My recent order hasn't arrived yet, what's going on?"

[1385] Input: [Audio data]

[1386] Output: [Text data]

[1387] Specific operation: The server converts the audio data into a string via a speech recognition engine and saves it as text data.

[1388] Step 3: Emotion Recognition

[1389] The server sends the converted text data to an NLP engine (e.g., Google Cloud Natural Language API) to run an emotion recognition algorithm. It analyzes the text "My recent order hasn't arrived yet, what's going on?" and extracts the emotion "anxiety" contained within it.

[1390] Input: [Text data]

[1391] Output: [emotion data]

[1392] Specific operation: The server uses an NLP engine to parse the text data and extracts emotional information from the text using an emotion recognition algorithm.

[1393] Step 4: Response Generation

[1394] The server uses an AI model (e.g., GPT-3) to generate an appropriate response based on the emotion recognition result "anxiety." The prompt used is "The user is anxious because their ordered item has not yet arrived. Please generate a response that includes empathy and a solution." The response generated is "We apologize for the inconvenience. We will check immediately and provide you with tracking information."

[1395] Input: [emotion data]

[1396] Output: [Response text data]

[1397] Specific operation: The response generation engine inputs a prompt sentence into the generation AI model and generates an appropriate response sentence based on emotion.

[1398] Step 5: Sending and displaying the response

[1399] The server sends the generated response text data to the terminal. The terminal receives this data and displays it to the user. If necessary, it converts it to speech using a text-to-speech engine (e.g., Google Cloud Text-to-Speech) and plays it back to the user.

[1400] Input: [Response text data]

[1401] Output: [Display or audio data]

[1402] Specific operation: The device displays text data using a display UI and plays it back as audio using a text-to-speech engine.

[1403] (Application Example 1)

[1404] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1405] Conventional dialogue systems lacked the ability to provide appropriate responses based on user emotions, making it difficult to fully improve the user experience. Furthermore, in specific applications such as food delivery services, there is a need to generate rapid and emotionally resonant responses to user inquiries. Existing technologies have not adequately addressed these challenges.

[1406] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1407] In this invention, the server includes means for collecting voice data input from the user, means for converting the voice data into text data, means for analyzing the text data and performing emotion recognition, and means for generating a response using a generative AI model based on the emotion recognition results. This makes it possible to accurately recognize the user's emotions and quickly generate an appropriate response that is in line with those emotions.

[1408] "Audio data" refers to the audio signal that a user produces using an audio input device such as a microphone.

[1409] "Text data" refers to audio data that has been converted into a string by a speech recognition engine, or string information directly entered by the user.

[1410] "Emotion recognition" refers to the process of analyzing collected text data to identify the user's emotional state (e.g., anxiety, joy, anger, etc.).

[1411] An "appropriate response" refers to a reply message that is generated based on the results of emotion recognition and is tailored to the user's emotions and the content of their inquiry.

[1412] A "generative AI model" refers to an artificial intelligence algorithm that uses natural language processing to generate appropriate responses to user input.

[1413] "Means of generating responses" refers to the process of creating appropriate responses in text or audio format using emotion recognition results and generative AI models.

[1414] "Synthesized speech" refers to speech synthesis technology that converts text data into audio data and plays it back for the user.

[1415] "Means for displaying or reproducing" means devices and methods for providing the generated response to the user visually or audibly.

[1416] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses. The system consists of a terminal used by the user, a server that processes the input data, and various software components that generate and provide responses to the user.

[1417] System Configuration

[1418] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1419] 2. Server: A computer system that receives, analyzes, and processes data transmitted from terminals. The server includes software components such as a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, and a generative AI model.

[1420] 3. Software Components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, a response generation engine, and a synthesized speech engine. These are necessary to analyze user input data and generate appropriate responses.

[1421] Program processing

[1422] When a user uses the application and says, "My pizza hasn't arrived yet," the device collects the audio data and sends it to the server. The server converts the audio data into text data through a speech recognition engine. This converted text data is then sent to an NLP engine, which applies an emotion recognition algorithm to recognize the emotion of "anxiety."

[1423] The server then uses a generative AI model to generate an appropriate response based on the emotion recognition results. For example, it might say, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." The generated response is sent to the device in text or synthesized speech format and displayed or played back to the user.

[1424] Hardware and software used

[1425] Speech recognition engine: This is an engine that converts speech data into text data, and commonly used examples include "Google Speech Recognition API" and "Microsoft Azure Speech Services".

[1426] Natural Language Processing (NLP): Used to analyze text data and recognize user intent and emotions. Models such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa are used for this purpose.

[1427] Emotion recognition algorithms: These are algorithms for identifying emotions from text data. For example, emotion recognition models such as "j-hartmann / emotion-english-distilroberta-base" are applied.

[1428] Generative AI models: These are models that generate appropriate responses in response to user input, and OpenAI's "GPT-3" is one example.

[1429] Specific example

[1430] Suppose a user voice-inquiries to the app saying, "My pizza order hasn't arrived yet, what's going on?" This voice input is converted to text, and the emotion of "anxiety" is recognized. The server then generates a response saying, "We apologize for the inconvenience. We will check immediately and provide you with tracking information," which is then displayed or played back to the user.

[1431] Example of a prompt:

[1432] A user has inquired, "My pizza hasn't arrived yet..." This inquiry has caused the user to feel anxious. Please generate an appropriate response.

[1433] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1434] Step 1:

[1435] A user uses a food delivery application and says, "My pizza hasn't arrived yet..." The device uses its built-in microphone to collect this voice input data. The input data is the user's voice signal and is sent to the server after collection.

[1436] Step 2:

[1437] The server receives the audio data sent from the terminal. The server uses a speech recognition engine to convert the audio data into text data. In this process, the speech recognition engine (e.g., "Google Speech Recognition API") converts the audio signal into string information, and the output is the text data "My pizza hasn't arrived yet..."

[1438] Step 3:

[1439] The server sends the converted text data to a natural language processing (NLP) engine. The NLP engine (e.g., "BERT" or "RoBERTa") analyzes this text data and processes it to identify the user's intent and emotions. This process includes using a language model to perform contextual analysis and generate the data necessary for the emotion recognition algorithm. As output, the text data is processed into data for emotion analysis.

[1440] Step 4:

[1441] The analyzed data is sent to an emotion recognition algorithm on the server. The emotion recognition algorithm (e.g., "j-hartmann / emotion-english-distilroberta-base") extracts emotions from the text data. This specific operation involves assigning specific emotion labels (e.g., "anxiety," "joy," "anger," etc.). The output is an emotion recognition result of "anxiety."

[1442] Step 5:

[1443] The server generates a response using a generative AI model based on the obtained emotion recognition results. The generative AI model (e.g., "GPT-3") creates an appropriate response based on emotion and text data. This specific operation involves a process of requesting the model to generate a response using a prompt statement. The output is the generated text response, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1444] Step 6:

[1445] The generated response is sent from the server to the terminal. The terminal then displays or plays this text response to the user. This specific action includes displaying the text on the screen and converting it to speech using a speech synthesis engine (e.g., "Microsoft Azure Speech Services") and playing it through the speaker.

[1446] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1447] This invention relates to a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. In particular, it features a "emotion engine" for emotion recognition, enabling highly accurate emotion recognition and empathetic responses. The system's operation and specific usage examples are described below.

[1448] System Configuration

[1449] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[1450] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1451] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[1452] 3. Software components: These include a speech recognition engine, a natural language processing (NLP) engine, an emotion recognition algorithm, an emotion engine, a response generation engine, a synthesized speech engine, and so on.

[1453] Program processing

[1454] 1. Collection of audio and text data

[1455] A user uses the application and says, "My recent order hasn't arrived yet, what's going on?"

[1456] The device collects voice data and sends it to the server.

[1457] 2. Converting audio data

[1458] The server converts the received audio data into text data via its speech recognition engine. The resulting text is "My recent order hasn't arrived yet, what's going on?"

[1459] 3. Emotion recognition

[1460] The server sends text data to the NLP engine, which then uses the emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[1461] 4. Response generation

[1462] Based on the emotion recognition results, the server uses a response generation engine to generate an appropriate response such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1463] 5. Sending and displaying responses

[1464] The server sends the generated response to the terminal as text data.

[1465] The terminal displays the received text to the user. If necessary, the terminal uses a text-to-speech engine to convert the text into speech and play the response.

[1466] Specific example

[1467] Examples of use in customer service

[1468] 1. The user enters a message into customer service saying, "My ordered item hasn't arrived yet, what's going on?"

[1469] 2. The terminal sends this input to the server.

[1470] 3. The server analyzes the input data and uses an emotion engine to recognize the emotion of "anxiety."

[1471] 4. The server generates an empathetic response such as, "We will check the situation immediately and provide tracking information."

[1472] 5. The terminal displays a response to the user.

[1473] Examples of use in the field of education

[1474] 1. The user (student) inputs "I don't understand this problem" by voice.

[1475] 2. The device collects audio and sends it to the server.

[1476] 3. The server converts the audio into text and uses an emotion engine to recognize the emotion "confusion."

[1477] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[1478] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[1479] This system enables appropriate dialogue based on the user's emotions, and the introduction of an emotion engine, in particular, improves the accuracy of emotion recognition. This greatly enhances the user experience.

[1480] The following describes the processing flow.

[1481] Step 1:

[1482] The user launches the application and uses voice or text input. For example, the user might say, "My recent order hasn't arrived yet, what's going on?"

[1483] Step 2:

[1484] The device collects the user's voice or text data. If voice input is performed, the voice data is temporarily stored.

[1485] Step 3:

[1486] The device sends the collected audio or text data to the server. Audio data is sent in audio file format (e.g., .wav or .mp3), and text data is sent in string format.

[1487] Step 4:

[1488] The server passes the received voice data to the speech recognition engine, which converts it into text data. For example, the voice data "My recent order hasn't arrived yet, what's going on?" is converted into text data.

[1489] Step 5:

[1490] The server sends the converted text data to a natural language processing (NLP) engine, which then uses an emotion engine to execute an emotion recognition algorithm. Through the analysis of the text data, emotions such as "anxiety" are identified.

[1491] Step 6:

[1492] The server sends data to the response generation engine based on the emotion recognition results, and generates an appropriate response. For example, if the emotion recognition indicates "anxiety," the response generated will be, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1493] Step 7:

[1494] The server sends the generated response to the terminal as text data.

[1495] Step 8:

[1496] The device displays the received text data to the user. If necessary, the device uses a text-to-speech engine to convert the text into speech and play a response. For example, a voice message such as, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information," might be played.

[1497] (Example 2)

[1498] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1499] In modern dialogue systems, accurately recognizing user emotions and providing appropriate, empathetic responses is challenging. Conventional systems suffer from low accuracy in emotion recognition, resulting in a poor user experience.

[1500] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1501] In this invention, the server includes means for collecting voice data input from a user, means for converting the voice data into text data, means for analyzing the text data and performing natural language processing, means for performing emotion recognition using the data analyzed by the natural language processing, means for generating an appropriate response based on the result of the emotion recognition, and means for displaying or playing the generated response to the user. This makes it possible to recognize the user's emotions with higher accuracy and provide an empathetic and appropriate response accordingly.

[1502] A "user" refers to a person who uses the system.

[1503] "Voice data" refers to digital data that electronically records the voice input by the user.

[1504] "Text data" refers to data obtained by converting audio data into written text.

[1505] "Natural language processing" refers to the technology that enables computers to understand and analyze human language.

[1506] "Emotion recognition" refers to the technology that identifies a user's emotions from text data.

[1507] "Response" refers to the reply or answer that the system generates to the user.

[1508] "Means of generation" refers to a function that generates a response using a specific process or algorithm.

[1509] "Means for displaying or reproducing" refers to technologies or devices for showing or letting the user hear the generated response.

[1510] Modes for carrying out the invention

[1511] This invention is a dialogue system that recognizes emotions based on user voice and text input and generates appropriate responses corresponding to those emotions. The actual operation and usage examples of the system are shown below.

[1512] System Configuration

[1513] This system consists of a user terminal, a server that processes the input data, and various software components that generate and provide responses to the user. It also integrates an emotion engine for emotion recognition.

[1514] 1. Terminal: A device that collects voice or text input from the user and sends it to the server. This includes smartphones, tablets, and personal computers.

[1515] 2. Server: A computer system that receives, analyzes, and processes data sent from a terminal.

[1516] 3. Software Components: These include speech recognition engines, natural language processing (NLP) engines, emotion recognition algorithms, emotion engines, response generation engines, and synthesized speech engines. Specifically, this includes software such as Google Cloud Speech-to-Text, spaCy, BERT, IBM Watson Tone Analyzer, OpenAI GPT-3, and Amazon Polly.

[1517] Overview of program processing

[1518] 1. Collection of audio and text data

[1519] Users can use the application to make inquiries via voice or text. For example, they might say, "My recent order hasn't arrived yet, what's going on?"

[1520] The device collects user input and sends it to a server via the internet.

[1521] 2. Converting audio data

[1522] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text).

[1523] The converted text data is, "My recent order hasn't arrived yet, what's going on?"

[1524] 3. Emotion recognition

[1525] The server sends text data to an NLP engine (spaCy, BERT), which then analyzes the emotions using an emotion recognition algorithm (IBM Watson Tone Analyzer).

[1526] For example, the text "My recent order hasn't arrived yet, what's going on?" can be used to identify the emotion of "anxiety."

[1527] 4. Response generation

[1528] The server uses a response generation engine (OpenAI GPT-3) to generate an appropriate response based on the emotion recognition results.

[1529] One example of a response that might be generated is, "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information."

[1530] 5. Sending and displaying responses

[1531] The server sends the generated response to the terminal as text data.

[1532] The device displays the received text to the user, and, if necessary, uses a text-to-speech engine (Amazon Polly) to convert the text into speech and play it back to the user.

[1533] Specific example

[1534] Examples of use in customer service

[1535] 1. The user enters, "My ordered item hasn't arrived yet. What's going on?"

[1536] 2. The terminal sends this input to the server.

[1537] 3. The server receives the input data, converts it to text using a speech recognition engine, and recognizes the emotion "anxiety" using an emotion engine.

[1538] 4. The server generates a response saying, "We will check the situation immediately and provide tracking information."

[1539] 5. The device displays the response to the user and plays it back as audio if necessary.

[1540] Examples of use in the field of education

[1541] 1. The user (student) inputs "I don't understand this problem" by voice.

[1542] 2. The device collects audio and sends it to the server.

[1543] 3. The server converts the audio to text, and the emotion engine recognizes the emotion as "confused."

[1544] 4. The server generates a response saying, "Please tell me specifically which part you don't understand. Let's solve it together."

[1545] 5. The device reads the text aloud using synthesized speech and plays it back to the user.

[1546] This invention enables appropriate dialogue based on the user's emotions, and the accuracy of emotion recognition is particularly improved by the introduction of an emotion engine. This greatly enhances the user experience.

[1547] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1548] Step 1:

[1549] Users use the application to make inquiries via voice or text. For example, they might say to customer service, "My recent order hasn't arrived yet, what's going on?"

[1550] Input: User voice or text input

[1551] Output: Collected audio or text data

[1552] Specific operation: The device collects the user's voice via the microphone, or text input via the keyboard. This data is temporarily stored on the device.

[1553] Step 2:

[1554] The device sends the voice or text data entered by the user to the server.

[1555] Input: Collected audio or text data

[1556] Output: Audio or text data sent to the server

[1557] Specific operation: The device sends voice or text data to the server via the internet. Encrypted communication (e.g., SSL / TLS) is often used in this process.

[1558] Step 3:

[1559] The server converts the received audio data into text data using a speech recognition engine (e.g., speech recognition engine).

[1560] Input: Audio data

[1561] Output: Text data

[1562] Specific operation: The server invokes the speech recognition engine and provides voice data as input. The engine analyzes the voice waveform and converts it into text. For example, the voice saying "My recent order hasn't arrived, what's going on?" becomes the text "My recent order hasn't arrived, what's going on?".

[1563] Step 4:

[1564] The server sends the converted text data to a natural language processing (NLP) engine (e.g., a natural language processing engine) for sentiment recognition.

[1565] Input: Text data

[1566] Output: Emotional data (e.g., anxiety)

[1567] Specific operation: The server sends text data to the NLP engine, which performs contextual analysis. The emotion engine is used to recognize the user's emotions from the analyzed data. For example, the emotion "anxiety" is identified from the text "My recent order hasn't arrived, what's going on?".

[1568] Step 5:

[1569] The server generates an appropriate response using a response generation engine (e.g., a generative AI model) based on the emotion recognition results.

[1570] Input: Sentiment data, text data

[1571] Output: Response text

[1572] Specific operation: The server takes sentiment data and text data as input and invokes the response generation engine. The engine generates a response based on the provided data. For example, a response such as "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information." might be generated.

[1573] Step 6:

[1574] The server sends the generated response text to the terminal.

[1575] Input: Response text

[1576] Output: Sent response text

[1577] Specific operation: The server sends the generated response text to the terminal. This also uses encrypted communication.

[1578] Step 7:

[1579] The terminal displays the received response text to the user. If necessary, it uses a text-to-speech engine (e.g., a speech synthesis engine) to convert the text to speech and play it back to the user.

[1580] Input: Response text

[1581] Output: Text displayed to the user or audio played back.

[1582] Specific operation: The device displays the received response text to the user. If necessary, it inputs the text into a text-to-speech engine and plays the generated audio through the speaker. For example, the user can see the text "We apologize for the inconvenience. We will investigate immediately and provide you with tracking information" on the device screen, or hear the same content in audio.

[1583] In this way, a series of processes from user input to response generation are executed, creating a mechanism that improves the user experience.

[1584] (Application Example 2)

[1585] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1586] Traditional dialogue systems struggle to accurately recognize user emotions and generate appropriate and empathetic responses based on them. Furthermore, low accuracy in emotion recognition is likely to impair the user experience. In particular, customer service requires the rapid resolution of user anxiety and confusion, a task often failing to be adequately achieved by conventional systems.

[1587] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data or text data input from the user, means for converting voice data into text data, means for analyzing text data and performing emotion recognition, means for generating an appropriate response using natural language generation technology, means for converting the generated response into synthesized speech and displaying or playing it back to the user, means for inputting prompt sentences using a generation AI model, and means for adjusting based on the user's emotions and providing an empathetic response. This enables highly accurate emotion recognition and empathetic response generation, greatly improving the user experience.

[1588] "Voice data input by the user" refers to voice information provided by the user through a voice input device.

[1589] "Methods for converting to text data" refers to the process of converting speech information into text data using speech recognition technology.

[1590] "Means for performing emotion recognition" refers to methods that use natural language processing technology to analyze and identify a user's emotions from text data.

[1591] "Means for generating appropriate responses using natural language generation technology" refers to a function that generates appropriate responses based on recognized emotions and input information using a generative AI model.

[1592] "Means for converting to synthesized speech and displaying or playing it for the user" refers to the process of converting generated text responses into speech and providing it to the user.

[1593] "A method for inputting prompt sentences using a generative AI model" refers to a technique that inputs prompt sentences suitable for a generative AI model and generates an appropriate response.

[1594] "Means of adjusting based on emotions and providing empathetic responses" refers to a function that modifies the response content based on recognized emotions and provides empathetic responses that take the user's feelings into consideration.

[1595] One embodiment of this invention is a dialogue system that recognizes emotion from user input data, generates an appropriate response using natural language generation technology, and provides it as synthesized speech. This system uses the following hardware and software.

[1596] 1. Hardware to be used

[1597] User devices: Devices such as smartphones, tablets, and personal computers. These devices collect voice or text input from users and send it to the server.

[1598] Server: A computer system that performs data analysis and processing.

[1599] 2. Software to use

[1600] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert speech data into text data.

[1601] Emotion Recognition Engine: Using HUME AI, the system analyzes user emotions from text data.

[1602] Natural language generation engine: Uses OpenAI GPT-4 to generate appropriate responses based on emotion recognition results.

[1603] Text-to-speech engine: Uses the Google Text-to-Speech API to convert generated text responses into speech.

[1604] 3. Data flow and operating procedures

[1605] The server collects and processes voice or text data entered by the user. Specifically, the process proceeds as follows:

[1606] 1. Collection and conversion of audio data

[1607] The user speaks into the device and says, "The item I ordered hasn't arrived yet."

[1608] The device collects voice data and sends it to the server.

[1609] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1610] 2. Emotion recognition

[1611] The server sends text data to HUME AI for emotion recognition.

[1612] The text "The item I ordered hasn't arrived yet" conveys the emotion of "anxiety."

[1613] 3. Response generation

[1614] The server sends a prompt message to OpenAI GPT-4 based on the emotion recognition results and text data, and generates a response.

[1615] The generated response is, "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1616] 4. Provision of synthesized speech and responses

[1617] The server converts the generated text response into speech using the Google Text-to-Speech API.

[1618] The device plays synthesized speech or displays text to the user.

[1619] Specific examples and prompt statements

[1620] Specific Scenario

[1621] 1. The user opens the app and voice-inputs, "My ordered item hasn't arrived yet."

[1622] 2. The app converts the voice to text and sends it to the server as "My ordered items have not yet arrived."

[1623] 3. The server uses HUME AI to recognize emotions in the text and identifies the emotion "anxiety."

[1624] 4. The server sends a prompt to OpenAI GPT-4 stating, "The user is feeling anxious. Please check the order status immediately," and generates a response.

[1625] 5. Provide the user with the generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1626] Examples of prompts for generative AI models

[1627] prompt:

[1628] User text input: "The item I ordered has not yet arrived."

[1629] Recognized emotion: "anxiety"

[1630] Please generate an appropriate response:

[1631] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1632] Step 1:

[1633] The user makes a voice input into the device. For example, the user might say, "The item I ordered hasn't arrived yet." This voice data is collected via the device's microphone.

[1634] Input: Voice data "The item I ordered has not yet arrived."

[1635] Output: Audio data stored on the device

[1636] Step 2:

[1637] The terminal sends the collected audio data to the server. This transfer of audio data to the server in real time uses a network connection.

[1638] Input: Audio data stored on the device

[1639] Output: Audio data sent to the server

[1640] Step 3:

[1641] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The audio data is passed to the API, and the converted text data is received as the result.

[1642] Input: Audio data sent to the server

[1643] Data processing: Converting audio data to text data.

[1644] Output: Text data "The item I ordered has not yet arrived."

[1645] Step 4:

[1646] The server sends text data to HUME AI for emotion recognition. HUME AI analyzes the text data and recognizes the user's emotion as "anxiety."

[1647] Input: Text data "The item I ordered has not yet arrived."

[1648] Data processing: Apply emotion recognition algorithms.

[1649] Output: Emotion tag "anxiety"

[1650] Step 5:

[1651] The server generates an appropriate response to the OpenAI GPT-4 model using prompts. These prompts include user text input and recognized emotions.

[1652] Example of a prompt:

[1653] User text input: "The item I ordered has not yet arrived."

[1654] Recognized emotion: "anxiety"

[1655] Please generate an appropriate response:

[1656] Input: Text data "The item I ordered has not yet arrived", emotion tag "anxiety", prompt text

[1657] Data processing: Uses a natural language generation AI model.

[1658] Output: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1659] Step 6:

[1660] The server uses the Google Text-to-Speech API to convert the generated response text into audio data. The generated response text is passed to the API and the audio data is received.

[1661] Input: Generated response: "We apologize for the inconvenience. We will investigate immediately and provide tracking information."

[1662] Data processing: Converting text data to audio data.

[1663] Output: Synthesized speech data

[1664] Step 7:

[1665] The server sends synthesized speech data and text data to the terminal. Data is transferred in real time over the network.

[1666] Input: synthesized speech data, generated response text

[1667] Output: Synthesized speech data and text data sent to the terminal

[1668] Step 8:

[1669] The device plays the received synthesized speech data to the user and displays the generated response text. This allows the user to receive a response both audibly and visually.

[1670] Input: Synthesized speech data and text data sent to the terminal.

[1671] Output: Played audio response, displayed text response

[1672] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1673] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1674] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1675] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1676] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1677] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1678] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1679] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1680] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1681] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1682] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1683] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1684] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1685] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1686] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1687] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1688] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1689] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1690] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1691] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1692] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1693] The following is further disclosed regarding the embodiments described above.

[1694] (Claim 1)

[1695] A means of collecting voice data input from the user,

[1696] Means for converting the aforementioned audio data into text data,

[1697] A means for analyzing the aforementioned text data and performing emotion recognition,

[1698] A means for generating an appropriate response based on the results of the aforementioned emotion recognition,

[1699] Means for displaying or playing the generated response to the user,

[1700] A system that includes this.

[1701] (Claim 2)

[1702] The system according to claim 1, further comprising means for collecting text data entered by a user.

[1703] (Claim 3)

[1704] The system according to claim 1 or 2, further comprising means for converting the generated response into synthesized speech and playing it back.

[1705] "Example 1"

[1706] (Claim 1)

[1707] A means of collecting voice data input from the user,

[1708] Means for converting the aforementioned audio data into text data,

[1709] A means for analyzing the aforementioned text data and performing emotion recognition,

[1710] A means for generating an appropriate response using a generative AI model based on the results of the emotion recognition,

[1711] Means for displaying or playing the generated response to the user,

[1712] A system that includes this.

[1713] (Claim 2)

[1714] The system according to claim 1, further comprising means for collecting text data entered by a user.

[1715] (Claim 3)

[1716] The system according to claim 1 or 2, further comprising means for converting the generated response into synthesized speech and playing it back.

[1717] "Application Example 1"

[1718] (Claim 1)

[1719] A means of collecting voice data input from the user,

[1720] Means for converting the aforementioned audio data into text data,

[1721] A means for analyzing the aforementioned text data and performing emotion recognition,

[1722] A means for generating an appropriate response based on the results of the aforementioned emotion recognition,

[1723] Means for displaying or playing the generated response to the user,

[1724] A means for generating a response using a generative AI model based on the emotion recognition result,

[1725] A system that includes this.

[1726] (Claim 2)

[1727] The system according to claim 1, further comprising means for collecting text data entered by a user.

[1728] (Claim 3)

[1729] The system according to claim 1, further comprising means for converting the generated response into synthesized speech and playing it back.

[1730] "Example 2 of combining an emotion engine"

[1731] (Claim 1)

[1732] A means of collecting voice data input from the user,

[1733] Means for converting the aforementioned audio data into text data,

[1734] A means for analyzing the aforementioned text data and performing natural language processing,

[1735] A means for performing emotion recognition using data analyzed by the aforementioned natural language processing,

[1736] A means for generating an appropriate response based on the results of the aforementioned emotion recognition,

[1737] Means for displaying or playing the generated response to the user,

[1738] A system that includes this.

[1739] (Claim 2)

[1740] The system according to claim 1, further comprising means for collecting text data entered by a user.

[1741] (Claim 3)

[1742] The system according to claim 1, further comprising means for converting the generated response into synthesized speech and playing it back.

[1743] "Application example 2 when combining with an emotional engine"

[1744] (Claim 1)

[1745] Means for collecting voice data or text data entered by the user,

[1746] Means for converting the aforementioned audio data into text data,

[1747] A means for analyzing the aforementioned text data and performing emotion recognition,

[1748] A means for generating an appropriate response using natural language generation technology based on the results of the aforementioned emotion recognition,

[1749] Means for converting the generated response into synthesized speech and displaying or playing it for the user,

[1750] A system that includes this.

[1751] (Claim 2)

[1752] The system according to claim 1, further comprising means for inputting a prompt sentence using a generation AI model when generating the generated response.

[1753] (Claim 3)

[1754] The system according to claim 1, further comprising means for adjusting the generated response based on the user's emotions to provide an empathetic response. [Explanation of symbols]

[1755] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting voice data input from the user, Means for converting the aforementioned audio data into text data, A means for analyzing the aforementioned text data and performing emotion recognition, A means for generating an appropriate response based on the results of the aforementioned emotion recognition, Means for displaying or playing the generated response to the user, A system that includes this.

2. The system according to claim 1, further comprising means for collecting text data entered by a user.

3. The system according to claim 1 or 2, further comprising means for converting the generated response into synthesized speech and playing it back.

Images