System

Abstract

A system is provided.SOLUTION: A system comprising: means for converting input voice data into text data; means for analyzing an emotion from the text data; means for generating a sentence including the emotion based on the emotion; means for converting the generated sentence into voice data; and means for transmitting the voice data to a speaking device to cause the speaking device to speak.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Recent advances in text generation and voice interface technologies have led to the widespread use of voice assistants and interactive robots. However, these devices primarily focus on providing information and responding to instructions, and lack the ability to provide emotional support or empathy to users. This leaves users who feel lonely or isolated without sufficient psychological support. The present invention aims to provide emotional support and encouragement to users who feel lonely or isolated by analyzing the user's emotions from input speech and automatically generating emotional utterances. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for converting input voice data into text data; means for analyzing emotions from the text data; means for generating emotional sentences based on the emotions; means for converting the generated sentences into voice data; and means for transmitting the voice data to a speech device to produce speech. This system provides appropriate speech according to the emotional state of the user, and can provide psychological support to users who feel lonely or isolated.

[0006] "Input voice data" refers to voice uttered by a user, and is data collected by a voice input device such as a microphone.

[0007] "Text data" is data that has been converted from voice data using voice recognition technology and expressed as a string of characters.

[0008] "Means for analyzing emotions" refers to algorithms or software modules that identify a user's emotions or psychological state from input text data.

[0009] "Emotional sentences" are texts of speech content that are generated in consideration of the user's emotional state and in a manner that corresponds to that emotion.

[0010] The "means for converting into voice data" refers to an algorithm or software module for converting the generated text data into voice data using voice synthesis technology.

[0011] A "speech device" is a hardware device that receives voice data and plays it back for the user to hear. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The system of the present invention converts a user's voice input into an emotionally charged speech in order to provide psychological support to a user who feels lonely or isolated. Specific embodiments of the system are described below.

[0034] System Overview

[0035] This system consists of a terminal that receives user voice input, a server that analyzes the voice data and identifies emotions, and a speech device that converts the generated text into voice data and speaks it.The main technical elements used are voice recognition technology, natural language processing technology, and voice synthesis technology.

[0036] Program processing

[0037] The program in the system of the present invention is processed in the following procedure.

[0038] 1. Acquiring voice input

[0039] The user makes a speech input such as "I'm tired today" into the terminal.

[0040] 2. Converting voice data to text

[0041] The device converts the voice data acquired from the user into text data using voice recognition software, for example, generating text data such as "I'm tired today."

[0042] 3. Sending text data

[0043] The terminal transmits the converted text data to the server.

[0044] 4. Emotion Analysis

[0045] The server then passes the received text data to emotion analysis software, which uses natural language processing techniques to analyze the user's emotions and identify the emotion "tired."

[0046] 5. Generating emotional sentences

[0047] The server uses sentence generation software based on the analyzed emotions to generate emotional sentences, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0048] 6. Text-to-speech

[0049] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0050] 7. Transmission to speaking device

[0051] The server transmits the generated voice data to the terminal.

[0052] 8. Executing the utterance

[0053] The terminal transfers the received voice data to a speech device, which then plays the voice data. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so good things will surely happen."

[0054] Specific examples

[0055] When a user voice-inputs "I'm tired today," the device converts the voice into text data ("I'm tired today") and sends it to the server. The server analyzes the emotion from the text and identifies the emotion category "tired." Next, it generates a sentence expressing the emotion ("I see, today was tough. But you always try your best, so good things will surely happen.") and converts this into voice data. Finally, the humanoid robot plays the voice back to the user via the device, providing psychological support.

[0056] The above is an embodiment of the system of the present invention. This system enables communication that is sensitive to the user's emotions, thereby reducing feelings of loneliness and isolation.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] The user inputs "I'm tired today" into the terminal by voice.

[0060] Step 2:

[0061] The device captures the user's voice through a microphone.

[0062] Step 3:

[0063] The device uses speech recognition software to convert the captured voice data into text, for example, "I'm tired today."

[0064] Step 4:

[0065] The terminal transmits the converted text data to the server.

[0066] Step 5:

[0067] The server passes the received text data to sentiment analysis software, which uses natural language processing techniques to analyze the user's emotions.

[0068] Step 6:

[0069] The server identifies the user's emotion (for example, "fatigue") from the output of the emotion analysis software.

[0070] Step 7:

[0071] The server uses sentence generation software to generate emotional sentences based on the user's emotions, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will come."

[0072] Step 8:

[0073] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0074] Step 9:

[0075] The server transmits the generated voice data to the terminal.

[0076] Step 10:

[0077] The terminal transfers the voice data received from the server to the speech device.

[0078] Step 11:

[0079] The speech device (part of the terminal) plays back the voice data and speaks encouraging words to the user, such as, "You had a hard day today, didn't you? But you always try your best, so I'm sure good things will come your way."

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] In modern society, the number of people feeling lonely and isolated is increasing. While systems exist to provide psychological support, many of these systems lack sufficient emotional response capabilities and are unable to respond in a way that is in tune with the user's actual feelings. Therefore, there is a need for a system that can analyze emotions based on the user's voice input, generate sentences containing appropriate emotional expressions, and respond via voice.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to a central processing unit, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a terminal and transferring it to a speech device for speech output. This makes it possible to provide a voice response that is in line with the user's emotions, thereby reducing feelings of loneliness and isolation.

[0085] "Audio data" means digital or analog data that is an electronic recording of sound.

[0086] "Text data" is digital data that includes character information.

[0087] A "central processing unit" is a device that performs the central processing of a server or computer.

[0088] "Means of emotion analysis" refers to algorithms or software that identify emotions from input text data.

[0089] "Means for generating emotional sentences" refers to algorithms or software for generating sentences that contain appropriate emotional expressions based on the analyzed emotions.

[0090] "Means for converting into audio data" refers to the algorithms or software used to convert the generated text data into audio format.

[0091] A "terminal" refers to a device that is directly operated by a user, and is an electronic device that has the function of inputting and outputting voice.

[0092] A "speech device" is a device or equipment for playing back generated voice data.

[0093] MODE FOR CARRYING OUT THE INVENTION

[0094] The system of the present invention is a dialogue system that combines emotion analysis and speech generation, designed to reduce users' feelings of loneliness and isolation. The system mainly performs a series of processes, including user voice input, speech recognition, emotion analysis, sentence generation, speech synthesis, and speech production.

[0095] Hardware and Software Configuration

[0096] The user inputs voice into the system using a voice input device (microphone or smartphone).

[0097] The device receives the user's voice and converts the voice data into text data using speech recognition software (e.g., a commonly used speech recognition API), such as the Google Speech-to-Text API.

[0098] The terminal sends the generated text data to a server, which is a central processing unit. At this time, the data is sent securely via the HTTPS protocol.

[0099] The server passes the received text data to sentiment analysis software (e.g., algorithms or APIs that use natural language processing technology) to analyze the sentiment. Here, IBM Watson Tone Analyzer is used as an example.

[0100] Based on the results of the sentiment analysis, the server uses sentence generation software (e.g., a generative AI model such as OpenAI's GPT-3) to generate emotional sentences.

[0101] The generated sentences are passed to speech synthesis software (such as Amazon Polly) and converted into speech data, which then adjusts the tone and expression of the voice according to the analyzed emotion.

[0102] The server transmits the generated voice data to the terminal again, and the terminal transfers the voice data to a speech device (for example, a humanoid robot).

[0103] The speech device plays back the received voice data and speaks to the user. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0104] Specific examples

[0105] Specifically, suppose a user says "I'm tired today" by voice input. In this case, the device converts the voice into text data via the Google Speech-to-Text API. The generated text data "I'm tired today" is sent to the server using HTTPS.

[0106] The server uses IBM Watson Tone Analyzer to analyze the emotion of "tiredness," and then uses OpenAI's GPT-3 to generate a sentence that expresses the emotion: "Today was tough, wasn't it? But you always work hard, so I'm sure good things will happen."

[0107] The sentence is converted into voice data using Amazon Polly and sent from the server to the device, which then forwards the voice data to the humanoid robot, which then speaks the sentence to the user.

[0108] Prompt Sentence Examples

[0109] If you want a generative AI model to analyze emotions based on a user's speech, such as "I'm tired today," and generate an emotional response based on the results, you might use the following prompt:

[0110] Analyze the user's emotions based on the phrase "I'm tired today," and generate a response containing encouraging words based on the emotion "tired."

[0111] The above is a specific embodiment for carrying out the present invention. This system allows users to receive emotionally responsive responses, thereby reducing feelings of loneliness and isolation.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user inputs voice data into the terminal. For example, the user might say, "I'm tired today." This voice data becomes the input.

[0115] Step 2:

[0116] The device converts the voice data acquired from the user into text data using voice recognition software (e.g., Google Speech-to-Text API). Specifically, the device's microphone captures the voice data and sends it to the Google Speech-to-Text API, obtaining text data such as "I'm tired today" as output.

[0117] Step 3:

[0118] The terminal sends the converted text data to the server. Specifically, the terminal sends the generated text data to the server as an HTTPS request, using the TLS protocol to ensure security. The input of this step is the text data "I'm tired today," and the output is the completion of sending the text data to the server.

[0119] Step 4:

[0120] The server passes the received text data to emotion analysis software (for example, IBM Watson Tone Analyzer) to analyze the emotion. The server then sends the text data in JSON format to the emotion analysis engine, and obtains the emotion label "tired" as the output of the analysis result. The input is the text data "I'm tired today," and the output is the emotion label "tired."

[0121] Step 5:

[0122] The server generates emotional sentences using sentence generation software (e.g., OpenAI's GPT-3) based on the analyzed emotions. Specifically, the server supplies the emotion label "tired" and the original text data as prompts to GPT-3, and obtains the sentence "Today was tough, wasn't it? But you always try your best, so good things will surely come." The input is the emotion label and the original text data, and the output is the generated sentence.

[0123] Step 6:

[0124] The server passes the generated sentence to speech synthesis software (e.g., Amazon Polly) and converts it into voice data. The server then sends the generated sentence in text format to the Amazon Polly API, which outputs voice data with tone and speed set according to the emotion. The input is the generated sentence, and the output is voice data.

[0125] Step 7:

[0126] The server sends the generated audio data to the device. Specifically, the server sends the audio data in binary format to the device as an HTTPS request, and the device confirms receipt of the audio data. The input of this step is the audio data, and the output is completion of sending the audio data to the device.

[0127] Step 8:

[0128] The terminal transfers the received voice data to a speech device (e.g., a humanoid robot), which then plays the voice data. Specifically, the terminal transmits the voice data to the speech device via Bluetooth or Wi-Fi, and the speaker of the speech device plays the following voice as output: "Today was tough, wasn't it? But you always try your best, so good things will surely happen." The input is the voice data, and the output is the spoken voice.

[0129] (Application example 1)

[0130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0131] In recent years, the number of users feeling lonely and isolated has been increasing, creating a need for psychological support. However, conventional systems have had difficulty providing appropriate feedback in real time that reflects the user's emotions. In particular, content distribution services lack a means to provide comments and encouragement that reflect the user's emotions, creating a need for improved user experience. The objective of this invention is to provide a system that converts voice input into emotional speech, thereby providing users with psychological support.

[0132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0133] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to produce speech, and means for providing appropriate comments and encouragement to users in a content distribution service based on the results of the emotion analysis. This enables communication that is sensitive to the user's emotions, reduces feelings of loneliness and isolation, and improves the content viewing experience.

[0134] "Audio data" is a sound signal expressed in digital or analog format, and is information including the user's voice.

[0135] "Text data" refers to character information expressed in digital form, and includes words obtained by analyzing voice data.

[0136] "Emotions" refer to a person's psychological state and include a variety of psychological states such as joy, sadness, and anger.

[0137] "Means of emotion analysis" refers to technologies and algorithms for identifying a user's emotions from input text data, and may utilize natural language processing technology or machine learning models.

[0138] "Emotional sentences" are sentences that express a context or a specific emotion generated based on the analyzed emotion, and are generated according to the user's psychological state.

[0139] A "speech device" is a device for reproducing and communicating audio data to a user, and includes a speaker or other audio reproduction device.

[0140] "Content distribution service" refers to a service that provides digital content to users, including various media formats such as movies, music, videos, and articles.

[0141] The system for implementing this invention converts a user's voice input into text data, analyzes emotions, generates emotional sentences based on the results, and finally provides feedback to the user as voice data. A specific configuration for implementing this system will be described below.

[0142] System configuration

[0143] The system consists of the following major components:

[0144] 1. Voice input acquisition device

[0145] This device, such as a smartphone or a head-mounted display, captures the user's voice.

[0146] 2. Voice Recognition Software

[0147] Speech recognition software converts captured voice data into text data using voice recognition technologies such as Google's Speech-to-Text API.

[0148] 3. Emotion Analysis Server

[0149] This server analyzes the user's emotions from the received text data, using natural language processing technologies such as Hugging Face's GPT-3.

[0150] 4. Text Generation Software

[0151] The sentence generation software generates emotional sentences based on the analyzed emotions. The generated sentences include concrete examples such as, "I see you had a hard day today. But you always work hard, so good things will surely happen."

[0152] 5. Text-to-Speech Software

[0153] Text-to-speech software converts the generated text into audio data, using voice synthesis technologies such as Google's Text-to-Speech API.

[0154] 6. Speaking Devices

[0155] This device plays back the generated audio data, similar to the audio output function of a smartphone speaker or a head-mounted display.

[0156] Processing method

[0157] A user initiates the system by providing a voice input. For example, the user might say, "I'm tired today." The voice input capture device records this speech and converts it into text data using speech recognition software. This text data is then sent to an emotion analysis server, which analyzes the emotion "tired" using natural language processing techniques. Based on the analysis results, sentence generation software generates an appropriate emotional response. Finally, speech synthesis software converts this sentence into speech data, which is played back to the user via a speech device.

[0158] Specific examples

[0159] As a concrete example, the following prompt sentence is used when the user inputs "I'm tired today." "User input: 'I'm tired today'. Generate an encouraging sentence based on this input." This prompt sentence generates appropriate feedback that is in line with the user's emotions.

[0160] In this way, the system can provide communication that is sensitive to the user's emotions, helping to reduce feelings of loneliness and isolation, and improving the user experience in content distribution services.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] The user performs voice input, for example, saying, "I'm tired today."

[0164] Input: User's voice data

[0165] Output: Audio data sent to the device

[0166] Specific operation: The microphone on the smartphone or head-mounted display picks up the user's voice.

[0167] Step 2:

[0168] The device uses voice recognition software (such as Google's Speech-to-Text API) to convert voice input into text data.

[0169] Input: User's voice data

[0170] Output: Text data generated by the speech recognition software (e.g., "I'm tired today")

[0171] Specific operation: A process is carried out to convert the voice data into text data.

[0172] Step 3:

[0173] The terminal transmits the converted text data to the emotion analysis server.

[0174] Input: Text data

[0175] Output: Text data sent to the sentiment analysis server

[0176] Specific operation: Send text data to the server using an HTTP request, etc.

[0177] Step 4:

[0178] The server uses natural language processing technology (such as GPT-3 in Hugging Face) to analyze emotions from the received text data.

[0179] Input: Text data (e.g., "I'm tired today")

[0180] Output: Analyzed emotion data (e.g., "tired")

[0181] Specific operations: Perform sentiment analysis of text data using natural language processing technology.

[0182] Step 5:

[0183] The server uses sentence generation software to generate emotional sentences based on the analyzed emotions.

[0184] Input: Emotion data (e.g., "tired")

[0185] Output: Generated sentence (e.g. "Today was tough, but you always try your best, so I'm sure good things will come out of it.")

[0186] Specific action: The sentence generation software generates sentences that correspond to the emotions.

[0187] Step 6:

[0188] The server uses speech synthesis software (such as Google's Text-to-Speech API) to convert the generated text into audio data.

[0189] Input: Text (e.g., "I know you had a hard day today. But you always work hard, so good things will come your way.")

[0190] Output: Audio data

[0191] Specific operation: The process converts text into audio data.

[0192] Step 7:

[0193] The server transmits the generated voice data to the terminal.

[0194] Input: Audio data

[0195] Output: Audio data sent to the device

[0196] Specific operation: Send audio data to the terminal using an HTTP request, etc.

[0197] Step 8:

[0198] The terminal transfers the received voice data to the speech device, which then reproduces the voice data.

[0199] Input: Audio data

[0200] Output: The audio played to the user

[0201] Specific operation: Play audio data through the speaker of a smartphone or head-mounted display.

[0202] In this way, a system is realized that provides emotional feedback in real time based on the user's voice input.

[0203] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0204] The system of the present invention combines the user's voice input with an emotion engine to more precisely recognize the user's emotions and generate emotional utterances based on those emotions. By adding an analysis function using the emotion engine, this system further improves the accuracy of recognizing the user's emotional state, realizing more empathetic and effective communication.

[0205] System Overview

[0206] The system consists of the following main components:

[0207] 1. A device that receives user voice input

[0208] 2. A server containing an emotion engine that analyzes voice data and identifies emotions

[0209] 3. Server that generates sentences based on emotions and converts them into voice data

[0210] 4. A speech device that produces speech using the generated voice data

[0211] Program processing

[0212] The processing flow of a typical program in the system of the present invention is shown below.

[0213] 1. Acquiring voice input

[0214] The user inputs into the terminal, "Today was very hard."

[0215] 2. Converting voice data to text

[0216] The device uses a microphone to capture voice data and uses voice recognition software to convert it into text data ("Today was very hard").

[0217] 3. Sending text data

[0218] The terminal transmits the converted text data to the server.

[0219] 4. Emotion Analysis

[0220] The server first uses an emotion engine to analyze the voice characteristics (pitch, tempo, intonation) of the voice data, and then, if necessary, analyzes non-voice data such as the user's facial expressions and movements, and combines these data to recognize the user's emotions.

[0221] 5. Information Integration

[0222] The server integrates the voice features and text data to more precisely identify the user's emotional state. For example, if the text data indicates "painful," and the voice features also support this, the emotion "very pain" can be identified.

[0223] 6. Generating emotional sentences

[0224] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0225] 7. Text-to-speech

[0226] The server then passes the generated text to speech synthesis software, which generates speech data that expresses the appropriate emotion, taking into account tone and intonation.

[0227] 8. Sending audio data

[0228] The server transmits the generated voice data to the terminal.

[0229] 9. Executing the utterance

[0230] The terminal transfers the voice data received from the server to the speech device, which then speaks, "Today must have been really tough. But your efforts were not in vain."

[0231] Specific examples

[0232] When a user voice-inputs "Today was really hard," the device converts the speech into text data ("Today was really hard") and sends it to the server. The server uses an emotion engine to analyze the speech features and further analyzes the text data to identify the "hard" emotion. Next, it generates a sentence such as "Today was really hard, wasn't it? But your efforts were worth it," and converts it into speech data. Finally, the device sends the speech data to a speech generator, which can provide the user with encouraging speech.

[0233] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide communication that is sensitive to the user's emotions.

[0234] The processing flow will be explained below.

[0235] Step 1:

[0236] The user inputs into the terminal, "Today was very hard."

[0237] Step 2:

[0238] The terminal uses a microphone to capture the user's voice data.

[0239] Step 3:

[0240] The device uses speech recognition software to convert the captured voice data into text, such as "Today was a very difficult day."

[0241] Step 4:

[0242] The terminal transmits the converted text data and speech feature data, which includes information such as pitch, tempo, and intonation, to the server.

[0243] Step 5:

[0244] The server first analyzes the voice feature data using an emotion engine and recognizes the emotion from the user's voice. For example, if the voice feature data shows that the user's voice is low and slow, the emotion "sad" is recognized.

[0245] Step 6:

[0246] The server analyzes the text data ("Today was very difficult") using natural language processing technology and identifies the emotion ("difficult") contained in the text data.

[0247] Step 7:

[0248] The server integrates the analysis results of the voice feature data and the text data to more accurately identify the user's emotional state. For example, if both the voice feature data and the text data indicate "sad" and "distressed," the overall emotion identified will be "very distressed."

[0249] Step 8:

[0250] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0251] Step 9:

[0252] The server then passes the generated text to speech synthesis software, which generates emotionally appropriate speech data with a tone and intonation that reflects the identified emotion.

[0253] Step 10:

[0254] The server transmits the generated voice data to the terminal.

[0255] Step 11:

[0256] The terminal transfers the voice data received from the server to the speech device.

[0257] Step 12:

[0258] The speech device of the terminal plays back the voice data and speaks encouraging words to the user, such as, "Today must have been really tough, but your efforts will not have been in vain."

[0259] Example 2

[0260] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] Conventional speech recognition and emotion analysis systems have faced challenges in accurately recognizing a user's emotions and responding appropriately. In particular, when identifying emotions based solely on speech data, the user's speech content and voice characteristics alone are insufficient, leading to concerns about reduced accuracy. Furthermore, when generating emotion-based responses, it has been difficult to provide the empathetic and appropriate communication users expect.

[0262] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to the server via a cloud service, means for identifying emotions by analyzing the text data and voice features, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a speech device to have it speak. This enables the user's emotions to be recognized with high accuracy, and emotional sentences to be generated based on the recognition, enabling more empathetic and effective communication.

[0263] The "means for converting input voice data into text data" refers to a device or algorithm for converting voice data input by a user into text format data.

[0264] "Means for sending to a server via a cloud service" refers to protocols and software for sending data from a local device to a remote server using cloud computing infrastructure.

[0265] The "means for analyzing text data and audio features to identify emotions" refers to an algorithm or engine that analyzes the content of text data and audio features (e.g., pitch, tempo, intonation) of audio data to identify the user's emotions.

[0266] The "means for generating emotional sentences based on emotions" refers to algorithms and natural language processing techniques for generating empathetic and emotional sentences in response to the identified emotions of a user.

[0267] The "means for converting the generated text into speech data" refers to speech synthesis software or technology for converting the generated text in text format into speech format data.

[0268] The "means for transmitting to a speech generating device and causing speech generation" refers to a protocol or interface for transmitting the generated voice data to a physical speech generating device and using that device to actually reproduce the voice.

[0269] "Audio features" are predetermined features such as pitch, tempo, and intonation contained in audio data, and are information that enables emotion recognition by analyzing these features.

[0270] "Non-voice data" refers to data other than voice data, such as the user's facial expressions, movements, heart rate, etc., which are used complementary to emotion recognition.

[0271] This invention is a system that recognizes emotions with high accuracy based on a user's voice input and provides speech that incorporates the emotions accordingly. This system is composed of a terminal that acquires the user's voice input, a server that analyzes the voice data and identifies the emotion, a server that generates sentences based on the emotion and converts them into voice data, and a speech device that speaks the generated voice data.

[0272] The user inputs voice into the device. For example, the user says, "Today was very hard." This voice is captured by the device's microphone. The device then converts the voice data into text data using voice recognition software (for example, a voice recognition API). The converted text data ("Today was very hard") is sent to the server via a cloud service.

[0273] The server uses an emotion engine (e.g., emotion analysis API) to analyze the voice features (pitch, tempo, intonation) of the voice data. If necessary, it also analyzes non-voice data such as the user's facial expressions and movements to comprehensively recognize emotions. This makes it possible to identify the user's emotions by integrating text data, voice features, and non-voice data. For example, if the text data contains the word "spicy" and the voice features support this, the emotion "very spicy" is identified.

[0274] Next, the server uses a generative AI model (e.g., a natural language generation model) to generate emotional sentences based on the identified emotions. For example, a sentence like, "Today must have been really tough, but your efforts are worth it." In this generation process, prompt sentences like the following are used:

[0275] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0276] The server then passes the generated sentences to speech synthesis software (e.g., a speech synthesis API) to generate emotionally appropriate speech data, including the appropriate tone and intonation, which is then transmitted to the device using a secure protocol (e.g., HTTPS).

[0277] Finally, the device transmits the received voice data to a speech device (e.g., a smart speaker or speaker system), which then speaks, "You've had a really hard day today, but your efforts are worth it." Through this series of processes, the user can experience empathetic and effective communication through the system.

[0278] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide an appropriate emotional response based on the recognition.

[0279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0280] Step 1:

[0281] The user speaks to the terminal, saying, "Today was a very hard day." This speech input is captured by the terminal's microphone. The captured speech data becomes the input.

[0282] Step 2:

[0283] The device converts the captured voice data into text using speech recognition software (e.g., a speech recognition API), where the voice signal is analyzed through digital signal processing and a corresponding string of characters ("Today was very hard") is output.

[0284] Step 3:

[0285] The device sends the converted text data to a server via a cloud service, using a secure protocol (e.g., HTTPS). The input of this transmission process is the text data, and the output is the completion of the transfer of the text data to the server.

[0286] Step 4:

[0287] The server analyzes emotions using an emotion engine (e.g., emotion analysis API) to analyze text data and voice features. This analysis is performed by extracting voice features (pitch, tempo, intonation) from the voice data and, if necessary, analyzing non-voice data (e.g., the user's facial expressions and movements). The input is text data and voice feature data, and the output is the analyzed user's emotion information.

[0288] Step 5:

[0289] The server integrates the voice features and text data to precisely identify the user's emotion. In this integration process, for example, if the text data indicates "spicy," the emotion "very spicy" can be identified by matching it with the voice features. The input is voice feature data and text data, and the output is the final emotion information.

[0290] Step 6:

[0291] The server uses a generative AI model (e.g., a natural language generation model) based on the identified emotion to generate emotional sentences. Specifically, it generates empathetic sentences that are in tune with the emotion based on the identified emotional information. The input is the emotional information, and the output is the generated sentences (e.g., "Today was really tough, wasn't it? But your hard work won't be in vain.").

[0292] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0293] Step 7:

[0294] The server passes the generated sentence to speech synthesis software (e.g., a speech synthesis API) to generate speech data that appropriately expresses emotion. The speech synthesis process adds a voice tone and intonation appropriate for the generated sentence. The input is the generated sentence, and the output is speech data.

[0295] Step 8:

[0296] The server sends the generated audio data to the terminal. Again, a secure protocol (e.g., HTTPS) is used. The input is the audio data, and the output is the completion of sending the audio data to the terminal.

[0297] Step 9:

[0298] The device then transfers the received voice data to a speech device (e.g., a smart speaker or speaker system), which ultimately speaks, "Today was really tough, wasn't it? But your efforts were worth it." The input is the voice data sent from the server, and the output is delivered to the user as physical sound.

[0299] (Application example 2)

[0300] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0301] Conventional systems were able to analyze emotions in response to user voice input and generate emotionally charged sentences, but lacked the ability to provide optimal content and actions based on the user's emotional state. As a result, users were unable to receive appropriate content or feedback based on their emotions, making it difficult to provide empathetic and effective communication and entertainment. Furthermore, the lack of a function to recommend content based on emotions made it impossible to provide personalized services.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0303] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to have it speak, and means for recommending personalized content based on the user's emotions. This makes it possible to provide optimal content tailored to the user's emotional state, thereby realizing the provision of more empathetic and personalized services.

[0304] "Voice data" refers to the digitized voice of the user.

[0305] "Text data" is voice data converted into a string of characters.

[0306] "Emotion analysis" is the process of identifying an emotional state from text data and voice data.

[0307] "Sentence generation" means creating sentences based on the results of sentiment analysis.

[0308] "Convert to voice data" means converting the generated text back into voice data.

[0309] A "speech device" is a device that outputs voice data as voice.

[0310] "Personalized content" refers to content that is individually recommended to a user based on their emotional state and preferences.

[0311] "Recommendation" is a function that selects and presents optimal content based on the user's emotional state.

[0312] The system for this application example recognizes emotions based on the user's voice input and provides the function of recommending personalized content. To achieve this, the following hardware and software configuration is required.

[0313] Hardware used

[0314] 1. Audio input device (microphone): A device required to capture the user's voice.

[0315] 2. Speaking device (speaker): A device for speaking the generated voice data to the user.

[0316] 3. Computing device (PC / server): A device for data processing and voice analysis and generation.

[0317] Software used

[0318] 1. Speech Recognition Library: A library for converting user speech into text data.

[0319] 2. Natural Language Processing Library (TextBlob): A library for analyzing emotions from converted text data.

[0320] 3. Speech synthesis software: Software that converts emotionally charged text into voice data.

[0321] 4. Server application: An application for integrating and managing the various libraries and software mentioned above.

[0322] Data processing and calculation

[0323] The server first converts the speech received from the voice input device into text data using a speech recognition library. It then uses the converted text data and speech data to analyze emotions using a natural language processing library. Based on the analyzed emotional information, it recommends personalized content tailored to the user's emotions. The recommended content information is then converted into emotionally charged speech data using text generation and speech synthesis software, and the speech data is then sent to the speech device.

[0324] Specific examples

[0325] If a user says, "I'm very tired today," the server captures this speech and converts it into text data using a speech recognition library. It then analyzes the emotion from the text data using a natural language processing library to identify "tiredness." Based on the analyzed emotion, personalized content such as relaxing music or videos is recommended. Finally, information about the recommended content is generated as text, converted into audio data using speech synthesis software, and provided to the user via a speech device.

[0326] Prompt Sentence Examples

[0327] Identify emotions from the text entered by the user and suggest content that corresponds to those emotions. For example, if a user says, "I've been working all day today and I'm really tired," recommend relaxing music or stress-relieving videos.

[0328] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0329] Step 1:

[0330] The user uses the device's microphone to input voice, such as saying, "I'm very tired today." The device digitizes this voice data (an analog signal) and stores it at a certain sample rate and bit depth.

[0331] Input: User's voice

[0332] Output: Digitized audio data

[0333] Step 2:

[0334] The device uses a speech recognition library (SpeechRecognition) to convert the digitized voice data into text data. Specifically, the speech recognition engine analyzes this digital data and outputs the string "I'm very tired today."

[0335] Input: Digitized audio data

[0336] Output: Text data ("I'm very tired today")

[0337] Step 3:

[0338] The device sends the converted text data to the server over a network connection, either via an HTTP request or WebSocket.

[0339] Input: Text data

[0340] Output: Sends text data to the server

[0341] Step 4:

[0342] The server analyzes the received text data using a natural language processing library (TextBlob) to identify the emotion. The analysis engine calculates the context of the text and the positive / negative level of the words, and identifies the emotion as "tired."

[0343] Input: Text data

[0344] Output: Emotion data ("tired")

[0345] Step 5:

[0346] The server runs a recommendation algorithm to recommend content appropriate for the user based on the identified emotion data. For example, if the user feels "tired," it will suggest relaxing music or videos. The specific content recommended is retrieved from a database.

[0347] Input: Emotion data

[0348] Output: Recommended content data

[0349] Step 6:

[0350] The server generates sentences containing information about recommended content, using a sentence generation engine to create sentences such as "You've had a tough day today. How about listening to some relaxing music?"

[0351] Input: Recommended content data

[0352] Output: Generated sentence data

[0353] Step 7:

[0354] The server then passes the generated text data to speech synthesis software, which converts it into speech data, taking into account the emotion and tone of the text and generating speech with the appropriate intonation.

[0355] Input: Generated sentence data

[0356] Output: Audio data

[0357] Step 8:

[0358] The server sends the generated audio data to the device over a network connection, either via HTTP responses or WebSockets.

[0359] Input: Audio data

[0360] Output: Sending audio data from the server to the device

[0361] Step 9:

[0362] The device receives the voice data from the server and plays it back on the speaker to the user. Specifically, the device plays a message such as, "You've had a tough day today. Why don't you listen to some relaxing music?"

[0363] Input: Audio data

[0364] Output: Audio played through speaking device

[0365] Through these steps, users can be recommended content that suits their emotional state and receive empathetic feedback.

[0366] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0367] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0368] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0369] [Second embodiment]

[0370] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0371] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0372] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0373] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0374] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0375] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0376] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0377] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0378] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0379] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0380] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0381] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0382] The system of the present invention converts a user's voice input into an emotionally charged speech in order to provide psychological support to a user who feels lonely or isolated. Specific embodiments of the system are described below.

[0383] System Overview

[0384] This system consists of a terminal that receives user voice input, a server that analyzes the voice data and identifies emotions, and a speech device that converts the generated text into voice data and speaks it.The main technical elements used are voice recognition technology, natural language processing technology, and voice synthesis technology.

[0385] Program processing

[0386] The program in the system of the present invention is processed in the following procedure.

[0387] 1. Acquiring voice input

[0388] The user makes a speech input such as "I'm tired today" into the terminal.

[0389] 2. Converting voice data to text

[0390] The device converts the voice data acquired from the user into text data using voice recognition software, for example, generating text data such as "I'm tired today."

[0391] 3. Sending text data

[0392] The terminal transmits the converted text data to the server.

[0393] 4. Emotion Analysis

[0394] The server then passes the received text data to emotion analysis software, which uses natural language processing techniques to analyze the user's emotions and identify the emotion "tired."

[0395] 5. Generating emotional sentences

[0396] The server uses sentence generation software based on the analyzed emotions to generate emotional sentences, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0397] 6. Text-to-speech

[0398] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0399] 7. Transmission to speaking device

[0400] The server transmits the generated voice data to the terminal.

[0401] 8. Executing the utterance

[0402] The terminal transfers the received voice data to a speech device, which then plays the voice data. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so good things will surely happen."

[0403] Specific examples

[0404] When a user voice-inputs "I'm tired today," the device converts the voice into text data ("I'm tired today") and sends it to the server. The server analyzes the emotion from the text and identifies the emotion category "tired." Next, it generates a sentence expressing the emotion ("I see, today was tough. But you always try your best, so good things will surely happen.") and converts this into voice data. Finally, the humanoid robot plays the voice back to the user via the device, providing psychological support.

[0405] The above is an embodiment of the system of the present invention. This system enables communication that is sensitive to the user's emotions, thereby reducing feelings of loneliness and isolation.

[0406] The processing flow will be explained below.

[0407] Step 1:

[0408] The user inputs "I'm tired today" into the terminal by voice.

[0409] Step 2:

[0410] The device captures the user's voice through a microphone.

[0411] Step 3:

[0412] The device uses speech recognition software to convert the captured voice data into text, for example, "I'm tired today."

[0413] Step 4:

[0414] The terminal transmits the converted text data to the server.

[0415] Step 5:

[0416] The server passes the received text data to sentiment analysis software, which uses natural language processing techniques to analyze the user's emotions.

[0417] Step 6:

[0418] The server identifies the user's emotion (for example, "fatigue") from the output of the emotion analysis software.

[0419] Step 7:

[0420] The server uses sentence generation software to generate emotional sentences based on the user's emotions, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will come."

[0421] Step 8:

[0422] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0423] Step 9:

[0424] The server transmits the generated voice data to the terminal.

[0425] Step 10:

[0426] The terminal transfers the voice data received from the server to the speech device.

[0427] Step 11:

[0428] The speech device (part of the terminal) plays back the voice data and speaks encouraging words to the user, such as, "You had a hard day today, didn't you? But you always try your best, so I'm sure good things will happen."

[0429] Example 1

[0430] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0431] In modern society, the number of people feeling lonely and isolated is increasing. While systems exist to provide psychological support, many of these systems lack sufficient emotional response capabilities and are unable to respond in a way that is in line with the user's actual feelings. Therefore, there is a need for a system that can analyze emotions based on the user's voice input, generate sentences containing appropriate emotional expressions, and respond via voice.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0433] In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to a central processing unit, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a terminal and transferring it to a speech device for speech generation. This makes it possible to provide a voice response that is in line with the user's emotions, thereby reducing feelings of loneliness and isolation.

[0434] "Audio data" means digital or analog data that is an electronic recording of sound.

[0435] "Text data" is digital data that includes character information.

[0436] A "central processing unit" is a device that performs the central processing of a server or computer.

[0437] "Means of emotion analysis" refers to algorithms or software that identify emotions from input text data.

[0438] "Means for generating emotional sentences" refers to algorithms or software for generating sentences that contain appropriate emotional expressions based on the analyzed emotions.

[0439] "Means for converting into audio data" refers to the algorithms or software used to convert the generated text data into audio format.

[0440] A "terminal" refers to a device that is directly operated by a user, and is an electronic device that has the function of inputting and outputting voice.

[0441] A "speech device" is a device or equipment for playing back generated voice data.

[0442] MODE FOR CARRYING OUT THE INVENTION

[0443] The system of the present invention is a dialogue system that combines emotion analysis and speech generation, designed to reduce users' feelings of loneliness and isolation. The system mainly performs a series of processes, including user voice input, speech recognition, emotion analysis, sentence generation, speech synthesis, and speech production.

[0444] Hardware and Software Configuration

[0445] The user inputs voice into the system using a voice input device (microphone or smartphone).

[0446] The device receives the user's voice and converts the voice data into text data using speech recognition software (e.g., a commonly used speech recognition API), such as the Google Speech-to-Text API.

[0447] The terminal sends the generated text data to a server, which is a central processing unit. At this time, the data is sent securely via the HTTPS protocol.

[0448] The server passes the received text data to sentiment analysis software (e.g., algorithms or APIs that use natural language processing technology) to analyze the sentiment. Here, IBM Watson Tone Analyzer is used as an example.

[0449] Based on the results of the sentiment analysis, the server uses sentence generation software (e.g., a generative AI model such as OpenAI's GPT-3) to generate emotional sentences.

[0450] The generated sentences are passed to speech synthesis software (such as Amazon Polly) and converted into speech data, which then adjusts the tone and expression of the voice according to the analyzed emotion.

[0451] The server transmits the generated voice data to the terminal again, and the terminal transfers the voice data to a speech device (for example, a humanoid robot).

[0452] The speech device plays back the received voice data and speaks to the user. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0453] Specific examples

[0454] Specifically, suppose a user says "I'm tired today" by voice input. In this case, the device converts the voice into text data via the Google Speech-to-Text API. The generated text data "I'm tired today" is sent to the server using HTTPS.

[0455] The server uses IBM Watson Tone Analyzer to analyze the emotion of "tiredness," and then uses OpenAI's GPT-3 to generate a sentence that expresses the emotion: "Today was tough, wasn't it? But you always work hard, so I'm sure good things will happen."

[0456] The sentence is converted into voice data using Amazon Polly and sent from the server to the device, which then forwards the voice data to the humanoid robot, which then speaks the sentence to the user.

[0457] Prompt Sentence Examples

[0458] If you want a generative AI model to analyze emotions based on a user's speech, such as "I'm tired today," and generate an emotional response based on the results, you might use the following prompt:

[0459] Analyze the user's emotions based on the phrase "I'm tired today," and generate a response containing encouraging words based on the emotion "tired."

[0460] The above is a specific embodiment for carrying out the present invention. This system allows users to receive emotionally responsive responses, thereby reducing feelings of loneliness and isolation.

[0461] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0462] Step 1:

[0463] The user inputs voice data into the terminal. For example, the user might say, "I'm tired today." This voice data becomes the input.

[0464] Step 2:

[0465] The device converts the voice data acquired from the user into text data using voice recognition software (e.g., Google Speech-to-Text API). Specifically, the device's microphone captures the voice data and sends it to the Google Speech-to-Text API, obtaining text data such as "I'm tired today" as output.

[0466] Step 3:

[0467] The terminal sends the converted text data to the server. Specifically, the terminal sends the generated text data to the server as an HTTPS request, using the TLS protocol to ensure security. The input of this step is the text data "I'm tired today," and the output is the completion of sending the text data to the server.

[0468] Step 4:

[0469] The server passes the received text data to emotion analysis software (for example, IBM Watson Tone Analyzer) to analyze the emotion. The server then sends the text data in JSON format to the emotion analysis engine, and obtains the emotion label "tired" as the output of the analysis result. The input is the text data "I'm tired today," and the output is the emotion label "tired."

[0470] Step 5:

[0471] The server generates emotional sentences using sentence generation software (e.g., OpenAI's GPT-3) based on the analyzed emotions. Specifically, the server supplies the emotion label "tired" and the original text data as prompts to GPT-3, and obtains the sentence "Today was tough, wasn't it? But you always try your best, so good things will surely come." The input is the emotion label and the original text data, and the output is the generated sentence.

[0472] Step 6:

[0473] The server passes the generated sentence to speech synthesis software (e.g., Amazon Polly) and converts it into voice data. The server then sends the generated sentence in text format to the Amazon Polly API, which outputs voice data with tone and speed set according to the emotion. The input is the generated sentence, and the output is voice data.

[0474] Step 7:

[0475] The server sends the generated audio data to the device. Specifically, the server sends the audio data in binary format to the device as an HTTPS request, and the device confirms receipt of the audio data. The input of this step is the audio data, and the output is completion of sending the audio data to the device.

[0476] Step 8:

[0477] The terminal transfers the received voice data to a speech device (e.g., a humanoid robot), which then plays the voice data. Specifically, the terminal transmits the voice data to the speech device via Bluetooth or Wi-Fi, and the speaker of the speech device plays the following voice as output: "Today was tough, wasn't it? But you always try your best, so good things will surely happen." The input is the voice data, and the output is the spoken voice.

[0478] (Application example 1)

[0479] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] In recent years, the number of users feeling lonely and isolated has been increasing, creating a need for psychological support. However, conventional systems have had difficulty providing appropriate feedback in real time that reflects the user's emotions. In particular, content distribution services lack a means to provide comments and encouragement that reflect the user's emotions, creating a need for improved user experience. The objective of this invention is to provide a system that converts voice input into emotional speech, thereby providing users with psychological support.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0482] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to produce speech, and means for providing appropriate comments and encouragement to users in a content distribution service based on the results of the emotion analysis. This enables communication that is sensitive to the user's emotions, reduces feelings of loneliness and isolation, and improves the content viewing experience.

[0483] "Audio data" is a sound signal expressed in digital or analog format, and is information including the user's voice.

[0484] "Text data" refers to character information expressed in digital form, and includes words obtained by analyzing voice data.

[0485] "Emotions" refer to a person's psychological state and include a variety of psychological states such as joy, sadness, and anger.

[0486] "Means of emotion analysis" refers to technologies and algorithms for identifying a user's emotions from input text data, and may utilize natural language processing technology or machine learning models.

[0487] "Emotional sentences" are sentences that express a context or a specific emotion generated based on the analyzed emotion, and are generated according to the user's psychological state.

[0488] A "speech device" is a device for reproducing and communicating audio data to a user, and includes a speaker or other audio reproduction device.

[0489] "Content distribution service" refers to a service that provides digital content to users, including various media formats such as movies, music, videos, and articles.

[0490] The system for implementing this invention converts a user's voice input into text data, analyzes emotions, generates emotional sentences based on the results, and finally provides feedback to the user as voice data. A specific configuration for implementing this system will be described below.

[0491] System configuration

[0492] The system consists of the following major components:

[0493] 1. Voice input acquisition device

[0494] This device, such as a smartphone or a head-mounted display, captures the user's voice.

[0495] 2. Voice Recognition Software

[0496] Speech recognition software converts captured voice data into text data using voice recognition technologies such as Google's Speech-to-Text API.

[0497] 3. Emotion Analysis Server

[0498] This server analyzes the user's emotions from the received text data, using natural language processing technologies such as Hugging Face's GPT-3.

[0499] 4. Text Generation Software

[0500] The sentence generation software generates emotional sentences based on the analyzed emotions. The generated sentences include concrete examples such as, "I see you had a hard day today. But you always work hard, so good things will surely happen."

[0501] 5. Text-to-Speech Software

[0502] Text-to-speech software converts the generated text into audio data, using voice synthesis technologies such as Google's Text-to-Speech API.

[0503] 6. Speaking Devices

[0504] This device plays back the generated audio data, similar to the audio output function of a smartphone speaker or a head-mounted display.

[0505] Processing method

[0506] A user initiates the system by providing voice input. For example, the user might say, "I'm tired today." The voice input capture device records this speech and converts it into text data using speech recognition software. This text data is then sent to an emotion analysis server, which analyzes the emotion "tired" using natural language processing techniques. Based on the analysis results, sentence generation software generates an appropriate emotional response. Finally, speech synthesis software converts this sentence into voice data, which is played back to the user via a speech device.

[0507] Specific examples

[0508] As a concrete example, the following prompt sentence is used when the user inputs "I'm tired today." "User input: 'I'm tired today'. Generate an encouraging sentence based on this input." This prompt sentence generates appropriate feedback that is in line with the user's emotions.

[0509] In this way, the system can provide communication that is sensitive to the user's emotions, helping to reduce feelings of loneliness and isolation, and improving the user experience in content distribution services.

[0510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0511] Step 1:

[0512] The user performs voice input, for example, saying, "I'm tired today."

[0513] Input: User's voice data

[0514] Output: Audio data sent to the device

[0515] Specific operation: The microphone on the smartphone or head-mounted display picks up the user's voice.

[0516] Step 2:

[0517] The device uses voice recognition software (such as Google's Speech-to-Text API) to convert voice input into text data.

[0518] Input: User's voice data

[0519] Output: Text data generated by the speech recognition software (e.g., "I'm tired today")

[0520] Specific operation: A process is carried out to convert the voice data into text data.

[0521] Step 3:

[0522] The terminal transmits the converted text data to the emotion analysis server.

[0523] Input: Text data

[0524] Output: Text data sent to the sentiment analysis server

[0525] Specific operation: Send text data to the server using an HTTP request, etc.

[0526] Step 4:

[0527] The server uses natural language processing technology (such as GPT-3 in Hugging Face) to analyze emotions from the received text data.

[0528] Input: Text data (e.g., "I'm tired today")

[0529] Output: Analyzed emotion data (e.g., "tired")

[0530] Specific operations: Perform sentiment analysis of text data using natural language processing technology.

[0531] Step 5:

[0532] The server uses sentence generation software to generate emotional sentences based on the analyzed emotions.

[0533] Input: Emotion data (e.g., "tired")

[0534] Output: Generated sentence (e.g. "Today was tough, but you always try your best, so I'm sure good things will come out of it.")

[0535] Specific action: The sentence generation software generates sentences that correspond to the emotions.

[0536] Step 6:

[0537] The server uses speech synthesis software (such as Google's Text-to-Speech API) to convert the generated text into audio data.

[0538] Input: Text (e.g., "I know you had a hard day today. But you always work hard, so good things will come your way.")

[0539] Output: Audio data

[0540] Specific operation: The process converts text into audio data.

[0541] Step 7:

[0542] The server transmits the generated voice data to the terminal.

[0543] Input: Audio data

[0544] Output: Audio data sent to the device

[0545] Specific operation: Send audio data to the terminal using an HTTP request, etc.

[0546] Step 8:

[0547] The terminal transfers the received voice data to the speech device, which then reproduces the voice data.

[0548] Input: Audio data

[0549] Output: The audio played to the user

[0550] Specific operation: Play audio data through the speaker of a smartphone or head-mounted display.

[0551] In this way, a system is realized that provides emotional feedback in real time based on the user's voice input.

[0552] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0553] The system of the present invention combines the user's voice input with an emotion engine to more precisely recognize the user's emotions and generate emotional utterances based on those emotions. By adding an analysis function using the emotion engine, this system further improves the accuracy of recognizing the user's emotional state, realizing more empathetic and effective communication.

[0554] System Overview

[0555] The system consists of the following main components:

[0556] 1. A device that receives user voice input

[0557] 2. A server containing an emotion engine that analyzes voice data and identifies emotions

[0558] 3. Server that generates sentences based on emotions and converts them into voice data

[0559] 4. A speech device that produces speech using the generated voice data

[0560] Program processing

[0561] The processing flow of a typical program in the system of the present invention is shown below.

[0562] 1. Acquiring voice input

[0563] The user inputs into the terminal, "Today was very hard."

[0564] 2. Converting voice data to text

[0565] The device uses a microphone to capture voice data and uses voice recognition software to convert it into text data ("Today was very hard").

[0566] 3. Sending text data

[0567] The terminal transmits the converted text data to the server.

[0568] 4. Emotion Analysis

[0569] The server first uses an emotion engine to analyze the voice characteristics (pitch, tempo, intonation) of the voice data, and then, if necessary, analyzes non-voice data such as the user's facial expressions and movements, and combines these data to recognize the user's emotions.

[0570] 5. Information Integration

[0571] The server integrates the voice features and text data to more precisely identify the user's emotional state. For example, if the text data indicates "painful," and the voice features also support this, the emotion "very pain" can be identified.

[0572] 6. Generating emotional sentences

[0573] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0574] 7. Text-to-speech

[0575] The server then passes the generated text to speech synthesis software, which generates speech data that expresses the appropriate emotion, taking into account tone and intonation.

[0576] 8. Sending audio data

[0577] The server transmits the generated voice data to the terminal.

[0578] 9. Executing the utterance

[0579] The terminal transfers the voice data received from the server to the speech device, which then speaks, "Today must have been really tough. But your efforts were not in vain."

[0580] Specific examples

[0581] When a user voice-inputs "Today was really hard," the device converts the speech into text data ("Today was really hard") and sends it to the server. The server uses an emotion engine to analyze the speech features and further analyzes the text data to identify the "hard" emotion. Next, it generates a sentence such as "Today was really hard, wasn't it? But your efforts were worth it," and converts it into speech data. Finally, the device sends the speech data to a speech generator, which can provide encouraging speech to the user.

[0582] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide communication that is sensitive to the user's emotions.

[0583] The processing flow will be explained below.

[0584] Step 1:

[0585] The user inputs into the terminal, "Today was very hard."

[0586] Step 2:

[0587] The terminal uses a microphone to capture the user's voice data.

[0588] Step 3:

[0589] The device uses speech recognition software to convert the captured voice data into text, such as "Today was a very difficult day."

[0590] Step 4:

[0591] The terminal transmits the converted text data and speech feature data, which includes information such as pitch, tempo, and intonation, to the server.

[0592] Step 5:

[0593] The server first analyzes the voice feature data using an emotion engine and recognizes the emotion from the user's voice. For example, if the voice feature data shows that the user's voice is low and slow, the emotion "sad" is recognized.

[0594] Step 6:

[0595] The server analyzes the text data ("Today was very difficult") using natural language processing technology and identifies the emotion ("difficult") contained in the text data.

[0596] Step 7:

[0597] The server integrates the analysis results of the voice feature data and the text data to more accurately identify the user's emotional state. For example, if both the voice feature data and the text data indicate "sad" and "distressed," the overall emotion identified will be "very distressed."

[0598] Step 8:

[0599] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0600] Step 9:

[0601] The server then passes the generated text to speech synthesis software, which generates emotionally appropriate speech data with a tone and intonation that reflects the identified emotion.

[0602] Step 10:

[0603] The server transmits the generated voice data to the terminal.

[0604] Step 11:

[0605] The terminal transfers the voice data received from the server to the speech device.

[0606] Step 12:

[0607] The speech device of the terminal plays back the voice data and speaks encouraging words to the user, such as, "Today must have been really tough, but your efforts will not have been in vain."

[0608] Example 2

[0609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] Conventional speech recognition and emotion analysis systems have faced challenges in accurately recognizing a user's emotions and responding appropriately. In particular, when identifying emotions based solely on speech data, the user's speech content and voice characteristics alone are insufficient, leading to concerns about reduced accuracy. Furthermore, when generating emotion-based responses, it has been difficult to provide the empathetic and appropriate communication users expect.

[0611] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to the server via a cloud service, means for identifying emotions by analyzing the text data and voice features, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a speech device to have it speak. This enables the user's emotions to be recognized with high accuracy, and emotional sentences to be generated based on the recognition, enabling more empathetic and effective communication.

[0612] The "means for converting input voice data into text data" refers to a device or algorithm for converting voice data input by a user into text format data.

[0613] "Means for sending to a server via a cloud service" refers to protocols and software for sending data from a local device to a remote server using cloud computing infrastructure.

[0614] The "means for analyzing text data and audio features to identify emotions" refers to an algorithm or engine that analyzes the content of text data and audio features (e.g., pitch, tempo, intonation) of audio data to identify a user's emotions.

[0615] The "means for generating emotional sentences based on emotions" refers to algorithms and natural language processing techniques for generating empathetic and emotional sentences in response to the identified emotions of a user.

[0616] The "means for converting the generated text into speech data" refers to speech synthesis software or technology for converting the generated text in text format into speech format data.

[0617] The "means for transmitting to a speech generating device and causing speech generation" refers to a protocol or interface for transmitting the generated voice data to a physical speech generating device and using that device to actually reproduce the voice.

[0618] "Audio features" are predetermined features such as pitch, tempo, and intonation contained in audio data, and are information that enables emotion recognition by analyzing these features.

[0619] "Non-voice data" refers to data other than voice data, such as the user's facial expressions, movements, heart rate, etc., which are used complementary to emotion recognition.

[0620] This invention is a system that recognizes emotions with high accuracy based on a user's voice input and provides speech that incorporates the emotions accordingly. This system is composed of a terminal that acquires the user's voice input, a server that analyzes the voice data and identifies the emotion, a server that generates sentences based on the emotion and converts them into voice data, and a speech device that speaks the generated voice data.

[0621] The user inputs voice into the device. For example, the user says, "Today was very hard." This voice is captured by the device's microphone. The device then converts the voice data into text data using voice recognition software (for example, a voice recognition API). The converted text data ("Today was very hard") is sent to the server via a cloud service.

[0622] The server uses an emotion engine (e.g., emotion analysis API) to analyze the voice features (pitch, tempo, intonation) of the voice data. If necessary, it also analyzes non-voice data such as the user's facial expressions and movements to comprehensively recognize emotions. This makes it possible to identify the user's emotions by integrating text data, voice features, and non-voice data. For example, if the text data contains the word "spicy" and the voice features support this, the emotion "very spicy" is identified.

[0623] Next, the server uses a generative AI model (e.g., a natural language generation model) to generate emotional sentences based on the identified emotions. For example, a sentence like, "Today must have been really tough, but your efforts are worth it." In this generation process, prompt sentences like the following are used:

[0624] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0625] The server then passes the generated sentences to speech synthesis software (e.g., a speech synthesis API) to generate emotionally appropriate speech data, including the appropriate tone and intonation, which is then transmitted to the device using a secure protocol (e.g., HTTPS).

[0626] Finally, the device transmits the received voice data to a speech device (e.g., a smart speaker or speaker system), which then speaks, "You've had a really hard day today, but your efforts are worth it." Through this series of processes, the user can experience empathetic and effective communication through the system.

[0627] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide an appropriate emotional response based on the recognition.

[0628] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0629] Step 1:

[0630] The user speaks to the terminal, saying, "Today was a very hard day." This speech input is captured by the terminal's microphone. The captured speech data becomes the input.

[0631] Step 2:

[0632] The device converts the captured voice data into text using speech recognition software (e.g., a speech recognition API), where the voice signal is analyzed through digital signal processing and a corresponding string of characters ("Today was very hard") is output.

[0633] Step 3:

[0634] The device sends the converted text data to a server via a cloud service, using a secure protocol (e.g., HTTPS). The input of this transmission process is the text data, and the output is the completion of the transfer of the text data to the server.

[0635] Step 4:

[0636] The server analyzes emotions using an emotion engine (e.g., emotion analysis API) to analyze text data and voice features. This analysis is performed by extracting voice features (pitch, tempo, intonation) from the voice data and, if necessary, analyzing non-voice data (e.g., the user's facial expressions and movements). The input is text data and voice feature data, and the output is the analyzed user's emotion information.

[0637] Step 5:

[0638] The server integrates the voice features and text data to precisely identify the user's emotion. In this integration process, for example, if the text data indicates "spicy," the emotion "very spicy" can be identified by matching it with the voice features. The input is voice feature data and text data, and the output is the final emotion information.

[0639] Step 6:

[0640] The server uses a generative AI model (e.g., a natural language generation model) based on the identified emotion to generate emotional sentences. Specifically, it generates empathetic sentences that are in tune with the emotion based on the identified emotional information. The input is the emotional information, and the output is the generated sentences (e.g., "Today was really tough, wasn't it? But your hard work won't be in vain.").

[0641] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0642] Step 7:

[0643] The server passes the generated sentence to speech synthesis software (e.g., a speech synthesis API) to generate speech data that appropriately expresses emotion. The speech synthesis process adds a voice tone and intonation appropriate for the generated sentence. The input is the generated sentence, and the output is speech data.

[0644] Step 8:

[0645] The server sends the generated audio data to the terminal. Again, a secure protocol (e.g., HTTPS) is used. The input is the audio data, and the output is the completion of sending the audio data to the terminal.

[0646] Step 9:

[0647] The device then transfers the received voice data to a speech device (e.g., a smart speaker or speaker system), which ultimately speaks, "Today was really tough, wasn't it? But your efforts were worth it." The input is the voice data sent from the server, and the output is delivered to the user as physical sound.

[0648] (Application example 2)

[0649] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Conventional systems were able to analyze emotions in response to user voice input and generate emotionally charged sentences, but lacked the ability to provide optimal content and actions based on the user's emotional state. As a result, users were unable to receive appropriate content or feedback based on their emotions, making it difficult to provide empathetic and effective communication and entertainment. Furthermore, the lack of a function to recommend content based on emotions made it impossible to provide personalized services.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0652] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to have it speak, and means for recommending personalized content based on the user's emotions. This makes it possible to provide optimal content tailored to the user's emotional state, thereby realizing the provision of more empathetic and personalized services.

[0653] "Voice data" refers to the digitized voice of the user.

[0654] "Text data" is voice data converted into a string of characters.

[0655] "Emotion analysis" is the process of identifying an emotional state from text data and voice data.

[0656] "Sentence generation" means creating sentences based on the results of sentiment analysis.

[0657] "Convert to voice data" means converting the generated text back into voice data.

[0658] A "speech device" is a device that outputs voice data as voice.

[0659] "Personalized content" refers to content that is individually recommended to a user based on their emotional state and preferences.

[0660] "Recommendation" is a function that selects and presents optimal content based on the user's emotional state.

[0661] The system for this application example recognizes emotions based on the user's voice input and provides the function of recommending personalized content. To achieve this, the following hardware and software configuration is required.

[0662] Hardware used

[0663] 1. Audio input device (microphone): A device required to capture the user's voice.

[0664] 2. Speaking device (speaker): A device for speaking the generated voice data to the user.

[0665] 3. Computing device (PC / server): A device for data processing and voice analysis and generation.

[0666] Software used

[0667] 1. Speech Recognition Library: A library for converting user speech into text data.

[0668] 2. Natural Language Processing Library (TextBlob): A library for analyzing emotions from converted text data.

[0669] 3. Speech synthesis software: Software that converts emotionally charged text into voice data.

[0670] 4. Server application: An application for integrating and managing the various libraries and software mentioned above.

[0671] Data processing and calculation

[0672] The server first converts the speech received from the voice input device into text data using a speech recognition library. It then uses the converted text data and speech data to analyze emotions using a natural language processing library. Based on the analyzed emotional information, it recommends personalized content tailored to the user's emotions. The recommended content information is then converted into emotionally charged speech data using text generation and speech synthesis software, and the speech data is then sent to the speech device.

[0673] Specific examples

[0674] If a user says, "I'm very tired today," the server captures this speech and converts it into text data using a speech recognition library. It then analyzes the emotion from the text data using a natural language processing library to identify "tiredness." Based on the analyzed emotion, personalized content such as relaxing music or videos is recommended. Finally, information about the recommended content is generated as text, converted into audio data using speech synthesis software, and provided to the user via a speech device.

[0675] Prompt Sentence Examples

[0676] Identify emotions from the text entered by the user and suggest content that corresponds to those emotions. For example, if a user says, "I've been working all day today and I'm really tired," recommend relaxing music or stress-relieving videos.

[0677] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0678] Step 1:

[0679] The user uses the device's microphone to input voice, such as saying, "I'm very tired today." The device digitizes this voice data (an analog signal) and stores it at a certain sample rate and bit depth.

[0680] Input: User's voice

[0681] Output: Digitized audio data

[0682] Step 2:

[0683] The device uses a speech recognition library (SpeechRecognition) to convert the digitized voice data into text data. Specifically, the speech recognition engine analyzes this digital data and outputs the string "I'm very tired today."

[0684] Input: Digitized audio data

[0685] Output: Text data ("I'm very tired today")

[0686] Step 3:

[0687] The device sends the converted text data to the server over a network connection, either via an HTTP request or WebSocket.

[0688] Input: Text data

[0689] Output: Sends text data to the server

[0690] Step 4:

[0691] The server analyzes the received text data using a natural language processing library (TextBlob) to identify the emotion. The analysis engine calculates the context of the text and the positive / negative level of the words, and identifies the emotion as "tired."

[0692] Input: Text data

[0693] Output: Emotion data ("tired")

[0694] Step 5:

[0695] The server runs a recommendation algorithm to recommend content appropriate for the user based on the identified emotion data. For example, if the user feels "tired," it will suggest relaxing music or videos. The specific content recommended is retrieved from a database.

[0696] Input: Emotion data

[0697] Output: Recommended content data

[0698] Step 6:

[0699] The server generates sentences containing information about recommended content, using a sentence generation engine to create sentences such as "You've had a tough day today. How about listening to some relaxing music?"

[0700] Input: Recommended content data

[0701] Output: Generated sentence data

[0702] Step 7:

[0703] The server then passes the generated text data to speech synthesis software, which converts it into speech data, taking into account the emotion and tone of the text and generating speech with the appropriate intonation.

[0704] Input: Generated sentence data

[0705] Output: Audio data

[0706] Step 8:

[0707] The server sends the generated audio data to the device over a network connection, either via HTTP responses or WebSockets.

[0708] Input: Audio data

[0709] Output: Sending audio data from the server to the device

[0710] Step 9:

[0711] The device receives the voice data from the server and plays it back on the speaker to the user. Specifically, the device plays a message such as, "You've had a tough day today. Why don't you listen to some relaxing music?"

[0712] Input: Audio data

[0713] Output: Audio played through speaking device

[0714] Through these steps, users can be recommended content that suits their emotional state and receive empathetic feedback.

[0715] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0717] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0718] [Third embodiment]

[0719] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0720] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0721] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0722] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0723] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0724] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0725] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0726] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0729] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0730] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0731] The system of the present invention converts a user's voice input into an emotionally charged speech in order to provide psychological support to a user who feels lonely or isolated. Specific embodiments of the system are described below.

[0732] System Overview

[0733] This system consists of a terminal that receives user voice input, a server that analyzes the voice data and identifies emotions, and a speech device that converts the generated text into voice data and speaks it.The main technical elements used are voice recognition technology, natural language processing technology, and voice synthesis technology.

[0734] Program processing

[0735] The program in the system of the present invention is processed in the following procedure.

[0736] 1. Acquiring voice input

[0737] The user makes a speech input such as "I'm tired today" into the terminal.

[0738] 2. Converting voice data to text

[0739] The device converts the voice data acquired from the user into text data using voice recognition software, for example, generating text data such as "I'm tired today."

[0740] 3. Sending text data

[0741] The terminal transmits the converted text data to the server.

[0742] 4. Emotion Analysis

[0743] The server then passes the received text data to emotion analysis software, which uses natural language processing techniques to analyze the user's emotions and identify the emotion "tired."

[0744] 5. Generating emotional sentences

[0745] The server uses sentence generation software based on the analyzed emotions to generate emotional sentences, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0746] 6. Text-to-speech

[0747] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0748] 7. Transmission to speaking device

[0749] The server transmits the generated voice data to the terminal.

[0750] 8. Executing the utterance

[0751] The terminal transfers the received voice data to a speech device, which then plays the voice data. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so good things will surely happen."

[0752] Specific examples

[0753] When a user voice-inputs "I'm tired today," the device converts the voice into text data ("I'm tired today") and sends it to the server. The server analyzes the emotion from the text and identifies the emotion category "tired." Next, it generates a sentence expressing the emotion ("I see, today was tough. But you always try your best, so good things will surely happen.") and converts this into voice data. Finally, the humanoid robot plays the voice back to the user via the device, providing psychological support.

[0754] The above is an embodiment of the system of the present invention. This system enables communication that is sensitive to the user's emotions, thereby reducing feelings of loneliness and isolation.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] The user inputs "I'm tired today" into the terminal by voice.

[0758] Step 2:

[0759] The device captures the user's voice through a microphone.

[0760] Step 3:

[0761] The device uses speech recognition software to convert the captured voice data into text, for example, "I'm tired today."

[0762] Step 4:

[0763] The terminal transmits the converted text data to the server.

[0764] Step 5:

[0765] The server passes the received text data to sentiment analysis software, which uses natural language processing techniques to analyze the user's emotions.

[0766] Step 6:

[0767] The server identifies the user's emotion (for example, "fatigue") from the output of the emotion analysis software.

[0768] Step 7:

[0769] The server uses sentence generation software to generate emotional sentences based on the user's emotions, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will come."

[0770] Step 8:

[0771] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[0772] Step 9:

[0773] The server transmits the generated voice data to the terminal.

[0774] Step 10:

[0775] The terminal transfers the voice data received from the server to the speech device.

[0776] Step 11:

[0777] The speech device (part of the terminal) plays back the voice data and speaks encouraging words to the user, such as, "You had a hard day today, didn't you? But you always try your best, so I'm sure good things will happen."

[0778] Example 1

[0779] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0780] In modern society, the number of people feeling lonely and isolated is increasing. While systems exist to provide psychological support, many of these systems lack sufficient emotional response capabilities and are unable to respond in a way that is in line with the user's actual feelings. Therefore, there is a need for a system that can analyze emotions based on the user's voice input, generate sentences containing appropriate emotional expressions, and respond via voice.

[0781] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0782] In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to a central processing unit, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a terminal and transferring it to a speech device for speech generation. This makes it possible to provide a voice response that is in line with the user's emotions, thereby reducing feelings of loneliness and isolation.

[0783] "Audio data" means digital or analog data that is an electronic recording of sound.

[0784] "Text data" is digital data that includes character information.

[0785] A "central processing unit" is a device that performs the central processing of a server or computer.

[0786] "Means of emotion analysis" refers to algorithms or software that identify emotions from input text data.

[0787] "Means for generating emotional sentences" refers to algorithms or software for generating sentences that contain appropriate emotional expressions based on the analyzed emotions.

[0788] "Means for converting into audio data" refers to the algorithms or software used to convert the generated text data into audio format.

[0789] A "terminal" refers to a device that is directly operated by a user, and is an electronic device that has the function of inputting and outputting voice.

[0790] A "speech device" is a device or equipment for playing back generated voice data.

[0791] MODE FOR CARRYING OUT THE INVENTION

[0792] The system of the present invention is a dialogue system that combines emotion analysis and speech generation, designed to reduce users' feelings of loneliness and isolation. The system mainly performs a series of processes, including user voice input, speech recognition, emotion analysis, sentence generation, speech synthesis, and speech production.

[0793] Hardware and Software Configuration

[0794] The user inputs voice into the system using a voice input device (microphone or smartphone).

[0795] The device receives the user's voice and converts the voice data into text data using speech recognition software (e.g., a commonly used speech recognition API), such as the Google Speech-to-Text API.

[0796] The terminal sends the generated text data to a server, which is a central processing unit. At this time, the data is sent securely via the HTTPS protocol.

[0797] The server passes the received text data to sentiment analysis software (e.g., algorithms or APIs that use natural language processing technology) to analyze the sentiment. Here, IBM Watson Tone Analyzer is used as an example.

[0798] Based on the results of the sentiment analysis, the server uses sentence generation software (e.g., a generative AI model such as OpenAI's GPT-3) to generate emotional sentences.

[0799] The generated sentences are passed to speech synthesis software (such as Amazon Polly) and converted into speech data, which then adjusts the tone and expression of the voice according to the analyzed emotion.

[0800] The server transmits the generated voice data to the terminal again, and the terminal transfers the voice data to a speech device (for example, a humanoid robot).

[0801] The speech device plays back the received voice data and speaks to the user. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[0802] Specific examples

[0803] Specifically, suppose a user says "I'm tired today" by voice input. In this case, the device converts the voice into text data via the Google Speech-to-Text API. The generated text data "I'm tired today" is sent to the server using HTTPS.

[0804] The server uses IBM Watson Tone Analyzer to analyze the emotion of "tiredness," and then uses OpenAI's GPT-3 to generate a sentence that expresses the emotion: "Today was tough, wasn't it? But you always work hard, so I'm sure good things will happen."

[0805] The sentence is converted into voice data using Amazon Polly and sent from the server to the device, which then forwards the voice data to the humanoid robot, which then speaks the sentence to the user.

[0806] Prompt Sentence Examples

[0807] If you want a generative AI model to analyze emotions based on a user's speech, such as "I'm tired today," and generate an emotional response based on the results, you might use the following prompt:

[0808] Analyze the user's emotions based on the phrase "I'm tired today," and generate a response containing encouraging words based on the emotion "tired."

[0809] The above is a specific embodiment for carrying out the present invention. This system allows users to receive emotionally responsive responses, thereby reducing feelings of loneliness and isolation.

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] Step 1:

[0812] The user inputs voice data into the terminal. For example, the user might say, "I'm tired today." This voice data becomes the input.

[0813] Step 2:

[0814] The device converts the voice data acquired from the user into text data using voice recognition software (e.g., Google Speech-to-Text API). Specifically, the device's microphone captures the voice data and sends it to the Google Speech-to-Text API, obtaining text data such as "I'm tired today" as output.

[0815] Step 3:

[0816] The terminal sends the converted text data to the server. Specifically, the terminal sends the generated text data to the server as an HTTPS request, using the TLS protocol to ensure security. The input of this step is the text data "I'm tired today," and the output is the completion of sending the text data to the server.

[0817] Step 4:

[0818] The server passes the received text data to emotion analysis software (for example, IBM Watson Tone Analyzer) to analyze the emotion. The server then sends the text data in JSON format to the emotion analysis engine, and obtains the emotion label "tired" as the output of the analysis result. The input is the text data "I'm tired today," and the output is the emotion label "tired."

[0819] Step 5:

[0820] The server generates emotional sentences using sentence generation software (e.g., OpenAI's GPT-3) based on the analyzed emotions. Specifically, the server supplies the emotion label "tired" and the original text data as prompts to GPT-3, and obtains the sentence "Today was tough, wasn't it? But you always try your best, so good things will surely come." The input is the emotion label and the original text data, and the output is the generated sentence.

[0821] Step 6:

[0822] The server passes the generated sentence to speech synthesis software (e.g., Amazon Polly) and converts it into voice data. The server then sends the generated sentence in text format to the Amazon Polly API, which outputs voice data with tone and speed set according to the emotion. The input is the generated sentence, and the output is voice data.

[0823] Step 7:

[0824] The server sends the generated audio data to the device. Specifically, the server sends the audio data in binary format to the device as an HTTPS request, and the device confirms receipt of the audio data. The input of this step is the audio data, and the output is completion of sending the audio data to the device.

[0825] Step 8:

[0826] The terminal transfers the received voice data to a speech device (e.g., a humanoid robot), which then plays the voice data. Specifically, the terminal transmits the voice data to the speech device via Bluetooth or Wi-Fi, and the speaker of the speech device plays the following voice as output: "Today was tough, wasn't it? But you always try your best, so good things will surely happen." The input is the voice data, and the output is the spoken voice.

[0827] (Application example 1)

[0828] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0829] In recent years, the number of users feeling lonely and isolated has been increasing, creating a need for psychological support. However, conventional systems have had difficulty providing appropriate feedback in real time that reflects the user's emotions. In particular, content distribution services lack a means to provide comments and encouragement that reflect the user's emotions, creating a need for improved user experience. The objective of this invention is to provide a system that converts voice input into emotional speech, thereby providing users with psychological support.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0831] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to produce speech, and means for providing appropriate comments and encouragement to users in a content distribution service based on the results of the emotion analysis. This enables communication that is sensitive to the user's emotions, reduces feelings of loneliness and isolation, and improves the content viewing experience.

[0832] "Audio data" is a sound signal expressed in digital or analog format, and is information including the user's voice.

[0833] "Text data" refers to character information expressed in digital form, and includes words obtained by analyzing voice data.

[0834] "Emotions" refer to a person's psychological state and include a variety of psychological states such as joy, sadness, and anger.

[0835] "Means of emotion analysis" refers to technologies and algorithms for identifying a user's emotions from input text data, and may utilize natural language processing technology or machine learning models.

[0836] "Emotional sentences" are sentences that express a context or a specific emotion generated based on the analyzed emotion, and are generated according to the user's psychological state.

[0837] A "speech device" is a device for reproducing and communicating audio data to a user, and includes a speaker or other audio reproduction device.

[0838] "Content distribution service" refers to a service that provides digital content to users, including various media formats such as movies, music, videos, and articles.

[0839] The system for implementing this invention converts a user's voice input into text data, analyzes emotions, generates emotional sentences based on the results, and finally provides feedback to the user as voice data. A specific configuration for implementing this system will be described below.

[0840] System configuration

[0841] The system consists of the following major components:

[0842] 1. Voice input acquisition device

[0843] This device, such as a smartphone or a head-mounted display, captures the user's voice.

[0844] 2. Voice Recognition Software

[0845] Speech recognition software converts captured voice data into text data using voice recognition technologies such as Google's Speech-to-Text API.

[0846] 3. Emotion Analysis Server

[0847] This server analyzes the user's emotions from the received text data, using natural language processing technologies such as Hugging Face's GPT-3.

[0848] 4. Text Generation Software

[0849] The sentence generation software generates emotional sentences based on the analyzed emotions. The generated sentences include concrete examples such as, "I see you had a hard day today. But you always work hard, so good things will surely happen."

[0850] 5. Text-to-Speech Software

[0851] Text-to-speech software converts the generated text into audio data, using voice synthesis technologies such as Google's Text-to-Speech API.

[0852] 6. Speaking Devices

[0853] This device plays back the generated audio data, similar to the audio output function of a smartphone speaker or a head-mounted display.

[0854] Processing method

[0855] A user initiates the system by providing voice input. For example, the user might say, "I'm tired today." The voice input capture device records this speech and converts it into text data using speech recognition software. This text data is then sent to an emotion analysis server, which analyzes the emotion "tired" using natural language processing techniques. Based on the analysis results, sentence generation software generates an appropriate emotional response. Finally, speech synthesis software converts this sentence into voice data, which is played back to the user via a speech device.

[0856] Specific examples

[0857] As a concrete example, the following prompt sentence is used when the user inputs "I'm tired today." "User input: 'I'm tired today'. Generate an encouraging sentence based on this input." This prompt sentence generates appropriate feedback that is in line with the user's emotions.

[0858] In this way, the system can provide communication that is sensitive to the user's emotions, helping to reduce feelings of loneliness and isolation, and improving the user experience in content distribution services.

[0859] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0860] Step 1:

[0861] The user performs voice input, for example, saying, "I'm tired today."

[0862] Input: User's voice data

[0863] Output: Audio data sent to the device

[0864] Specific operation: The microphone on the smartphone or head-mounted display picks up the user's voice.

[0865] Step 2:

[0866] The device uses voice recognition software (such as Google's Speech-to-Text API) to convert voice input into text data.

[0867] Input: User's voice data

[0868] Output: Text data generated by the speech recognition software (e.g., "I'm tired today")

[0869] Specific operation: A process is carried out to convert the voice data into text data.

[0870] Step 3:

[0871] The terminal transmits the converted text data to the emotion analysis server.

[0872] Input: Text data

[0873] Output: Text data sent to the sentiment analysis server

[0874] Specific operation: Send text data to the server using an HTTP request, etc.

[0875] Step 4:

[0876] The server uses natural language processing technology (such as GPT-3 in Hugging Face) to analyze emotions from the received text data.

[0877] Input: Text data (e.g., "I'm tired today")

[0878] Output: Analyzed emotion data (e.g., "tired")

[0879] Specific operations: Perform sentiment analysis of text data using natural language processing technology.

[0880] Step 5:

[0881] The server uses sentence generation software to generate emotional sentences based on the analyzed emotions.

[0882] Input: Emotion data (e.g., "tired")

[0883] Output: Generated sentence (e.g. "Today was tough, but you always try your best, so I'm sure good things will come out of it.")

[0884] Specific action: The sentence generation software generates sentences that correspond to the emotions.

[0885] Step 6:

[0886] The server uses speech synthesis software (such as Google's Text-to-Speech API) to convert the generated text into audio data.

[0887] Input: Text (e.g., "I know you had a hard day today. But you always work hard, so good things will come your way.")

[0888] Output: Audio data

[0889] Specific operation: The process converts text into audio data.

[0890] Step 7:

[0891] The server transmits the generated voice data to the terminal.

[0892] Input: Audio data

[0893] Output: Audio data sent to the device

[0894] Specific operation: Send audio data to the terminal using an HTTP request, etc.

[0895] Step 8:

[0896] The terminal transfers the received voice data to the speech device, which then reproduces the voice data.

[0897] Input: Audio data

[0898] Output: The audio played to the user

[0899] Specific operation: Play audio data through the speaker of a smartphone or head-mounted display.

[0900] In this way, a system is realized that provides emotional feedback in real time based on the user's voice input.

[0901] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0902] The system of the present invention combines the user's voice input with an emotion engine to more precisely recognize the user's emotions and generate emotional utterances based on those emotions. By adding an analysis function using the emotion engine, this system further improves the accuracy of recognizing the user's emotional state, realizing more empathetic and effective communication.

[0903] System Overview

[0904] The system consists of the following main components:

[0905] 1. A device that receives user voice input

[0906] 2. A server containing an emotion engine that analyzes voice data and identifies emotions

[0907] 3. Server that generates sentences based on emotions and converts them into voice data

[0908] 4. A speech device that produces speech using the generated voice data

[0909] Program processing

[0910] The processing flow of a typical program in the system of the present invention is shown below.

[0911] 1. Acquiring voice input

[0912] The user inputs into the terminal, "Today was very hard."

[0913] 2. Converting voice data to text

[0914] The device uses a microphone to capture voice data and uses voice recognition software to convert it into text data ("Today was very hard").

[0915] 3. Sending text data

[0916] The terminal transmits the converted text data to the server.

[0917] 4. Emotion Analysis

[0918] The server first uses an emotion engine to analyze the voice characteristics (pitch, tempo, intonation) of the voice data, and then, if necessary, analyzes non-voice data such as the user's facial expressions and movements, and combines these data to recognize the user's emotions.

[0919] 5. Information Integration

[0920] The server integrates the voice features and text data to more precisely identify the user's emotional state. For example, if the text data indicates "painful," and the voice features also support this, the emotion "very pain" can be identified.

[0921] 6. Generating emotional sentences

[0922] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0923] 7. Text-to-speech

[0924] The server then passes the generated text to speech synthesis software, which generates speech data that expresses the appropriate emotion, taking into account tone and intonation.

[0925] 8. Sending audio data

[0926] The server transmits the generated voice data to the terminal.

[0927] 9. Executing the utterance

[0928] The terminal transfers the voice data received from the server to the speech device, which then speaks, "Today must have been really tough. But your efforts were not in vain."

[0929] Specific examples

[0930] When a user voice-inputs "Today was really hard," the device converts the speech into text data ("Today was really hard") and sends it to the server. The server uses an emotion engine to analyze the speech features and further analyzes the text data to identify the "hard" emotion. Next, it generates a sentence such as "Today was really hard, wasn't it? But your efforts were worth it," and converts it into speech data. Finally, the device sends the speech data to a speech generator, which can provide encouraging speech to the user.

[0931] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide communication that is sensitive to the user's emotions.

[0932] The processing flow will be explained below.

[0933] Step 1:

[0934] The user inputs into the terminal, "Today was very hard."

[0935] Step 2:

[0936] The terminal uses a microphone to capture the user's voice data.

[0937] Step 3:

[0938] The device uses speech recognition software to convert the captured voice data into text, such as "Today was a very difficult day."

[0939] Step 4:

[0940] The terminal transmits the converted text data and speech feature data, which includes information such as pitch, tempo, and intonation, to the server.

[0941] Step 5:

[0942] The server first analyzes the voice feature data using an emotion engine and recognizes the emotion from the user's voice. For example, if the voice feature data shows that the user's voice is low and slow, the emotion "sad" is recognized.

[0943] Step 6:

[0944] The server analyzes the text data ("Today was very difficult") using natural language processing technology and identifies the emotion ("difficult") contained in the text data.

[0945] Step 7:

[0946] The server integrates the analysis results of the voice feature data and the text data to more accurately identify the user's emotional state. For example, if both the voice feature data and the text data indicate "sad" and "distressed," the overall emotion identified will be "very distressed."

[0947] Step 8:

[0948] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[0949] Step 9:

[0950] The server then passes the generated text to speech synthesis software, which generates emotionally appropriate speech data with a tone and intonation that reflects the identified emotion.

[0951] Step 10:

[0952] The server transmits the generated voice data to the terminal.

[0953] Step 11:

[0954] The terminal transfers the voice data received from the server to the speech device.

[0955] Step 12:

[0956] The speech device of the terminal plays back the voice data and speaks encouraging words to the user, such as, "Today must have been really tough, but your efforts will not have been in vain."

[0957] Example 2

[0958] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0959] Conventional speech recognition and emotion analysis systems have faced challenges in accurately recognizing a user's emotions and responding appropriately. In particular, when identifying emotions based solely on speech data, the user's speech content and voice characteristics alone are insufficient, leading to concerns about reduced accuracy. Furthermore, when generating emotion-based responses, it has been difficult to provide the empathetic and appropriate communication users expect.

[0960] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to the server via a cloud service, means for identifying emotions by analyzing the text data and voice features, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a speech device to have it speak. This enables the user's emotions to be recognized with high accuracy, and emotional sentences to be generated based on the recognition, enabling more empathetic and effective communication.

[0961] The "means for converting input voice data into text data" refers to a device or algorithm for converting voice data input by a user into text format data.

[0962] "Means for sending to a server via a cloud service" refers to protocols and software for sending data from a local device to a remote server using cloud computing infrastructure.

[0963] The "means for analyzing text data and audio features to identify emotions" refers to an algorithm or engine that analyzes the content of text data and audio features (e.g., pitch, tempo, intonation) of audio data to identify a user's emotions.

[0964] The "means for generating emotional sentences based on emotions" refers to algorithms and natural language processing techniques for generating empathetic and emotional sentences in response to the identified emotions of a user.

[0965] The "means for converting the generated text into speech data" refers to speech synthesis software or technology for converting the generated text in text format into speech format data.

[0966] The "means for transmitting to a speech generating device and causing speech generation" refers to a protocol or interface for transmitting the generated voice data to a physical speech generating device and using that device to actually reproduce the voice.

[0967] "Audio features" are predetermined features such as pitch, tempo, and intonation contained in audio data, and are information that enables emotion recognition by analyzing these features.

[0968] "Non-voice data" refers to data other than voice data, such as the user's facial expressions, movements, heart rate, etc., which are used complementary to emotion recognition.

[0969] This invention is a system that recognizes emotions with high accuracy based on a user's voice input and provides speech that incorporates the emotions accordingly. This system is composed of a terminal that acquires the user's voice input, a server that analyzes the voice data and identifies the emotion, a server that generates sentences based on the emotion and converts them into voice data, and a speech device that speaks the generated voice data.

[0970] The user inputs voice into the device. For example, the user says, "Today was very hard." This voice is captured by the device's microphone. The device then converts the voice data into text data using voice recognition software (for example, a voice recognition API). The converted text data ("Today was very hard") is sent to the server via a cloud service.

[0971] The server uses an emotion engine (e.g., emotion analysis API) to analyze the voice features (pitch, tempo, intonation) of the voice data. If necessary, it also analyzes non-voice data such as the user's facial expressions and movements to comprehensively recognize emotions. This makes it possible to identify the user's emotions by integrating text data, voice features, and non-voice data. For example, if the text data contains the word "spicy" and the voice features support this, the emotion "very spicy" is identified.

[0972] Next, the server uses a generative AI model (e.g., a natural language generation model) to generate emotional sentences based on the identified emotions. For example, a sentence like, "Today must have been really tough, but your efforts are worth it." In this generation process, prompt sentences like the following are used:

[0973] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0974] The server then passes the generated sentences to speech synthesis software (e.g., a speech synthesis API) to generate emotionally appropriate speech data, including the appropriate tone and intonation, which is then transmitted to the device using a secure protocol (e.g., HTTPS).

[0975] Finally, the device transmits the received voice data to a speech device (e.g., a smart speaker or speaker system), which then speaks, "You've had a really hard day today, but your efforts are worth it." Through this series of processes, the user can experience empathetic and effective communication through the system.

[0976] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide an appropriate emotional response based on the recognition.

[0977] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0978] Step 1:

[0979] The user speaks to the terminal, saying, "Today was a very hard day." This speech input is captured by the terminal's microphone. The captured speech data becomes the input.

[0980] Step 2:

[0981] The device converts the captured voice data into text using speech recognition software (e.g., a speech recognition API), where the voice signal is analyzed through digital signal processing and a corresponding string of characters ("Today was very hard") is output.

[0982] Step 3:

[0983] The device sends the converted text data to a server via a cloud service, using a secure protocol (e.g., HTTPS). The input of this transmission process is the text data, and the output is the completion of the transfer of the text data to the server.

[0984] Step 4:

[0985] The server analyzes emotions using an emotion engine (e.g., emotion analysis API) to analyze text data and voice features. This analysis is performed by extracting voice features (pitch, tempo, intonation) from the voice data and, if necessary, analyzing non-voice data (e.g., the user's facial expressions and movements). The input is text data and voice feature data, and the output is the analyzed user's emotion information.

[0986] Step 5:

[0987] The server integrates the voice features and text data to precisely identify the user's emotion. In this integration process, for example, if the text data indicates "spicy," the emotion "very spicy" can be identified by matching it with the voice features. The input is voice feature data and text data, and the output is the final emotion information.

[0988] Step 6:

[0989] The server uses a generative AI model (e.g., a natural language generation model) based on the identified emotion to generate emotional sentences. Specifically, it generates empathetic sentences that are in tune with the emotion based on the identified emotional information. The input is the emotional information, and the output is the generated sentences (e.g., "Today was really tough, wasn't it? But your hard work won't be in vain.").

[0990] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[0991] Step 7:

[0992] The server passes the generated sentence to speech synthesis software (e.g., a speech synthesis API) to generate speech data that appropriately expresses emotion. The speech synthesis process adds a voice tone and intonation appropriate for the generated sentence. The input is the generated sentence, and the output is speech data.

[0993] Step 8:

[0994] The server sends the generated audio data to the terminal. Again, a secure protocol (e.g., HTTPS) is used. The input is the audio data, and the output is the completion of sending the audio data to the terminal.

[0995] Step 9:

[0996] The device then transfers the received voice data to a speech device (e.g., a smart speaker or speaker system), which ultimately speaks, "Today was really tough, wasn't it? But your efforts were worth it." The input is the voice data sent from the server, and the output is delivered to the user as physical sound.

[0997] (Application example 2)

[0998] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0999] Conventional systems were able to analyze emotions in response to user voice input and generate emotionally charged sentences, but lacked the ability to provide optimal content and actions based on the user's emotional state. As a result, users were unable to receive appropriate content or feedback based on their emotions, making it difficult to provide empathetic and effective communication and entertainment. Furthermore, the lack of a function to recommend content based on emotions made it impossible to provide personalized services.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1001] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to have it speak, and means for recommending personalized content based on the user's emotions. This makes it possible to provide optimal content tailored to the user's emotional state, thereby realizing the provision of more empathetic and personalized services.

[1002] "Voice data" refers to the digitized voice of the user.

[1003] "Text data" is voice data converted into a string of characters.

[1004] "Emotion analysis" is the process of identifying an emotional state from text data and voice data.

[1005] "Sentence generation" means creating sentences based on the results of sentiment analysis.

[1006] "Convert to voice data" means converting the generated text back into voice data.

[1007] A "speech device" is a device that outputs voice data as voice.

[1008] "Personalized content" refers to content that is individually recommended to a user based on their emotional state and preferences.

[1009] "Recommendation" is a function that selects and presents optimal content based on the user's emotional state.

[1010] The system for this application example recognizes emotions based on the user's voice input and provides the function of recommending personalized content. To achieve this, the following hardware and software configuration is required.

[1011] Hardware used

[1012] 1. Audio input device (microphone): A device required to capture the user's voice.

[1013] 2. Speaking device (speaker): A device for speaking the generated voice data to the user.

[1014] 3. Computing device (PC / server): A device for data processing and voice analysis and generation.

[1015] Software used

[1016] 1. Speech Recognition Library: A library for converting user speech into text data.

[1017] 2. Natural Language Processing Library (TextBlob): A library for analyzing emotions from converted text data.

[1018] 3. Speech synthesis software: Software that converts emotionally charged text into voice data.

[1019] 4. Server application: An application for integrating and managing the various libraries and software mentioned above.

[1020] Data processing and calculation

[1021] The server first converts the speech received from the voice input device into text data using a speech recognition library. It then uses the converted text data and speech data to analyze emotions using a natural language processing library. Based on the analyzed emotional information, it recommends personalized content tailored to the user's emotions. The recommended content information is then converted into emotionally charged speech data using text generation and speech synthesis software, and the speech data is then sent to the speech device.

[1022] Specific examples

[1023] If a user says, "I'm very tired today," the server captures this speech and converts it into text data using a speech recognition library. It then analyzes the emotion from the text data using a natural language processing library to identify "tiredness." Based on the analyzed emotion, personalized content such as relaxing music or videos is recommended. Finally, information about the recommended content is generated as text, converted into audio data using speech synthesis software, and provided to the user via a speech device.

[1024] Prompt Sentence Examples

[1025] Identify emotions from the text entered by the user and suggest content that corresponds to those emotions. For example, if a user says, "I've been working all day today and I'm really tired," recommend relaxing music or stress-relieving videos.

[1026] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1027] Step 1:

[1028] The user uses the device's microphone to input voice, such as saying, "I'm very tired today." The device digitizes this voice data (an analog signal) and stores it at a certain sample rate and bit depth.

[1029] Input: User's voice

[1030] Output: Digitized audio data

[1031] Step 2:

[1032] The device uses a speech recognition library (SpeechRecognition) to convert the digitized voice data into text data. Specifically, the speech recognition engine analyzes this digital data and outputs the string "I'm very tired today."

[1033] Input: Digitized audio data

[1034] Output: Text data ("I'm very tired today")

[1035] Step 3:

[1036] The device sends the converted text data to the server over a network connection, either via an HTTP request or WebSocket.

[1037] Input: Text data

[1038] Output: Sends text data to the server

[1039] Step 4:

[1040] The server analyzes the received text data using a natural language processing library (TextBlob) to identify the emotion. The analysis engine calculates the context of the text and the positive / negative level of the words, and identifies the emotion as "tired."

[1041] Input: Text data

[1042] Output: Emotion data ("tired")

[1043] Step 5:

[1044] The server runs a recommendation algorithm to recommend content appropriate for the user based on the identified emotion data. For example, if the user feels "tired," it will suggest relaxing music or videos. The specific content recommended is retrieved from a database.

[1045] Input: Emotion data

[1046] Output: Recommended content data

[1047] Step 6:

[1048] The server generates sentences containing information about recommended content, using a sentence generation engine to create sentences such as "You've had a tough day today. How about listening to some relaxing music?"

[1049] Input: Recommended content data

[1050] Output: Generated sentence data

[1051] Step 7:

[1052] The server then passes the generated text data to speech synthesis software, which converts it into speech, taking into account the emotion and tone of the text and generating speech with the appropriate intonation.

[1053] Input: Generated sentence data

[1054] Output: Audio data

[1055] Step 8:

[1056] The server sends the generated audio data to the device over a network connection, either via HTTP responses or WebSockets.

[1057] Input: Audio data

[1058] Output: Sending audio data from the server to the device

[1059] Step 9:

[1060] The device receives the voice data from the server and plays it back on the speaker to the user. Specifically, the device plays a message such as, "You've had a tough day today. Why don't you listen to some relaxing music?"

[1061] Input: Audio data

[1062] Output: Audio played through speaking device

[1063] Through these steps, users can be recommended content that suits their emotional state and receive empathetic feedback.

[1064] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1065] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1066] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1067] [Fourth embodiment]

[1068] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1069] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1070] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1071] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1072] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1073] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1074] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1075] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1076] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1077] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1078] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1079] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1080] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1081] The system of the present invention converts a user's voice input into an emotionally charged speech in order to provide psychological support to a user who feels lonely or isolated. Specific embodiments of the system are described below.

[1082] System Overview

[1083] This system consists of a terminal that receives user voice input, a server that analyzes the voice data and identifies emotions, and a speech device that converts the generated text into voice data and speaks it.The main technical elements used are voice recognition technology, natural language processing technology, and voice synthesis technology.

[1084] Program processing

[1085] The program in the system of the present invention is processed in the following procedure.

[1086] 1. Acquiring voice input

[1087] The user makes a speech input such as "I'm tired today" into the terminal.

[1088] 2. Converting voice data to text

[1089] The device converts the voice data acquired from the user into text data using voice recognition software, for example, generating text data such as "I'm tired today."

[1090] 3. Sending text data

[1091] The terminal transmits the converted text data to the server.

[1092] 4. Emotion Analysis

[1093] The server then passes the received text data to emotion analysis software, which uses natural language processing techniques to analyze the user's emotions and identify the emotion "tired."

[1094] 5. Generating emotional sentences

[1095] The server uses sentence generation software based on the analyzed emotions to generate emotional sentences, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[1096] 6. Text-to-speech

[1097] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[1098] 7. Transmission to speaking device

[1099] The server transmits the generated voice data to the terminal.

[1100] 8. Executing the utterance

[1101] The terminal transfers the received voice data to a speech device, which then plays the voice data. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so good things will surely happen."

[1102] Specific examples

[1103] When a user voice-inputs "I'm tired today," the device converts the voice into text data ("I'm tired today") and sends it to the server. The server analyzes the emotion from the text and identifies the emotion category "tired." Next, it generates a sentence expressing the emotion ("I see, today was tough. But you always try your best, so good things will surely happen.") and converts this into voice data. Finally, the humanoid robot plays the voice back to the user via the device, providing psychological support.

[1104] The above is an embodiment of the system of the present invention. This system enables communication that is sensitive to the user's emotions, thereby reducing feelings of loneliness and isolation.

[1105] The processing flow will be explained below.

[1106] Step 1:

[1107] The user inputs "I'm tired today" into the terminal by voice.

[1108] Step 2:

[1109] The device captures the user's voice through a microphone.

[1110] Step 3:

[1111] The device uses speech recognition software to convert the captured voice data into text, for example, "I'm tired today."

[1112] Step 4:

[1113] The terminal transmits the converted text data to the server.

[1114] Step 5:

[1115] The server passes the received text data to sentiment analysis software, which uses natural language processing techniques to analyze the user's emotions.

[1116] Step 6:

[1117] The server identifies the user's emotion (for example, "fatigue") from the output of the emotion analysis software.

[1118] Step 7:

[1119] The server uses sentence generation software to generate emotional sentences based on the user's emotions, such as "Today was tough, wasn't it? But you always try your best, so I'm sure good things will come."

[1120] Step 8:

[1121] The server then passes the generated text to speech synthesis software, which converts it into voice data, which is generated in a tone that appropriately expresses emotion.

[1122] Step 9:

[1123] The server transmits the generated voice data to the terminal.

[1124] Step 10:

[1125] The terminal transfers the voice data received from the server to the speech device.

[1126] Step 11:

[1127] The speech device (part of the terminal) plays back the voice data and speaks encouraging words to the user, such as, "You had a hard day today, didn't you? But you always try your best, so I'm sure good things will happen."

[1128] Example 1

[1129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1130] In modern society, the number of people feeling lonely and isolated is increasing. While systems exist to provide psychological support, many of these systems lack sufficient emotional response capabilities and are unable to respond in a way that is in line with the user's actual feelings. Therefore, there is a need for a system that can analyze emotions based on the user's voice input, generate sentences containing appropriate emotional expressions, and respond via voice.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1132] In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to a central processing unit, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a terminal and transferring it to a speech device for speech generation. This makes it possible to provide a voice response that is in line with the user's emotions, thereby reducing feelings of loneliness and isolation.

[1133] "Audio data" means digital or analog data that is an electronic recording of sound.

[1134] "Text data" is digital data that includes character information.

[1135] A "central processing unit" is a device that performs the central processing of a server or computer.

[1136] "Means of emotion analysis" refers to algorithms or software that identify emotions from input text data.

[1137] "Means for generating emotional sentences" refers to algorithms or software for generating sentences that contain appropriate emotional expressions based on the analyzed emotions.

[1138] "Means for converting into audio data" refers to the algorithms or software used to convert the generated text data into audio format.

[1139] A "terminal" refers to a device that is directly operated by a user, and is an electronic device that has the function of inputting and outputting voice.

[1140] A "speech device" is a device or equipment for playing back generated voice data.

[1141] MODE FOR CARRYING OUT THE INVENTION

[1142] The system of the present invention is a dialogue system that combines emotion analysis and speech generation, designed to reduce users' feelings of loneliness and isolation. The system mainly performs a series of processes, including user voice input, speech recognition, emotion analysis, sentence generation, speech synthesis, and speech production.

[1143] Hardware and Software Configuration

[1144] The user inputs voice into the system using a voice input device (microphone or smartphone).

[1145] The device receives the user's voice and converts the voice data into text data using speech recognition software (e.g., a commonly used speech recognition API), such as the Google Speech-to-Text API.

[1146] The terminal sends the generated text data to a server, which is a central processing unit. At this time, the data is sent securely via the HTTPS protocol.

[1147] The server passes the received text data to sentiment analysis software (e.g., algorithms or APIs that use natural language processing technology) to analyze the sentiment. Here, IBM Watson Tone Analyzer is used as an example.

[1148] Based on the results of the sentiment analysis, the server uses sentence generation software (e.g., a generative AI model such as OpenAI's GPT-3) to generate emotional sentences.

[1149] The generated sentences are passed to speech synthesis software (such as Amazon Polly) and converted into speech data, which then adjusts the tone and expression of the voice according to the analyzed emotion.

[1150] The server transmits the generated voice data to the terminal again, and the terminal transfers the voice data to a speech device (for example, a humanoid robot).

[1151] The speech device plays back the received voice data and speaks to the user. For example, a humanoid robot might say, "Today was tough, wasn't it? But you always try your best, so I'm sure good things will happen."

[1152] Specific examples

[1153] Specifically, suppose a user says "I'm tired today" by voice input. In this case, the device converts the voice into text data via the Google Speech-to-Text API. The generated text data "I'm tired today" is sent to the server using HTTPS.

[1154] The server uses IBM Watson Tone Analyzer to analyze the emotion of "tiredness," and then uses OpenAI's GPT-3 to generate a sentence that expresses the emotion: "Today was tough, wasn't it? But you always work hard, so I'm sure good things will happen."

[1155] The sentence is converted into voice data using Amazon Polly and sent from the server to the device, which then forwards the voice data to the humanoid robot, which then speaks the sentence to the user.

[1156] Prompt Sentence Examples

[1157] If you want a generative AI model to analyze emotions based on a user's speech, such as "I'm tired today," and generate an emotional response based on the results, you might use the following prompt:

[1158] Analyze the user's emotions based on the phrase "I'm tired today," and generate a response containing encouraging words based on the emotion "tired."

[1159] The above is a specific embodiment for carrying out the present invention. This system allows users to receive emotionally responsive responses, thereby reducing feelings of loneliness and isolation.

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] The user inputs voice data into the terminal. For example, the user might say, "I'm tired today." This voice data becomes the input.

[1163] Step 2:

[1164] The device converts the voice data acquired from the user into text data using voice recognition software (e.g., Google Speech-to-Text API). Specifically, the device's microphone captures the voice data and sends it to the Google Speech-to-Text API, obtaining text data such as "I'm tired today" as output.

[1165] Step 3:

[1166] The terminal sends the converted text data to the server. Specifically, the terminal sends the generated text data to the server as an HTTPS request, using the TLS protocol to ensure security. The input of this step is the text data "I'm tired today," and the output is the completion of sending the text data to the server.

[1167] Step 4:

[1168] The server passes the received text data to emotion analysis software (for example, IBM Watson Tone Analyzer) to analyze the emotion. The server then sends the text data in JSON format to the emotion analysis engine, and obtains the emotion label "tired" as the output of the analysis result. The input is the text data "I'm tired today," and the output is the emotion label "tired."

[1169] Step 5:

[1170] The server generates emotional sentences using sentence generation software (e.g., OpenAI's GPT-3) based on the analyzed emotions. Specifically, the server supplies the emotion label "tired" and the original text data as prompts to GPT-3, and obtains the sentence "Today was tough, wasn't it? But you always try your best, so good things will surely come." The input is the emotion label and the original text data, and the output is the generated sentence.

[1171] Step 6:

[1172] The server passes the generated sentence to speech synthesis software (e.g., Amazon Polly) and converts it into voice data. The server then sends the generated sentence in text format to the Amazon Polly API, which outputs voice data with tone and speed set according to the emotion. The input is the generated sentence, and the output is voice data.

[1173] Step 7:

[1174] The server sends the generated audio data to the device. Specifically, the server sends the audio data in binary format to the device as an HTTPS request, and the device confirms receipt of the audio data. The input of this step is the audio data, and the output is completion of sending the audio data to the device.

[1175] Step 8:

[1176] The terminal transfers the received voice data to a speech device (e.g., a humanoid robot), which then plays the voice data. Specifically, the terminal transmits the voice data to the speech device via Bluetooth or Wi-Fi, and the speaker of the speech device plays the following voice as output: "Today was tough, wasn't it? But you always try your best, so good things will surely happen." The input is the voice data, and the output is the spoken voice.

[1177] (Application example 1)

[1178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1179] In recent years, the number of users feeling lonely and isolated has been increasing, creating a need for psychological support. However, conventional systems have had difficulty providing appropriate feedback in real time that reflects the user's emotions. In particular, content distribution services lack a means to provide comments and encouragement that reflect the user's emotions, creating a need for improved user experience. The objective of this invention is to provide a system that converts voice input into emotional speech, thereby providing users with psychological support.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1181] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to produce speech, and means for providing appropriate comments and encouragement to users in a content distribution service based on the results of the emotion analysis. This enables communication that is sensitive to the user's emotions, reduces feelings of loneliness and isolation, and improves the content viewing experience.

[1182] "Audio data" is a sound signal expressed in digital or analog format, and is information including the user's voice.

[1183] "Text data" refers to character information expressed in digital form, and includes words obtained by analyzing voice data.

[1184] "Emotions" refer to a person's psychological state and include a variety of psychological states such as joy, sadness, and anger.

[1185] "Means of emotion analysis" refers to technologies and algorithms for identifying a user's emotions from input text data, and may utilize natural language processing technology or machine learning models.

[1186] "Emotional sentences" are sentences that express a context or a specific emotion generated based on the analyzed emotion, and are generated according to the user's psychological state.

[1187] A "speech device" is a device for reproducing and communicating audio data to a user, and includes a speaker or other audio reproduction device.

[1188] "Content distribution service" refers to a service that provides digital content to users, including various media formats such as movies, music, videos, and articles.

[1189] The system for implementing this invention converts a user's voice input into text data, analyzes emotions, generates emotional sentences based on the results, and finally provides feedback to the user as voice data. A specific configuration for implementing this system will be described below.

[1190] System configuration

[1191] The system consists of the following major components:

[1192] 1. Voice input acquisition device

[1193] This device, such as a smartphone or a head-mounted display, captures the user's voice.

[1194] 2. Voice Recognition Software

[1195] Speech recognition software converts captured voice data into text data using voice recognition technologies such as Google's Speech-to-Text API.

[1196] 3. Emotion Analysis Server

[1197] This server analyzes the user's emotions from the received text data, using natural language processing technologies such as Hugging Face's GPT-3.

[1198] 4. Text Generation Software

[1199] The sentence generation software generates emotional sentences based on the analyzed emotions. The generated sentences include concrete examples such as, "I see you had a hard day today. But you always work hard, so good things will surely happen."

[1200] 5. Text-to-Speech Software

[1201] Text-to-speech software converts the generated text into audio data, using voice synthesis technologies such as Google's Text-to-Speech API.

[1202] 6. Speaking Devices

[1203] This device plays back the generated audio data, similar to the audio output function of a smartphone speaker or a head-mounted display.

[1204] Processing method

[1205] A user initiates the system by providing voice input. For example, the user might say, "I'm tired today." The voice input capture device records this speech and converts it into text data using speech recognition software. This text data is then sent to an emotion analysis server, which analyzes the emotion "tired" using natural language processing techniques. Based on the analysis results, sentence generation software generates an appropriate emotional response. Finally, speech synthesis software converts this sentence into voice data, which is played back to the user via a speech device.

[1206] Specific examples

[1207] As a concrete example, the following prompt sentence is used when the user inputs "I'm tired today." "User input: 'I'm tired today'. Generate an encouraging sentence based on this input." This prompt sentence generates appropriate feedback that is in line with the user's emotions.

[1208] In this way, the system can provide communication that is sensitive to the user's emotions, helping to reduce feelings of loneliness and isolation, and improving the user experience in content distribution services.

[1209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1210] Step 1:

[1211] The user performs voice input, for example, saying, "I'm tired today."

[1212] Input: User's voice data

[1213] Output: Audio data sent to the device

[1214] Specific operation: The microphone on the smartphone or head-mounted display picks up the user's voice.

[1215] Step 2:

[1216] The device uses voice recognition software (such as Google's Speech-to-Text API) to convert voice input into text data.

[1217] Input: User's voice data

[1218] Output: Text data generated by the speech recognition software (e.g., "I'm tired today")

[1219] Specific operation: A process is carried out to convert the voice data into text data.

[1220] Step 3:

[1221] The terminal transmits the converted text data to the emotion analysis server.

[1222] Input: Text data

[1223] Output: Text data sent to the sentiment analysis server

[1224] Specific operation: Send text data to the server using an HTTP request, etc.

[1225] Step 4:

[1226] The server uses natural language processing technology (such as GPT-3 in Hugging Face) to analyze emotions from the received text data.

[1227] Input: Text data (e.g., "I'm tired today")

[1228] Output: Analyzed emotion data (e.g., "tired")

[1229] Specific operations: Perform sentiment analysis of text data using natural language processing technology.

[1230] Step 5:

[1231] The server uses sentence generation software to generate emotional sentences based on the analyzed emotions.

[1232] Input: Emotion data (e.g., "tired")

[1233] Output: Generated sentence (e.g. "Today was tough, but you always try your best, so I'm sure good things will come out of it.")

[1234] Specific action: The sentence generation software generates sentences that correspond to the emotions.

[1235] Step 6:

[1236] The server uses speech synthesis software (such as Google's Text-to-Speech API) to convert the generated text into audio data.

[1237] Input: Text (e.g., "I know you had a hard day today. But you always work hard, so good things will come your way.")

[1238] Output: Audio data

[1239] Specific operation: The process converts text into audio data.

[1240] Step 7:

[1241] The server transmits the generated voice data to the terminal.

[1242] Input: Audio data

[1243] Output: Audio data sent to the device

[1244] Specific operation: Send audio data to the terminal using an HTTP request, etc.

[1245] Step 8:

[1246] The terminal transfers the received voice data to the speech device, which then reproduces the voice data.

[1247] Input: Audio data

[1248] Output: The audio played to the user

[1249] Specific operation: Play audio data through the speaker of a smartphone or head-mounted display.

[1250] In this way, a system is realized that provides emotional feedback in real time based on the user's voice input.

[1251] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1252] The system of the present invention combines the user's voice input with an emotion engine to more precisely recognize the user's emotions and generate emotional utterances based on those emotions. By adding an analysis function using the emotion engine, this system further improves the accuracy of recognizing the user's emotional state, realizing more empathetic and effective communication.

[1253] System Overview

[1254] The system consists of the following main components:

[1255] 1. A device that receives user voice input

[1256] 2. A server containing an emotion engine that analyzes voice data and identifies emotions

[1257] 3. Server that generates sentences based on emotions and converts them into voice data

[1258] 4. A speech device that produces speech using the generated voice data

[1259] Program processing

[1260] The processing flow of a typical program in the system of the present invention is shown below.

[1261] 1. Acquiring voice input

[1262] The user inputs into the terminal, "Today was very hard."

[1263] 2. Converting voice data to text

[1264] The device uses a microphone to capture voice data and uses voice recognition software to convert it into text data ("Today was very hard").

[1265] 3. Sending text data

[1266] The terminal transmits the converted text data to the server.

[1267] 4. Emotion Analysis

[1268] The server first uses an emotion engine to analyze the voice characteristics (pitch, tempo, intonation) of the voice data, and then, if necessary, analyzes non-voice data such as the user's facial expressions and movements, and combines these data to recognize the user's emotions.

[1269] 5. Information Integration

[1270] The server integrates the voice features and text data to more precisely identify the user's emotional state. For example, if the text data indicates "painful," and the voice features also support this, the emotion "very pain" can be identified.

[1271] 6. Generating emotional sentences

[1272] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[1273] 7. Text-to-speech

[1274] The server then passes the generated text to speech synthesis software, which generates speech data that expresses the appropriate emotion, taking into account tone and intonation.

[1275] 8. Sending audio data

[1276] The server transmits the generated voice data to the terminal.

[1277] 9. Executing the utterance

[1278] The terminal transfers the voice data received from the server to the speech device, which then speaks, "Today must have been really tough. But your efforts were not in vain."

[1279] Specific examples

[1280] When a user voice-inputs "Today was really hard," the device converts the speech into text data ("Today was really hard") and sends it to the server. The server uses an emotion engine to analyze the speech features and further analyzes the text data to identify the "hard" emotion. Next, it generates a sentence such as "Today was really hard, wasn't it? But your efforts were worth it," and converts it into speech data. Finally, the device sends the speech data to a speech generator, which can provide encouraging speech to the user.

[1281] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide communication that is sensitive to the user's emotions.

[1282] The processing flow will be explained below.

[1283] Step 1:

[1284] The user inputs into the terminal, "Today was very hard."

[1285] Step 2:

[1286] The terminal uses a microphone to capture the user's voice data.

[1287] Step 3:

[1288] The device uses speech recognition software to convert the captured voice data into text, such as "Today was a very difficult day."

[1289] Step 4:

[1290] The terminal transmits the converted text data and speech feature data, which includes information such as pitch, tempo, and intonation, to the server.

[1291] Step 5:

[1292] The server first analyzes the voice feature data using an emotion engine and recognizes the emotion from the user's voice. For example, if the voice feature data shows that the user's voice is low and slow, the emotion "sad" is recognized.

[1293] Step 6:

[1294] The server analyzes the text data ("Today was very difficult") using natural language processing technology and identifies the emotion ("difficult") contained in the text data.

[1295] Step 7:

[1296] The server integrates the analysis results of the voice feature data and the text data to more accurately identify the user's emotional state. For example, if both the voice feature data and the text data indicate "sad" and "distressed," the overall emotion identified will be "very distressed."

[1297] Step 8:

[1298] The server uses sentence generation software to generate emotional sentences based on the identified emotions, such as "You had a really hard day today, but your efforts were worth it."

[1299] Step 9:

[1300] The server then passes the generated text to speech synthesis software, which generates emotionally appropriate speech data with a tone and intonation that reflects the identified emotion.

[1301] Step 10:

[1302] The server transmits the generated voice data to the terminal.

[1303] Step 11:

[1304] The terminal transfers the voice data received from the server to the speech device.

[1305] Step 12:

[1306] The speech device of the terminal plays back the voice data and speaks encouraging words to the user, such as, "Today must have been really tough, but your efforts will not have been in vain."

[1307] Example 2

[1308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] Conventional speech recognition and emotion analysis systems have faced challenges in accurately recognizing a user's emotions and responding appropriately. In particular, when identifying emotions based solely on speech data, the user's speech content and voice characteristics alone are insufficient, leading to concerns about reduced accuracy. Furthermore, when generating emotion-based responses, it has been difficult to provide the empathetic and appropriate communication users expect.

[1310] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for converting input voice data into text data, means for transmitting the text data to the server via a cloud service, means for identifying emotions by analyzing the text data and voice features, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, and means for transmitting the voice data to a speech device to have it speak. This enables the user's emotions to be recognized with high accuracy, and emotional sentences to be generated based on the recognition, enabling more empathetic and effective communication.

[1311] The "means for converting input voice data into text data" refers to a device or algorithm for converting voice data input by a user into text format data.

[1312] "Means for sending to a server via a cloud service" refers to protocols and software for sending data from a local device to a remote server using cloud computing infrastructure.

[1313] The "means for analyzing text data and audio features to identify emotions" refers to an algorithm or engine that analyzes the content of text data and audio features (e.g., pitch, tempo, intonation) of audio data to identify a user's emotions.

[1314] The "means for generating emotional sentences based on emotions" refers to algorithms and natural language processing techniques for generating empathetic and emotional sentences in response to the identified emotions of a user.

[1315] The "means for converting the generated text into speech data" refers to speech synthesis software or technology for converting the generated text in text format into speech format data.

[1316] The "means for transmitting to a speech generating device and causing speech generation" refers to a protocol or interface for transmitting the generated voice data to a physical speech generating device and using that device to actually reproduce the voice.

[1317] "Audio features" are predetermined features such as pitch, tempo, and intonation contained in audio data, and are information that enables emotion recognition by analyzing these features.

[1318] "Non-voice data" refers to data other than voice data, such as the user's facial expressions, movements, heart rate, etc., which are used complementary to emotion recognition.

[1319] This invention is a system that recognizes emotions with high accuracy based on a user's voice input and provides speech that incorporates the emotions accordingly. This system is composed of a terminal that acquires the user's voice input, a server that analyzes the voice data and identifies the emotion, a server that generates sentences based on the emotion and converts them into voice data, and a speech device that speaks the generated voice data.

[1320] The user inputs voice into the device. For example, the user says, "Today was very hard." This voice is captured by the device's microphone. The device then converts the voice data into text data using voice recognition software (for example, a voice recognition API). The converted text data ("Today was very hard") is sent to the server via a cloud service.

[1321] The server uses an emotion engine (e.g., emotion analysis API) to analyze the voice features (pitch, tempo, intonation) of the voice data. If necessary, it also analyzes non-voice data such as the user's facial expressions and movements to comprehensively recognize emotions. This makes it possible to identify the user's emotions by integrating text data, voice features, and non-voice data. For example, if the text data contains the word "spicy" and the voice features support this, the emotion "very spicy" is identified.

[1322] Next, the server uses a generative AI model (e.g., a natural language generation model) to generate emotional sentences based on the identified emotions. For example, a sentence like, "Today must have been really tough, but your efforts are worth it." In this generation process, prompt sentences like the following are used:

[1323] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[1324] The server then passes the generated sentences to speech synthesis software (e.g., a speech synthesis API) to generate emotionally appropriate speech data, including the appropriate tone and intonation, which is then transmitted to the device using a secure protocol (e.g., HTTPS).

[1325] Finally, the device transmits the received voice data to a speech device (e.g., a smart speaker or speaker system), which then speaks, "You've had a really hard day today, but your efforts are worth it." Through this series of processes, the user can experience empathetic and effective communication through the system.

[1326] The above is an embodiment of the system of the present invention. This system makes it possible to recognize the user's emotions with high accuracy and provide an appropriate emotional response based on the recognition.

[1327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1328] Step 1:

[1329] The user speaks to the terminal, saying, "Today was a very hard day." This speech input is captured by the terminal's microphone. The captured speech data becomes the input.

[1330] Step 2:

[1331] The device converts the captured voice data into text using speech recognition software (e.g., a speech recognition API), where the voice signal is analyzed through digital signal processing and a corresponding string of characters ("Today was very hard") is output.

[1332] Step 3:

[1333] The device sends the converted text data to a server via a cloud service, using a secure protocol (e.g., HTTPS). The input of this transmission process is the text data, and the output is the completion of the transfer of the text data to the server.

[1334] Step 4:

[1335] The server analyzes emotions using an emotion engine (e.g., emotion analysis API) to analyze text data and voice features. This analysis is performed by extracting voice features (pitch, tempo, intonation) from the voice data and, if necessary, analyzing non-voice data (e.g., the user's facial expressions and movements). The input is text data and voice feature data, and the output is the analyzed user's emotion information.

[1336] Step 5:

[1337] The server integrates the voice features and text data to precisely identify the user's emotion. In this integration process, for example, if the text data indicates "spicy," the emotion "very spicy" can be identified by matching it with the voice features. The input is voice feature data and text data, and the output is the final emotion information.

[1338] Step 6:

[1339] The server uses a generative AI model (e.g., a natural language generation model) based on the identified emotion to generate emotional sentences. Specifically, it generates empathetic sentences that are in tune with the emotion based on the identified emotional information. The input is the emotional information, and the output is the generated sentences (e.g., "Today was really tough, wasn't it? But your hard work won't be in vain.").

[1340] Prompt: "The user says, 'I had a really hard day today.' Generate a sentence that is comforting and encouraging to the user."

[1341] Step 7:

[1342] The server passes the generated sentence to speech synthesis software (e.g., a speech synthesis API) to generate speech data that appropriately expresses emotion. The speech synthesis process adds a voice tone and intonation appropriate for the generated sentence. The input is the generated sentence, and the output is speech data.

[1343] Step 8:

[1344] The server sends the generated audio data to the terminal. Again, a secure protocol (e.g., HTTPS) is used. The input is the audio data, and the output is the completion of sending the audio data to the terminal.

[1345] Step 9:

[1346] The device then transfers the received voice data to a speech device (e.g., a smart speaker or speaker system), which ultimately speaks, "Today was really tough, wasn't it? But your efforts were worth it." The input is the voice data sent from the server, and the output is delivered to the user as physical sound.

[1347] (Application example 2)

[1348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1349] Conventional systems were able to analyze emotions in response to user voice input and generate emotionally charged sentences, but lacked the ability to provide optimal content and actions based on the user's emotional state. As a result, users were unable to receive appropriate content or feedback based on their emotions, making it difficult to provide empathetic and effective communication and entertainment. Furthermore, the lack of a function to recommend content based on emotions made it impossible to provide personalized services.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1351] In this invention, the server includes means for converting input voice data into text data, means for analyzing emotions from the text data, means for generating emotional sentences based on the emotions, means for converting the generated sentences into voice data, means for transmitting the voice data to a speech device to have it speak, and means for recommending personalized content based on the user's emotions. This makes it possible to provide optimal content tailored to the user's emotional state, thereby realizing the provision of more empathetic and personalized services.

[1352] "Voice data" refers to the digitized voice of the user.

[1353] "Text data" is voice data converted into a string of characters.

[1354] "Emotion analysis" is the process of identifying an emotional state from text data and voice data.

[1355] "Sentence generation" means creating sentences based on the results of sentiment analysis.

[1356] "Convert to voice data" means converting the generated text back into voice data.

[1357] A "speech device" is a device that outputs voice data as voice.

[1358] "Personalized content" refers to content that is individually recommended to a user based on their emotional state and preferences.

[1359] "Recommendation" is a function that selects and presents optimal content based on the user's emotional state.

[1360] The system for this application example recognizes emotions based on the user's voice input and provides the function of recommending personalized content. To achieve this, the following hardware and software configuration is required.

[1361] Hardware used

[1362] 1. Audio input device (microphone): A device required to capture the user's voice.

[1363] 2. Speaking device (speaker): A device for speaking the generated voice data to the user.

[1364] 3. Computing device (PC / server): A device for data processing and voice analysis and generation.

[1365] Software used

[1366] 1. Speech Recognition Library: A library for converting user speech into text data.

[1367] 2. Natural Language Processing Library (TextBlob): A library for analyzing emotions from converted text data.

[1368] 3. Speech synthesis software: Software that converts emotionally charged text into voice data.

[1369] 4. Server application: An application for integrating and managing the various libraries and software mentioned above.

[1370] Data processing and calculation

[1371] The server first converts the speech received from the voice input device into text data using a speech recognition library. It then uses the converted text data and speech data to analyze emotions using a natural language processing library. Based on the analyzed emotional information, it recommends personalized content tailored to the user's emotions. The recommended content information is then converted into emotionally charged speech data using text generation and speech synthesis software, and the speech data is then sent to the speech device.

[1372] Specific examples

[1373] If a user says, "I'm very tired today," the server captures this speech and converts it into text data using a speech recognition library. It then analyzes the emotion from the text data using a natural language processing library to identify "tiredness." Based on the analyzed emotion, personalized content such as relaxing music or videos is recommended. Finally, information about the recommended content is generated as text, converted into audio data using speech synthesis software, and provided to the user via a speech device.

[1374] Prompt Sentence Examples

[1375] Identify emotions from the text entered by the user and suggest content that corresponds to those emotions. For example, if a user says, "I've been working all day today and I'm really tired," recommend relaxing music or stress-relieving videos.

[1376] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1377] Step 1:

[1378] The user uses the device's microphone to input voice, such as saying, "I'm very tired today." The device digitizes this voice data (an analog signal) and stores it at a certain sample rate and bit depth.

[1379] Input: User's voice

[1380] Output: Digitized audio data

[1381] Step 2:

[1382] The device uses a speech recognition library (SpeechRecognition) to convert the digitized voice data into text data. Specifically, the speech recognition engine analyzes this digital data and outputs the string "I'm very tired today."

[1383] Input: Digitized audio data

[1384] Output: Text data ("I'm very tired today")

[1385] Step 3:

[1386] The device sends the converted text data to the server over a network connection, either via an HTTP request or WebSocket.

[1387] Input: Text data

[1388] Output: Sends text data to the server

[1389] Step 4:

[1390] The server analyzes the received text data using a natural language processing library (TextBlob) to identify the emotion. The analysis engine calculates the context of the text and the positive / negative level of the words, and identifies the emotion as "tired."

[1391] Input: Text data

[1392] Output: Emotion data ("tired")

[1393] Step 5:

[1394] The server runs a recommendation algorithm to recommend content appropriate for the user based on the identified emotion data. For example, if the user feels "tired," it will suggest relaxing music or videos. The specific content recommended is retrieved from a database.

[1395] Input: Emotion data

[1396] Output: Recommended content data

[1397] Step 6:

[1398] The server generates sentences containing information about recommended content, using a sentence generation engine to create sentences such as "You've had a tough day today. How about listening to some relaxing music?"

[1399] Input: Recommended content data

[1400] Output: Generated sentence data

[1401] Step 7:

[1402] The server then passes the generated text data to speech synthesis software, which converts it into speech, taking into account the emotion and tone of the text and generating speech with the appropriate intonation.

[1403] Input: Generated sentence data

[1404] Output: Audio data

[1405] Step 8:

[1406] The server sends the generated audio data to the device over a network connection, either via HTTP responses or WebSockets.

[1407] Input: Audio data

[1408] Output: Sending audio data from the server to the device

[1409] Step 9:

[1410] The device receives the voice data from the server and plays it back on the speaker to the user. Specifically, the device plays a message such as, "You've had a tough day today. Why don't you listen to some relaxing music?"

[1411] Input: Audio data

[1412] Output: Audio played through speaking device

[1413] Through these steps, users can be recommended content that suits their emotional state and receive empathetic feedback.

[1414] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1415] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1416] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1417] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1418] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1419] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1420] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1421] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1422] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1423] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1424] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1425] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1426] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1427] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1428] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1429] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1430] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1431] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1432] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1433] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1434] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1435] The following is further disclosed regarding the above embodiment.

[1436] (Claim 1)

[1437] means for converting input voice data into text data;

[1438] means for analyzing emotions from the text data;

[1439] means for generating emotional sentences based on the emotions;

[1440] means for converting the generated sentence into voice data;

[1441] means for transmitting the voice data to a speech device and causing the speech device to speak;

[1442] A system including:

[1443] (Claim 2)

[1444] 2. The system of claim 1, wherein the input voice data is a voice input of a user.

[1445] (Claim 3)

[1446] 2. The system according to claim 1, wherein the emotion analyzing means uses natural language processing technology.

[1447] "Example 1"

[1448] (Claim 1)

[1449] means for converting input voice data into text data;

[1450] means for transmitting the text data to a central processing unit;

[1451] means for analyzing emotions from the text data;

[1452] means for generating emotional sentences based on the emotions;

[1453] means for converting the generated sentence into voice data;

[1454] means for transmitting the voice data to a terminal and transferring the voice data to a speech device for speech generation;

[1455] A system including:

[1456] (Claim 2)

[1457] 2. The system of claim 1, wherein the input voice data is a voice input of a user.

[1458] (Claim 3)

[1459] 2. The system according to claim 1, wherein the emotion analyzing means uses natural language processing technology.

[1460] "Application Example 1"

[1461] (Claim 1)

[1462] means for converting input voice data into text data;

[1463] means for analyzing emotions from the text data;

[1464] means for generating emotional sentences based on the emotions;

[1465] means for converting the generated sentence into voice data;

[1466] means for transmitting the voice data to a speech device and causing the speech device to speak;

[1467] The system includes a means for providing appropriate comments and encouragement to users in a content distribution service based on the results of analyzing the emotions.

[1468] (Claim 2)

[1469] 2. The system of claim 1, wherein the input voice data is a voice input of a user.

[1470] (Claim 3)

[1471] 2. The system according to claim 1, wherein the emotion analyzing means uses natural language processing technology.

[1472] "Example 2: Combining Emotion Engines"

[1473] (Claim 1)

[1474] means for converting input voice data into text data;

[1475] means for transmitting the text data to a server via a cloud service;

[1476] means for analyzing the text data and speech features to identify emotions;

[1477] means for generating emotional sentences based on the emotions;

[1478] means for converting the generated sentence into voice data;

[1479] means for transmitting the voice data to a speech device and causing the speech device to speak;

[1480] A system including:

[1481] (Claim 2)

[1482] 2. The system of claim 1, wherein the input voice data is a voice input of a user.

[1483] (Claim 3)

[1484] 10. The system of claim 1, wherein the emotion analyzing means integrates voice features and non-voice data to identify emotions.

[1485] "Application example 2 when combining emotion engines"

[1486] (Claim 1)

[1487] means for converting input voice data into text data;

[1488] means for analyzing emotions from the text data;

[1489] means for generating emotional sentences based on the emotions;

[1490] means for converting the generated sentence into voice data;

[1491] means for transmitting the voice data to a speech device and causing the speech device to speak;

[1492] A means for recommending personalized content based on user sentiment;

[1493] A system including:

[1494] (Claim 2)

[1495] 2. The system of claim 1, wherein the input voice data is a voice input of a user.

[1496] (Claim 3)

[1497] 2. The system according to claim 1, wherein the emotion analyzing means uses natural language processing technology. [Explanation of symbols]

[1498] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for converting input voice data into text data; means for analyzing emotions from the text data; means for generating emotional sentences based on the emotions; means for converting the generated sentence into voice data; means for transmitting the voice data to a speech device and causing the speech device to speak; A system including:

2. 2. The system of claim 1, wherein the input voice data is a user's voice input.

3. 2. The system of claim 1, wherein the emotion analysis means uses natural language processing techniques.

Images