system

Abstract

Provide a system. means for receiving a user's voice input; means for converting the received voice input into text data using a voice recognition engine; means for transmitting the converted text data to a server; A means for generating a response text using generative artificial intelligence on the server side; means for receiving the generated response text from the server; A means for converting the received response text into voice data using a voice synthesis engine; means for outputting the generated voice data to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to cope with the increasing number of dementia patients in an aging society, reducing the burden on caregivers is an urgent issue. In particular, the behavior of dementia patients repeatedly repeating or asking the same questions creates high stress for caregivers. There is a need to solve this problem and reduce the mental and physical burden on caregivers. [Means for solving the problem]

[0005] This invention provides a system including: means for receiving a user's voice input; means for converting the received voice input into text data using a voice recognition engine; means for transmitting the converted text data to a server; means for generating a response text using generative artificial intelligence on the server side; means for receiving the generated response text from the server; means for converting the received response text into voice data using a voice synthesis engine; and means for outputting the generated voice data to the user. This system reduces the burden on caregivers by enabling the AI ​​speaker to continue to respond appropriately even when a dementia patient repeatedly asks the same question. This system can also generate more personalized responses based on the user's past speech history, thereby improving the quality of care. Furthermore, the voice recognition engine and voice synthesis engine operate on a cloud-based system, enabling improved processing power and flexible scalability.

[0006] "User" refers to a dementia patient or their caregiver who uses the system.

[0007] "Voice input" refers to the voice signals spoken by a user to the system.

[0008] "Speech recognition engine" refers to software or hardware for analyzing voice input and converting it into text data.

[0009] "Text data" refers to character string information that represents the content of a user's speech converted by a voice recognition engine.

[0010] "Server" refers to a computer system that runs on a cloud or network and is primarily responsible for generating response text.

[0011] "Generative artificial intelligence" refers to algorithms or software for generating appropriate response text based on text data submitted by a user.

[0012] "Response text" refers to character string information generated by generative artificial intelligence to respond to the user.

[0013] A "speech synthesis engine" refers to software or hardware for analyzing text data and converting it into voice data.

[0014] "Speech data" refers to the speech signals generated by a speech synthesis engine and spoken to a user.

[0015] "Output means" refers to a speaker or similar device for communicating generated audio data to a user. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] ---

[0038] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0039] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0040] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0041] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., a generative AI model). For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go outside for a walk."

[0042] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back from the speaker.

[0043] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and tells the user, "I had delicious curry yesterday."

[0044] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0045] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0049] Step 2:

[0050] The terminal receives the user's voice input through a microphone.

[0051] Step 3:

[0052] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0053] Step 4:

[0054] The terminal sends the converted text data to the server as an HTTP POST request.

[0055] Step 5:

[0056] The server analyzes the received text data, "It's nice weather today," and uses generative artificial intelligence to generate the response text, "It's really nice weather! Maybe it would be nice to go for a walk outside."

[0057] Step 6:

[0058] The server generates a response text and sends it back to the terminal as an HTTP response.

[0059] Step 7:

[0060] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! It might be nice to go for a walk outside."

[0061] Step 8:

[0062] The terminal makes the generated voice data available to the user through a speaker.

[0063] Example 1

[0064] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0065] The problem that this invention aims to solve is to reduce the burden on caregivers for dementia patients and to improve the effectiveness of dialogue between patients and AI speakers. Another problem is to achieve highly accurate speech recognition and natural voice response generation to improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, we aim to provide a system that can be adapted to a wide range of users.

[0066] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0067] In this invention, the server includes means for receiving the user's voice using a highly sensitive noise-canceling microphone, means for the speech recognition engine to always use the latest algorithms and be updated to improve the accuracy of proper noun recognition, means for the generative artificial intelligence to refer to the user's past speech history and generate personalized responses, and means for the speech synthesis engine to support multiple languages ​​and generate responses in different languages ​​and dialects with natural pronunciation. This enables highly accurate speech recognition and natural voice responses, and the personalized responses improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, a system that can be adapted to a wide range of users is realized.

[0068] "User" refers to a person who uses the system.

[0069] "Voice input" refers to speech uttered by a user.

[0070] "Means for receiving" refers to a combination of hardware and software for capturing audio input.

[0071] "Speech recognition engine" refers to software or a service for converting voice data into text data.

[0072] "Text data" refers to character string data converted by a voice recognition engine.

[0073] "Server" refers to a remote computing device for analyzing text data generated from speech input and generating appropriate responses.

[0074] "Generative AI" refers to an AI model that generates appropriate responses based on text data.

[0075] "Response text" refers to the string data of a response generated by generative artificial intelligence.

[0076] "Speech synthesis engine" refers to software or a service for converting text data into speech data.

[0077] "Voice data" refers to sound data generated by a speech synthesis engine.

[0078] "Output means" refers to a combination of hardware and software, such as a speaker, that allows the user to hear the audio data.

[0079] A "noise-canceling microphone" refers to a microphone equipped with technology that eliminates ambient noise.

[0080] "Proper noun recognition accuracy" refers to the accuracy in recognizing proper nouns such as specific people's names, place names, and brand names.

[0081] "Means for being updated" refers to the update mechanism for keeping the software and algorithms used up to date.

[0082] "Utterance history" refers to a record of the contents of utterances made by a user in the past.

[0083] "Personalized responses" refer to customized responses generated based on the characteristics and history of individual users.

[0084] "Multilingual" refers to the ability to support multiple languages.

[0085] A "dialect" refers to the use of words and pronunciation that is unique to a particular region or community.

[0086] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0087] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the device, which is equipped with a highly sensitive noise-canceling microphone. This microphone filters out ambient noise and captures accurate voice data.

[0088] The device then sends the received voice input to a cloud-based speech recognition engine (e.g., Google® Cloud Speech-to-Text) and converts it into text data. This speech recognition engine always uses the latest algorithms and is regularly updated to improve recognition accuracy, especially for proper nouns. For example, a user's utterance, "It's a nice day today," is converted into text data, "It's a nice day today."

[0089] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using a generative artificial intelligence (e.g., a model from OpenAI (registered trademark)). This generative artificial intelligence references the user's past speech history to generate a personalized response. For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe we should go for a walk outside."

[0090] The generated response text is then sent back to the device. The device then sends the received response text to a cloud-based speech synthesis engine (e.g., Amazon Polly), which converts it into voice data. This speech synthesis engine supports multiple languages ​​and can generate natural-sounding responses in different languages ​​and dialects. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back through the speaker.

[0091] As a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data saying, "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response saying, "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and speaks to the user, saying, "I had delicious curry yesterday."

[0092] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0093] An example of a prompt sentence that can be input to a generative AI model is "Tell me about what you had for dinner last night."

[0094] As described above, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1:

[0097] The user speaks to the AI ​​speaker, for example, saying, "It's nice weather today."

[0098] Input: User's spoken utterance "It's a nice day today."

[0099] Output: Captured as audio data

[0100] Step 2:

[0101] The device receives the user's voice using a highly sensitive noise-canceling microphone, which filters out ambient noise and captures accurate voice data.

[0102] Input: User's voice data

[0103] Output: Clear audio data after noise removal

[0104] Step 3:

[0105] The device sends the received voice data to a cloud-based speech recognition engine (e.g., a speech recognition API) that converts the speech into text data. This speech recognition engine uses the latest algorithms to improve the accuracy of proper noun recognition.

[0106] Input: Clear audio data after noise removal

[0107] Data processing: Converting voice data into text data

[0108] Output: Text data "It's a nice day today."

[0109] Step 4:

[0110] The terminal transmits the converted text data to the server.

[0111] Input: Text data "It's a nice day today"

[0112] Output: Send text data to the server

[0113] Step 5:

[0114] The server analyzes the received text data and generates a response text using generative artificial intelligence (e.g., a generative AI model). It references past speech history to generate a personalized response.

[0115] Input: Text data "It's a nice day today"

[0116] Data calculations: term analysis, context understanding, and reference to past speech history

[0117] Output: Response text "What lovely weather! Maybe we should go for a walk outside."

[0118] Step 6:

[0119] The server returns the generated response text to the terminal.

[0120] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0121] Output: Response text sent to terminal

[0122] Step 7:

[0123] The device sends the received response text to a cloud-based speech synthesis engine (e.g., speech synthesis API) and converts it into voice data. This speech synthesis engine supports multiple languages ​​and generates voice with natural pronunciation.

[0124] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0125] Data processing: Convert text data into audio data

[0126] Output: Audio data

[0127] Step 8:

[0128] The terminal reproduces the generated voice data from a speaker and responds to the user.

[0129] Input: Audio data

[0130] Output: Voice response "What lovely weather! Maybe we should go for a walk outside."

[0131] The above is a detailed description of the program processing for this system. By clearly indicating the specific operations performed at each step, as well as the inputs and outputs, the processing flow of the entire system becomes clearer.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] In recent years, with the progress of aging society, caring for dementia patients has become a social issue. In particular, providing psychological support to dementia patients through daily conversations places a heavy burden on caregivers. Therefore, there is a need for effective conversation support systems to help dementia patients live their daily lives independently. Furthermore, there is a need for technology that can provide personalized content based on the individual preferences and past speech history of each patient.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for converting a user's speech into text data using a speech recognition engine, means for transmitting the converted text data to a network server, means for generating a response text using generative artificial intelligence on the network server side, means for converting the generated response text into voice data using a speech synthesis engine, and means for providing personalized content such as audiobooks, music, podcasts, and news based on the user's speech history and preference history. This enables personalized dialogue and content provision for dementia patients, thereby supporting the patients' daily lives while reducing the burden on caregivers.

[0137] "User's voice input" refers to a voice message uttered by a dementia patient.

[0138] "Speech recognition engine" refers to a program or device for receiving voice data and converting it into character string data.

[0139] "Text data" refers to character string data converted by a voice recognition engine.

[0140] "Network server" refers to a central computing unit that processes data and generates responses via the cloud or the internet.

[0141] "Generative artificial intelligence" refers to an AI engine that uses pre-trained models to generate natural-sounding response text based on user input data.

[0142] "Response text" is character string data generated by the generative artificial intelligence, and includes a response to the user.

[0143] "Speech synthesis engine" refers to a program or device for converting text data into speech data.

[0144] "Audio Data" refers to audio files that are generated by a speech synthesis engine and can be played through a speaker.

[0145] "Speech history" refers to a record of voice inputs made by the user in the past.

[0146] "Preference history" refers to a past record of a user's preferences and hobbies.

[0147] "Audiobook" refers to digital content in which the contents of a book are presented in audio form.

[0148] "Music" refers to audio data recorded from musical instruments or singing.

[0149] A "podcast" is an audio program distributed over the Internet.

[0150] "News" refers to audio or text data that conveys current events or information.

[0151] "Personalization" refers to changing content based on an individual user's specific preferences and history.

[0152] The present invention is a personalized content delivery system for dementia patients, which aims to reduce the burden on caregivers and provide patients with a sense of security. The system executes a series of processes: receiving voice input from a user, converting it into text data, generating a response, and providing it to the user again as voice.

[0153] Specifically, the main components of the system include:

[0154] 1. Receiving audio input:

[0155] The user (a dementia patient) speaks to the system, and the voice input is received by a microphone installed in the device, which can be a smartphone, smart glasses, or a virtual reality head-mounted display.

[0156] 2. Speech Recognition:

[0157] The received voice input is converted into text data by a speech recognition engine, which can be a cloud-based service such as Google Speech Recognition API.

[0158] 3. Sending text data:

[0159] The converted text data is sent over the network to a server, which is a cloud server connected to the Internet.

[0160] 4. Generate response:

[0161] The server analyzes the received text data and generates an appropriate response text using a generative AI model (e.g., GPT-3 (registered trademark)). This also takes into account the user's past speech history and preference history. For example, in response to the input text "Play me some old hit songs," the server generates a response such as "Playing hit songs from the 1960s."

[0162] 5. Sending a response text:

[0163] The generated response text is sent back to the terminal.

[0164] 6. Speech synthesis:

[0165] The terminal sends the received response text to a speech synthesis engine (such as Pyttsx3) and converts it into voice data.

[0166] 7. Audio Output:

[0167] The generated voice data is output to the user through the device's speaker. For example, a voice saying "Playing a hit song from the 1960s" is played.

[0168] Examples and prompts

[0169] As a concrete example, consider the case where a user says, "Play me an old hit song." When the user says, "Play me an old hit song," the speech recognition engine converts this into text data. The converted text data is then sent to the server, where a generative AI model generates a response such as, "Playing a hit song from the 1960s." The device receives this response, converts it into audio data using a speech synthesis engine, and plays it back.

[0170] An example prompt to use with a generative AI model might look like this:

[0171] User Input: Play me some old hit songs

[0172] Generate an appropriate response.

[0173] The system provides personalized content based on the user's speech history and preferences to support the daily lives of dementia patients. It also uses a cloud-based speech recognition and speech synthesis engine to improve processing power and provide flexible scalability.

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1:

[0176] The user speaks a voice input. The user's voice message is received through the device's microphone. This input is processed as raw voice data.

[0177] Step 2:

[0178] The device sends the received voice data to a voice recognition engine. Specifically, this voice data is sent to a cloud-based service such as Google Speech Recognition API, where it is converted into text data. The input here is voice data, and the output is text data. The voice recognition engine analyzes the voice data and recognizes its content as a string of characters.

[0179] Step 3:

[0180] The terminal sends the converted text data to the server via the network. At this stage, the input is text data and the output is string data sent to the server. The terminal makes the appropriate API call to send the text data.

[0181] Step 4:

[0182] The server passes the received text data to a generative AI model. The input is text data, and the output is the generated response text. The server uses a pre-trained generative AI model (e.g., GPT-3) to generate an appropriate response to the spoken content. For example, in response to the input "Play me an old hit song," a response such as "I'll play a hit song from the 1960s" is generated.

[0183] Step 5:

[0184] The server returns the generated response text to the terminal. The input here is the generated response text, and the output is the response text sent to the terminal. The server returns the generated response in real time.

[0185] Step 6:

[0186] The device sends the received response text to a speech synthesis engine. The input here is the response text, and the output is voice data. Specifically, a speech synthesis library such as Pyttsx3 is used to convert the text data into natural-sounding voice.

[0187] Step 7:

[0188] The device outputs the generated voice data to the user through a speaker. At this stage, the input is voice data from the speech synthesis engine, and the output is a voice message that the user can hear. For example, "Playing a hit song from the 1960s."

[0189] Through these steps, appropriate responses and content are provided in real time based on the user's speech.This system can provide personalized information support and psychological comfort to dementia patients.

[0190] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0191] This invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between patients and AI speakers. The system incorporates an emotion engine that recognizes the user's emotions, providing more personalized responses.

[0192] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0193] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0194] The device then sends this text data to the emotion engine to recognize the user's emotion. The emotion engine can detect emotions not only from text data but also from acoustic features such as the user's voice tone and speed. For example, if the user speaks excitedly, the emotion engine will recognize the user as being in an "excited" state.

[0195] The device then sends the converted text data and the recognized emotion data to the server. The server analyzes this data and generates an appropriate response text using a generative artificial intelligence (generative AI model). For example, if the input text is "What a nice day today" and the emotion data is "Excited," the server generates the response "What a nice day! Let's go for a walk outside!"

[0196] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Let's go for a walk outside!" is converted into natural-sounding speech and played back from the speaker.

[0197] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." Next, an emotion engine recognizes the user's emotion from their tone and speed of speech and determines it as "confused." The device then sends this text data and emotion data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday, everything's fine." Finally, the device synthesizes this response into voice and tells the user, "I had delicious curry yesterday, everything's fine."

[0198] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition, emotion, and speech synthesis engines are all cloud-based, enabling increased processing power and flexible scalability.

[0199] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0200] The processing flow will be explained below.

[0201] Step 1:

[0202] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0203] Step 2:

[0204] The terminal receives the user's voice input through a microphone.

[0205] Step 3:

[0206] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0207] Step 4:

[0208] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0209] Step 5:

[0210] The emotion engine generates emotion data "joy," and the terminal transmits this data together with the text data to the server.

[0211] Step 6:

[0212] The server receives the text data "It's a nice day today" and the emotion data "joy."

[0213] Step 7:

[0214] The server uses generative artificial intelligence to generate a response text, "What lovely weather! Maybe we should go for a walk outside." based on the user's speech content and emotions.

[0215] Step 8:

[0216] The server returns the generated response text to the terminal.

[0217] Step 9:

[0218] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! Maybe we should go for a walk outside."

[0219] Step 10:

[0220] The terminal outputs the generated voice data to the user from a speaker.

[0221] For example, if a user says "I forgot what I had for dinner last night" the process would be:

[0222] Step 1:

[0223] The user speaks to the AI ​​speaker, saying, "I forgot what I had for dinner last night."

[0224] Step 2:

[0225] The terminal receives the user's voice input through a microphone.

[0226] Step 3:

[0227] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "I forgot what I had for dinner last night."

[0228] Step 4:

[0229] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0230] Step 5:

[0231] The emotion engine generates emotion data "confusion," and the terminal transmits this data together with the text data to the server.

[0232] Step 6:

[0233] The server receives the text data "I forgot what I had for dinner last night" and the emotion data "confused."

[0234] Step 7:

[0235] The server uses generative artificial intelligence to generate a response text, "I had delicious curry yesterday, so it's fine," based on the user's speech content and emotions.

[0236] Step 8:

[0237] The server returns the generated response text to the terminal.

[0238] Step 9:

[0239] The response text received by the device is sent to a speech synthesis engine and converted into voice data such as "I had delicious curry yesterday, it's fine."

[0240] Step 10:

[0241] The terminal outputs the generated voice data to the user from a speaker.

[0242] Example 2

[0243] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] The purpose of this invention is to realize a system that reduces the burden on caregivers when interacting with dementia patients and provides more appropriate and personalized responses to the patients. Conventional voice dialogue systems can only generate simple responses without taking the user's emotions into consideration, resulting in a lack of communication that reflects the user's emotional state. This makes it difficult to provide a sense of security to dementia patients and build a relationship of trust.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0246] In this invention, the server includes means for sending received text data to an emotion engine and recognizing the user's emotion, means for using the recognized emotion data and text data with generative artificial intelligence to generate an appropriate response text, and means for converting the generated response text into voice data with a voice synthesis engine, thereby enabling the generation of an individualized response according to the user's emotion.

[0247] A "user" is a person with whom the system is used to interact.

[0248] "Voice input" is voice data that a user utters to the system.

[0249] A "terminal" is a device that receives a user's voice input and performs subsequent processing.

[0250] A "voice recognition engine" is a technology that converts received voice data into text data.

[0251] "Text data" is character string data that represents a user's voice input and is converted by a voice recognition engine.

[0252] An "emotion engine" is a technology that analyzes a user's emotions from text data and acoustic features and generates emotion data.

[0253] "Emotion data" is data that represents the user's emotion recognized by the emotion engine.

[0254] A "server" is a remote computing device that processes text and emotion data and generates and manages response text.

[0255] "Generative AI" is an AI technology that generates appropriate text based on input data.

[0256] "Response text" is text data that represents a response to a user, generated by generative artificial intelligence.

[0257] A "speech synthesis engine" is a technology that converts response text into voice data.

[0258] "Voice data" refers to a voice file generated by a voice synthesis engine and output to the user.

[0259] The present invention is a system that reduces the burden on caregivers of dementia patients and allows them to communicate with the patients via a voice interface. The system incorporates an emotion analysis engine that recognizes the user's emotions and provides more personalized responses. A systematic explanation for implementing the present invention is provided below.

[0260] System configuration

[0261] First, a device with a voice receiving function is required for users to input voice. This device captures the user's speech through a microphone. Specifically, this applies to a typical smart speaker, tablet, or smartphone.

[0262] Next, a cloud-based speech recognition engine, such as the Google Cloud Speech-to-Text API, is used to convert the voice input into text data. The received voice input is sent to this speech recognition engine, which converts it into text data.

[0263] Next, this text data needs to be sent to an emotion engine to analyze the user's emotions. For example, a cloud-based emotion analysis engine such as IBM Watson® Tone Analyzer can be used. The emotion analysis engine detects the user's emotions from the text data and acoustic features such as the tone and speed of the voice.

[0264] After recognizing the emotion, the device sends the converted text data and the recognized emotion data to a server running a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response text.

[0265] The generated response text is sent back to the device, which then sends it to a cloud-based speech synthesis engine (e.g., Amazon Polly) to convert it into voice data, which is then played back through the speaker to respond to the user.

[0266] Specific examples

[0267] As a concrete example, consider a scenario in which a user says, "It's a nice day today." The user's speech is received by the device and sent to a speech recognition engine. The speech recognition engine converts this utterance into text data saying, "It's a nice day today." The emotion engine then analyzes this text data and recognizes that the user is "excited." The text data, along with the emotion data, is sent to the server, and the generative artificial intelligence generates a response saying, "It's really nice weather! Let's go outside for a walk!" Finally, this response text is converted into audio data by a speech synthesis engine and played back to the user.

[0268] As another example, consider the case where a user utters, "I forgot what I had for dinner last night." The speech recognition engine converts this into text data, "I forgot what I had for dinner last night," and the emotion engine recognizes this as "confused." On the server side, the generative AI generates a response, "I had delicious curry yesterday, so it's okay," which is then converted into speech by the speech synthesis engine. This response is then played back to the user.

[0269] Prompt Sentence Examples

[0270] User: "It's a nice day today."

[0271] Server: "What beautiful weather! Let's go outside for a walk!"

[0272] User: "I forgot what I had for dinner last night."

[0273] Server: "I had a delicious curry yesterday, it's fine."

[0274] As such, the present invention is a system that utilizes a cloud-based speech recognition engine, emotion analysis engine, generative artificial intelligence, and speech synthesis engine to provide personalized dialogue based on the user's emotions, thereby enabling more appropriate and reassuring dialogue for dementia patients and helping to reduce the burden on caregivers.

[0275] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0276] Step 1:

[0277] The user gives a voice input, for example, saying "It's a nice day today."

[0278] Input: User's voice

[0279] Output: None (audio is input to the device)

[0280] Specifically, the microphone of the terminal captures the user's voice.

[0281] Step 2:

[0282] The device receives the audio.

[0283] Input: User's voice

[0284] Output: Audio data

[0285] The received audio is temporarily stored on the terminal as audio data.

[0286] Step 3:

[0287] The device sends the voice input to a speech recognition engine, which converts it into text data.

[0288] Input: Audio data

[0289] Output: Text data

[0290] The device sends this voice data to a cloud-based speech recognition engine, which converts it into text data such as "It's nice weather today." The device uses the Google Cloud Speech-to-Text API.

[0291] Step 4:

[0292] The terminal sends the converted text data to an emotion engine to recognize the user's emotion.

[0293] Input: Text data

[0294] Output: Emotion data

[0295] The text data is sent to an emotion engine, which analyzes the user's emotion from the tone and speed of the voice. The emotion engine uses tools such as IBM Watson Tone Analyzer. For example, if the user is speaking excitedly, emotion data of "excited" is generated.

[0296] Step 5:

[0297] The device transmits text data and emotion data to the server.

[0298] Input: Text data, emotion data

[0299] Output: Data sent to the server

[0300] The converted text data and the recognized emotion data are transmitted from the terminal to the server.

[0301] Step 6:

[0302] The server uses a generative AI model to generate appropriate response text.

[0303] Input: Text data, emotion data

[0304] Output: Response text

[0305] The server analyzes the received text data and emotion data and generates a response text using generative artificial intelligence (e.g., OpenAI GPT-3). For example, based on the text "What a nice day today" and the emotion data "excitement," the response text generated is "What a nice day! Let's go for a walk outside!"

[0306] Step 7:

[0307] The server generates a response text and sends it to the terminal.

[0308] Input: Response text

[0309] Output: Data sent to the terminal

[0310] The generated response text is sent from the server to the terminal.

[0311] Step 8:

[0312] The terminal sends the response text to a speech synthesis engine, which converts it into voice data.

[0313] Input: Response text

[0314] Output: Audio data

[0315] The device sends the received response text to a cloud-based speech synthesis engine, which converts it into voice data. The speech synthesis engine is Amazon Polly or similar. For example, a response text like "What beautiful weather! Let's go for a walk outside!" is converted into voice data.

[0316] Step 9:

[0317] The terminal plays the generated audio to the user.

[0318] Input: Audio data

[0319] Output: Played audio

[0320] The device plays the generated voice data from a speaker and provides a response to the user. For example, the user can hear the voice response, "What a lovely day! Let's go for a walk outside!"

[0321] (Application example 2)

[0322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] The purpose of this invention is to provide a system that reduces the burden on caregivers of dementia patients and enables more effective and personalized dialogue when patients use food delivery services. Dementia patients often have difficulty selecting and ordering food, so the challenge is to alleviate this difficulty and enable them to eat with peace of mind.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0325] In this invention, the server includes a means for receiving voice input, a means for converting the voice into text data, a means for recognizing the user's emotion with an emotion engine, and a means for generating suggested text based on the past speech history and emotion data, thereby enabling appropriate and personalized meal suggestions and delivery arrangements for dementia patients.

[0326] A "voice recognition engine" is software or hardware that converts a user's voice input into text data.

[0327] An "emotion engine" is software or hardware for recognizing a user's emotions from text data and voice characteristics.

[0328] "Generative AI" is an AI model that can generate appropriate responses or suggestions based on the data it receives.

[0329] "Text data" is character string information based on the user's voice input, converted by a voice recognition engine.

[0330] A "server" is a computer system that receives data, performs analytical processing, and sends the results to a client.

[0331] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[0332] The "past utterance history" is a record of utterances previously made by the user, and is data that can be used as reference for future conversations.

[0333] "Emotion data" is information about the user's emotional state as recognized by the emotion engine.

[0334] "Suggested text" is text data generated by generative artificial intelligence and containing content to be suggested to the user.

[0335] A "food delivery service" is a service that aims to deliver meals to a user's location.

[0336] A "client" is a user terminal that sends and receives data to and from a server.

[0337] This invention is a system that reduces the burden on caregivers of dementia patients and provides effective and personalized dialogue when patients use food delivery services. This system is composed of a combination of a speech recognition engine, an emotion engine, a generative artificial intelligence, a speech synthesis engine, etc.

[0338] System Components

[0339] 1. Receiving voice input

[0340] A device (e.g., a smart speaker) receives a user's voice input.

[0341] 2. Speech Recognition Engine

[0342] A speech recognition engine (e.g., a cloud-based speech recognition API) converts the received voice input into text data.

[0343] 3. Emotion recognition

[0344] An emotion engine (e.g., IBM Watson Tone Analyzer) is used to recognize user emotions from text data and voice features. This emotion data indicates the user's emotional state and is used to generate suggestions.

[0345] 4. Data transmission

[0346] The terminal transmits the converted text data and the recognized emotion data to the server.

[0347] 5. Proposal Generation Using Generative AI

[0348] The server uses a generative artificial intelligence model (e.g., GPT-4 (registered trademark)) to generate optimal suggested text based on past speech history and emotional data.

[0349] 6. Speech synthesis engine

[0350] The generated suggested text is converted into voice data by a speech synthesis engine (e.g., Amazon Polly).

[0351] 7. Audio Output

[0352] The terminal outputs the generated voice data to the user, and an actual conversation takes place.

[0353] Explanations and examples

[0354] When a user requests food delivery, the device may say, for example, "I don't know what I want to eat today." The device receives this voice input and converts it into text data using a speech recognition engine. This text data is "I don't know what I want to eat today." The emotion engine analyzes this text data and the tone and speed of the user's voice, and determines that the user is "confused."

[0355] Next, this text data and emotion data are sent to the server, where the server's generative AI refers to the user's past speech history (for example, the last time they ordered teriyaki salmon) and generates the suggested text, "How about teriyaki salmon?"

[0356] This suggested text is converted into voice data by a speech synthesis engine, and the device suggests to the user, "How about teriyaki salmon?"

[0357] For illustrative purposes, the following prompt sentence example will be used:

[0358] User utterance: I don't remember what I had for dinner last night. I'm hungry, but I don't know what to eat (emotional state: confused). Please suggest the best food based on that.

[0359] In this way, the system provides dementia patients with appropriate and personalized suggestions and support, making food delivery easier to use.

[0360] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0361] Step 1:

[0362] The device (e.g., a smart speaker) receives the user's voice input, which becomes the initial information for the system.

[0363] Step 2:

[0364] The device sends the received voice input to a speech recognition engine (e.g., a cloud-based speech recognition API) to convert it into text data. The input is voice data, and the output is text data. Here, the speech recognition engine analyzes the voice waveform and generates a corresponding text string.

[0365] Step 3:

[0366] The device sends the converted text data to an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data and speech characteristics, and the output is emotion data. The emotion engine analyzes the emotional state from the text content, speech tone, speed, etc.

[0367] Step 4:

[0368] The terminal transmits the recognized emotion data and text data to the server. The input is the text data and emotion data, and the output is the transmission data. Here, the terminal performs the data transmission operation.

[0369] Step 5:

[0370] The server analyzes the received text data and emotion data and generates optimal suggested text using a generative AI model (e.g., GPT-4). The input is text data, emotion data, and past speech history, and the output is suggested text. The generative AI model analyzes this data and generates an appropriate response.

[0371] Step 6:

[0372] The server sends the generated suggested text to the terminal. The input is the suggested text and the output is the transmitted data. The server performs data communication and sends the generated text back to the terminal.

[0373] Step 7:

[0374] The device sends the received suggested text to a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The input is the suggested text and the output is speech data. The speech synthesis engine analyzes the text content and generates natural-sounding speech.

[0375] Step 8:

[0376] The device plays the generated voice data from the speaker and outputs it to the user. The input is voice data and the output is voice output. Specifically, the voice is played through the device's speaker and the suggestion is delivered to the user.

[0377] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0378] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0379] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0380] [Second embodiment]

[0381] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0382] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0383] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0384] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0385] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0386] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0387] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0388] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0389] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0390] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0391] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0392] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0393] ---

[0394] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0395] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0396] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0397] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., a generative AI model). For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go outside for a walk."

[0398] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back from the speaker.

[0399] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and tells the user, "I had delicious curry yesterday."

[0400] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0401] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0402] The processing flow will be explained below.

[0403] Step 1:

[0404] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0405] Step 2:

[0406] The terminal receives the user's voice input through a microphone.

[0407] Step 3:

[0408] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0409] Step 4:

[0410] The terminal sends the converted text data to the server as an HTTP POST request.

[0411] Step 5:

[0412] The server analyzes the received text data, "It's nice weather today," and uses generative artificial intelligence to generate the response text, "It's really nice weather! Maybe it would be nice to go for a walk outside."

[0413] Step 6:

[0414] The server generates a response text and sends it back to the terminal as an HTTP response.

[0415] Step 7:

[0416] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! It might be nice to go for a walk outside."

[0417] Step 8:

[0418] The terminal makes the generated voice data available to the user through a speaker.

[0419] Example 1

[0420] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0421] The problem that this invention aims to solve is to reduce the burden on caregivers for dementia patients and to improve the effectiveness of dialogue between patients and AI speakers. Another problem is to achieve highly accurate speech recognition and natural voice response generation to improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, we aim to provide a system that can be adapted to a wide range of users.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0423] In this invention, the server includes means for receiving the user's voice using a highly sensitive noise-canceling microphone, means for the speech recognition engine to always use the latest algorithms and be updated to improve the accuracy of proper noun recognition, means for the generative artificial intelligence to refer to the user's past speech history and generate personalized responses, and means for the speech synthesis engine to support multiple languages ​​and generate responses in different languages ​​and dialects with natural pronunciation. This enables highly accurate speech recognition and natural voice responses, and the personalized responses improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, a system that can be adapted to a wide range of users is realized.

[0424] "User" refers to a person who uses the system.

[0425] "Voice input" refers to speech uttered by a user.

[0426] "Means for receiving" refers to a combination of hardware and software for capturing audio input.

[0427] "Speech recognition engine" refers to software or a service for converting voice data into text data.

[0428] "Text data" refers to character string data converted by a voice recognition engine.

[0429] "Server" refers to a remote computing device for analyzing text data generated from speech input and generating appropriate responses.

[0430] "Generative AI" refers to an AI model that generates appropriate responses based on text data.

[0431] "Response text" refers to the string data of a response generated by generative artificial intelligence.

[0432] "Speech synthesis engine" refers to software or a service for converting text data into speech data.

[0433] "Voice data" refers to sound data generated by a speech synthesis engine.

[0434] "Output means" refers to a combination of hardware and software, such as a speaker, that allows the user to hear the audio data.

[0435] A "noise-canceling microphone" refers to a microphone equipped with technology that eliminates ambient noise.

[0436] "Proper noun recognition accuracy" refers to the accuracy in recognizing proper nouns such as specific people's names, place names, and brand names.

[0437] "Means for being updated" refers to the update mechanism for keeping the software and algorithms used up to date.

[0438] "Utterance history" refers to a record of the contents of utterances made by a user in the past.

[0439] "Personalized responses" refer to customized responses generated based on the characteristics and history of individual users.

[0440] "Multilingual" refers to the ability to support multiple languages.

[0441] A "dialect" refers to the use of words and pronunciation that is unique to a particular region or community.

[0442] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0443] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the device, which is equipped with a highly sensitive noise-canceling microphone. This microphone filters out ambient noise and captures accurate voice data.

[0444] The device then sends the received voice input to a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. This speech recognition engine always uses the latest algorithms and is regularly updated to improve recognition accuracy, especially for proper nouns. For example, the user's speech, "It's a nice day today," is converted into text data, "It's a nice day today."

[0445] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., an OpenAI model). This generative artificial intelligence references the user's past speech history to generate a personalized response. For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go for a walk outside."

[0446] The generated response text is then sent back to the device. The device then sends the received response text to a cloud-based speech synthesis engine (e.g., Amazon Polly), which converts it into voice data. This speech synthesis engine supports multiple languages ​​and can generate natural-sounding responses in different languages ​​and dialects. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back through the speaker.

[0447] As a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data saying, "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response saying, "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and speaks to the user, saying, "I had delicious curry yesterday."

[0448] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0449] An example of a prompt sentence that can be input to a generative AI model is "Tell me about what you had for dinner last night."

[0450] As described above, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0451] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0452] Step 1:

[0453] The user speaks to the AI ​​speaker, for example, saying, "It's nice weather today."

[0454] Input: User's spoken utterance "It's a nice day today."

[0455] Output: Captured as audio data

[0456] Step 2:

[0457] The device receives the user's voice using a highly sensitive noise-canceling microphone, which filters out ambient noise and captures accurate voice data.

[0458] Input: User's voice data

[0459] Output: Clear audio data after noise removal

[0460] Step 3:

[0461] The device sends the received voice data to a cloud-based speech recognition engine (e.g., a speech recognition API) that converts the speech into text data. This speech recognition engine uses the latest algorithms to improve the accuracy of proper noun recognition.

[0462] Input: Clear audio data after noise removal

[0463] Data processing: Converting voice data into text data

[0464] Output: Text data "It's a nice day today."

[0465] Step 4:

[0466] The terminal transmits the converted text data to the server.

[0467] Input: Text data "It's a nice day today"

[0468] Output: Send text data to the server

[0469] Step 5:

[0470] The server analyzes the received text data and generates a response text using generative artificial intelligence (e.g., a generative AI model). It references past speech history to generate a personalized response.

[0471] Input: Text data "It's a nice day today"

[0472] Data calculations: term analysis, context understanding, and reference to past speech history

[0473] Output: Response text "What lovely weather! Maybe we should go for a walk outside."

[0474] Step 6:

[0475] The server returns the generated response text to the terminal.

[0476] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0477] Output: Response text sent to terminal

[0478] Step 7:

[0479] The device sends the received response text to a cloud-based speech synthesis engine (e.g., speech synthesis API) and converts it into voice data. This speech synthesis engine supports multiple languages ​​and generates voice with natural pronunciation.

[0480] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0481] Data processing: Convert text data into audio data

[0482] Output: Audio data

[0483] Step 8:

[0484] The terminal reproduces the generated voice data from a speaker and responds to the user.

[0485] Input: Audio data

[0486] Output: Voice response "What lovely weather! Maybe we should go for a walk outside."

[0487] The above is a detailed description of the program processing for this system. By clearly indicating the specific operations performed at each step, as well as the inputs and outputs, the processing flow of the entire system becomes clearer.

[0488] (Application example 1)

[0489] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] In recent years, with the progress of aging society, caring for dementia patients has become a social issue. In particular, providing psychological support to dementia patients through daily conversations places a heavy burden on caregivers. Therefore, there is a need for effective conversation support systems to help dementia patients live their daily lives independently. Furthermore, there is a need for technology that can provide personalized content based on the individual preferences and past speech history of each patient.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0492] In this invention, the server includes means for converting a user's speech into text data using a speech recognition engine, means for transmitting the converted text data to a network server, means for generating a response text using generative artificial intelligence on the network server side, means for converting the generated response text into voice data using a speech synthesis engine, and means for providing personalized content such as audiobooks, music, podcasts, and news based on the user's speech history and preference history. This enables personalized dialogue and content provision for dementia patients, thereby supporting the patients' daily lives while reducing the burden on caregivers.

[0493] "User's voice input" refers to a voice message uttered by a dementia patient.

[0494] "Speech recognition engine" refers to a program or device for receiving voice data and converting it into character string data.

[0495] "Text data" refers to character string data converted by a voice recognition engine.

[0496] "Network server" refers to a central computing unit that processes data and generates responses via the cloud or the internet.

[0497] "Generative artificial intelligence" refers to an AI engine that uses pre-trained models to generate natural-sounding response text based on user input data.

[0498] "Response text" is character string data generated by the generative artificial intelligence, and includes a response to the user.

[0499] "Speech synthesis engine" refers to a program or device for converting text data into speech data.

[0500] "Audio Data" refers to audio files that are generated by a speech synthesis engine and can be played through a speaker.

[0501] "Speech history" refers to a record of voice inputs made by the user in the past.

[0502] "Preference history" refers to a past record of a user's preferences and hobbies.

[0503] "Audiobook" refers to digital content in which the contents of a book are presented in audio form.

[0504] "Music" refers to audio data recorded from musical instruments or singing.

[0505] A "podcast" is an audio program distributed over the Internet.

[0506] "News" refers to audio or text data that conveys current events or information.

[0507] "Personalization" refers to changing content based on an individual user's specific preferences and history.

[0508] The present invention is a personalized content delivery system for dementia patients, which aims to reduce the burden on caregivers and provide patients with a sense of security. The system executes a series of processes: receiving voice input from a user, converting it into text data, generating a response, and providing it to the user again as voice.

[0509] Specifically, the main components of the system include:

[0510] 1. Receiving audio input:

[0511] The user (a dementia patient) speaks to the system, and the voice input is received by a microphone installed in the device, which can be a smartphone, smart glasses, or a virtual reality head-mounted display.

[0512] 2. Speech Recognition:

[0513] The received voice input is converted into text data by a speech recognition engine, which can be a cloud-based service such as Google Speech Recognition API.

[0514] 3. Sending text data:

[0515] The converted text data is sent over the network to a server, which is a cloud server connected to the Internet.

[0516] 4. Generate response:

[0517] On the server side, the received text data is analyzed and an appropriate response text is generated using a generative AI model (e.g., GPT-3). The user's past speech history and preference history are also taken into consideration. For example, in response to the input text "Play me some old hit songs," the server generates a response such as "Playing hit songs from the 1960s."

[0518] 5. Sending a response text:

[0519] The generated response text is sent back to the terminal.

[0520] 6. Speech synthesis:

[0521] The terminal sends the received response text to a speech synthesis engine (such as Pyttsx3) and converts it into voice data.

[0522] 7. Audio Output:

[0523] The generated voice data is output to the user through the device's speaker. For example, a voice saying "Playing a hit song from the 1960s" is played.

[0524] Examples and prompts

[0525] As a concrete example, consider the case where a user says, "Play me an old hit song." When the user says, "Play me an old hit song," the speech recognition engine converts this into text data. The converted text data is then sent to the server, where a generative AI model generates a response such as, "Playing a hit song from the 1960s." The device receives this response, converts it into audio data using a speech synthesis engine, and plays it back.

[0526] An example prompt to use with a generative AI model might look like this:

[0527] User Input: Play me some old hit songs

[0528] Generate an appropriate response.

[0529] The system provides personalized content based on the user's speech history and preferences to support the daily lives of dementia patients. It also uses a cloud-based speech recognition and speech synthesis engine to improve processing power and provide flexible scalability.

[0530] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0531] Step 1:

[0532] The user speaks a voice input. The user's voice message is received through the device's microphone. This input is processed as raw voice data.

[0533] Step 2:

[0534] The device sends the received voice data to a voice recognition engine. Specifically, this voice data is sent to a cloud-based service such as Google Speech Recognition API, where it is converted into text data. The input here is voice data, and the output is text data. The voice recognition engine analyzes the voice data and recognizes its content as a string of characters.

[0535] Step 3:

[0536] The terminal sends the converted text data to the server via the network. At this stage, the input is text data and the output is string data sent to the server. The terminal makes the appropriate API call to send the text data.

[0537] Step 4:

[0538] The server passes the received text data to a generative AI model. The input is text data, and the output is the generated response text. The server uses a pre-trained generative AI model (e.g., GPT-3) to generate an appropriate response to the spoken content. For example, in response to the input "Play me an old hit song," a response such as "I'll play a hit song from the 1960s" is generated.

[0539] Step 5:

[0540] The server returns the generated response text to the terminal. The input here is the generated response text, and the output is the response text sent to the terminal. The server returns the generated response in real time.

[0541] Step 6:

[0542] The device sends the received response text to a speech synthesis engine. The input here is the response text, and the output is voice data. Specifically, a speech synthesis library such as Pyttsx3 is used to convert the text data into natural-sounding voice.

[0543] Step 7:

[0544] The device outputs the generated voice data to the user through a speaker. At this stage, the input is voice data from the speech synthesis engine, and the output is a voice message that the user can hear. For example, "Playing a hit song from the 1960s."

[0545] Through these steps, appropriate responses and content are provided in real time based on the user's speech.This system can provide personalized information support and psychological comfort to dementia patients.

[0546] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0547] This invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between patients and AI speakers. The system incorporates an emotion engine that recognizes the user's emotions, providing more personalized responses.

[0548] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0549] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0550] The device then sends this text data to the emotion engine to recognize the user's emotion. The emotion engine can detect emotions not only from text data but also from acoustic features such as the user's voice tone and speed. For example, if the user speaks excitedly, the emotion engine will recognize the user as being in an "excited" state.

[0551] The device then sends the converted text data and the recognized emotion data to the server. The server analyzes this data and generates an appropriate response text using a generative artificial intelligence (generative AI model). For example, if the input text is "What a nice day today" and the emotion data is "Excited," the server generates the response "What a nice day! Let's go for a walk outside!"

[0552] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Let's go for a walk outside!" is converted into natural-sounding speech and played back from the speaker.

[0553] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." Next, an emotion engine recognizes the user's emotion from their tone and speed of speech and determines it as "confused." The device then sends this text data and emotion data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday, everything's fine." Finally, the device synthesizes this response into voice and tells the user, "I had delicious curry yesterday, everything's fine."

[0554] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition, emotion, and speech synthesis engines are all cloud-based, enabling increased processing power and flexible scalability.

[0555] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0559] Step 2:

[0560] The terminal receives the user's voice input through a microphone.

[0561] Step 3:

[0562] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0563] Step 4:

[0564] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0565] Step 5:

[0566] The emotion engine generates emotion data "joy," and the terminal transmits this data together with the text data to the server.

[0567] Step 6:

[0568] The server receives the text data "It's a nice day today" and the emotion data "joy."

[0569] Step 7:

[0570] The server uses generative artificial intelligence to generate a response text, "What lovely weather! Maybe we should go for a walk outside." based on the user's speech content and emotions.

[0571] Step 8:

[0572] The server returns the generated response text to the terminal.

[0573] Step 9:

[0574] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! Maybe we should go for a walk outside."

[0575] Step 10:

[0576] The terminal outputs the generated voice data to the user from a speaker.

[0577] For example, if a user says "I forgot what I had for dinner last night" the process would be:

[0578] Step 1:

[0579] The user speaks to the AI ​​speaker, saying, "I forgot what I had for dinner last night."

[0580] Step 2:

[0581] The terminal receives the user's voice input through a microphone.

[0582] Step 3:

[0583] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "I forgot what I had for dinner last night."

[0584] Step 4:

[0585] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0586] Step 5:

[0587] The emotion engine generates emotion data "confusion," and the terminal transmits this data together with the text data to the server.

[0588] Step 6:

[0589] The server receives the text data "I forgot what I had for dinner last night" and the emotion data "confused."

[0590] Step 7:

[0591] The server uses generative artificial intelligence to generate a response text, "I had delicious curry yesterday, so it's fine," based on the user's speech content and emotions.

[0592] Step 8:

[0593] The server returns the generated response text to the terminal.

[0594] Step 9:

[0595] The response text received by the device is sent to a speech synthesis engine and converted into voice data such as "I had delicious curry yesterday, it's fine."

[0596] Step 10:

[0597] The terminal outputs the generated voice data to the user from a speaker.

[0598] Example 2

[0599] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] The purpose of this invention is to realize a system that reduces the burden on caregivers when interacting with dementia patients and provides more appropriate and personalized responses to the patients. Conventional voice dialogue systems can only generate simple responses without taking the user's emotions into consideration, resulting in a lack of communication that reflects the user's emotional state. This makes it difficult to provide a sense of security to dementia patients and build a relationship of trust.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0602] In this invention, the server includes means for sending received text data to an emotion engine and recognizing the user's emotion, means for using the recognized emotion data and text data with generative artificial intelligence to generate an appropriate response text, and means for converting the generated response text into voice data with a voice synthesis engine, thereby enabling the generation of an individualized response according to the user's emotion.

[0603] A "user" is a person with whom the system is used to interact.

[0604] "Voice input" is voice data that a user utters to the system.

[0605] A "terminal" is a device that receives a user's voice input and performs subsequent processing.

[0606] A "voice recognition engine" is a technology that converts received voice data into text data.

[0607] "Text data" is character string data that represents a user's voice input and is converted by a voice recognition engine.

[0608] An "emotion engine" is a technology that analyzes a user's emotions from text data and acoustic features and generates emotion data.

[0609] "Emotion data" is data that represents the user's emotion recognized by the emotion engine.

[0610] A "server" is a remote computing device that processes text and emotion data and generates and manages response text.

[0611] "Generative AI" is an AI technology that generates appropriate text based on input data.

[0612] "Response text" is text data that represents a response to a user, generated by generative artificial intelligence.

[0613] A "speech synthesis engine" is a technology that converts response text into voice data.

[0614] "Voice data" refers to a voice file generated by a voice synthesis engine and output to the user.

[0615] The present invention is a system that reduces the burden on caregivers of dementia patients and allows them to communicate with the patients via a voice interface. The system incorporates an emotion analysis engine that recognizes the user's emotions and provides more personalized responses. A systematic explanation for implementing the present invention is provided below.

[0616] System configuration

[0617] First, a device with a voice receiving function is required for users to input voice. This device captures the user's speech through a microphone. Specifically, this applies to a typical smart speaker, tablet, or smartphone.

[0618] Next, a cloud-based speech recognition engine, such as the Google Cloud Speech-to-Text API, is used to convert the voice input into text data. The received voice input is sent to this speech recognition engine, which converts it into text data.

[0619] Next, this text data must be sent to an emotion engine to analyze the user's emotions. For example, a cloud-based emotion analysis engine such as IBM Watson Tone Analyzer can be used. The emotion analysis engine detects the user's emotions from the text data and acoustic features such as the tone and speed of the voice.

[0620] After recognizing the emotion, the device sends the converted text data and the recognized emotion data to a server running a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response text.

[0621] The generated response text is sent back to the device, which then sends it to a cloud-based speech synthesis engine (e.g., Amazon Polly) to convert it into voice data, which is then played back through the speaker to respond to the user.

[0622] Specific examples

[0623] As a concrete example, consider a scenario in which a user says, "It's a nice day today." The user's speech is received by the device and sent to a speech recognition engine. The speech recognition engine converts this utterance into text data saying, "It's a nice day today." The emotion engine then analyzes this text data and recognizes that the user is "excited." The text data, along with the emotion data, is sent to the server, and the generative artificial intelligence generates a response saying, "It's really nice weather! Let's go outside for a walk!" Finally, this response text is converted into audio data by a speech synthesis engine and played back to the user.

[0624] As another example, consider the case where a user utters, "I forgot what I had for dinner last night." The speech recognition engine converts this into text data, "I forgot what I had for dinner last night," and the emotion engine recognizes this as "confused." On the server side, the generative AI generates a response, "I had delicious curry yesterday, so it's okay," which is then converted into speech by the speech synthesis engine. This response is then played back to the user.

[0625] Prompt Sentence Examples

[0626] User: "It's a nice day today."

[0627] Server: "What beautiful weather! Let's go outside for a walk!"

[0628] User: "I forgot what I had for dinner last night."

[0629] Server: "I had a delicious curry yesterday, it's fine."

[0630] As such, the present invention is a system that utilizes a cloud-based speech recognition engine, emotion analysis engine, generative artificial intelligence, and speech synthesis engine to provide personalized dialogue based on the user's emotions, thereby enabling more appropriate and reassuring dialogue for dementia patients and helping to reduce the burden on caregivers.

[0631] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0632] Step 1:

[0633] The user gives a voice input, for example, saying "It's a nice day today."

[0634] Input: User's voice

[0635] Output: None (audio is input to the device)

[0636] Specifically, the microphone of the terminal captures the user's voice.

[0637] Step 2:

[0638] The device receives the audio.

[0639] Input: User's voice

[0640] Output: Audio data

[0641] The received audio is temporarily stored on the terminal as audio data.

[0642] Step 3:

[0643] The device sends the voice input to a speech recognition engine, which converts it into text data.

[0644] Input: Audio data

[0645] Output: Text data

[0646] The device sends this voice data to a cloud-based speech recognition engine, which converts it into text data such as "It's nice weather today." The device uses the Google Cloud Speech-to-Text API.

[0647] Step 4:

[0648] The terminal sends the converted text data to an emotion engine to recognize the user's emotion.

[0649] Input: Text data

[0650] Output: Emotion data

[0651] The text data is sent to an emotion engine, which analyzes the user's emotion from the tone and speed of the voice. The emotion engine uses tools such as IBM Watson Tone Analyzer. For example, if the user is speaking excitedly, emotion data of "excited" is generated.

[0652] Step 5:

[0653] The device transmits text data and emotion data to the server.

[0654] Input: Text data, emotion data

[0655] Output: Data sent to the server

[0656] The converted text data and the recognized emotion data are transmitted from the terminal to the server.

[0657] Step 6:

[0658] The server uses a generative AI model to generate appropriate response text.

[0659] Input: Text data, emotion data

[0660] Output: Response text

[0661] The server analyzes the received text data and emotion data and generates a response text using generative artificial intelligence (e.g., OpenAI GPT-3). For example, based on the text "What a nice day today" and the emotion data "excitement," the response text generated is "What a nice day! Let's go for a walk outside!"

[0662] Step 7:

[0663] The server generates a response text and sends it to the terminal.

[0664] Input: Response text

[0665] Output: Data sent to the terminal

[0666] The generated response text is sent from the server to the terminal.

[0667] Step 8:

[0668] The terminal sends the response text to a speech synthesis engine, which converts it into voice data.

[0669] Input: Response text

[0670] Output: Audio data

[0671] The device sends the received response text to a cloud-based speech synthesis engine, which converts it into voice data. The speech synthesis engine is Amazon Polly or similar. For example, a response text like "What beautiful weather! Let's go for a walk outside!" is converted into voice data.

[0672] Step 9:

[0673] The terminal plays the generated audio to the user.

[0674] Input: Audio data

[0675] Output: Played audio

[0676] The device plays the generated voice data from a speaker and provides a response to the user. For example, the user can hear the voice response, "What a lovely day! Let's go for a walk outside!"

[0677] (Application example 2)

[0678] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0679] The purpose of this invention is to provide a system that reduces the burden on caregivers of dementia patients and enables more effective and personalized dialogue when patients use food delivery services. Dementia patients often have difficulty selecting and ordering food, so the challenge is to alleviate this difficulty and enable them to eat with peace of mind.

[0680] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0681] In this invention, the server includes a means for receiving voice input, a means for converting the voice into text data, a means for recognizing the user's emotion with an emotion engine, and a means for generating suggested text based on the past speech history and emotion data, thereby enabling appropriate and personalized meal suggestions and delivery arrangements for dementia patients.

[0682] A "voice recognition engine" is software or hardware that converts a user's voice input into text data.

[0683] An "emotion engine" is software or hardware for recognizing a user's emotions from text data and voice characteristics.

[0684] "Generative AI" is an AI model that can generate appropriate responses or suggestions based on the data it receives.

[0685] "Text data" is character string information based on the user's voice input, converted by a voice recognition engine.

[0686] A "server" is a computer system that receives data, performs analytical processing, and sends the results to a client.

[0687] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[0688] The "past utterance history" is a record of utterances previously made by the user, and is data that can be used as reference for future conversations.

[0689] "Emotion data" is information about the user's emotional state as recognized by the emotion engine.

[0690] "Suggested text" is text data generated by generative artificial intelligence and containing content to be suggested to the user.

[0691] A "food delivery service" is a service that aims to deliver meals to a user's location.

[0692] A "client" is a user terminal that sends and receives data to and from a server.

[0693] This invention is a system that reduces the burden on caregivers of dementia patients and provides effective and personalized dialogue when patients use food delivery services. This system is composed of a combination of a speech recognition engine, an emotion engine, a generative artificial intelligence, a speech synthesis engine, etc.

[0694] System Components

[0695] 1. Receiving voice input

[0696] A device (e.g., a smart speaker) receives a user's voice input.

[0697] 2. Speech Recognition Engine

[0698] A speech recognition engine (e.g., a cloud-based speech recognition API) converts the received voice input into text data.

[0699] 3. Emotion recognition

[0700] An emotion engine (e.g., IBM Watson Tone Analyzer) is used to recognize user emotions from text data and voice features. This emotion data indicates the user's emotional state and is used to generate suggestions.

[0701] 4. Data transmission

[0702] The terminal transmits the converted text data and the recognized emotion data to the server.

[0703] 5. Proposal Generation Using Generative AI

[0704] The server uses a generative artificial intelligence model (e.g., GPT-4) to generate optimal suggested text based on past speech history and emotional data.

[0705] 6. Speech synthesis engine

[0706] The generated suggested text is converted into voice data by a speech synthesis engine (e.g., Amazon Polly).

[0707] 7. Audio Output

[0708] The terminal outputs the generated voice data to the user, and an actual conversation takes place.

[0709] Explanations and examples

[0710] When a user requests food delivery, the device may say, for example, "I don't know what I want to eat today." The device receives this voice input and converts it into text data using a speech recognition engine. This text data is "I don't know what I want to eat today." The emotion engine analyzes this text data and the tone and speed of the user's voice, and determines that the user is "confused."

[0711] Next, this text data and emotion data are sent to the server, where the server's generative AI refers to the user's past speech history (for example, the last time they ordered teriyaki salmon) and generates the suggested text, "How about teriyaki salmon?"

[0712] This suggested text is converted into voice data by a speech synthesis engine, and the device suggests to the user, "How about teriyaki salmon?"

[0713] For illustrative purposes, the following prompt sentence example will be used:

[0714] User utterance: I don't remember what I had for dinner last night. I'm hungry, but I don't know what to eat (emotional state: confused). Please suggest the best food based on that.

[0715] In this way, the system provides dementia patients with appropriate and personalized suggestions and support, making food delivery easier to use.

[0716] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0717] Step 1:

[0718] The device (e.g., a smart speaker) receives the user's voice input, which becomes the initial information for the system.

[0719] Step 2:

[0720] The device sends the received voice input to a speech recognition engine (e.g., a cloud-based speech recognition API) to convert it into text data. The input is voice data, and the output is text data. Here, the speech recognition engine analyzes the voice waveform and generates a corresponding text string.

[0721] Step 3:

[0722] The device sends the converted text data to an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data and speech characteristics, and the output is emotion data. The emotion engine analyzes the emotional state from the text content, speech tone, speed, etc.

[0723] Step 4:

[0724] The terminal transmits the recognized emotion data and text data to the server. The input is the text data and emotion data, and the output is the transmission data. Here, the terminal performs the data transmission operation.

[0725] Step 5:

[0726] The server analyzes the received text data and emotion data and generates optimal suggested text using a generative AI model (e.g., GPT-4). The input is text data, emotion data, and past speech history, and the output is suggested text. The generative AI model analyzes this data and generates an appropriate response.

[0727] Step 6:

[0728] The server sends the generated suggested text to the terminal. The input is the suggested text and the output is the transmitted data. The server performs data communication and sends the generated text back to the terminal.

[0729] Step 7:

[0730] The device sends the received suggested text to a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The input is the suggested text and the output is speech data. The speech synthesis engine analyzes the text content and generates natural-sounding speech.

[0731] Step 8:

[0732] The device plays the generated voice data from the speaker and outputs it to the user. The input is voice data and the output is voice output. Specifically, the voice is played through the device's speaker and the suggestion is delivered to the user.

[0733] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0734] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0735] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0736] [Third embodiment]

[0737] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0738] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0739] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0740] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0741] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0742] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0743] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0744] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0745] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0746] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0747] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0748] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0749] ---

[0750] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0751] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0752] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0753] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., a generative AI model). For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go outside for a walk."

[0754] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back from the speaker.

[0755] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and tells the user, "I had delicious curry yesterday."

[0756] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0757] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0758] The processing flow will be explained below.

[0759] Step 1:

[0760] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0761] Step 2:

[0762] The terminal receives the user's voice input through a microphone.

[0763] Step 3:

[0764] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0765] Step 4:

[0766] The terminal sends the converted text data to the server as an HTTP POST request.

[0767] Step 5:

[0768] The server analyzes the received text data, "It's nice weather today," and uses generative artificial intelligence to generate the response text, "It's really nice weather! Maybe it would be nice to go for a walk outside."

[0769] Step 6:

[0770] The server generates a response text and sends it back to the terminal as an HTTP response.

[0771] Step 7:

[0772] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! It might be nice to go for a walk outside."

[0773] Step 8:

[0774] The terminal makes the generated voice data available to the user through a speaker.

[0775] Example 1

[0776] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0777] The problem that this invention aims to solve is to reduce the burden on caregivers for dementia patients and to improve the effectiveness of dialogue between patients and AI speakers. Another problem is to achieve highly accurate speech recognition and natural voice response generation to improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, we aim to provide a system that can be adapted to a wide range of users.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0779] In this invention, the server includes means for receiving the user's voice using a highly sensitive noise-canceling microphone, means for the speech recognition engine to always use the latest algorithms and be updated to improve the accuracy of proper noun recognition, means for the generative artificial intelligence to refer to the user's past speech history and generate personalized responses, and means for the speech synthesis engine to support multiple languages ​​and generate responses in different languages ​​and dialects with natural pronunciation. This enables highly accurate speech recognition and natural voice responses, and the personalized responses improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, a system that can be adapted to a wide range of users is realized.

[0780] "User" refers to a person who uses the system.

[0781] "Voice input" refers to speech uttered by a user.

[0782] "Means for receiving" refers to a combination of hardware and software for capturing audio input.

[0783] "Speech recognition engine" refers to software or a service for converting voice data into text data.

[0784] "Text data" refers to character string data converted by a voice recognition engine.

[0785] "Server" refers to a remote computing device for analyzing text data generated from speech input and generating appropriate responses.

[0786] "Generative AI" refers to an AI model that generates appropriate responses based on text data.

[0787] "Response text" refers to the string data of a response generated by generative artificial intelligence.

[0788] "Speech synthesis engine" refers to software or a service for converting text data into speech data.

[0789] "Voice data" refers to sound data generated by a speech synthesis engine.

[0790] "Output means" refers to a combination of hardware and software, such as a speaker, that allows the user to hear the audio data.

[0791] A "noise-canceling microphone" refers to a microphone equipped with technology that eliminates ambient noise.

[0792] "Proper noun recognition accuracy" refers to the accuracy in recognizing proper nouns such as specific people's names, place names, and brand names.

[0793] "Means for being updated" refers to the update mechanism for keeping the software and algorithms used up to date.

[0794] "Utterance history" refers to a record of the contents of utterances made by a user in the past.

[0795] "Personalized responses" refer to customized responses generated based on the characteristics and history of individual users.

[0796] "Multilingual" refers to the ability to support multiple languages.

[0797] A "dialect" refers to the use of words and pronunciation that is unique to a particular region or community.

[0798] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[0799] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the device, which is equipped with a highly sensitive noise-canceling microphone. This microphone filters out ambient noise and captures accurate voice data.

[0800] The device then sends the received voice input to a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. This speech recognition engine always uses the latest algorithms and is regularly updated to improve recognition accuracy, especially for proper nouns. For example, the user's speech, "It's a nice day today," is converted into text data, "It's a nice day today."

[0801] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., an OpenAI model). This generative artificial intelligence references the user's past speech history to generate a personalized response. For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go for a walk outside."

[0802] The generated response text is then sent back to the device. The device then sends the received response text to a cloud-based speech synthesis engine (e.g., Amazon Polly), which converts it into voice data. This speech synthesis engine supports multiple languages ​​and can generate natural-sounding responses in different languages ​​and dialects. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back through the speaker.

[0803] As a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data saying, "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response saying, "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and speaks to the user, saying, "I had delicious curry yesterday."

[0804] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[0805] An example of a prompt sentence that can be input to a generative AI model is "Tell me about what you had for dinner last night."

[0806] As described above, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] The user speaks to the AI ​​speaker, for example, saying, "It's nice weather today."

[0810] Input: User's spoken utterance "It's a nice day today."

[0811] Output: Captured as audio data

[0812] Step 2:

[0813] The device receives the user's voice using a highly sensitive noise-canceling microphone, which filters out ambient noise and captures accurate voice data.

[0814] Input: User's voice data

[0815] Output: Clear audio data after noise removal

[0816] Step 3:

[0817] The device sends the received voice data to a cloud-based speech recognition engine (e.g., a speech recognition API) that converts the speech into text data. This speech recognition engine uses the latest algorithms to improve the accuracy of proper noun recognition.

[0818] Input: Clear audio data after noise removal

[0819] Data processing: Converting voice data into text data

[0820] Output: Text data "It's a nice day today."

[0821] Step 4:

[0822] The terminal transmits the converted text data to the server.

[0823] Input: Text data "It's a nice day today"

[0824] Output: Send text data to the server

[0825] Step 5:

[0826] The server analyzes the received text data and generates a response text using generative artificial intelligence (e.g., a generative AI model). It references past speech history to generate a personalized response.

[0827] Input: Text data "It's a nice day today"

[0828] Data calculations: term analysis, context understanding, and reference to past speech history

[0829] Output: Response text "What lovely weather! Maybe we should go for a walk outside."

[0830] Step 6:

[0831] The server returns the generated response text to the terminal.

[0832] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0833] Output: Response text sent to terminal

[0834] Step 7:

[0835] The device sends the received response text to a cloud-based speech synthesis engine (e.g., speech synthesis API) and converts it into voice data. This speech synthesis engine supports multiple languages ​​and generates voice with natural pronunciation.

[0836] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[0837] Data processing: Convert text data into audio data

[0838] Output: Audio data

[0839] Step 8:

[0840] The terminal reproduces the generated voice data from a speaker and responds to the user.

[0841] Input: Audio data

[0842] Output: Voice response "What lovely weather! Maybe we should go for a walk outside."

[0843] The above is a detailed description of the program processing for this system. By clearly indicating the specific operations performed at each step, as well as the inputs and outputs, the processing flow of the entire system becomes clearer.

[0844] (Application example 1)

[0845] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0846] In recent years, with the progress of aging society, caring for dementia patients has become a social issue. In particular, providing psychological support to dementia patients through daily conversations places a heavy burden on caregivers. Therefore, there is a need for effective conversation support systems to help dementia patients live their daily lives independently. Furthermore, there is a need for technology that can provide personalized content based on the individual preferences and past speech history of each patient.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0848] In this invention, the server includes means for converting a user's speech into text data using a speech recognition engine, means for transmitting the converted text data to a network server, means for generating a response text using generative artificial intelligence on the network server side, means for converting the generated response text into voice data using a speech synthesis engine, and means for providing personalized content such as audiobooks, music, podcasts, and news based on the user's speech history and preference history. This enables personalized dialogue and content provision for dementia patients, thereby supporting the patients' daily lives while reducing the burden on caregivers.

[0849] "User's voice input" refers to a voice message uttered by a dementia patient.

[0850] "Speech recognition engine" refers to a program or device for receiving voice data and converting it into character string data.

[0851] "Text data" refers to character string data converted by a voice recognition engine.

[0852] "Network server" refers to a central computing unit that processes data and generates responses via the cloud or the internet.

[0853] "Generative artificial intelligence" refers to an AI engine that uses pre-trained models to generate natural-sounding response text based on user input data.

[0854] "Response text" is character string data generated by the generative artificial intelligence, and includes a response to the user.

[0855] "Speech synthesis engine" refers to a program or device for converting text data into speech data.

[0856] "Audio Data" refers to audio files that are generated by a speech synthesis engine and can be played through a speaker.

[0857] "Speech history" refers to a record of voice inputs made by the user in the past.

[0858] "Preference history" refers to a past record of a user's preferences and hobbies.

[0859] "Audiobook" refers to digital content in which the contents of a book are presented in audio form.

[0860] "Music" refers to audio data recorded from musical instruments or singing.

[0861] A "podcast" is an audio program distributed over the Internet.

[0862] "News" refers to audio or text data that conveys current events or information.

[0863] "Personalization" refers to changing content based on an individual user's specific preferences and history.

[0864] The present invention is a personalized content delivery system for dementia patients, which aims to reduce the burden on caregivers and provide patients with a sense of security. The system executes a series of processes: receiving voice input from a user, converting it into text data, generating a response, and providing it to the user again as voice.

[0865] Specifically, the main components of the system include:

[0866] 1. Receiving audio input:

[0867] The user (a dementia patient) speaks to the system, and the voice input is received by a microphone installed in the device, which can be a smartphone, smart glasses, or a virtual reality head-mounted display.

[0868] 2. Speech Recognition:

[0869] The received voice input is converted into text data by a speech recognition engine, which can be a cloud-based service such as Google Speech Recognition API.

[0870] 3. Sending text data:

[0871] The converted text data is sent over the network to a server, which is a cloud server connected to the Internet.

[0872] 4. Generate response:

[0873] On the server side, the received text data is analyzed and an appropriate response text is generated using a generative AI model (e.g., GPT-3). The user's past speech history and preference history are also taken into consideration. For example, in response to the input text "Play me some old hit songs," the server generates a response such as "Playing hit songs from the 1960s."

[0874] 5. Sending a response text:

[0875] The generated response text is sent back to the terminal.

[0876] 6. Speech synthesis:

[0877] The terminal sends the received response text to a speech synthesis engine (such as Pyttsx3) and converts it into voice data.

[0878] 7. Audio Output:

[0879] The generated voice data is output to the user through the device's speaker. For example, a voice saying "Playing a hit song from the 1960s" is played.

[0880] Examples and prompts

[0881] As a concrete example, consider the case where a user says, "Play me an old hit song." When the user says, "Play me an old hit song," the speech recognition engine converts this into text data. The converted text data is then sent to the server, where a generative AI model generates a response such as, "Playing a hit song from the 1960s." The device receives this response, converts it into audio data using a speech synthesis engine, and plays it back.

[0882] An example prompt to use with a generative AI model might look like this:

[0883] User Input: Play me some old hit songs

[0884] Generate an appropriate response.

[0885] The system provides personalized content based on the user's speech history and preferences to support the daily lives of dementia patients. It also uses a cloud-based speech recognition and speech synthesis engine to improve processing power and provide flexible scalability.

[0886] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0887] Step 1:

[0888] The user speaks a voice input. The user's voice message is received through the device's microphone. This input is processed as raw voice data.

[0889] Step 2:

[0890] The device sends the received voice data to a voice recognition engine. Specifically, this voice data is sent to a cloud-based service such as Google Speech Recognition API, where it is converted into text data. The input here is voice data, and the output is text data. The voice recognition engine analyzes the voice data and recognizes its content as a string of characters.

[0891] Step 3:

[0892] The terminal sends the converted text data to the server via the network. At this stage, the input is text data and the output is string data sent to the server. The terminal makes the appropriate API call to send the text data.

[0893] Step 4:

[0894] The server passes the received text data to a generative AI model. The input is text data, and the output is the generated response text. The server uses a pre-trained generative AI model (e.g., GPT-3) to generate an appropriate response to the spoken content. For example, in response to the input "Play me an old hit song," a response such as "I'll play a hit song from the 1960s" is generated.

[0895] Step 5:

[0896] The server returns the generated response text to the terminal. The input here is the generated response text, and the output is the response text sent to the terminal. The server returns the generated response in real time.

[0897] Step 6:

[0898] The device sends the received response text to a speech synthesis engine. The input here is the response text, and the output is voice data. Specifically, a speech synthesis library such as Pyttsx3 is used to convert the text data into natural-sounding voice.

[0899] Step 7:

[0900] The device outputs the generated voice data to the user through a speaker. At this stage, the input is voice data from the speech synthesis engine, and the output is a voice message that the user can hear. For example, "Playing a hit song from the 1960s."

[0901] Through these steps, appropriate responses and content are provided in real time based on the user's speech.This system can provide personalized information support and psychological comfort to dementia patients.

[0902] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0903] This invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between patients and AI speakers. The system incorporates an emotion engine that recognizes the user's emotions, providing more personalized responses.

[0904] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[0905] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[0906] The device then sends this text data to the emotion engine to recognize the user's emotion. The emotion engine can detect emotions not only from text data but also from acoustic features such as the user's voice tone and speed. For example, if the user speaks excitedly, the emotion engine will recognize the user as being in an "excited" state.

[0907] The device then sends the converted text data and the recognized emotion data to the server. The server analyzes this data and generates an appropriate response text using a generative artificial intelligence (generative AI model). For example, if the input text is "What a nice day today" and the emotion data is "Excited," the server generates the response "What a nice day! Let's go for a walk outside!"

[0908] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Let's go for a walk outside!" is converted into natural-sounding speech and played back from the speaker.

[0909] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." Next, an emotion engine recognizes the user's emotion from their tone and speed of speech and determines it as "confused." The device then sends this text data and emotion data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday, everything's fine." Finally, the device synthesizes this response into voice and tells the user, "I had delicious curry yesterday, everything's fine."

[0910] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition, emotion, and speech synthesis engines are all cloud-based, enabling increased processing power and flexible scalability.

[0911] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[0912] The processing flow will be explained below.

[0913] Step 1:

[0914] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[0915] Step 2:

[0916] The terminal receives the user's voice input through a microphone.

[0917] Step 3:

[0918] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[0919] Step 4:

[0920] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0921] Step 5:

[0922] The emotion engine generates emotion data "joy," and the terminal transmits this data together with the text data to the server.

[0923] Step 6:

[0924] The server receives the text data "It's a nice day today" and the emotion data "joy."

[0925] Step 7:

[0926] The server uses generative artificial intelligence to generate a response text, "What lovely weather! Maybe we should go for a walk outside." based on the user's speech content and emotions.

[0927] Step 8:

[0928] The server returns the generated response text to the terminal.

[0929] Step 9:

[0930] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! Maybe we should go for a walk outside."

[0931] Step 10:

[0932] The terminal outputs the generated voice data to the user from a speaker.

[0933] For example, if a user says "I forgot what I had for dinner last night" the process would be:

[0934] Step 1:

[0935] The user speaks to the AI ​​speaker, saying, "I forgot what I had for dinner last night."

[0936] Step 2:

[0937] The terminal receives the user's voice input through a microphone.

[0938] Step 3:

[0939] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "I forgot what I had for dinner last night."

[0940] Step 4:

[0941] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[0942] Step 5:

[0943] The emotion engine generates emotion data "confusion," and the terminal transmits this data together with the text data to the server.

[0944] Step 6:

[0945] The server receives the text data "I forgot what I had for dinner last night" and the emotion data "confused."

[0946] Step 7:

[0947] The server uses generative artificial intelligence to generate a response text, "I had delicious curry yesterday, so it's fine," based on the user's speech content and emotions.

[0948] Step 8:

[0949] The server returns the generated response text to the terminal.

[0950] Step 9:

[0951] The response text received by the device is sent to a speech synthesis engine and converted into voice data such as "I had delicious curry yesterday, it's fine."

[0952] Step 10:

[0953] The terminal outputs the generated voice data to the user from a speaker.

[0954] Example 2

[0955] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0956] The purpose of this invention is to realize a system that reduces the burden on caregivers when interacting with dementia patients and provides more appropriate and personalized responses to the patients. Conventional voice dialogue systems can only generate simple responses without taking the user's emotions into consideration, resulting in a lack of communication that reflects the user's emotional state. This makes it difficult to provide a sense of security to dementia patients and build a relationship of trust.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0958] In this invention, the server includes means for sending received text data to an emotion engine and recognizing the user's emotion, means for using the recognized emotion data and text data with generative artificial intelligence to generate an appropriate response text, and means for converting the generated response text into voice data with a voice synthesis engine, thereby enabling the generation of an individualized response according to the user's emotion.

[0959] A "user" is a person with whom the system is used to interact.

[0960] "Voice input" is voice data that a user utters to the system.

[0961] A "terminal" is a device that receives a user's voice input and performs subsequent processing.

[0962] A "voice recognition engine" is a technology that converts received voice data into text data.

[0963] "Text data" is character string data that represents a user's voice input and is converted by a voice recognition engine.

[0964] An "emotion engine" is a technology that analyzes a user's emotions from text data and acoustic features and generates emotion data.

[0965] "Emotion data" is data that represents the user's emotion recognized by the emotion engine.

[0966] A "server" is a remote computing device that processes text and emotion data and generates and manages response text.

[0967] "Generative AI" is an AI technology that generates appropriate text based on input data.

[0968] "Response text" is text data that represents a response to a user, generated by generative artificial intelligence.

[0969] A "speech synthesis engine" is a technology that converts response text into voice data.

[0970] "Voice data" refers to a voice file generated by a voice synthesis engine and output to the user.

[0971] The present invention is a system that reduces the burden on caregivers of dementia patients and allows them to communicate with the patients via a voice interface. The system incorporates an emotion analysis engine that recognizes the user's emotions and provides more personalized responses. A systematic explanation for implementing the present invention is provided below.

[0972] System configuration

[0973] First, a device with a voice receiving function is required for users to input voice. This device captures the user's speech through a microphone. Specifically, this applies to a typical smart speaker, tablet, or smartphone.

[0974] Next, a cloud-based speech recognition engine, such as the Google Cloud Speech-to-Text API, is used to convert the voice input into text data. The received voice input is sent to this speech recognition engine, which converts it into text data.

[0975] Next, this text data must be sent to an emotion engine to analyze the user's emotions. For example, a cloud-based emotion analysis engine such as IBM Watson Tone Analyzer can be used. The emotion analysis engine detects the user's emotions from the text data and acoustic features such as the tone and speed of the voice.

[0976] After recognizing the emotion, the device sends the converted text data and the recognized emotion data to a server running a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response text.

[0977] The generated response text is sent back to the device, which then sends it to a cloud-based speech synthesis engine (e.g., Amazon Polly) to convert it into voice data, which is then played back through the speaker to respond to the user.

[0978] Specific examples

[0979] As a concrete example, consider a scenario in which a user says, "It's a nice day today." The user's speech is received by the device and sent to a speech recognition engine. The speech recognition engine converts this utterance into text data saying, "It's a nice day today." The emotion engine then analyzes this text data and recognizes that the user is "excited." The text data, along with the emotion data, is sent to the server, and the generative artificial intelligence generates a response saying, "It's really nice weather! Let's go outside for a walk!" Finally, this response text is converted into audio data by a speech synthesis engine and played back to the user.

[0980] As another example, consider the case where a user utters, "I forgot what I had for dinner last night." The speech recognition engine converts this into text data, "I forgot what I had for dinner last night," and the emotion engine recognizes this as "confused." On the server side, the generative AI generates a response, "I had delicious curry yesterday, so it's okay," which is then converted into speech by the speech synthesis engine. This response is then played back to the user.

[0981] Prompt Sentence Examples

[0982] User: "It's a nice day today."

[0983] Server: "What beautiful weather! Let's go outside for a walk!"

[0984] User: "I forgot what I had for dinner last night."

[0985] Server: "I had a delicious curry yesterday, it's fine."

[0986] As such, the present invention is a system that utilizes a cloud-based speech recognition engine, emotion analysis engine, generative artificial intelligence, and speech synthesis engine to provide personalized dialogue based on the user's emotions, thereby enabling more appropriate and reassuring dialogue for dementia patients and helping to reduce the burden on caregivers.

[0987] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0988] Step 1:

[0989] The user gives a voice input, for example, saying "It's a nice day today."

[0990] Input: User's voice

[0991] Output: None (audio is input to the device)

[0992] Specifically, the microphone of the terminal captures the user's voice.

[0993] Step 2:

[0994] The device receives the audio.

[0995] Input: User's voice

[0996] Output: Audio data

[0997] The received audio is temporarily stored on the terminal as audio data.

[0998] Step 3:

[0999] The device sends the voice input to a speech recognition engine, which converts it into text data.

[1000] Input: Audio data

[1001] Output: Text data

[1002] The device sends this voice data to a cloud-based speech recognition engine, which converts it into text data such as "It's nice weather today." The device uses the Google Cloud Speech-to-Text API.

[1003] Step 4:

[1004] The terminal sends the converted text data to an emotion engine to recognize the user's emotion.

[1005] Input: Text data

[1006] Output: Emotion data

[1007] The text data is sent to an emotion engine, which analyzes the user's emotion from the tone and speed of the voice. The emotion engine uses tools such as IBM Watson Tone Analyzer. For example, if the user is speaking excitedly, emotion data of "excited" is generated.

[1008] Step 5:

[1009] The device transmits text data and emotion data to the server.

[1010] Input: Text data, emotion data

[1011] Output: Data sent to the server

[1012] The converted text data and the recognized emotion data are transmitted from the terminal to the server.

[1013] Step 6:

[1014] The server uses a generative AI model to generate appropriate response text.

[1015] Input: Text data, emotion data

[1016] Output: Response text

[1017] The server analyzes the received text data and emotion data and generates a response text using generative artificial intelligence (e.g., OpenAI GPT-3). For example, based on the text "What a nice day today" and the emotion data "excitement," the response text generated is "What a nice day! Let's go for a walk outside!"

[1018] Step 7:

[1019] The server generates a response text and sends it to the terminal.

[1020] Input: Response text

[1021] Output: Data sent to the terminal

[1022] The generated response text is sent from the server to the terminal.

[1023] Step 8:

[1024] The terminal sends the response text to a speech synthesis engine, which converts it into voice data.

[1025] Input: Response text

[1026] Output: Audio data

[1027] The device sends the received response text to a cloud-based speech synthesis engine, which converts it into voice data. The speech synthesis engine is Amazon Polly or similar. For example, a response text like "What beautiful weather! Let's go for a walk outside!" is converted into voice data.

[1028] Step 9:

[1029] The terminal plays the generated audio to the user.

[1030] Input: Audio data

[1031] Output: Played audio

[1032] The device plays the generated voice data from a speaker and provides a response to the user. For example, the user can hear the voice response, "What a lovely day! Let's go for a walk outside!"

[1033] (Application example 2)

[1034] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] The purpose of this invention is to provide a system that reduces the burden on caregivers of dementia patients and enables more effective and personalized dialogue when patients use food delivery services. Dementia patients often have difficulty selecting and ordering food, so the challenge is to alleviate this difficulty and enable them to eat with peace of mind.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1037] In this invention, the server includes a means for receiving voice input, a means for converting the voice into text data, a means for recognizing the user's emotion with an emotion engine, and a means for generating suggested text based on the past speech history and emotion data, thereby enabling appropriate and personalized meal suggestions and delivery arrangements for dementia patients.

[1038] A "voice recognition engine" is software or hardware that converts a user's voice input into text data.

[1039] An "emotion engine" is software or hardware for recognizing a user's emotions from text data and voice characteristics.

[1040] "Generative AI" is an AI model that can generate appropriate responses or suggestions based on the data it receives.

[1041] "Text data" is character string information based on the user's voice input, converted by a voice recognition engine.

[1042] A "server" is a computer system that receives data, performs analytical processing, and sends the results to a client.

[1043] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[1044] The "past utterance history" is a record of utterances previously made by the user, and is data that can be used as reference for future conversations.

[1045] "Emotion data" is information about the user's emotional state as recognized by the emotion engine.

[1046] "Suggested text" is text data generated by generative artificial intelligence and containing content to be suggested to the user.

[1047] A "food delivery service" is a service that aims to deliver meals to a user's location.

[1048] A "client" is a user terminal that sends and receives data to and from a server.

[1049] This invention is a system that reduces the burden on caregivers of dementia patients and provides effective and personalized dialogue when patients use food delivery services. This system is composed of a combination of a speech recognition engine, an emotion engine, a generative artificial intelligence, a speech synthesis engine, etc.

[1050] System Components

[1051] 1. Receiving voice input

[1052] A device (e.g., a smart speaker) receives a user's voice input.

[1053] 2. Speech Recognition Engine

[1054] A speech recognition engine (e.g., a cloud-based speech recognition API) converts the received voice input into text data.

[1055] 3. Emotion recognition

[1056] An emotion engine (e.g., IBM Watson Tone Analyzer) is used to recognize user emotions from text data and voice features. This emotion data indicates the user's emotional state and is used to generate suggestions.

[1057] 4. Data transmission

[1058] The terminal transmits the converted text data and the recognized emotion data to the server.

[1059] 5. Proposal Generation Using Generative AI

[1060] The server uses a generative artificial intelligence model (e.g., GPT-4) to generate optimal suggested text based on past speech history and emotional data.

[1061] 6. Speech synthesis engine

[1062] The generated suggested text is converted into voice data by a speech synthesis engine (e.g., Amazon Polly).

[1063] 7. Audio Output

[1064] The terminal outputs the generated voice data to the user, and an actual conversation takes place.

[1065] Explanations and examples

[1066] When a user requests food delivery, the device may say, for example, "I don't know what I want to eat today." The device receives this voice input and converts it into text data using a speech recognition engine. This text data is "I don't know what I want to eat today." The emotion engine analyzes this text data and the tone and speed of the user's voice, and determines that the user is "confused."

[1067] Next, this text data and emotion data are sent to the server, where the server's generative AI refers to the user's past speech history (for example, the last time they ordered teriyaki salmon) and generates the suggested text, "How about teriyaki salmon?"

[1068] This suggested text is converted into voice data by a speech synthesis engine, and the device suggests to the user, "How about teriyaki salmon?"

[1069] For illustrative purposes, the following prompt sentence example will be used:

[1070] User utterance: I don't remember what I had for dinner last night. I'm hungry, but I don't know what to eat (emotional state: confused). Please suggest the best food based on that.

[1071] In this way, the system provides dementia patients with appropriate and personalized suggestions and support, making food delivery easier to use.

[1072] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1073] Step 1:

[1074] The device (e.g., a smart speaker) receives the user's voice input, which becomes the initial information for the system.

[1075] Step 2:

[1076] The device sends the received voice input to a speech recognition engine (e.g., a cloud-based speech recognition API) to convert it into text data. The input is voice data, and the output is text data. Here, the speech recognition engine analyzes the voice waveform and generates a corresponding text string.

[1077] Step 3:

[1078] The device sends the converted text data to an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data and speech characteristics, and the output is emotion data. The emotion engine analyzes the emotional state from the text content, speech tone, speed, etc.

[1079] Step 4:

[1080] The terminal transmits the recognized emotion data and text data to the server. The input is the text data and emotion data, and the output is the transmission data. Here, the terminal performs the data transmission operation.

[1081] Step 5:

[1082] The server analyzes the received text data and emotion data and generates optimal suggested text using a generative AI model (e.g., GPT-4). The input is text data, emotion data, and past speech history, and the output is suggested text. The generative AI model analyzes this data and generates an appropriate response.

[1083] Step 6:

[1084] The server sends the generated suggested text to the terminal. The input is the suggested text and the output is the transmitted data. The server performs data communication and sends the generated text back to the terminal.

[1085] Step 7:

[1086] The device sends the received suggested text to a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The input is the suggested text and the output is speech data. The speech synthesis engine analyzes the text content and generates natural-sounding speech.

[1087] Step 8:

[1088] The device plays the generated voice data from the speaker and outputs it to the user. The input is voice data and the output is voice output. Specifically, the voice is played through the device's speaker and the suggestion is delivered to the user.

[1089] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1090] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1091] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1092] [Fourth embodiment]

[1093] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1094] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1095] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1096] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1097] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1098] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1099] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1100] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1101] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1102] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1103] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1104] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1105] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1106] ---

[1107] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[1108] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[1109] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[1110] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., a generative AI model). For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go outside for a walk."

[1111] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back from the speaker.

[1112] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and tells the user, "I had delicious curry yesterday."

[1113] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[1114] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[1115] The processing flow will be explained below.

[1116] Step 1:

[1117] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[1118] Step 2:

[1119] The terminal receives the user's voice input through a microphone.

[1120] Step 3:

[1121] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[1122] Step 4:

[1123] The terminal sends the converted text data to the server as an HTTP POST request.

[1124] Step 5:

[1125] The server analyzes the received text data, "It's nice weather today," and uses generative artificial intelligence to generate the response text, "It's really nice weather! Maybe it would be nice to go for a walk outside."

[1126] Step 6:

[1127] The server generates a response text and sends it back to the terminal as an HTTP response.

[1128] Step 7:

[1129] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! It might be nice to go for a walk outside."

[1130] Step 8:

[1131] The terminal makes the generated voice data available to the user through a speaker.

[1132] Example 1

[1133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1134] The problem that this invention aims to solve is to reduce the burden on caregivers for dementia patients and to improve the effectiveness of dialogue between patients and AI speakers. Another problem is to achieve highly accurate speech recognition and natural voice response generation to improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, we aim to provide a system that can be adapted to a wide range of users.

[1135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1136] In this invention, the server includes means for receiving the user's voice using a highly sensitive noise-canceling microphone, means for the speech recognition engine to always use the latest algorithms and be updated to improve the accuracy of proper noun recognition, means for the generative artificial intelligence to refer to the user's past speech history and generate personalized responses, and means for the speech synthesis engine to support multiple languages ​​and generate responses in different languages ​​and dialects with natural pronunciation. This enables highly accurate speech recognition and natural voice responses, and the personalized responses improve the user experience. Furthermore, by supporting multiple languages ​​and dialects, a system that can be adapted to a wide range of users is realized.

[1137] "User" refers to a person who uses the system.

[1138] "Voice input" refers to speech uttered by a user.

[1139] "Means for receiving" refers to a combination of hardware and software for capturing audio input.

[1140] "Speech recognition engine" refers to software or a service for converting voice data into text data.

[1141] "Text data" refers to character string data converted by a voice recognition engine.

[1142] "Server" refers to a remote computing device for analyzing text data generated from speech input and generating appropriate responses.

[1143] "Generative AI" refers to an AI model that generates appropriate responses based on text data.

[1144] "Response text" refers to the string data of a response generated by generative artificial intelligence.

[1145] "Speech synthesis engine" refers to software or a service for converting text data into speech data.

[1146] "Voice data" refers to sound data generated by a speech synthesis engine.

[1147] "Output means" refers to a combination of hardware and software, such as a speaker, that allows the user to hear the audio data.

[1148] A "noise-canceling microphone" refers to a microphone equipped with technology that eliminates ambient noise.

[1149] "Proper noun recognition accuracy" refers to the accuracy in recognizing proper nouns such as specific people's names, place names, and brand names.

[1150] "Means for being updated" refers to the update mechanism for keeping the software and algorithms used up to date.

[1151] "Utterance history" refers to a record of the contents of utterances made by a user in the past.

[1152] "Personalized responses" refer to customized responses generated based on the characteristics and history of individual users.

[1153] "Multilingual" refers to the ability to support multiple languages.

[1154] A "dialect" refers to the use of words and pronunciation that is unique to a particular region or community.

[1155] The present invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between the patient and an AI speaker. This system includes the following elements:

[1156] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the device, which is equipped with a highly sensitive noise-canceling microphone. This microphone filters out ambient noise and captures accurate voice data.

[1157] The device then sends the received voice input to a cloud-based speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. This speech recognition engine always uses the latest algorithms and is regularly updated to improve recognition accuracy, especially for proper nouns. For example, the user's speech, "It's a nice day today," is converted into text data, "It's a nice day today."

[1158] The device then sends the converted text data to the server. The server analyzes the received text data and generates an appropriate response text using generative artificial intelligence (e.g., an OpenAI model). This generative artificial intelligence references the user's past speech history to generate a personalized response. For example, in response to the input text "It's a nice day today," the server generates the response "It's really nice weather! Maybe it would be nice to go for a walk outside."

[1159] The generated response text is then sent back to the device. The device then sends the received response text to a cloud-based speech synthesis engine (e.g., Amazon Polly), which converts it into voice data. This speech synthesis engine supports multiple languages ​​and can generate natural-sounding responses in different languages ​​and dialects. For example, the text "What a lovely day! Maybe we should go for a walk outside." is converted into natural-sounding speech and played back through the speaker.

[1160] As a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data saying, "I forgot what I had for dinner last night." The device then sends this text data to a server, which uses generative artificial intelligence to generate a response saying, "I had delicious curry yesterday." Finally, the device synthesizes this response into speech and speaks to the user, saying, "I had delicious curry yesterday."

[1161] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition and synthesis engines are cloud-based, enabling increased processing power and flexible scalability.

[1162] An example of a prompt sentence that can be input to a generative AI model is "Tell me about what you had for dinner last night."

[1163] As described above, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[1164] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1165] Step 1:

[1166] The user speaks to the AI ​​speaker, for example, saying, "It's nice weather today."

[1167] Input: User's spoken utterance "It's a nice day today."

[1168] Output: Captured as audio data

[1169] Step 2:

[1170] The device receives the user's voice using a highly sensitive noise-canceling microphone, which filters out ambient noise and captures accurate voice data.

[1171] Input: User's voice data

[1172] Output: Clear audio data after noise removal

[1173] Step 3:

[1174] The device sends the received voice data to a cloud-based speech recognition engine (e.g., a speech recognition API) that converts the speech into text data. This speech recognition engine uses the latest algorithms to improve the accuracy of proper noun recognition.

[1175] Input: Clear audio data after noise removal

[1176] Data processing: Converting voice data into text data

[1177] Output: Text data "It's a nice day today."

[1178] Step 4:

[1179] The terminal transmits the converted text data to the server.

[1180] Input: Text data "It's a nice day today"

[1181] Output: Send text data to the server

[1182] Step 5:

[1183] The server analyzes the received text data and generates a response text using generative artificial intelligence (e.g., a generative AI model). It references past speech history to generate a personalized response.

[1184] Input: Text data "It's a nice day today"

[1185] Data calculations: term analysis, context understanding, and reference to past speech history

[1186] Output: Response text "What lovely weather! Maybe we should go for a walk outside."

[1187] Step 6:

[1188] The server returns the generated response text to the terminal.

[1189] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[1190] Output: Response text sent to terminal

[1191] Step 7:

[1192] The device sends the received response text to a cloud-based speech synthesis engine (e.g., speech synthesis API) and converts it into voice data. This speech synthesis engine supports multiple languages ​​and generates voice with natural pronunciation.

[1193] Input: Response text "What lovely weather! Maybe we should go for a walk outside."

[1194] Data processing: Convert text data into audio data

[1195] Output: Audio data

[1196] Step 8:

[1197] The terminal reproduces the generated voice data from a speaker and responds to the user.

[1198] Input: Audio data

[1199] Output: Voice response "What lovely weather! Maybe we should go for a walk outside."

[1200] The above is a detailed description of the program processing for this system. By clearly indicating the specific operations performed at each step, as well as the inputs and outputs, the processing flow of the entire system becomes clearer.

[1201] (Application example 1)

[1202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] In recent years, with the progress of aging society, caring for dementia patients has become a social issue. In particular, providing psychological support to dementia patients through daily conversations places a heavy burden on caregivers. Therefore, there is a need for effective conversation support systems to help dementia patients live their daily lives independently. Furthermore, there is a need for technology that can provide personalized content based on the individual preferences and past speech history of each patient.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1205] In this invention, the server includes means for converting a user's speech into text data using a speech recognition engine, means for transmitting the converted text data to a network server, means for generating a response text using generative artificial intelligence on the network server side, means for converting the generated response text into voice data using a speech synthesis engine, and means for providing personalized content such as audiobooks, music, podcasts, and news based on the user's speech history and preference history. This enables personalized dialogue and content provision for dementia patients, thereby supporting the patients' daily lives while reducing the burden on caregivers.

[1206] "User's voice input" refers to a voice message uttered by a dementia patient.

[1207] "Speech recognition engine" refers to a program or device for receiving voice data and converting it into character string data.

[1208] "Text data" refers to character string data converted by a voice recognition engine.

[1209] "Network server" refers to a central computing unit that processes data and generates responses via the cloud or the internet.

[1210] "Generative artificial intelligence" refers to an AI engine that uses pre-trained models to generate natural-sounding response text based on user input data.

[1211] "Response text" is character string data generated by the generative artificial intelligence, and includes a response to the user.

[1212] "Speech synthesis engine" refers to a program or device for converting text data into speech data.

[1213] "Audio Data" refers to audio files that are generated by a speech synthesis engine and can be played through a speaker.

[1214] "Speech history" refers to a record of voice inputs made by the user in the past.

[1215] "Preference history" refers to a past record of a user's preferences and hobbies.

[1216] "Audiobook" refers to digital content in which the contents of a book are presented in audio form.

[1217] "Music" refers to audio data recorded from musical instruments or singing.

[1218] A "podcast" is an audio program distributed over the Internet.

[1219] "News" refers to audio or text data that conveys current events or information.

[1220] "Personalization" refers to changing content based on an individual user's specific preferences and history.

[1221] The present invention is a personalized content delivery system for dementia patients, which aims to reduce the burden on caregivers and provide patients with a sense of security. The system executes a series of processes: receiving voice input from a user, converting it into text data, generating a response, and providing it to the user again as voice.

[1222] Specifically, the main components of the system include:

[1223] 1. Receiving audio input:

[1224] The user (a dementia patient) speaks to the system, and the voice input is received by a microphone installed in the device, which can be a smartphone, smart glasses, or a virtual reality head-mounted display.

[1225] 2. Speech Recognition:

[1226] The received voice input is converted into text data by a speech recognition engine, which can be a cloud-based service such as Google Speech Recognition API.

[1227] 3. Sending text data:

[1228] The converted text data is sent over the network to a server, which is a cloud server connected to the Internet.

[1229] 4. Generate response:

[1230] On the server side, the received text data is analyzed and an appropriate response text is generated using a generative AI model (e.g., GPT-3). The user's past speech history and preference history are also taken into consideration. For example, in response to the input text "Play me some old hit songs," the server generates a response such as "Playing hit songs from the 1960s."

[1231] 5. Sending a response text:

[1232] The generated response text is sent back to the terminal.

[1233] 6. Speech synthesis:

[1234] The terminal sends the received response text to a speech synthesis engine (such as Pyttsx3) and converts it into voice data.

[1235] 7. Audio Output:

[1236] The generated voice data is output to the user through the device's speaker. For example, a voice saying "Playing a hit song from the 1960s" is played.

[1237] Examples and prompts

[1238] As a concrete example, consider the case where a user says, "Play me an old hit song." When the user says, "Play me an old hit song," the speech recognition engine converts this into text data. The converted text data is then sent to the server, where a generative AI model generates a response such as, "Playing a hit song from the 1960s." The device receives this response, converts it into audio data using a speech synthesis engine, and plays it back.

[1239] An example prompt to use with a generative AI model might look like this:

[1240] User Input: Play me some old hit songs

[1241] Generate an appropriate response.

[1242] The system provides personalized content based on the user's speech history and preferences to support the daily lives of dementia patients. It also uses a cloud-based speech recognition and speech synthesis engine to improve processing power and provide flexible scalability.

[1243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1244] Step 1:

[1245] The user speaks a voice input. The user's voice message is received through the device's microphone. This input is processed as raw voice data.

[1246] Step 2:

[1247] The device sends the received voice data to a voice recognition engine. Specifically, this voice data is sent to a cloud-based service such as Google Speech Recognition API, where it is converted into text data. The input here is voice data, and the output is text data. The voice recognition engine analyzes the voice data and recognizes its content as a string of characters.

[1248] Step 3:

[1249] The terminal sends the converted text data to the server via the network. At this stage, the input is text data and the output is string data sent to the server. The terminal makes the appropriate API call to send the text data.

[1250] Step 4:

[1251] The server passes the received text data to a generative AI model. The input is text data, and the output is the generated response text. The server uses a pre-trained generative AI model (e.g., GPT-3) to generate an appropriate response to the spoken content. For example, in response to the input "Play me an old hit song," a response such as "I'll play a hit song from the 1960s" is generated.

[1252] Step 5:

[1253] The server returns the generated response text to the terminal. The input here is the generated response text, and the output is the response text sent to the terminal. The server returns the generated response in real time.

[1254] Step 6:

[1255] The device sends the received response text to a speech synthesis engine. The input here is the response text, and the output is voice data. Specifically, a speech synthesis library such as Pyttsx3 is used to convert the text data into natural-sounding voice.

[1256] Step 7:

[1257] The device outputs the generated voice data to the user through a speaker. At this stage, the input is voice data from the speech synthesis engine, and the output is a voice message that the user can hear. For example, "Playing a hit song from the 1960s."

[1258] Through these steps, appropriate responses and content are provided in real time based on the user's speech.This system can provide personalized information support and psychological comfort to dementia patients.

[1259] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1260] This invention is a system for reducing the burden on caregivers of dementia patients and for enabling effective dialogue between patients and AI speakers. The system incorporates an emotion engine that recognizes the user's emotions, providing more personalized responses.

[1261] First, the user speaks to the AI ​​speaker. For example, the user might say, "It's a nice day today." This voice input is received by the AI ​​speaker device.

[1262] The device sends the received voice input to a voice recognition engine, which converts it into text data. A commonly available voice recognition service (e.g., a cloud-based voice recognition API) can be used as the voice recognition engine. For example, a user's utterance "It's a nice day today" is converted into text data such as "It's a nice day today."

[1263] The device then sends this text data to the emotion engine to recognize the user's emotion. The emotion engine can detect emotions not only from text data but also from acoustic features such as the user's voice tone and speed. For example, if the user speaks excitedly, the emotion engine will recognize the user as being in an "excited" state.

[1264] The device then sends the converted text data and the recognized emotion data to the server. The server analyzes this data and generates an appropriate response text using a generative artificial intelligence (generative AI model). For example, if the input text is "What a nice day today" and the emotion data is "Excited," the server generates the response "What a nice day! Let's go for a walk outside!"

[1265] The generated response text is sent back to the device. The device then sends the received response text to a speech synthesis engine, which converts it into voice data. A commonly available speech synthesis service (e.g., a cloud-based speech synthesis API) can be used as the speech synthesis engine. For example, the text "What a lovely day! Let's go for a walk outside!" is converted into natural-sounding speech and played back from the speaker.

[1266] To give a specific example, a similar process is applied when a user says, "I forgot what I had for dinner last night." First, a speech recognition engine converts this utterance into text data: "I forgot what I had for dinner last night." Next, an emotion engine recognizes the user's emotion from their tone and speed of speech and determines it as "confused." The device then sends this text data and emotion data to a server, which uses generative artificial intelligence to generate a response: "I had delicious curry yesterday, everything's fine." Finally, the device synthesizes this response into voice and tells the user, "I had delicious curry yesterday, everything's fine."

[1267] The system can also generate responses based on the user's past speech history, enabling more personalized interactions. The speech recognition, emotion, and speech synthesis engines are all cloud-based, enabling increased processing power and flexible scalability.

[1268] In this way, the present invention realizes a system that reduces the burden on caregivers and provides patients with a sense of security through dialogue with dementia patients.

[1269] The processing flow will be explained below.

[1270] Step 1:

[1271] The user speaks to the AI ​​speaker, saying, "It's nice weather today."

[1272] Step 2:

[1273] The terminal receives the user's voice input through a microphone.

[1274] Step 3:

[1275] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "It's nice weather today."

[1276] Step 4:

[1277] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[1278] Step 5:

[1279] The emotion engine generates emotion data "joy," and the terminal transmits this data together with the text data to the server.

[1280] Step 6:

[1281] The server receives the text data "It's a nice day today" and the emotion data "joy."

[1282] Step 7:

[1283] The server uses generative artificial intelligence to generate a response text, "What lovely weather! Maybe we should go for a walk outside." based on the user's speech content and emotions.

[1284] Step 8:

[1285] The server returns the generated response text to the terminal.

[1286] Step 9:

[1287] The device sends the received response text to a speech synthesis engine, which converts it into voice data such as, "What lovely weather! Maybe we should go for a walk outside."

[1288] Step 10:

[1289] The terminal outputs the generated voice data to the user from a speaker.

[1290] For example, if a user says "I forgot what I had for dinner last night" the process would be:

[1291] Step 1:

[1292] The user speaks to the AI ​​speaker, saying, "I forgot what I had for dinner last night."

[1293] Step 2:

[1294] The terminal receives the user's voice input through a microphone.

[1295] Step 3:

[1296] The voice input received by the device is sent to a voice recognition engine and converted into text data such as "I forgot what I had for dinner last night."

[1297] Step 4:

[1298] The device sends the converted text data to an emotion engine, which recognizes emotions from the user's tone and speed of speech.

[1299] Step 5:

[1300] The emotion engine generates emotion data "confusion," and the terminal transmits this data together with the text data to the server.

[1301] Step 6:

[1302] The server receives the text data "I forgot what I had for dinner last night" and the emotion data "confused."

[1303] Step 7:

[1304] The server uses generative artificial intelligence to generate a response text, "I had delicious curry yesterday, so it's fine," based on the user's speech content and emotions.

[1305] Step 8:

[1306] The server returns the generated response text to the terminal.

[1307] Step 9:

[1308] The response text received by the device is sent to a speech synthesis engine and converted into voice data such as "I had delicious curry yesterday, it's fine."

[1309] Step 10:

[1310] The terminal outputs the generated voice data to the user from a speaker.

[1311] Example 2

[1312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1313] The purpose of this invention is to realize a system that reduces the burden on caregivers when interacting with dementia patients and provides more appropriate and personalized responses to the patients. Conventional voice dialogue systems can only generate simple responses without taking the user's emotions into consideration, resulting in a lack of communication that reflects the user's emotional state. This makes it difficult to provide a sense of security to dementia patients and build a relationship of trust.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1315] In this invention, the server includes means for sending received text data to an emotion engine and recognizing the user's emotion, means for using the recognized emotion data and text data with generative artificial intelligence to generate an appropriate response text, and means for converting the generated response text into voice data with a voice synthesis engine, thereby enabling the generation of an individualized response according to the user's emotion.

[1316] A "user" is a person with whom the system is used to interact.

[1317] "Voice input" is voice data that a user utters to the system.

[1318] A "terminal" is a device that receives a user's voice input and performs subsequent processing.

[1319] A "voice recognition engine" is a technology that converts received voice data into text data.

[1320] "Text data" is character string data that represents a user's voice input and is converted by a voice recognition engine.

[1321] An "emotion engine" is a technology that analyzes a user's emotions from text data and acoustic features and generates emotion data.

[1322] "Emotion data" is data that represents the user's emotion recognized by the emotion engine.

[1323] A "server" is a remote computing device that processes text and emotion data and generates and manages response text.

[1324] "Generative AI" is an AI technology that generates appropriate text based on input data.

[1325] "Response text" is text data that represents a response to a user, generated by generative artificial intelligence.

[1326] A "speech synthesis engine" is a technology that converts response text into voice data.

[1327] "Voice data" refers to a voice file generated by a voice synthesis engine and output to the user.

[1328] The present invention is a system that reduces the burden on caregivers of dementia patients and allows them to communicate with the patients via a voice interface. The system incorporates an emotion analysis engine that recognizes the user's emotions and provides more personalized responses. A systematic explanation for implementing the present invention is provided below.

[1329] System configuration

[1330] First, a device with a voice receiving function is required for users to input voice. This device captures the user's speech through a microphone. Specifically, this applies to a typical smart speaker, tablet, or smartphone.

[1331] Next, a cloud-based speech recognition engine, such as the Google Cloud Speech-to-Text API, is used to convert the voice input into text data. The received voice input is sent to this speech recognition engine, which converts it into text data.

[1332] Next, this text data must be sent to an emotion engine to analyze the user's emotions. For example, a cloud-based emotion analysis engine such as IBM Watson Tone Analyzer can be used. The emotion analysis engine detects the user's emotions from the text data and acoustic features such as the tone and speed of the voice.

[1333] After recognizing the emotion, the device sends the converted text data and the recognized emotion data to a server running a generative AI model (e.g., OpenAI GPT-3), which generates an appropriate response text.

[1334] The generated response text is sent back to the device, which then sends it to a cloud-based speech synthesis engine (e.g., Amazon Polly) to convert it into voice data, which is then played back through the speaker to respond to the user.

[1335] Specific examples

[1336] As a concrete example, consider a scenario in which a user says, "It's a nice day today." The user's speech is received by the device and sent to a speech recognition engine. The speech recognition engine converts this utterance into text data saying, "It's a nice day today." The emotion engine then analyzes this text data and recognizes that the user is "excited." The text data, along with the emotion data, is sent to the server, and the generative artificial intelligence generates a response saying, "It's really nice weather! Let's go outside for a walk!" Finally, this response text is converted into audio data by a speech synthesis engine and played back to the user.

[1337] As another example, consider the case where a user utters, "I forgot what I had for dinner last night." The speech recognition engine converts this into text data, "I forgot what I had for dinner last night," and the emotion engine recognizes this as "confused." On the server side, the generative AI generates a response, "I had delicious curry yesterday, so it's okay," which is then converted into speech by the speech synthesis engine. This response is then played back to the user.

[1338] Prompt Sentence Examples

[1339] User: "It's a nice day today."

[1340] Server: "What beautiful weather! Let's go outside for a walk!"

[1341] User: "I forgot what I had for dinner last night."

[1342] Server: "I had a delicious curry yesterday, it's fine."

[1343] As such, the present invention is a system that utilizes a cloud-based speech recognition engine, emotion analysis engine, generative artificial intelligence, and speech synthesis engine to provide personalized dialogue based on the user's emotions, thereby enabling more appropriate and reassuring dialogue for dementia patients and helping to reduce the burden on caregivers.

[1344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1345] Step 1:

[1346] The user gives a voice input, for example, saying "It's a nice day today."

[1347] Input: User's voice

[1348] Output: None (audio is input to the device)

[1349] Specifically, the microphone of the terminal captures the user's voice.

[1350] Step 2:

[1351] The device receives the audio.

[1352] Input: User's voice

[1353] Output: Audio data

[1354] The received audio is temporarily stored on the terminal as audio data.

[1355] Step 3:

[1356] The device sends the voice input to a speech recognition engine, which converts it into text data.

[1357] Input: Audio data

[1358] Output: Text data

[1359] The device sends this voice data to a cloud-based speech recognition engine, which converts it into text data such as "It's nice weather today." The device uses the Google Cloud Speech-to-Text API.

[1360] Step 4:

[1361] The terminal sends the converted text data to an emotion engine to recognize the user's emotion.

[1362] Input: Text data

[1363] Output: Emotion data

[1364] The text data is sent to an emotion engine, which analyzes the user's emotion from the tone and speed of the voice. The emotion engine uses tools such as IBM Watson Tone Analyzer. For example, if the user is speaking excitedly, emotion data of "excited" is generated.

[1365] Step 5:

[1366] The device transmits text data and emotion data to the server.

[1367] Input: Text data, emotion data

[1368] Output: Data sent to the server

[1369] The converted text data and the recognized emotion data are transmitted from the terminal to the server.

[1370] Step 6:

[1371] The server uses a generative AI model to generate appropriate response text.

[1372] Input: Text data, emotion data

[1373] Output: Response text

[1374] The server analyzes the received text data and emotion data and generates a response text using generative artificial intelligence (e.g., OpenAI GPT-3). For example, based on the text "What a nice day today" and the emotion data "excitement," the response text generated is "What a nice day! Let's go for a walk outside!"

[1375] Step 7:

[1376] The server generates a response text and sends it to the terminal.

[1377] Input: Response text

[1378] Output: Data sent to the terminal

[1379] The generated response text is sent from the server to the terminal.

[1380] Step 8:

[1381] The terminal sends the response text to a speech synthesis engine, which converts it into voice data.

[1382] Input: Response text

[1383] Output: Audio data

[1384] The device sends the received response text to a cloud-based speech synthesis engine, which converts it into voice data. The speech synthesis engine is Amazon Polly or similar. For example, a response text like "What beautiful weather! Let's go for a walk outside!" is converted into voice data.

[1385] Step 9:

[1386] The terminal plays the generated audio to the user.

[1387] Input: Audio data

[1388] Output: Played audio

[1389] The device plays the generated voice data from a speaker and provides a response to the user. For example, the user can hear the voice response, "What a lovely day! Let's go for a walk outside!"

[1390] (Application example 2)

[1391] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1392] The purpose of this invention is to provide a system that reduces the burden on caregivers of dementia patients and enables more effective and personalized dialogue when patients use food delivery services. Dementia patients often have difficulty selecting and ordering food, so the challenge is to alleviate this difficulty and enable them to eat with peace of mind.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1394] In this invention, the server includes a means for receiving voice input, a means for converting the voice into text data, a means for recognizing the user's emotion with an emotion engine, and a means for generating suggested text based on the past speech history and emotion data, thereby enabling appropriate and personalized meal suggestions and delivery arrangements for dementia patients.

[1395] A "voice recognition engine" is software or hardware that converts a user's voice input into text data.

[1396] An "emotion engine" is software or hardware for recognizing a user's emotions from text data and voice characteristics.

[1397] "Generative AI" is an AI model that can generate appropriate responses or suggestions based on the data it receives.

[1398] "Text data" is character string information based on the user's voice input, converted by a voice recognition engine.

[1399] A "server" is a computer system that receives data, performs analytical processing, and sends the results to a client.

[1400] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[1401] The "past utterance history" is a record of utterances previously made by the user, and is data that can be used as reference for future conversations.

[1402] "Emotion data" is information about the user's emotional state as recognized by the emotion engine.

[1403] "Suggested text" is text data generated by generative artificial intelligence and containing content to be suggested to the user.

[1404] A "food delivery service" is a service that aims to deliver meals to a user's location.

[1405] A "client" is a user terminal that sends and receives data to and from a server.

[1406] This invention is a system that reduces the burden on caregivers of dementia patients and provides effective and personalized dialogue when patients use food delivery services. This system is composed of a combination of a speech recognition engine, an emotion engine, a generative artificial intelligence, a speech synthesis engine, etc.

[1407] System Components

[1408] 1. Receiving voice input

[1409] A device (e.g., a smart speaker) receives a user's voice input.

[1410] 2. Speech Recognition Engine

[1411] A speech recognition engine (e.g., a cloud-based speech recognition API) converts the received voice input into text data.

[1412] 3. Emotion recognition

[1413] An emotion engine (e.g., IBM Watson Tone Analyzer) is used to recognize user emotions from text data and voice features. This emotion data indicates the user's emotional state and is used to generate suggestions.

[1414] 4. Data transmission

[1415] The terminal transmits the converted text data and the recognized emotion data to the server.

[1416] 5. Proposal Generation Using Generative AI

[1417] The server uses a generative artificial intelligence model (e.g., GPT-4) to generate optimal suggested text based on past speech history and emotional data.

[1418] 6. Speech synthesis engine

[1419] The generated suggested text is converted into voice data by a speech synthesis engine (e.g., Amazon Polly).

[1420] 7. Audio Output

[1421] The terminal outputs the generated voice data to the user, and an actual conversation takes place.

[1422] Explanations and examples

[1423] When a user requests food delivery, the device may say, for example, "I don't know what I want to eat today." The device receives this voice input and converts it into text data using a speech recognition engine. This text data is "I don't know what I want to eat today." The emotion engine analyzes this text data and the tone and speed of the user's voice, and determines that the user is "confused."

[1424] Next, this text data and emotion data are sent to the server, where the server's generative AI refers to the user's past speech history (for example, the last time they ordered teriyaki salmon) and generates the suggested text, "How about teriyaki salmon?"

[1425] This suggested text is converted into voice data by a speech synthesis engine, and the device suggests to the user, "How about teriyaki salmon?"

[1426] For illustrative purposes, the following prompt sentence example will be used:

[1427] User utterance: I don't remember what I had for dinner last night. I'm hungry, but I don't know what to eat (emotional state: confused). Please suggest the best food based on that.

[1428] In this way, the system provides dementia patients with appropriate and personalized suggestions and support, making food delivery easier to use.

[1429] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1430] Step 1:

[1431] The device (e.g., a smart speaker) receives the user's voice input, which becomes the initial information for the system.

[1432] Step 2:

[1433] The device sends the received voice input to a speech recognition engine (e.g., a cloud-based speech recognition API) to convert it into text data. The input is voice data, and the output is text data. Here, the speech recognition engine analyzes the voice waveform and generates a corresponding text string.

[1434] Step 3:

[1435] The device sends the converted text data to an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data and speech characteristics, and the output is emotion data. The emotion engine analyzes the emotional state from the text content, speech tone, speed, etc.

[1436] Step 4:

[1437] The terminal transmits the recognized emotion data and text data to the server. The input is the text data and emotion data, and the output is the transmission data. Here, the terminal performs the data transmission operation.

[1438] Step 5:

[1439] The server analyzes the received text data and emotion data and generates optimal suggested text using a generative AI model (e.g., GPT-4). The input is text data, emotion data, and past speech history, and the output is suggested text. The generative AI model analyzes this data and generates an appropriate response.

[1440] Step 6:

[1441] The server sends the generated suggested text to the terminal. The input is the suggested text and the output is the transmitted data. The server performs data communication and sends the generated text back to the terminal.

[1442] Step 7:

[1443] The device sends the received suggested text to a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The input is the suggested text and the output is speech data. The speech synthesis engine analyzes the text content and generates natural-sounding speech.

[1444] Step 8:

[1445] The device plays the generated voice data from the speaker and outputs it to the user. The input is voice data and the output is voice output. Specifically, the voice is played through the device's speaker and the suggestion is delivered to the user.

[1446] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1447] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1448] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1449] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1450] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1451] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1452] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1453] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1454] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1455] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1456] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1457] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1458] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1459] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1460] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1461] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1462] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1463] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1464] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1465] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1466] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1467] The following is further disclosed regarding the above embodiment.

[1468] (Claim 1)

[1469] means for receiving a user's voice input;

[1470] means for converting the received voice input into text data using a voice recognition engine;

[1471] means for transmitting the converted text data to a server;

[1472] A means for generating a response text using generative artificial intelligence on the server side;

[1473] means for receiving the generated response text from the server;

[1474] A means for converting the received response text into voice data using a voice synthesis engine;

[1475] means for outputting the generated voice data to a user;

[1476] A system including:

[1477] (Claim 2)

[1478] 2. The system of claim 1, wherein the generative artificial intelligence further comprises means for generating a response text based on a user's past speech history.

[1479] (Claim 3)

[1480] The system of claim 1 , wherein the speech recognition engine and the speech synthesis engine are cloud-based.

[1481] "Example 1"

[1482] (Claim 1)

[1483] means for receiving a user's voice input;

[1484] means for converting the received voice input into text data using a voice recognition engine;

[1485] means for transmitting the converted text data to a server;

[1486] A means for generating a response text using generative artificial intelligence on the server side;

[1487] means for receiving the generated response text from the server;

[1488] A means for converting the received response text into voice data using a voice synthesis engine;

[1489] means for outputting the generated voice data to a user;

[1490] means for receiving a user's voice using a sensitive noise-canceling microphone;

[1491] The voice recognition engine is constantly updated to use the latest algorithms and improve the accuracy of proper noun recognition.

[1492] A means for the generative artificial intelligence to refer to the user's past speech history and generate a personalized response;

[1493] The speech synthesis engine is multilingual, providing a means to generate natural-sounding responses in different languages ​​and dialects;

[1494] A system including:

[1495] (Claim 2)

[1496] 2. The system of claim 1, wherein the generative artificial intelligence further comprises means for generating a response text based on a user's past speech history.

[1497] (Claim 3)

[1498] The system of claim 1 , wherein the speech recognition engine and the speech synthesis engine are cloud-based.

[1499] "Application Example 1"

[1500] (Claim 1)

[1501] means for receiving a user's voice input;

[1502] means for converting the received voice input into text data using a voice recognition engine;

[1503] means for transmitting the converted text data to a network server;

[1504] A means for generating a response text using generative artificial intelligence on the network server side;

[1505] means for receiving the generated response text from the network server;

[1506] A means for converting the received response text into voice data using a voice synthesis engine;

[1507] means for outputting the generated voice data to a user;

[1508] A means for providing personalized content such as audiobooks, music, podcasts, and news based on a user's speech history and preference history;

[1509] A system including:

[1510] (Claim 2)

[1511] 2. The system of claim 1, wherein the generative artificial intelligence further comprises means for generating response text based on the user's past speech history and preference history.

[1512] (Claim 3)

[1513] The system of claim 1 , wherein the speech recognition engine and the speech synthesis engine are cloud-based.

[1514] "Example 2: Combining Emotion Engines"

[1515] (Claim 1)

[1516] means for receiving a user's voice input;

[1517] means for converting the received voice input into text data using a voice recognition engine;

[1518] a means for transmitting the converted text data to an emotion engine to recognize the emotion of the user;

[1519] means for transmitting the recognized emotion data and text data to a server;

[1520] A means for generating a response text using generative artificial intelligence on the server side;

[1521] means for receiving the generated response text from the server;

[1522] A means for converting the received response text into voice data using a voice synthesis engine;

[1523] means for outputting the generated voice data to a user;

[1524] A system including:

[1525] (Claim 2)

[1526] 2. The system of claim 1, wherein the generative artificial intelligence further comprises means for generating a response text based on a user's past speech history and recognized emotion data.

[1527] (Claim 3)

[1528] The system of claim 1 , wherein the speech recognition engine, the emotion engine, and the speech synthesis engine are cloud-based.

[1529] "Application example 2 when combining emotion engines"

[1530] (Claim 1)

[1531] means for receiving a user's voice input;

[1532] means for converting the received voice input into text data using a voice recognition engine;

[1533] a means for transmitting the converted text data to an emotion engine to recognize the emotion of the user;

[1534] means for transmitting emotion data and text data recognized by the emotion engine to a server;

[1535] A means for generating suggested text based on past behavioral history and emotion data using generative artificial intelligence on the server side;

[1536] means for receiving the generated suggested text from a server;

[1537] means for converting the received proposal text into voice data by a voice synthesis engine;

[1538] means for outputting the generated voice data to a user;

[1539] A system including:

[1540] (Claim 2)

[1541] 2. The system of claim 1, wherein the generative artificial intelligence further comprises means for generating a response text based on the user's past speech history and emotion data.

[1542] (Claim 3)

[1543] The system of claim 1 , wherein the speech recognition engine, the emotion engine, and the speech synthesis engine are cloud-based. [Explanation of symbols]

[1544] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a user's voice input; means for converting the received voice input into text data using a voice recognition engine; means for transmitting the converted text data to a server; A means for generating a response text using generative artificial intelligence on the server side; means for receiving the generated response text from the server; A means for converting the received response text into voice data using a voice synthesis engine; means for outputting the generated voice data to a user; A system including:

2. The system of claim 1 , wherein the generative artificial intelligence further comprises means for generating a response text based on a user's past speech history.

3. The system of claim 1 , wherein the speech recognition engine and the speech synthesis engine are cloud-based.

Images