System

Abstract

A system is provided.SOLUTION: A system comprising: an information device used by an elderly person; means for causing the information device to perform AI recognition; means for transmitting the text data to a server via a communication network; means for generating a response to the text data using a generative speech model in the server; and means for transmitting the response to the information device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In preventing dementia in the elderly, the decrease in opportunities for daily conversation is a problem, particularly for those living alone or with limited social contact. This decrease in opportunities for conversation leads to decreased brain activity and an increased risk of dementia. Furthermore, due to a shortage of staff, elderly care facilities and nursing homes currently face difficulties in ensuring sufficient conversation with residents. There is a need to resolve these issues and encourage the elderly to enjoy daily conversation, thereby activating their brains and contributing to the prevention of dementia. [Means for solving the problem]

[0005] The present invention provides a system that uses a generative AI model to enable elderly people to enjoy natural conversations through an information terminal. The information terminal recognizes the user's voice, converts the voice data into text data, and sends it to a server. The server uses the generative AI model to generate an appropriate response to the text data and sends it to the information terminal. The response is then synthesized and provided to the user as voice data, creating an experience that feels as if the user is actually speaking with a human. The generative AI model also learns the user's conversation data and periodically retrains to provide more natural and rich conversations. This allows elderly people to enjoy daily conversations, promotes brain activation, and contributes to the prevention of dementia.

[0006] An "information terminal" is a device that is operated by a user and captures voice data, including smartphones and smart speakers.

[0007] "Speech recognition" refers to the process of capturing audio data and converting it into text data.

[0008] "Text data" is a digital representation of a body of text converted by speech recognition.

[0009] "Server" means a computer system for hosting generative AI models and processing audio and text data.

[0010] A "communications network" is an infrastructure for transmitting and receiving data between information terminals and servers, and includes the Internet and the like.

[0011] A "generative AI model" is an artificial intelligence model that analyzes text data received from a user and generates an appropriate response.

[0012] A "response" is a reply message that the generative AI model generates based on the user's text data.

[0013] "Speech synthesis" refers to the process of converting text data into speech data.

[0014] "Speech Data" means a playable digital representation of speech produced by speech synthesis.

[0015] "Users" refer to people who use the system of the present invention, and particularly to elderly people.

[0016] A "log" is data recorded during the system's operation, and includes the content of conversations and timestamps.

[0017] "Retraining" is the process by which a generative AI model uses new data to improve its performance.

[0018] "Human resource shortage" refers to a situation in which hospitals and nursing homes do not have the necessary personnel. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention provides a system that uses a generative AI model to enable elderly people to enjoy natural conversations. This system is composed of an information terminal, a server, and a communication network.

[0041] System configuration

[0042] server

[0043] The server hosts the generative AI model and manages conversation data with the user. The server receives text data sent by the user and generates appropriate responses using the generative AI model.

[0044] The server also stores user profile information (such as name, age, and hobbies) in a database, records conversation history, and periodically retrains the generative AI model to improve response accuracy.

[0045] Terminal

[0046] A device is a user-operated device, such as a smartphone or smart speaker, that has a microphone and speaker to capture and play back the user's voice.

[0047] The device has a voice recognition function and converts the user's voice data into text data. This text data is sent to the server, and the response received from the server is played back using a voice synthesis function.

[0048] user

[0049] The users are elderly people who enjoy conversations by speaking directly into the information terminal. Users can use the system in a natural manner without any special operations.

[0050] System Operation

[0051] 1. Start a conversation

[0052] The user speaks to the information terminal, asking questions such as, "Hello, what's in the news today?"

[0053] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the text data to a server via a communications network.

[0054] 2. Generating a Response

[0055] The server logs the received text data and inputs it into the generative AI model, which then analyzes the text data and generates an appropriate response.

[0056] The server transmits the generated response to the terminal as text data.

[0057] 3. Providing a Response

[0058] The device converts the received text data into voice data using a speech synthesis engine, which is then played back to the user through the speaker.

[0059] For example, if the response from the server is "Today, major newspapers are reporting the following news: XXX, XXX, XXX," this will be provided as audio.

[0060] Specific examples

[0061] Specific examples of conversations

[0062] 1. The user says, "Good morning, what's on my schedule today?"

[0063] 2. The device converts the voice into text and sends the text data, "Good morning, what are your plans for today?" to the server.

[0064] 3. The server uses the generative AI model to generate a response: "Good morning, we have plans to go for a walk at 10 o'clock today."

[0065] 4. The device converts the received response into audio and plays it back to the user.

[0066] Feedback and Improvements

[0067] 1. The server stores all conversation data in a log and uses it as training data for the generative AI model.

[0068] 2. The generative AI model undergoes periodic retraining, gradually achieving more natural and richer conversations.

[0069] The system of this invention allows elderly people to enjoy daily conversations, which is expected to promote brain activity and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user speaks into the terminal.

[0073] For example, say, "Hello, what can I do for you today?"

[0074] Step 2:

[0075] The device captures the audio.

[0076] A microphone collects the user's voice.

[0077] Step 3:

[0078] The device converts the speech to text.

[0079] The voice recognition engine converts the voice data into text data.

[0080] The converted text becomes "Hello, what can I do for you today?"

[0081] Step 4:

[0082] The terminal transmits the text data to the server.

[0083] A POST request is sent to the server's API via a communication network.

[0084] The request includes the converted text data.

[0085] Step 5:

[0086] The server receives the text data.

[0087] The server receives the request and extracts the text data.

[0088] Step 6:

[0089] The server logs the text data.

[0090] The received text data is saved in a database and recorded as a log.

[0091] Step 7:

[0092] The server inputs text data into the generative AI model.

[0093] A generative AI model analyzes the text data "Hello, what should I do today?"

[0094] Step 8:

[0095] A generative AI model generates the appropriate response.

[0096] The model uses internal data and algorithms to generate a response like, "Hello, how about enjoying some reading or a short walk today?"

[0097] Step 9:

[0098] The server sends the generated response to the terminal.

[0099] The generated text data is sent as an API response to be returned to the device.

[0100] Step 10:

[0101] The terminal receives the response text.

[0102] Receive the response to the request and extract the text data.

[0103] Step 11:

[0104] The device converts the received text data into voice.

[0105] The speech synthesis engine converts the text "Hello, how about enjoying some reading or a short walk today?" into speech data.

[0106] Step 12:

[0107] The terminal plays the audio to the user.

[0108] The audio data is played through the speaker to provide a response to the user.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] There is a demand for systems that allow elderly people to enjoy natural conversations in their daily lives. However, many current systems have low accuracy in speech recognition and response generation, making them difficult for elderly people to use. Furthermore, continuous learning is necessary for generative AI models to maintain a certain level of response quality, but this is often not achieved. Furthermore, conversation data is not managed, and logs are not kept to generate appropriate responses. This makes it difficult for elderly people to enjoy comfortable communication.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes means for converting voice data into text data by performing voice recognition, means for transmitting the text data to the server via a communication network, means for generating a response to the text data using a generative AI model, means for converting the generated response into voice data by voice synthesis and providing the voice data to the user from the communication device, and means for recording the response in a log and periodically retraining the generative AI model. This allows elderly people to enjoy natural conversations and also enables improvement in the accuracy of the generative AI model and maintenance of response quality.

[0114] The term "elderly" generally refers to older people, especially those aged 65 and over.

[0115] A "communication device" is a device for inputting and outputting voice, and includes, for example, smartphones and smart speakers.

[0116] "Speech recognition" refers to the technology that converts a user's speech into text data.

[0117] "Text data" refers to character data converted using voice recognition technology.

[0118] "Communication network" refers to the network infrastructure for data communication, and specifically includes the Internet.

[0119] "Computer" refers to a device that has server functionality and processes and stores data.

[0120] "Generative AI model" refers to an artificial intelligence model that generates appropriate responses to input text data.

[0121] "Response" refers to the reply generated by the generative AI model in response to text data.

[0122] "Speech synthesis" refers to the technology of converting text data into voice data.

[0123] "Voice data" refers to digital data of voice generated by voice synthesis technology.

[0124] "Log" refers to data that records the system's operation history and conversation data exchanged.

[0125] "Relearning" refers to the learning process of improving the accuracy of a generative AI model based on previously logged data.

[0126] The present invention provides a system that allows elderly people to enjoy natural conversations. This system is configured using communication devices, a server, and a communication network. The following describes how this system is configured and how it functions.

[0127] System Overview

[0128] server

[0129] The server plays a central role in hosting the generative AI model and managing conversation data with users. The server has the following functions:

[0130] 1. Receiving and logging

[0131] The server receives the text data sent from the device and records it in a log, which includes the conversation history and user profile information (such as name, age, and hobbies).

[0132] 2. Response generation using a generative AI model

[0133] The received text data is input into a generative AI model to generate an appropriate response. The generative AI model can be, for example, a generative AI model.

[0134] 3. Sending the Response

[0135] The server transmits the generated response to the terminal as text data.

[0136] 4. Periodic Retraining

[0137] The server collects all conversation data and periodically retrains the generative AI model based on this data, improving the accuracy and naturalness of responses.

[0138] Terminal

[0139] A terminal is a device that is directly operated by a user and has the following functions:

[0140] 1. Voice Recognition

[0141] The microphone captures the user's speech into the device and converts it into text data using a speech recognition engine, such as voice recognition software.

[0142] 2. Texting

[0143] The converted text data is transmitted to a server via a communication network.

[0144] 3. Speech Synthesis

[0145] The text data received from the server is converted into voice data using a speech synthesis engine, which uses speech synthesis software.

[0146] 4. Audio playback

[0147] The converted audio data is played back to the user through a speaker.

[0148] user

[0149] The users are elderly people who can use the system in a natural manner without requiring any special technical knowledge or operation.

[0150] Specific examples

[0151] Specific examples of conversations

[0152] 1. User utterances

[0153] The user says, "Good morning, how's the weather today?"

[0154] 2. Voice recognition on the device

[0155] The device converts the voice into text and sends the text data, such as "Good morning, how's the weather today?" to the server.

[0156] 3. Server Response Generation

[0157] The server uses the generative AI model to generate a response: "Good morning, it's sunny today."

[0158] 4. Device speech synthesis

[0159] The device converts the received response into speech and plays it to the user: "Good morning, it's sunny today."

[0160] Prompt Sentence Examples

[0161] Here is an example of inputting the following prompt sentence into the generative AI model:

[0162] User input: "Good morning, how's the weather today?"

[0163] Sample prompt: "The user is asking, 'Good morning, how's the weather today?' Generate an appropriate response."

[0164] The system of this invention is expected to enable elderly people to enjoy natural conversations on a daily basis, promote brain activity, and contribute to the prevention of dementia. This system will also reduce the burden on staff at hospitals and nursing homes and improve communication with residents.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] Voice input

[0168] A user speaks into a communication device. For example, they say something like, "Hello, what should I do today?" The input data is the user's voice. The output is the voice data captured by the communication device. Specifically, the user speaks into the microphone of the communication device, and the voice is recorded as digital voice data.

[0169] Step 2:

[0170] Voice Recognition

[0171] The device converts the captured voice data into text data using a voice recognition engine. For example, general voice recognition software is used as the voice recognition engine. The input data is voice data, and the output is text data converted by voice recognition. Specifically, the voice data is converted into the text data "Hello, what should I do today?"

[0172] Step 3:

[0173] Text sending

[0174] The terminal transmits the converted text data to the server via a communication network. The input data is the text data obtained by speech recognition, and the output is the status of the text data being transmitted to the server. Specifically, the text data is securely transmitted to the server via the Internet.

[0175] Step 4:

[0176] Response Generation

[0177] The server records the received text data in a log and inputs it into the generative AI model. The generative AI model analyzes the text data and generates an appropriate response. The input data is the received text data, and the output is the response text data generated by the generative AI model. Specifically, from the input "Hello, what should I do today?", a response such as "I recommend reading today" is generated.

[0178] Step 5:

[0179] Response Send

[0180] The server sends the generated response to the terminal as text data. The input data is the generated response text data, and the output is the status of the response text data being sent to the terminal. Specifically, the response text "I recommend reading today" is sent to the terminal.

[0181] Step 6:

[0182] Speech synthesis

[0183] The device converts the received response text data into voice data using a voice synthesis engine. For example, general voice synthesis software is used as the voice synthesis engine. The input data is the response text data, and the output is voice data generated by voice synthesis. Specifically, voice data is generated from the text "I recommend reading today."

[0184] Step 7:

[0185] Audio Output

[0186] The device plays the audio data to the user through the speaker. The input data is the audio data generated by speech synthesis, and the output is the audio that is played back to the user. Specifically, the speaker plays the audio "We recommend you read today," and the user can hear it.

[0187] The above are the specific processing steps of the program of this system.

[0188] (Application example 1)

[0189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] Elderly people often rely on their family members or nursing home staff to ensure their safety and respond to emergencies. However, if these support services are insufficient, the elderly are at greater risk of facing emergencies. Elderly people may also feel lonely, which is becoming a social problem. Therefore, there is a need for a system that allows elderly people to easily call for help using their voice and receive prompt, appropriate assistance.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0192] In this invention, the server includes means for generating appropriate suggestions for emergency responses when an elderly person utters an emergency word requesting help, means for providing the appropriate suggestions to the user at the information terminal, and means for notifying emergency contacts based on the emergency response suggestions. This enables a prompt and appropriate response when an elderly person faces an emergency, ensuring the safety of the elderly and reducing their sense of loneliness.

[0193] "Elderly people" refers to people who are over a certain age as defined by the Elderly People Act, and who often require special assistance in daily life.

[0194] An "information terminal" is a device operated by a user, including smartphones and smart speakers.

[0195] "Speech recognition" refers to the process by which an information device captures a user's voice and converts that voice data into text data.

[0196] "Generative AI model" refers to an algorithm or system that uses AI technology to analyze text data and generate natural-sounding conversational responses.

[0197] "Emergency response" refers to a series of actions and procedures to provide appropriate and prompt assistance to elderly people when they face an emergency.

[0198] "Appropriate Recommendations" refer to recommendations that direct appropriate and effective actions or responses to be taken in a particular situation.

[0199] "Communication network" refers to a network system for exchanging data between information terminals and servers, and includes the Internet.

[0200] "Speech synthesis" refers to the technology of converting text data into audio data and playing that audio back to the user.

[0201] "Emergency contacts" refers to pre-registered contacts, such as family members or care staff, who should be contacted in the event of an emergency.

[0202] "Safety checks" refer to regularly checking on the current status of elderly people and asking questions and making checks to ensure their safety.

[0203] "Means for generating a response" refers to the function or method by which the server uses a generative AI model to generate a response to the received text data.

[0204] "Voice data" refers to data generated using a voice synthesis engine for provision to the user by an information terminal.

[0205] This invention relates to a system that checks the safety of elderly people and responds to emergencies using information terminals, servers, and communication networks used by elderly people. The system aims to utilize generative AI models to have natural conversations with elderly people and provide appropriate responses in emergencies.

[0206] System configuration

[0207] server

[0208] The server hosts a generative AI model that generates responses based on the text data sent by the user. The generative AI model consists of an algorithm that analyzes the user's input text and generates the optimal response.

[0209] The server includes a natural language generation model such as GPT-2 as a means to generate responses, as well as an algorithm for detecting urgent words.

[0210] Information terminal

[0211] An information terminal is a device operated by a user, such as a smartphone or a smart speaker. An information terminal is equipped with a microphone and a speaker to capture and play back the user's voice.

[0212] The device is equipped with a voice recognition function that converts the user's speech into text, which is then sent to a server via a communications network.

[0213] communication network

[0214] Data communication between the information terminal and the server is carried out over the Internet, which allows for real-time communication.

[0215] System Operation

[0216] 1. Regular safety checks

[0217] The device periodically asks the elderly person questions to check their safety, such as "How are you?" and "Is there anything I can help you with?" This allows the device to periodically check the elderly person's condition.

[0218] The user responds to these questions verbally, which is converted into text by the device and sent to the server.

[0219] 2. Conversational Response Generation

[0220] The server analyzes the received text data and generates a response using a generative AI model, which is then sent to the device as text.

[0221] The device converts the received response into voice data using a voice synthesis engine and plays it back to the user.

[0222] 3. Emergency Response

[0223] When a user utters an emergency word such as "help," the device detects it and sends it to the server, which then generates appropriate suggestions for emergency response, such as "Should we call an ambulance?", and sends them to the device.

[0224] If necessary, the system will also notify pre-registered emergency contacts via email, SMS, or other means.

[0225] Program processing explanation

[0226] Speech recognition: Using the speech_recognition library, the information terminal analyzes the user's voice and converts it into text data.

[0227] Generative AI model: Implement a GPT-2 model using the transformers library to generate appropriate responses from text data received on the server.

[0228] Speech synthesis: Uses the pyttsx3 library to convert the generated text response into audio data and play it back to the user on the device.

[0229] Emergency response: Uses algorithms to detect specific emergency words, generates appropriate suggestions in an emergency, and notifies emergency contacts as needed.

[0230] Specific examples

[0231] Regular safety checks:

[0232] The device periodically asks, "How are you?", and the user responds, "I'm fine." This response is converted into text and analyzed by the server to generate follow-up questions such as, "That's great. Is there anything I can help you with today?"

[0233] Example prompt sentence:

[0234] Senior citizen: "What's the news today?"

[0235] Prompt: "Today's news"

[0236] The generative AI model responds, "Today's news is that XX was reported in major newspapers."

[0237] This allows elderly people to receive emergency response in real time while enjoying natural conversation.

[0238] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0239] Step 1:

[0240] The information terminal uses a microphone to capture the user's voice. The captured voice data is input. The voice data is converted into text data using a speech recognition engine (speech_recognition library). The converted text data is output.

[0241] Step 2:

[0242] The information terminal transmits the converted text data to a server via a communication network (Internet). At this time, data calculation is performed in which the text data is input and transmitted to the server using a communication protocol.

[0243] Step 3:

[0244] The server inputs the received text data into a generative AI model (GPT-2) and generates an appropriate response. The generative AI model performs text analysis and generates a response based on the prompt. This response is output. For example, in response to the prompt "Today's news is...", the response generated is "Today's news is that XX was reported in major newspapers."

[0245] Step 4:

[0246] The server transmits the generated response as text data to the information terminal. The text data is input and then processed by transmitting it to the information terminal via a communication network.

[0247] Step 5:

[0248] The information terminal converts the received text data into voice data using a speech synthesis engine (pyttsx3 library). Text data is input and voice data is output. The response converted into voice data is played back to the user through the speaker.

[0249] Step 6:

[0250] The device periodically asks the user safety confirmation questions and converts the responses to those questions into text data using a speech recognition engine. For example, the question "How are you?" is treated as a prompt, and the user responds "I'm fine." This response is then converted into text data.

[0251] Step 7:

[0252] When a user utters an emergency word (e.g., "Help"), the device detects the data and sends it to the server. The server generates appropriate suggestions for emergency response (e.g., "Should I call an ambulance?") and sends them to the device. The device then converts the generated suggestions into audio data and plays them back to the user.

[0253] Step 8:

[0254] If necessary, the server notifies pre-registered emergency contacts by entering their information and sending a notification using a communication protocol.

[0255] This will enable elderly people to enjoy natural conversation while receiving emergency response in real time, ensuring their safety.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] This invention provides a system that can provide more natural and emotional responses by combining a conversation system using a generative AI model with an emotion engine that recognizes the user's emotions. The entire system is composed of information terminals, a server, and a communication network.

[0258] System configuration

[0259] server

[0260] The server hosts the generative AI model and emotion engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[0261] The emotion engine analyzes the received voice data to recognize and evaluate the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[0262] The server also stores user profile information and conversation history in a database for periodic retraining of the generative AI model.

[0263] Terminal

[0264] A device is a device operated by a user, such as a smartphone or a smart speaker. A device has a microphone and a speaker and can capture and play back the user's voice.

[0265] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[0266] Furthermore, the terminal can display the user's emotional state transmitted from the server.

[0267] user

[0268] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[0269] System Operation

[0270] 1. Start a conversation

[0271] The user speaks to the information terminal, for example, saying, "Hello, I'm a little tired today."

[0272] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the converted text and voice data to a server via a communications network.

[0273] 2. Emotion recognition

[0274] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state (e.g., tired, fatigued tone).

[0275] The emotion engine evaluates the emotional state and returns the result to the server.

[0276] 3. Generating a Response

[0277] Based on the evaluation results from the emotion engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[0278] For example, if the user says, "Hello, I'm a little tired today," the response might be, "Hello, I suggest you take a little rest and then think about what you want to do next."

[0279] 4. Providing a Response

[0280] The server transmits the generated text data to the terminal, which converts the received text data into voice data using a voice synthesis engine and plays it back to the user.

[0281] The terminal can also visually display the evaluation results by the emotion engine to the user.

[0282] Specific examples

[0283] Specific examples of conversations

[0284] 1. The user says, "I'm very happy today."

[0285] 2. The device converts the voice into text and sends the text data, "I'm so happy today," and the voice data to the server.

[0286] 3. The server inputs the voice data into the emotion engine and evaluates the emotional state as "happy."

[0287] 4. The server uses the generative AI model to generate a response: "That's great! What did you do today?"

[0288] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[0289] Feedback and Improvements

[0290] 1. The server stores all conversation data and emotion evaluation data in logs and uses them as training data for the generative AI model and emotion engine.

[0291] 2. The generative AI model and emotion engine are periodically retrained to gradually achieve more natural and emotionally relevant responses.

[0292] The system of this invention allows elderly people to enjoy daily conversations that take their emotions into consideration, which is expected to promote brain activation and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[0293] The processing flow will be explained below.

[0294] Step 1:

[0295] The user speaks into the terminal.

[0296] For example, say, "Hello, I'm very happy today."

[0297] Step 2:

[0298] The device captures the audio.

[0299] The device's microphone collects the user's voice.

[0300] Step 3:

[0301] The device converts the speech to text.

[0302] The voice recognition engine converts the voice data into text data.

[0303] The converted text becomes "Hello, I'm very happy today."

[0304] Step 4:

[0305] The terminal transmits text data and voice data to the server.

[0306] A POST request is sent to the server's API via a communication network.

[0307] The request includes the converted text data and audio data.

[0308] Step 5:

[0309] The server receives the text data and the voice data.

[0310] The server receives the request and extracts this data.

[0311] Step 6:

[0312] The server inputs the voice data into the emotion engine.

[0313] An emotion engine analyzes the voice data and assesses the user's emotional state.

[0314] For example, the emotion engine recognizes the emotional state "happy."

[0315] Step 7:

[0316] The server inputs the emotion engine's evaluation results into the generative AI model.

[0317] The text data "Hello, I'm very happy today" and the emotional information "happy" are input into the generative AI model.

[0318] Step 8:

[0319] A generative AI model generates the appropriate response.

[0320] The model uses internal data and algorithms to generate a response: "That's great! What did you do today?"

[0321] Step 9:

[0322] The server sends the generated response to the terminal.

[0323] Send an API response to return the generated text data to the device.

[0324] Step 10:

[0325] The terminal receives the response text.

[0326] Receive the response to the request and extract the text data.

[0327] Step 11:

[0328] The device converts the received text data into voice.

[0329] The speech synthesis engine converts the text "That's great! What did you do today?" into speech data.

[0330] Step 12:

[0331] The terminal plays the audio to the user.

[0332] The audio data is played through the speaker to provide a response to the user.

[0333] Step 13:

[0334] The terminal displays the emotion evaluation result to the user.

[0335] For example, the screen might display "Emotion: Happy."

[0336] Step 14:

[0337] The server stores the conversation data and emotion evaluation data in a log.

[0338] All conversations and sentiment assessment details are recorded in a database.

[0339] Step 15:

[0340] The server periodically retrains the generative AI model and emotion engine.

[0341] The accuracy of the model is improved using collected conversational and emotional data.

[0342] Example 2

[0343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] In conversation systems using information terminals used by elderly people, conventional systems have had the problem of not taking the user's emotions into sufficient consideration when responding, making the conversation feel unnatural. Furthermore, because the system does not reflect the user's emotional state, the conversation may not be very satisfying. Furthermore, these systems do not fully utilize the user's profile information or conversation history, which means that they are unable to provide responses based on the user's past conversations and their emotions at the time.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0346] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for recognizing and evaluating the user's emotional state from the received voice data using an emotion engine, and means for adjusting the response based on the user's emotional state. This enables a natural and satisfying response that reflects the user's emotions. Furthermore, the generated response is converted into voice data using voice synthesis and provided to the user from the information terminal, making the conversation smoother. Furthermore, the server stores the user's profile information and conversation history, and the generative AI model receives feedback based on this information and re-learns, improving the quality of the conversation.

[0347] "Elderly" refers to people who are relatively old and require physical or cognitive assistance.

[0348] An "information terminal" is a device operated by a user that has input, output, and communication functions. Examples include smartphones and smart speakers.

[0349] "Speech recognition" refers to the technology of analyzing voice data and converting it into text data.

[0350] "Text data" refers to text information converted from voice data using voice recognition technology.

[0351] "Communications network" refers to the network infrastructure for data communication between information terminals and servers, including the Internet.

[0352] "Server" refers to the computer system that hosts the generative AI model and emotion engine, receives data sent by users, analyzes it, and responds.

[0353] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-sounding sentences based on input data.

[0354] "Emotion engine" refers to technology that analyzes voice data and recognizes and evaluates the user's emotional state.

[0355] "Speech synthesis" refers to the technology of generating voice data based on text data.

[0356] "Profile Information" refers to data that records personal information and characteristics of a user.

[0357] "Conversation history" refers to data that records the contents of past conversations that a user has had with the system.

[0358] "Relearning" refers to the training process used to improve the performance of generative AI models and emotion engines based on stored data.

[0359] "Response" refers to the reply that the generative AI model gives in response to input from the user.

[0360] "Audio data" refers to data that represents audio in digital form.

[0361] This invention is a system that realizes natural conversations using information terminals used by elderly people and generative AI models. This invention can provide responses that take into account the user's emotions, enabling more human-like communication.

[0362] System Configuration

[0363] The system consists of the following main components: information terminals, servers, and communication networks.

[0364] Information terminal

[0365] An information terminal is a device operated by a user, and includes smartphones, smart speakers, etc. The terminal is equipped with a microphone and speaker, captures the user's voice, and plays a response voice using voice synthesis.

[0366] server

[0367] The server hosts the generative AI model and emotion engine, which play a central role in receiving voice and text data sent by users and generating appropriate responses. The generative AI model uses a natural language generation algorithm such as GPT-3. The emotion engine recognizes the user's emotional state from the voice data using, for example, an emotion analysis API.

[0368] communication network

[0369] A communication network enables data communication between information terminals and servers, and typically the Internet is used.

[0370] Specific examples of processing

[0371] The specific operation of the system will now be described.

[0372] 1. Start a conversation

[0373] User

[0374] The user speaks to the information terminal, for example, saying, "I'm very happy today."

[0375] Terminal

[0376] The device uses a microphone to capture voice data, converts it into text data using a voice recognition engine, and then transmits the text data and voice data to a server via a communication network.

[0377] 2. Emotion recognition

[0378] server

[0379] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state. For example, in this case, the emotional state of "happy" is recognized.

[0380] 3. Generating a Response

[0381] server

[0382] The server inputs the emotion engine's evaluation results and text data into the generative AI model to generate an appropriate response, such as "That's great! What did you do today?"

[0383] 4. Providing a Response

[0384] Terminal

[0385] The terminal inputs the response text sent from the server into a speech synthesis engine, converts it into voice data, and plays it back to the user through the speaker.

[0386] Feedback and Improvements

[0387] server

[0388] The server stores all conversation data and emotion evaluation data in logs, which are used to retrain the generative AI model and emotion engine, allowing the system to gradually develop more natural and emotionally relevant responses, improving the user experience.

[0389] Prompt Sentence Examples

[0390] Here is an example prompt:

[0391] "When a user inputs words, the voice data and emotion data are sent to the server. The server uses an emotion engine to recognize the emotion, and based on the results, a generative AI model generates an appropriate response. The generated text is sent to the device and played back to the user by the speech synthesis engine."

[0392] This system allows elderly people to enjoy natural conversations that take their emotions into consideration, and is expected to be highly effective when used in hospitals and nursing homes.

[0393] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0394] Step 1:

[0395] User initiated conversation

[0396] Input: User's spoken utterance

[0397] Specific action: The user speaks to the information terminal. For example, he says, "Hello, I'm a little tired today."

[0398] Output: User's voice data

[0399] Step 2:

[0400] Device capture and conversion of audio data

[0401] Input: User's voice data

[0402] What it does: The device uses the built-in microphone to capture the user's voice and converts it into text using a speech recognition engine (e.g., Google Speech-to-Text API).

[0403] Output: Converted text data and original audio data

[0404] Step 3:

[0405] Sending data from the device to the server

[0406] Input: Text data, audio data

[0407] Specific operation: The terminal transmits the converted text data and the original voice data to a server via a communication network (e.g., the Internet).

[0408] Output: Text and audio data sent to the server

[0409] Step 4:

[0410] Emotion recognition by server

[0411] Input: Transmitted audio data

[0412] Specific operation: The server inputs the received voice data into an emotion engine (e.g., Microsoft Azure Emotion API) to analyze and evaluate the user's emotional state. In this case, the emotional state of "tired" is recognized from the voice data.

[0413] Output: Emotional state data (e.g., "Tired")

[0414] Step 5:

[0415] Server response generation

[0416] Input: Text data, emotional state data

[0417] Specific operation: The server inputs the emotional state data and text data obtained from the emotion engine into a generative AI model (e.g., GPT-3). The generative AI model generates an appropriate response based on this data. For example, it generates a response such as, "Hello, I recommend you take a short break. Then, think about what you want to do next."

[0418] Output: The generated response text

[0419] Step 6:

[0420] Sending response data from the server to the terminal

[0421] Input: Generated response text

[0422] Specific operation: The server sends the generated response text to the terminal.

[0423] Output: Response text sent to the terminal

[0424] Step 7:

[0425] Voice generation and playback of responses by the terminal

[0426] Input: Response text data

[0427] Specific operation: The device inputs the received response text into a speech synthesis engine (e.g., Amazon Polly) to convert the text into voice data, which the device then plays back to the user through the speaker.

[0428] Output: A spoken response to be played to the user (e.g., "Hello, I suggest you take a break and think about what you want to do next.")

[0429] Step 8:

[0430] Feedback and Data Storage

[0431] Input: Conversation data, emotion evaluation data

[0432] Specific operation: The server stores all conversation data and emotion evaluation data in a log, which is used to retrain the generative AI model and emotion engine.

[0433] Output: Updated training data

[0434] Through the above steps, users can enjoy natural conversations that take emotions into consideration.

[0435] (Application example 2)

[0436] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0437] To enable elderly people to enjoy more natural and emotionally appropriate conversations on information devices, simply generating responses based on text data is not enough. Conventional systems are unable to generate responses that take the user's emotional state into account, and are therefore unable to provide satisfying interactions for users. Furthermore, particularly when serving customers in brick-and-mortar stores, staff are required to grasp the customer's emotional state in real time and provide services accordingly. Therefore, a more accurate conversation system that also takes the user's emotions into account is needed.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0439] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for using an emotion recognition engine to perform emotion analysis on the voice data from the user, means for adjusting the response of the generative AI model based on the emotion analysis results, and means for visually displaying the response. This enables natural and satisfying conversations that correspond to the user's emotions, and makes it possible to provide flexible customer service based on the emotional state of customers, particularly in physical stores.

[0440] 1. An "information terminal" is a device operated by a user that has the ability to capture voice and convert it into text data.

[0441] 2. "Speech recognition" is a technology that converts a user's voice data into text data.

[0442] 3. "Communications network" means the infrastructure for transmitting and receiving data between information terminals and servers, including the Internet.

[0443] 4. "Server" means a computing system that hosts the generative AI model and emotion recognition engine and manages user conversation data and emotion data.

[0444] 5. A "generative AI model" is an algorithm that generates appropriate responses based on input text data.

[0445] 6. An "emotion recognition engine" is a technology that analyzes a user's voice data and evaluates their emotional state.

[0446] 7. “Sentiment analysis” is the process of identifying a user’s emotional state based on audio data.

[0447] 8. "Visually displaying" means presenting the results of an analysis or generated response to the user as text or graphics on a screen.

[0448] 9. "Speech synthesis" is a technology that converts text data into voice data.

[0449] 10. "Brick and Mortar Store" means a store that exists in a physical location and offers goods and services.

[0450] 11. "Customer service" refers to the service and interaction provided by staff to customers in physical stores.

[0451] This invention is a conversation system using an information terminal and a server for use by elderly people, which can provide natural conversation according to the user's emotions. This system is configured using an information terminal, a server, and a communication network. Specific embodiments for implementing this invention are described in detail below.

[0452] System configuration

[0453] server

[0454] The server hosts the generative AI model and emotion recognition engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[0455] The emotion recognition engine analyzes the received voice data to assess the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[0456] The server also stores user profile information and conversation history in a database, which is used to periodically retrain the generative AI model.

[0457] Terminal

[0458] A device is a user-operated device, such as smart glasses, a smartphone, or a smart speaker. A device has a microphone and a speaker to capture and play back the user's voice.

[0459] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[0460] Furthermore, the terminal can visually display the user's emotional state transmitted from the server.

[0461] user

[0462] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[0463] System Operation

[0464] 1. Start a conversation

[0465] The user speaks to the information terminal, for example, saying, "I want to know more about this product."

[0466] The device captures the voice with a microphone and converts the voice into text using voice recognition, which is then sent to a server via a communications network.

[0467] 2. Emotion recognition

[0468] The server inputs the received voice data into an emotion recognition engine to evaluate the user's emotional state (e.g., interested, happy, etc.).

[0469] The emotion recognition engine evaluates the emotional state and returns the results to the server.

[0470] 3. Generating a Response

[0471] Based on the evaluation results from the emotion recognition engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[0472] For example, if a user says, "I'd like to know more about this product," the response might be, "I'd like to tell you more about this product. Is there anything in particular you'd like to know?"

[0473] 4. Providing a Response

[0474] The server sends the generated text data to the terminal, which then converts the received text data into voice data using a speech synthesis engine and plays it back to the user.

[0475] The device can also visually display the evaluation results from the emotion recognition engine to the user.

[0476] Specific examples

[0477] Specific examples of conversations

[0478] 1. A user says, "I'd like to know more about this product."

[0479] 2. The device converts the voice into text and sends the text data, "I want to know more about this product," and the voice data to the server.

[0480] 3. The server inputs the voice data into an emotion recognition engine and evaluates the emotional state as "interested."

[0481] 4. The server uses the generative AI model to generate a response that says, "We'll explain this product in detail. Is there anything in particular you'd like to know?"

[0482] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[0483] Prompt Sentence Examples

[0484] The user's sentiment is 'interested'. Generate a sentiment-based response to the following text: "I'd like to know more about this product"

[0485] In this way, natural and meaningful conversations that take the user's emotions into account are realized.

[0486] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0487] Step 1:

[0488] A user speaks to an information terminal, for example, saying, "I'd like to know more about this product." The input is the user's voice, which is captured via a microphone. The output is raw voice data.

[0489] Step 2:

[0490] The device inputs the captured voice data into a voice recognition engine, which converts the voice data into text data. The voice recognition engine processes the voice data and outputs text data. Here, the input is raw voice data, and the output is text data.

[0491] Step 3:

[0492] The terminal transmits the converted text data and voice data to the server via the communication network. The input of this step is the text data and voice data, and the output is the data transmitted to the server.

[0493] Step 4:

[0494] The server inputs the received voice data into the emotion recognition engine to evaluate the user's emotional state. In this step, the emotion recognition engine analyzes the voice data to identify the emotional state and outputs the result. The input is the voice data, and the output is the evaluation result of the emotional state.

[0495] Step 5:

[0496] The server inputs the text data into the generative AI model along with the evaluation results from the emotion recognition engine. The generative AI model generates an optimal response based on the text data and the emotion evaluation results. The input for this step is the text data and the emotion evaluation results, and the output is the generated response text.

[0497] Step 6:

[0498] The server inputs the generated text data into a speech synthesis engine and converts it into voice data. The speech synthesis engine processes the text data and outputs voice data. The input is the response text data, and the output is the generated voice data.

[0499] Step 7:

[0500] The device plays the audio data sent from the server and provides it to the user. The device plays this audio data through a speaker so that the user can hear it. The device also visually displays the evaluation results from the emotion recognition engine. The input here is the generated audio data and the emotion evaluation result, and the output is the audio played to the user and the displayed emotional state.

[0501] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0502] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0503] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0504] [Second embodiment]

[0505] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0506] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0507] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0508] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0509] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0510] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0511] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0512] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0513] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0514] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0515] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0516] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0517] This invention provides a system that uses a generative AI model to enable elderly people to enjoy natural conversations. This system is composed of an information terminal, a server, and a communication network.

[0518] System configuration

[0519] server

[0520] The server hosts the generative AI model and manages conversation data with the user. The server receives text data sent by the user and generates appropriate responses using the generative AI model.

[0521] The server also stores user profile information (such as name, age, and hobbies) in a database, records conversation history, and periodically retrains the generative AI model to improve response accuracy.

[0522] Terminal

[0523] A device is a user-operated device, such as a smartphone or smart speaker, that has a microphone and speaker to capture and play back the user's voice.

[0524] The device has a voice recognition function and converts the user's voice data into text data. This text data is sent to the server, and the response received from the server is played back using a voice synthesis function.

[0525] user

[0526] The users are elderly people who enjoy conversations by speaking directly into the information terminal. Users can use the system in a natural manner without any special operations.

[0527] System Operation

[0528] 1. Start a conversation

[0529] The user speaks to the information terminal, asking questions such as, "Hello, what's in the news today?"

[0530] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the text data to a server via a communications network.

[0531] 2. Generating a Response

[0532] The server logs the received text data and inputs it into the generative AI model, which then analyzes the text data and generates an appropriate response.

[0533] The server transmits the generated response to the terminal as text data.

[0534] 3. Providing a Response

[0535] The device converts the received text data into voice data using a speech synthesis engine, which is then played back to the user through the speaker.

[0536] For example, if the response from the server is "Today, major newspapers are reporting the following news: XXX, XXX, XXX," this will be provided as audio.

[0537] Specific examples

[0538] Specific examples of conversations

[0539] 1. The user says, "Good morning, what's on my schedule today?"

[0540] 2. The device converts the voice into text and sends the text data, "Good morning, what are your plans for today?" to the server.

[0541] 3. The server uses the generative AI model to generate a response: "Good morning, we have plans to go for a walk at 10 o'clock today."

[0542] 4. The device converts the received response into audio and plays it back to the user.

[0543] Feedback and Improvements

[0544] 1. The server stores all conversation data in a log and uses it as training data for the generative AI model.

[0545] 2. The generative AI model undergoes periodic retraining, gradually achieving more natural and richer conversations.

[0546] The system of this invention allows elderly people to enjoy daily conversations, which is expected to promote brain activity and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[0547] The processing flow will be explained below.

[0548] Step 1:

[0549] The user speaks into the terminal.

[0550] For example, say, "Hello, what can I do for you today?"

[0551] Step 2:

[0552] The device captures the audio.

[0553] A microphone collects the user's voice.

[0554] Step 3:

[0555] The device converts the speech to text.

[0556] The voice recognition engine converts the voice data into text data.

[0557] The converted text becomes "Hello, what can I do for you today?"

[0558] Step 4:

[0559] The terminal transmits the text data to the server.

[0560] A POST request is sent to the server's API via a communication network.

[0561] The request includes the converted text data.

[0562] Step 5:

[0563] The server receives the text data.

[0564] The server receives the request and extracts the text data.

[0565] Step 6:

[0566] The server logs the text data.

[0567] The received text data is saved in a database and recorded as a log.

[0568] Step 7:

[0569] The server inputs text data into the generative AI model.

[0570] A generative AI model analyzes the text data "Hello, what should I do today?"

[0571] Step 8:

[0572] A generative AI model generates the appropriate response.

[0573] The model uses internal data and algorithms to generate a response like, "Hello, how about enjoying some reading or a short walk today?"

[0574] Step 9:

[0575] The server sends the generated response to the terminal.

[0576] The generated text data is sent as an API response to be returned to the device.

[0577] Step 10:

[0578] The terminal receives the response text.

[0579] Receive the response to the request and extract the text data.

[0580] Step 11:

[0581] The device converts the received text data into voice.

[0582] The speech synthesis engine converts the text "Hello, how about enjoying some reading or a short walk today?" into speech data.

[0583] Step 12:

[0584] The terminal plays the audio to the user.

[0585] The audio data is played through the speaker to provide a response to the user.

[0586] Example 1

[0587] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] There is a demand for systems that allow elderly people to enjoy natural conversations in their daily lives. However, many current systems have low accuracy in speech recognition and response generation, making them difficult for elderly people to use. Furthermore, continuous learning is necessary for generative AI models to maintain a certain level of response quality, but this is often not achieved. Furthermore, conversation data is not managed, and logs are not kept to generate appropriate responses. This makes it difficult for elderly people to enjoy comfortable communication.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0590] In this invention, the server includes means for converting voice data into text data by performing voice recognition, means for transmitting the text data to the server via a communication network, means for generating a response to the text data using a generative AI model, means for converting the generated response into voice data by voice synthesis and providing the voice data to the user from the communication device, and means for recording the response in a log and periodically retraining the generative AI model. This allows elderly people to enjoy natural conversations and also enables improvement in the accuracy of the generative AI model and maintenance of response quality.

[0591] The term "elderly" generally refers to older people, especially those aged 65 and over.

[0592] A "communication device" is a device for inputting and outputting voice, and includes, for example, smartphones and smart speakers.

[0593] "Speech recognition" refers to the technology that converts a user's speech into text data.

[0594] "Text data" refers to character data converted using voice recognition technology.

[0595] "Communication network" refers to the network infrastructure for data communication, and specifically includes the Internet.

[0596] "Computer" refers to a device that has server functionality and processes and stores data.

[0597] "Generative AI model" refers to an artificial intelligence model that generates appropriate responses to input text data.

[0598] "Response" refers to the reply generated by the generative AI model in response to text data.

[0599] "Speech synthesis" refers to the technology of converting text data into voice data.

[0600] "Voice data" refers to digital data of voice generated by voice synthesis technology.

[0601] "Log" refers to data that records the system's operation history and conversation data exchanged.

[0602] "Relearning" refers to the learning process of improving the accuracy of a generative AI model based on previously logged data.

[0603] The present invention provides a system that allows elderly people to enjoy natural conversations. This system is configured using communication devices, a server, and a communication network. The following describes how this system is configured and how it functions.

[0604] System Overview

[0605] server

[0606] The server plays a central role in hosting the generative AI model and managing conversation data with users. The server has the following functions:

[0607] 1. Receiving and logging

[0608] The server receives the text data sent from the device and records it in a log, which includes the conversation history and user profile information (such as name, age, and hobbies).

[0609] 2. Response generation using a generative AI model

[0610] The received text data is input into a generative AI model to generate an appropriate response. The generative AI model can be, for example, a generative AI model.

[0611] 3. Sending the Response

[0612] The server transmits the generated response to the terminal as text data.

[0613] 4. Periodic Retraining

[0614] The server collects all conversation data and periodically retrains the generative AI model based on this data, improving the accuracy and naturalness of responses.

[0615] Terminal

[0616] A terminal is a device that is directly operated by a user and has the following functions:

[0617] 1. Voice Recognition

[0618] The microphone captures the user's speech into the device and converts it into text data using a speech recognition engine, such as voice recognition software.

[0619] 2. Texting

[0620] The converted text data is transmitted to a server via a communication network.

[0621] 3. Speech Synthesis

[0622] The text data received from the server is converted into voice data using a speech synthesis engine, which uses speech synthesis software.

[0623] 4. Audio playback

[0624] The converted audio data is played back to the user through a speaker.

[0625] user

[0626] The users are elderly people who can use the system in a natural manner without requiring any special technical knowledge or operation.

[0627] Specific examples

[0628] Specific examples of conversations

[0629] 1. User utterances

[0630] The user says, "Good morning, how's the weather today?"

[0631] 2. Voice recognition on the device

[0632] The device converts the voice into text and sends the text data, such as "Good morning, how's the weather today?" to the server.

[0633] 3. Server Response Generation

[0634] The server uses the generative AI model to generate a response: "Good morning, it's sunny today."

[0635] 4. Device speech synthesis

[0636] The device converts the received response into speech and plays it to the user: "Good morning, it's sunny today."

[0637] Prompt Sentence Examples

[0638] Here is an example of inputting the following prompt sentence into the generative AI model:

[0639] User input: "Good morning, how's the weather today?"

[0640] Sample prompt: "The user is asking, 'Good morning, how's the weather today?' Generate an appropriate response."

[0641] The system of this invention is expected to enable elderly people to enjoy natural conversations on a daily basis, promote brain activity, and contribute to the prevention of dementia. This system will also reduce the burden on staff at hospitals and nursing homes and improve communication with residents.

[0642] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0643] Step 1:

[0644] Voice input

[0645] A user speaks into a communication device. For example, they say something like, "Hello, what should I do today?" The input data is the user's voice. The output is the voice data captured by the communication device. Specifically, the user speaks into the microphone of the communication device, and the voice is recorded as digital voice data.

[0646] Step 2:

[0647] Voice Recognition

[0648] The device converts the captured voice data into text data using a voice recognition engine. For example, general voice recognition software is used as the voice recognition engine. The input data is voice data, and the output is text data converted by voice recognition. Specifically, the voice data is converted into the text data "Hello, what should I do today?"

[0649] Step 3:

[0650] Text sending

[0651] The terminal transmits the converted text data to the server via a communication network. The input data is the text data obtained by speech recognition, and the output is the status of the text data being transmitted to the server. Specifically, the text data is securely transmitted to the server via the Internet.

[0652] Step 4:

[0653] Response Generation

[0654] The server records the received text data in a log and inputs it into the generative AI model. The generative AI model analyzes the text data and generates an appropriate response. The input data is the received text data, and the output is the response text data generated by the generative AI model. Specifically, from the input "Hello, what should I do today?", a response such as "I recommend reading today" is generated.

[0655] Step 5:

[0656] Response Send

[0657] The server sends the generated response to the terminal as text data. The input data is the generated response text data, and the output is the status of the response text data being sent to the terminal. Specifically, the response text "I recommend reading today" is sent to the terminal.

[0658] Step 6:

[0659] Speech synthesis

[0660] The device converts the received response text data into voice data using a voice synthesis engine. For example, general voice synthesis software is used as the voice synthesis engine. The input data is the response text data, and the output is voice data generated by voice synthesis. Specifically, voice data is generated from the text "I recommend reading today."

[0661] Step 7:

[0662] Audio Output

[0663] The device plays the audio data to the user through the speaker. The input data is the audio data generated by speech synthesis, and the output is the audio that is played back to the user. Specifically, the speaker plays the audio "We recommend you read today," and the user can hear it.

[0664] The above are the specific processing steps of the program of this system.

[0665] (Application example 1)

[0666] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0667] Elderly people often rely on their family members or nursing home staff to ensure their safety and respond to emergencies. However, if these support services are insufficient, the elderly are at greater risk of facing emergencies. Elderly people may also feel lonely, which is becoming a social problem. Therefore, there is a need for a system that allows elderly people to easily call for help using their voice and receive prompt, appropriate assistance.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0669] In this invention, the server includes means for generating appropriate suggestions for emergency responses when an elderly person utters an emergency word requesting help, means for providing the appropriate suggestions to the user at the information terminal, and means for notifying emergency contacts based on the emergency response suggestions. This enables a prompt and appropriate response when an elderly person faces an emergency, ensuring the safety of the elderly and reducing their sense of loneliness.

[0670] "Elderly people" refers to people who are over a certain age as defined by the Elderly People Act, and who often require special assistance in daily life.

[0671] An "information terminal" is a device operated by a user, including smartphones and smart speakers.

[0672] "Speech recognition" refers to the process by which an information device captures a user's voice and converts that voice data into text data.

[0673] "Generative AI model" refers to an algorithm or system that uses AI technology to analyze text data and generate natural-sounding conversational responses.

[0674] "Emergency response" refers to a series of actions and procedures to provide appropriate and prompt assistance to elderly people when they face an emergency.

[0675] "Appropriate Recommendations" refer to recommendations that direct appropriate and effective actions or responses to be taken in a particular situation.

[0676] "Communication network" refers to a network system for exchanging data between information terminals and servers, and includes the Internet.

[0677] "Speech synthesis" refers to the technology of converting text data into audio data and playing that audio back to the user.

[0678] "Emergency contacts" refers to pre-registered contacts, such as family members or care staff, who should be contacted in the event of an emergency.

[0679] "Safety checks" refer to regularly checking on the current status of elderly people and asking questions and making checks to ensure their safety.

[0680] "Means for generating a response" refers to the function or method by which the server uses a generative AI model to generate a response to the received text data.

[0681] "Voice data" refers to data generated using a voice synthesis engine for provision to the user by an information terminal.

[0682] This invention relates to a system that checks the safety of elderly people and responds to emergencies using information terminals, servers, and communication networks used by elderly people. The system aims to utilize generative AI models to have natural conversations with elderly people and provide appropriate responses in emergencies.

[0683] System configuration

[0684] server

[0685] The server hosts a generative AI model that generates responses based on the text data sent by the user. The generative AI model consists of an algorithm that analyzes the user's input text and generates the optimal response.

[0686] The server includes a natural language generation model such as GPT-2 as a means to generate responses, as well as an algorithm for detecting urgent words.

[0687] Information terminal

[0688] An information terminal is a device operated by a user, such as a smartphone or a smart speaker. An information terminal is equipped with a microphone and a speaker to capture and play back the user's voice.

[0689] The device is equipped with a voice recognition function that converts the user's speech into text, which is then sent to a server via a communications network.

[0690] communication network

[0691] Data communication between the information terminal and the server is carried out over the Internet, which allows for real-time communication.

[0692] System Operation

[0693] 1. Regular safety checks

[0694] The device periodically asks the elderly person questions to check their safety, such as "How are you?" and "Is there anything I can help you with?" This allows the device to periodically check the elderly person's condition.

[0695] The user responds to these questions verbally, which is converted into text by the device and sent to the server.

[0696] 2. Conversational Response Generation

[0697] The server analyzes the received text data and generates a response using a generative AI model, which is then sent to the device as text.

[0698] The device converts the received response into voice data using a voice synthesis engine and plays it back to the user.

[0699] 3. Emergency Response

[0700] When a user utters an emergency word such as "help," the device detects it and sends it to the server, which then generates appropriate suggestions for emergency response, such as "Should we call an ambulance?", and sends them to the device.

[0701] If necessary, the system will also notify pre-registered emergency contacts via email, SMS, or other means.

[0702] Program processing explanation

[0703] Speech recognition: Using the speech_recognition library, the information terminal analyzes the user's voice and converts it into text data.

[0704] Generative AI model: Implement a GPT-2 model using the transformers library to generate appropriate responses from text data received on the server.

[0705] Speech synthesis: Uses the pyttsx3 library to convert the generated text response into audio data and play it back to the user on the device.

[0706] Emergency response: Uses algorithms to detect specific emergency words, generates appropriate suggestions in an emergency, and notifies emergency contacts as needed.

[0707] Specific examples

[0708] Regular safety checks:

[0709] The device periodically asks, "How are you?", and the user responds, "I'm fine." This response is converted into text and analyzed by the server to generate follow-up questions such as, "That's great. Is there anything I can help you with today?"

[0710] Example prompt sentence:

[0711] Senior citizen: "What's the news today?"

[0712] Prompt: "Today's news"

[0713] The generative AI model responds, "Today's news is that XX was reported in major newspapers."

[0714] This allows elderly people to receive emergency response in real time while enjoying natural conversation.

[0715] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0716] Step 1:

[0717] The information terminal uses a microphone to capture the user's voice. The captured voice data is input. The voice data is converted into text data using a speech recognition engine (speech_recognition library). The converted text data is output.

[0718] Step 2:

[0719] The information terminal transmits the converted text data to a server via a communication network (Internet). At this time, data calculation is performed in which the text data is input and transmitted to the server using a communication protocol.

[0720] Step 3:

[0721] The server inputs the received text data into a generative AI model (GPT-2) and generates an appropriate response. The generative AI model performs text analysis and generates a response based on the prompt. This response is output. For example, in response to the prompt "Today's news is...", the response generated is "Today's news is that XX was reported in major newspapers."

[0722] Step 4:

[0723] The server transmits the generated response as text data to the information terminal. The text data is input and then processed by transmitting it to the information terminal via a communication network.

[0724] Step 5:

[0725] The information terminal converts the received text data into voice data using a speech synthesis engine (pyttsx3 library). Text data is input and voice data is output. The response converted into voice data is played back to the user through the speaker.

[0726] Step 6:

[0727] The device periodically asks the user safety confirmation questions and converts the responses to those questions into text data using a speech recognition engine. For example, the question "How are you?" is treated as a prompt, and the user responds "I'm fine." This response is then converted into text data.

[0728] Step 7:

[0729] When a user utters an emergency word (e.g., "Help"), the device detects the data and sends it to the server. The server generates appropriate suggestions for emergency response (e.g., "Should I call an ambulance?") and sends them to the device. The device then converts the generated suggestions into audio data and plays them back to the user.

[0730] Step 8:

[0731] If necessary, the server notifies pre-registered emergency contacts by entering their information and sending a notification using a communication protocol.

[0732] This will enable elderly people to enjoy natural conversation while receiving emergency response in real time, ensuring their safety.

[0733] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0734] This invention provides a system that can provide more natural and emotional responses by combining a conversation system using a generative AI model with an emotion engine that recognizes the user's emotions. The entire system is composed of information terminals, a server, and a communication network.

[0735] System configuration

[0736] server

[0737] The server hosts the generative AI model and emotion engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[0738] The emotion engine analyzes the received voice data to recognize and evaluate the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[0739] The server also stores user profile information and conversation history in a database for periodic retraining of the generative AI model.

[0740] Terminal

[0741] A device is a device operated by a user, such as a smartphone or a smart speaker. A device has a microphone and a speaker and can capture and play back the user's voice.

[0742] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[0743] Furthermore, the terminal can display the user's emotional state transmitted from the server.

[0744] user

[0745] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[0746] System Operation

[0747] 1. Start a conversation

[0748] The user speaks to the information terminal, for example, saying, "Hello, I'm a little tired today."

[0749] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the converted text and voice data to a server via a communications network.

[0750] 2. Emotion recognition

[0751] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state (e.g., tired, fatigued tone).

[0752] The emotion engine evaluates the emotional state and returns the result to the server.

[0753] 3. Generating a Response

[0754] Based on the evaluation results from the emotion engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[0755] For example, if the user says, "Hello, I'm a little tired today," the response might be, "Hello, I suggest you take a little rest and then think about what you want to do next."

[0756] 4. Providing a Response

[0757] The server transmits the generated text data to the terminal, which converts the received text data into voice data using a voice synthesis engine and plays it back to the user.

[0758] The terminal can also visually display the evaluation results by the emotion engine to the user.

[0759] Specific examples

[0760] Specific examples of conversations

[0761] 1. The user says, "I'm very happy today."

[0762] 2. The device converts the voice into text and sends the text data, "I'm so happy today," and the voice data to the server.

[0763] 3. The server inputs the voice data into the emotion engine and evaluates the emotional state as "happy."

[0764] 4. The server uses the generative AI model to generate a response: "That's great! What did you do today?"

[0765] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[0766] Feedback and Improvements

[0767] 1. The server stores all conversation data and emotion evaluation data in logs and uses them as training data for the generative AI model and emotion engine.

[0768] 2. The generative AI model and emotion engine are periodically retrained to gradually achieve more natural and emotionally relevant responses.

[0769] The system of this invention allows elderly people to enjoy daily conversations that take their emotions into consideration, which is expected to promote brain activation and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[0770] The processing flow will be explained below.

[0771] Step 1:

[0772] The user speaks into the terminal.

[0773] For example, say, "Hello, I'm very happy today."

[0774] Step 2:

[0775] The device captures the audio.

[0776] The device's microphone collects the user's voice.

[0777] Step 3:

[0778] The device converts the speech to text.

[0779] The voice recognition engine converts the voice data into text data.

[0780] The converted text becomes "Hello, I'm very happy today."

[0781] Step 4:

[0782] The terminal transmits text data and voice data to the server.

[0783] A POST request is sent to the server's API via a communication network.

[0784] The request includes the converted text data and audio data.

[0785] Step 5:

[0786] The server receives the text data and the voice data.

[0787] The server receives the request and extracts this data.

[0788] Step 6:

[0789] The server inputs the voice data into the emotion engine.

[0790] An emotion engine analyzes the voice data and assesses the user's emotional state.

[0791] For example, the emotion engine recognizes the emotional state "happy."

[0792] Step 7:

[0793] The server inputs the emotion engine's evaluation results into the generative AI model.

[0794] The text data "Hello, I'm very happy today" and the emotional information "happy" are input into the generative AI model.

[0795] Step 8:

[0796] A generative AI model generates the appropriate response.

[0797] The model uses internal data and algorithms to generate a response: "That's great! What did you do today?"

[0798] Step 9:

[0799] The server sends the generated response to the terminal.

[0800] Send an API response to return the generated text data to the device.

[0801] Step 10:

[0802] The terminal receives the response text.

[0803] Receive the response to the request and extract the text data.

[0804] Step 11:

[0805] The device converts the received text data into voice.

[0806] The speech synthesis engine converts the text "That's great! What did you do today?" into speech data.

[0807] Step 12:

[0808] The terminal plays the audio to the user.

[0809] The audio data is played through the speaker to provide a response to the user.

[0810] Step 13:

[0811] The terminal displays the emotion evaluation result to the user.

[0812] For example, the screen might display "Emotion: Happy."

[0813] Step 14:

[0814] The server stores the conversation data and emotion evaluation data in a log.

[0815] All conversations and sentiment assessment details are recorded in a database.

[0816] Step 15:

[0817] The server periodically retrains the generative AI model and emotion engine.

[0818] The accuracy of the model is improved using collected conversational and emotional data.

[0819] Example 2

[0820] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0821] In conversation systems using information terminals used by elderly people, conventional systems have had the problem of not taking the user's emotions into sufficient consideration when responding, making the conversation feel unnatural. Furthermore, because the system does not reflect the user's emotional state, the conversation may not be very satisfying. Furthermore, these systems do not fully utilize the user's profile information or conversation history, which means that they are unable to provide responses based on the user's past conversations and their emotions at the time.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0823] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for recognizing and evaluating the user's emotional state from the received voice data using an emotion engine, and means for adjusting the response based on the user's emotional state. This enables a natural and satisfying response that reflects the user's emotions. Furthermore, the generated response is converted into voice data using voice synthesis and provided to the user from the information terminal, making the conversation smoother. Furthermore, the server stores the user's profile information and conversation history, and the generative AI model receives feedback based on this information and re-learns, improving the quality of the conversation.

[0824] "Elderly" refers to people who are relatively old and require physical or cognitive assistance.

[0825] An "information terminal" is a device operated by a user that has input, output, and communication functions. Examples include smartphones and smart speakers.

[0826] "Speech recognition" refers to the technology of analyzing voice data and converting it into text data.

[0827] "Text data" refers to text information converted from voice data using voice recognition technology.

[0828] "Communications network" refers to the network infrastructure for data communication between information terminals and servers, including the Internet.

[0829] "Server" refers to the computer system that hosts the generative AI model and emotion engine, receives data sent by users, analyzes it, and responds.

[0830] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-sounding sentences based on input data.

[0831] "Emotion engine" refers to technology that analyzes voice data and recognizes and evaluates the user's emotional state.

[0832] "Speech synthesis" refers to the technology of generating voice data based on text data.

[0833] "Profile Information" refers to data that records personal information and characteristics of a user.

[0834] "Conversation history" refers to data that records the contents of past conversations that a user has had with the system.

[0835] "Relearning" refers to the training process used to improve the performance of generative AI models and emotion engines based on stored data.

[0836] "Response" refers to the reply that the generative AI model gives in response to input from the user.

[0837] "Audio data" refers to data that represents audio in digital form.

[0838] This invention is a system that realizes natural conversations using information terminals used by elderly people and generative AI models. This invention can provide responses that take into account the user's emotions, enabling more human-like communication.

[0839] System Configuration

[0840] The system consists of the following main components: information terminals, servers, and communication networks.

[0841] Information terminal

[0842] An information terminal is a device operated by a user, and includes smartphones, smart speakers, etc. The terminal is equipped with a microphone and speaker, captures the user's voice, and plays a response voice using voice synthesis.

[0843] server

[0844] The server hosts the generative AI model and emotion engine, which play a central role in receiving voice and text data sent by users and generating appropriate responses. The generative AI model uses a natural language generation algorithm such as GPT-3. The emotion engine recognizes the user's emotional state from the voice data using, for example, an emotion analysis API.

[0845] communication network

[0846] A communication network enables data communication between information terminals and servers, and typically the Internet is used.

[0847] Specific examples of processing

[0848] The specific operation of the system will now be described.

[0849] 1. Start a conversation

[0850] User

[0851] The user speaks to the information terminal, for example, saying, "I'm very happy today."

[0852] Terminal

[0853] The device uses a microphone to capture voice data, converts it into text data using a voice recognition engine, and then transmits the text data and voice data to a server via a communication network.

[0854] 2. Emotion recognition

[0855] server

[0856] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state. For example, in this case, the emotional state of "happy" is recognized.

[0857] 3. Generating a Response

[0858] server

[0859] The server inputs the emotion engine's evaluation results and text data into the generative AI model to generate an appropriate response, such as "That's great! What did you do today?"

[0860] 4. Providing a Response

[0861] Terminal

[0862] The terminal inputs the response text sent from the server into a speech synthesis engine, converts it into voice data, and plays it back to the user through the speaker.

[0863] Feedback and Improvements

[0864] server

[0865] The server stores all conversation data and emotion evaluation data in logs, which are used to retrain the generative AI model and emotion engine, allowing the system to gradually develop more natural and emotionally relevant responses, improving the user experience.

[0866] Prompt Sentence Examples

[0867] Here is an example prompt:

[0868] "When a user inputs words, the voice data and emotion data are sent to the server. The server uses an emotion engine to recognize the emotion, and based on the results, a generative AI model generates an appropriate response. The generated text is sent to the device and played back to the user by the speech synthesis engine."

[0869] This system allows elderly people to enjoy natural conversations that take their emotions into consideration, and is expected to be highly effective when used in hospitals and nursing homes.

[0870] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] User initiated conversation

[0873] Input: User's spoken utterance

[0874] Specific action: The user speaks to the information terminal. For example, he says, "Hello, I'm a little tired today."

[0875] Output: User's voice data

[0876] Step 2:

[0877] Device capture and conversion of audio data

[0878] Input: User's voice data

[0879] What it does: The device uses the built-in microphone to capture the user's voice and converts it into text using a speech recognition engine (e.g., Google Speech-to-Text API).

[0880] Output: Converted text data and original audio data

[0881] Step 3:

[0882] Sending data from the device to the server

[0883] Input: Text data, audio data

[0884] Specific operation: The terminal transmits the converted text data and the original voice data to a server via a communication network (e.g., the Internet).

[0885] Output: Text and audio data sent to the server

[0886] Step 4:

[0887] Emotion recognition by server

[0888] Input: Transmitted audio data

[0889] Specific operation: The server inputs the received voice data into an emotion engine (e.g., Microsoft Azure Emotion API) to analyze and evaluate the user's emotional state. In this case, the emotional state of "tired" is recognized from the voice data.

[0890] Output: Emotional state data (e.g., "Tired")

[0891] Step 5:

[0892] Server response generation

[0893] Input: Text data, emotional state data

[0894] Specific operation: The server inputs the emotional state data and text data obtained from the emotion engine into a generative AI model (e.g., GPT-3). The generative AI model generates an appropriate response based on this data. For example, it generates a response such as, "Hello, I recommend you take a short break. Then, think about what you want to do next."

[0895] Output: The generated response text

[0896] Step 6:

[0897] Sending response data from the server to the terminal

[0898] Input: Generated response text

[0899] Specific operation: The server sends the generated response text to the terminal.

[0900] Output: Response text sent to the terminal

[0901] Step 7:

[0902] Voice generation and playback of responses by the terminal

[0903] Input: Response text data

[0904] Specific operation: The device inputs the received response text into a speech synthesis engine (e.g., Amazon Polly) to convert the text into voice data, which the device then plays back to the user through the speaker.

[0905] Output: A spoken response to be played to the user (e.g., "Hello, I suggest you take a break and think about what you want to do next.")

[0906] Step 8:

[0907] Feedback and Data Storage

[0908] Input: Conversation data, emotion evaluation data

[0909] Specific operation: The server stores all conversation data and emotion evaluation data in a log, which is used to retrain the generative AI model and emotion engine.

[0910] Output: Updated training data

[0911] Through the above steps, users can enjoy natural conversations that take emotions into consideration.

[0912] (Application example 2)

[0913] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0914] To enable elderly people to enjoy more natural and emotionally appropriate conversations on information devices, simply generating responses based on text data is not enough. Conventional systems are unable to generate responses that take the user's emotional state into account, and are therefore unable to provide satisfying interactions for users. Furthermore, particularly when serving customers in brick-and-mortar stores, staff are required to grasp the customer's emotional state in real time and provide services accordingly. Therefore, a more accurate conversation system that also takes the user's emotions into account is needed.

[0915] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0916] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for using an emotion recognition engine to perform emotion analysis on the voice data from the user, means for adjusting the response of the generative AI model based on the emotion analysis results, and means for visually displaying the response. This enables natural and satisfying conversations that correspond to the user's emotions, and makes it possible to provide flexible customer service based on the emotional state of customers, particularly in physical stores.

[0917] 1. An "information terminal" is a device operated by a user that has the ability to capture voice and convert it into text data.

[0918] 2. "Speech recognition" is a technology that converts a user's voice data into text data.

[0919] 3. "Communications network" means the infrastructure for transmitting and receiving data between information terminals and servers, including the Internet.

[0920] 4. "Server" means a computing system that hosts the generative AI model and emotion recognition engine and manages user conversation data and emotion data.

[0921] 5. A "generative AI model" is an algorithm that generates appropriate responses based on input text data.

[0922] 6. An "emotion recognition engine" is a technology that analyzes a user's voice data and evaluates their emotional state.

[0923] 7. “Sentiment analysis” is the process of identifying a user’s emotional state based on audio data.

[0924] 8. "Visually displaying" means presenting the results of an analysis or generated response to the user as text or graphics on a screen.

[0925] 9. "Speech synthesis" is a technology that converts text data into voice data.

[0926] 10. "Brick and Mortar Store" means a store that exists in a physical location and offers goods and services.

[0927] 11. "Customer service" refers to the service and interaction provided by staff to customers in physical stores.

[0928] This invention is a conversation system using an information terminal and a server for use by elderly people, which can provide natural conversation according to the user's emotions. This system is configured using an information terminal, a server, and a communication network. Specific embodiments for implementing this invention are described in detail below.

[0929] System configuration

[0930] server

[0931] The server hosts the generative AI model and emotion recognition engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[0932] The emotion recognition engine analyzes the received voice data to assess the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[0933] The server also stores user profile information and conversation history in a database, which is used to periodically retrain the generative AI model.

[0934] Terminal

[0935] A device is a user-operated device, such as smart glasses, a smartphone, or a smart speaker. A device has a microphone and a speaker to capture and play back the user's voice.

[0936] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[0937] Furthermore, the terminal can visually display the user's emotional state transmitted from the server.

[0938] user

[0939] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[0940] System Operation

[0941] 1. Start a conversation

[0942] The user speaks to the information terminal, for example, saying, "I want to know more about this product."

[0943] The device captures the voice with a microphone and converts the voice into text using voice recognition, which is then sent to a server via a communications network.

[0944] 2. Emotion recognition

[0945] The server inputs the received voice data into an emotion recognition engine to evaluate the user's emotional state (e.g., interested, happy, etc.).

[0946] The emotion recognition engine evaluates the emotional state and returns the results to the server.

[0947] 3. Generating a Response

[0948] Based on the evaluation results from the emotion recognition engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[0949] For example, if a user says, "I'd like to know more about this product," the response might be, "I'd like to tell you more about this product. Is there anything in particular you'd like to know?"

[0950] 4. Providing a Response

[0951] The server sends the generated text data to the terminal, which then converts the received text data into voice data using a speech synthesis engine and plays it back to the user.

[0952] The device can also visually display the evaluation results from the emotion recognition engine to the user.

[0953] Specific examples

[0954] Specific examples of conversations

[0955] 1. A user says, "I'd like to know more about this product."

[0956] 2. The device converts the voice into text and sends the text data, "I want to know more about this product," and the voice data to the server.

[0957] 3. The server inputs the voice data into an emotion recognition engine and evaluates the emotional state as "interested."

[0958] 4. The server uses the generative AI model to generate a response that says, "We'll explain this product in detail. Is there anything in particular you'd like to know?"

[0959] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[0960] Prompt Sentence Examples

[0961] The user's sentiment is 'interested'. Generate a sentiment-based response to the following text: "I'd like to know more about this product"

[0962] In this way, natural and meaningful conversations that take the user's emotions into account are realized.

[0963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0964] Step 1:

[0965] A user speaks to an information terminal, for example, saying, "I'd like to know more about this product." The input is the user's voice, which is captured via a microphone. The output is raw voice data.

[0966] Step 2:

[0967] The device inputs the captured voice data into a voice recognition engine, which converts the voice data into text data. The voice recognition engine processes the voice data and outputs text data. Here, the input is raw voice data, and the output is text data.

[0968] Step 3:

[0969] The terminal transmits the converted text data and voice data to the server via the communication network. The input of this step is the text data and voice data, and the output is the data transmitted to the server.

[0970] Step 4:

[0971] The server inputs the received voice data into the emotion recognition engine to evaluate the user's emotional state. In this step, the emotion recognition engine analyzes the voice data to identify the emotional state and outputs the result. The input is the voice data, and the output is the evaluation result of the emotional state.

[0972] Step 5:

[0973] The server inputs the text data into the generative AI model along with the evaluation results from the emotion recognition engine. The generative AI model generates an optimal response based on the text data and the emotion evaluation results. The input for this step is the text data and the emotion evaluation results, and the output is the generated response text.

[0974] Step 6:

[0975] The server inputs the generated text data into a speech synthesis engine and converts it into voice data. The speech synthesis engine processes the text data and outputs voice data. The input is the response text data, and the output is the generated voice data.

[0976] Step 7:

[0977] The device plays the audio data sent from the server and provides it to the user. The device plays this audio data through a speaker so that the user can hear it. The device also visually displays the evaluation results from the emotion recognition engine. The input here is the generated audio data and the emotion evaluation result, and the output is the audio played to the user and the displayed emotional state.

[0978] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0979] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0981] [Third embodiment]

[0982] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0983] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0989] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0993] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0994] This invention provides a system that uses a generative AI model to enable elderly people to enjoy natural conversations. This system is composed of an information terminal, a server, and a communication network.

[0995] System configuration

[0996] server

[0997] The server hosts the generative AI model and manages conversation data with the user. The server receives text data sent by the user and generates appropriate responses using the generative AI model.

[0998] The server also stores user profile information (such as name, age, and hobbies) in a database, records conversation history, and periodically retrains the generative AI model to improve response accuracy.

[0999] Terminal

[1000] A device is a user-operated device, such as a smartphone or smart speaker, that has a microphone and speaker to capture and play back the user's voice.

[1001] The device has a voice recognition function and converts the user's voice data into text data. This text data is sent to the server, and the response received from the server is played back using a voice synthesis function.

[1002] user

[1003] The users are elderly people who enjoy conversations by speaking directly into the information terminal. Users can use the system in a natural manner without any special operations.

[1004] System Operation

[1005] 1. Start a conversation

[1006] The user speaks to the information terminal, asking questions such as, "Hello, what's in the news today?"

[1007] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the text data to a server via a communications network.

[1008] 2. Generating a Response

[1009] The server logs the received text data and inputs it into the generative AI model, which then analyzes the text data and generates an appropriate response.

[1010] The server transmits the generated response to the terminal as text data.

[1011] 3. Providing a Response

[1012] The device converts the received text data into voice data using a speech synthesis engine, which is then played back to the user through the speaker.

[1013] For example, if the response from the server is "Today, major newspapers are reporting the following news: XXX, XXX, XXX," this will be provided as audio.

[1014] Specific examples

[1015] Specific examples of conversations

[1016] 1. The user says, "Good morning, what's on my schedule today?"

[1017] 2. The device converts the voice into text and sends the text data, "Good morning, what are your plans for today?" to the server.

[1018] 3. The server uses the generative AI model to generate a response: "Good morning, we have plans to go for a walk at 10 o'clock today."

[1019] 4. The device converts the received response into audio and plays it back to the user.

[1020] Feedback and Improvements

[1021] 1. The server stores all conversation data in a log and uses it as training data for the generative AI model.

[1022] 2. The generative AI model undergoes periodic retraining, gradually achieving more natural and richer conversations.

[1023] The system of this invention allows elderly people to enjoy daily conversations, which is expected to promote brain activity and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[1024] The processing flow will be explained below.

[1025] Step 1:

[1026] The user speaks into the terminal.

[1027] For example, say, "Hello, what can I do for you today?"

[1028] Step 2:

[1029] The device captures the audio.

[1030] A microphone collects the user's voice.

[1031] Step 3:

[1032] The device converts the speech to text.

[1033] The voice recognition engine converts the voice data into text data.

[1034] The converted text becomes "Hello, what can I do for you today?"

[1035] Step 4:

[1036] The terminal transmits the text data to the server.

[1037] A POST request is sent to the server's API via a communication network.

[1038] The request includes the converted text data.

[1039] Step 5:

[1040] The server receives the text data.

[1041] The server receives the request and extracts the text data.

[1042] Step 6:

[1043] The server logs the text data.

[1044] The received text data is saved in a database and recorded as a log.

[1045] Step 7:

[1046] The server inputs text data into the generative AI model.

[1047] A generative AI model analyzes the text data "Hello, what should I do today?"

[1048] Step 8:

[1049] A generative AI model generates the appropriate response.

[1050] The model uses internal data and algorithms to generate a response like, "Hello, how about enjoying some reading or a short walk today?"

[1051] Step 9:

[1052] The server sends the generated response to the terminal.

[1053] The generated text data is sent as an API response to be returned to the device.

[1054] Step 10:

[1055] The terminal receives the response text.

[1056] Receive the response to the request and extract the text data.

[1057] Step 11:

[1058] The device converts the received text data into voice.

[1059] The speech synthesis engine converts the text "Hello, how about enjoying some reading or a short walk today?" into speech data.

[1060] Step 12:

[1061] The terminal plays the audio to the user.

[1062] The audio data is played through the speaker to provide a response to the user.

[1063] Example 1

[1064] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] There is a demand for systems that allow elderly people to enjoy natural conversations in their daily lives. However, many current systems have low accuracy in speech recognition and response generation, making them difficult for elderly people to use. Furthermore, continuous learning is necessary for generative AI models to maintain a certain level of response quality, but this is often not achieved. Furthermore, conversation data is not managed, and logs are not kept to generate appropriate responses. This makes it difficult for elderly people to enjoy comfortable communication.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1067] In this invention, the server includes means for converting voice data into text data by performing voice recognition, means for transmitting the text data to the server via a communication network, means for generating a response to the text data using a generative AI model, means for converting the generated response into voice data by voice synthesis and providing the voice data to the user from the communication device, and means for recording the response in a log and periodically retraining the generative AI model. This allows elderly people to enjoy natural conversations and also enables improvement in the accuracy of the generative AI model and maintenance of response quality.

[1068] The term "elderly" generally refers to older people, especially those aged 65 and over.

[1069] A "communication device" is a device for inputting and outputting voice, and includes, for example, smartphones and smart speakers.

[1070] "Speech recognition" refers to the technology that converts a user's speech into text data.

[1071] "Text data" refers to character data converted using voice recognition technology.

[1072] "Communication network" refers to the network infrastructure for data communication, and specifically includes the Internet.

[1073] "Computer" refers to a device that has server functionality and processes and stores data.

[1074] "Generative AI model" refers to an artificial intelligence model that generates appropriate responses to input text data.

[1075] "Response" refers to the reply generated by the generative AI model in response to text data.

[1076] "Speech synthesis" refers to the technology of converting text data into voice data.

[1077] "Voice data" refers to digital data of voice generated by voice synthesis technology.

[1078] "Log" refers to data that records the system's operation history and conversation data exchanged.

[1079] "Relearning" refers to the learning process of improving the accuracy of a generative AI model based on previously logged data.

[1080] The present invention provides a system that allows elderly people to enjoy natural conversations. This system is configured using communication devices, a server, and a communication network. The following describes how this system is configured and how it functions.

[1081] System Overview

[1082] server

[1083] The server plays a central role in hosting the generative AI model and managing conversation data with users. The server has the following functions:

[1084] 1. Receiving and logging

[1085] The server receives the text data sent from the device and records it in a log, which includes the conversation history and user profile information (such as name, age, and hobbies).

[1086] 2. Response generation using a generative AI model

[1087] The received text data is input into a generative AI model to generate an appropriate response. The generative AI model can be, for example, a generative AI model.

[1088] 3. Sending the Response

[1089] The server transmits the generated response to the terminal as text data.

[1090] 4. Periodic Retraining

[1091] The server collects all conversation data and periodically retrains the generative AI model based on this data, improving the accuracy and naturalness of responses.

[1092] Terminal

[1093] A terminal is a device that is directly operated by a user and has the following functions:

[1094] 1. Voice Recognition

[1095] The microphone captures the user's speech into the device and converts it into text data using a speech recognition engine, such as voice recognition software.

[1096] 2. Texting

[1097] The converted text data is transmitted to a server via a communication network.

[1098] 3. Speech Synthesis

[1099] The text data received from the server is converted into voice data using a speech synthesis engine, which uses speech synthesis software.

[1100] 4. Audio playback

[1101] The converted audio data is played back to the user through a speaker.

[1102] user

[1103] The users are elderly people who can use the system in a natural manner without requiring any special technical knowledge or operation.

[1104] Specific examples

[1105] Specific examples of conversations

[1106] 1. User utterances

[1107] The user says, "Good morning, how's the weather today?"

[1108] 2. Voice recognition on the device

[1109] The device converts the voice into text and sends the text data, such as "Good morning, how's the weather today?" to the server.

[1110] 3. Server Response Generation

[1111] The server uses the generative AI model to generate a response: "Good morning, it's sunny today."

[1112] 4. Device speech synthesis

[1113] The device converts the received response into speech and plays it to the user: "Good morning, it's sunny today."

[1114] Prompt Sentence Examples

[1115] Here is an example of inputting the following prompt sentence into the generative AI model:

[1116] User input: "Good morning, how's the weather today?"

[1117] Sample prompt: "The user is asking, 'Good morning, how's the weather today?' Generate an appropriate response."

[1118] The system of this invention is expected to enable elderly people to enjoy natural conversations on a daily basis, promote brain activity, and contribute to the prevention of dementia. This system will also reduce the burden on staff at hospitals and nursing homes and improve communication with residents.

[1119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1120] Step 1:

[1121] Voice input

[1122] A user speaks into a communication device. For example, they say something like, "Hello, what should I do today?" The input data is the user's voice. The output is the voice data captured by the communication device. Specifically, the user speaks into the microphone of the communication device, and the voice is recorded as digital voice data.

[1123] Step 2:

[1124] Voice Recognition

[1125] The device converts the captured voice data into text data using a voice recognition engine. For example, general voice recognition software is used as the voice recognition engine. The input data is voice data, and the output is text data converted by voice recognition. Specifically, the voice data is converted into the text data "Hello, what should I do today?"

[1126] Step 3:

[1127] Text sending

[1128] The terminal transmits the converted text data to the server via a communication network. The input data is the text data obtained by speech recognition, and the output is the status of the text data being transmitted to the server. Specifically, the text data is securely transmitted to the server via the Internet.

[1129] Step 4:

[1130] Response Generation

[1131] The server records the received text data in a log and inputs it into the generative AI model. The generative AI model analyzes the text data and generates an appropriate response. The input data is the received text data, and the output is the response text data generated by the generative AI model. Specifically, from the input "Hello, what should I do today?", a response such as "I recommend reading today" is generated.

[1132] Step 5:

[1133] Response Send

[1134] The server sends the generated response to the terminal as text data. The input data is the generated response text data, and the output is the status of the response text data being sent to the terminal. Specifically, the response text "I recommend reading today" is sent to the terminal.

[1135] Step 6:

[1136] Speech synthesis

[1137] The device converts the received response text data into voice data using a voice synthesis engine. For example, general voice synthesis software is used as the voice synthesis engine. The input data is the response text data, and the output is voice data generated by voice synthesis. Specifically, voice data is generated from the text "I recommend reading today."

[1138] Step 7:

[1139] Audio Output

[1140] The device plays the audio data to the user through the speaker. The input data is the audio data generated by speech synthesis, and the output is the audio that is played back to the user. Specifically, the speaker plays the audio "We recommend you read today," and the user can hear it.

[1141] The above are the specific processing steps of the program of this system.

[1142] (Application example 1)

[1143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] Elderly people often rely on their family members or nursing home staff to ensure their safety and respond to emergencies. However, if these support services are insufficient, the elderly are at greater risk of facing emergencies. Elderly people may also feel lonely, which is becoming a social problem. Therefore, there is a need for a system that allows elderly people to easily call for help using their voice and receive prompt, appropriate assistance.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1146] In this invention, the server includes means for generating appropriate suggestions for emergency responses when an elderly person utters an emergency word requesting help, means for providing the appropriate suggestions to the user at the information terminal, and means for notifying emergency contacts based on the emergency response suggestions. This enables a prompt and appropriate response when an elderly person faces an emergency, ensuring the safety of the elderly and reducing their sense of loneliness.

[1147] "Elderly people" refers to people who are over a certain age as defined by the Elderly People Act, and who often require special assistance in daily life.

[1148] An "information terminal" is a device operated by a user, including smartphones and smart speakers.

[1149] "Speech recognition" refers to the process by which an information device captures a user's voice and converts that voice data into text data.

[1150] "Generative AI model" refers to an algorithm or system that uses AI technology to analyze text data and generate natural-sounding conversational responses.

[1151] "Emergency response" refers to a series of actions and procedures to provide appropriate and prompt assistance to elderly people when they face an emergency.

[1152] "Appropriate Recommendations" refer to recommendations that direct appropriate and effective actions or responses to be taken in a particular situation.

[1153] "Communication network" refers to a network system for exchanging data between information terminals and servers, and includes the Internet.

[1154] "Speech synthesis" refers to the technology of converting text data into audio data and playing that audio back to the user.

[1155] "Emergency contacts" refers to pre-registered contacts, such as family members or care staff, who should be contacted in the event of an emergency.

[1156] "Safety checks" refer to regularly checking on the current status of elderly people and asking questions and making checks to ensure their safety.

[1157] "Means for generating a response" refers to the function or method by which the server uses a generative AI model to generate a response to the received text data.

[1158] "Voice data" refers to data generated using a voice synthesis engine for provision to the user by an information terminal.

[1159] This invention relates to a system that checks the safety of elderly people and responds to emergencies using information terminals, servers, and communication networks used by elderly people. The system aims to utilize generative AI models to have natural conversations with elderly people and provide appropriate responses in emergencies.

[1160] System configuration

[1161] server

[1162] The server hosts a generative AI model that generates responses based on the text data sent by the user. The generative AI model consists of an algorithm that analyzes the user's input text and generates the optimal response.

[1163] The server includes a natural language generation model such as GPT-2 as a means to generate responses, as well as an algorithm for detecting urgent words.

[1164] Information terminal

[1165] An information terminal is a device operated by a user, such as a smartphone or a smart speaker. An information terminal is equipped with a microphone and a speaker to capture and play back the user's voice.

[1166] The device is equipped with a voice recognition function that converts the user's speech into text, which is then sent to a server via a communications network.

[1167] communication network

[1168] Data communication between the information terminal and the server is carried out over the Internet, which allows for real-time communication.

[1169] System Operation

[1170] 1. Regular safety checks

[1171] The device periodically asks the elderly person questions to check their safety, such as "How are you?" and "Is there anything I can help you with?" This allows the device to periodically check the elderly person's condition.

[1172] The user responds to these questions verbally, which is converted into text by the device and sent to the server.

[1173] 2. Conversational Response Generation

[1174] The server analyzes the received text data and generates a response using a generative AI model, which is then sent to the device as text.

[1175] The device converts the received response into voice data using a voice synthesis engine and plays it back to the user.

[1176] 3. Emergency Response

[1177] When a user utters an emergency word such as "help," the device detects it and sends it to the server, which then generates appropriate suggestions for emergency response, such as "Should we call an ambulance?", and sends them to the device.

[1178] If necessary, the system will also notify pre-registered emergency contacts via email, SMS, or other means.

[1179] Program processing explanation

[1180] Speech recognition: Using the speech_recognition library, the information terminal analyzes the user's voice and converts it into text data.

[1181] Generative AI model: Implement a GPT-2 model using the transformers library to generate appropriate responses from text data received on the server.

[1182] Speech synthesis: Uses the pyttsx3 library to convert the generated text response into audio data and play it back to the user on the device.

[1183] Emergency response: Uses algorithms to detect specific emergency words, generates appropriate suggestions in an emergency, and notifies emergency contacts as needed.

[1184] Specific examples

[1185] Regular safety checks:

[1186] The device periodically asks, "How are you?", and the user responds, "I'm fine." This response is converted into text and analyzed by the server to generate follow-up questions such as, "That's great. Is there anything I can help you with today?"

[1187] Example prompt sentence:

[1188] Senior citizen: "What's the news today?"

[1189] Prompt: "Today's news"

[1190] The generative AI model responds, "Today's news is that XX was reported in major newspapers."

[1191] This allows elderly people to receive emergency response in real time while enjoying natural conversation.

[1192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1193] Step 1:

[1194] The information terminal uses a microphone to capture the user's voice. The captured voice data is input. The voice data is converted into text data using a speech recognition engine (speech_recognition library). The converted text data is output.

[1195] Step 2:

[1196] The information terminal transmits the converted text data to a server via a communication network (Internet). At this time, data calculation is performed in which the text data is input and transmitted to the server using a communication protocol.

[1197] Step 3:

[1198] The server inputs the received text data into a generative AI model (GPT-2) and generates an appropriate response. The generative AI model performs text analysis and generates a response based on the prompt. This response is output. For example, in response to the prompt "Today's news is...", the response generated is "Today's news is that XX was reported in major newspapers."

[1199] Step 4:

[1200] The server transmits the generated response as text data to the information terminal. The text data is input and then processed by transmitting it to the information terminal via a communication network.

[1201] Step 5:

[1202] The information terminal converts the received text data into voice data using a speech synthesis engine (pyttsx3 library). Text data is input and voice data is output. The response converted into voice data is played back to the user through the speaker.

[1203] Step 6:

[1204] The device periodically asks the user safety confirmation questions and converts the responses to those questions into text data using a speech recognition engine. For example, the question "How are you?" is treated as a prompt, and the user responds "I'm fine." This response is then converted into text data.

[1205] Step 7:

[1206] When a user utters an emergency word (e.g., "Help"), the device detects the data and sends it to the server. The server generates appropriate suggestions for emergency response (e.g., "Should I call an ambulance?") and sends them to the device. The device then converts the generated suggestions into audio data and plays them back to the user.

[1207] Step 8:

[1208] If necessary, the server notifies pre-registered emergency contacts by entering their information and sending a notification using a communication protocol.

[1209] This will enable elderly people to enjoy natural conversation while receiving emergency response in real time, ensuring their safety.

[1210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1211] This invention provides a system that can provide more natural and emotional responses by combining a conversation system using a generative AI model with an emotion engine that recognizes the user's emotions. The entire system is composed of information terminals, a server, and a communication network.

[1212] System configuration

[1213] server

[1214] The server hosts the generative AI model and emotion engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[1215] The emotion engine analyzes the received voice data to recognize and evaluate the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[1216] The server also stores user profile information and conversation history in a database for periodic retraining of the generative AI model.

[1217] Terminal

[1218] A device is a device operated by a user, such as a smartphone or a smart speaker. A device has a microphone and a speaker and can capture and play back the user's voice.

[1219] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[1220] Furthermore, the terminal can display the user's emotional state transmitted from the server.

[1221] user

[1222] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[1223] System Operation

[1224] 1. Start a conversation

[1225] The user speaks to the information terminal, for example, saying, "Hello, I'm a little tired today."

[1226] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the converted text and voice data to a server via a communications network.

[1227] 2. Emotion recognition

[1228] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state (e.g., tired, fatigued tone).

[1229] The emotion engine evaluates the emotional state and returns the result to the server.

[1230] 3. Generating a Response

[1231] Based on the evaluation results from the emotion engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[1232] For example, if the user says, "Hello, I'm a little tired today," the response might be, "Hello, I suggest you take a little rest and then think about what you want to do next."

[1233] 4. Providing a Response

[1234] The server transmits the generated text data to the terminal, which converts the received text data into voice data using a voice synthesis engine and plays it back to the user.

[1235] The terminal can also visually display the evaluation results by the emotion engine to the user.

[1236] Specific examples

[1237] Specific examples of conversations

[1238] 1. The user says, "I'm very happy today."

[1239] 2. The device converts the voice into text and sends the text data, "I'm so happy today," and the voice data to the server.

[1240] 3. The server inputs the voice data into the emotion engine and evaluates the emotional state as "happy."

[1241] 4. The server uses the generative AI model to generate a response: "That's great! What did you do today?"

[1242] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[1243] Feedback and Improvements

[1244] 1. The server stores all conversation data and emotion evaluation data in logs and uses them as training data for the generative AI model and emotion engine.

[1245] 2. The generative AI model and emotion engine are periodically retrained to gradually achieve more natural and emotionally relevant responses.

[1246] The system of this invention allows elderly people to enjoy daily conversations that take their emotions into consideration, which is expected to promote brain activation and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[1247] The processing flow will be explained below.

[1248] Step 1:

[1249] The user speaks into the terminal.

[1250] For example, say, "Hello, I'm very happy today."

[1251] Step 2:

[1252] The device captures the audio.

[1253] The device's microphone collects the user's voice.

[1254] Step 3:

[1255] The device converts the speech to text.

[1256] The voice recognition engine converts the voice data into text data.

[1257] The converted text becomes "Hello, I'm very happy today."

[1258] Step 4:

[1259] The terminal transmits text data and voice data to the server.

[1260] A POST request is sent to the server's API via a communication network.

[1261] The request includes the converted text data and audio data.

[1262] Step 5:

[1263] The server receives the text data and the voice data.

[1264] The server receives the request and extracts this data.

[1265] Step 6:

[1266] The server inputs the voice data into the emotion engine.

[1267] An emotion engine analyzes the voice data and assesses the user's emotional state.

[1268] For example, the emotion engine recognizes the emotional state "happy."

[1269] Step 7:

[1270] The server inputs the emotion engine's evaluation results into the generative AI model.

[1271] The text data "Hello, I'm very happy today" and the emotional information "happy" are input into the generative AI model.

[1272] Step 8:

[1273] A generative AI model generates the appropriate response.

[1274] The model uses internal data and algorithms to generate a response: "That's great! What did you do today?"

[1275] Step 9:

[1276] The server sends the generated response to the terminal.

[1277] Send an API response to return the generated text data to the device.

[1278] Step 10:

[1279] The terminal receives the response text.

[1280] Receive the response to the request and extract the text data.

[1281] Step 11:

[1282] The device converts the received text data into voice.

[1283] The speech synthesis engine converts the text "That's great! What did you do today?" into speech data.

[1284] Step 12:

[1285] The terminal plays the audio to the user.

[1286] The audio data is played through the speaker to provide a response to the user.

[1287] Step 13:

[1288] The terminal displays the emotion evaluation result to the user.

[1289] For example, the screen might display "Emotion: Happy."

[1290] Step 14:

[1291] The server stores the conversation data and emotion evaluation data in a log.

[1292] All conversations and sentiment assessment details are recorded in a database.

[1293] Step 15:

[1294] The server periodically retrains the generative AI model and emotion engine.

[1295] The accuracy of the model is improved using collected conversational and emotional data.

[1296] Example 2

[1297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1298] In conversation systems using information terminals used by elderly people, conventional systems have had the problem of not taking the user's emotions into sufficient consideration when responding, making the conversation feel unnatural. Furthermore, because the system does not reflect the user's emotional state, the conversation may not be very satisfying. Furthermore, these systems do not fully utilize the user's profile information or conversation history, which means that they are unable to provide responses based on the user's past conversations and their emotions at the time.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1300] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for recognizing and evaluating the user's emotional state from the received voice data using an emotion engine, and means for adjusting the response based on the user's emotional state. This enables a natural and satisfying response that reflects the user's emotions. Furthermore, the generated response is converted into voice data using voice synthesis and provided to the user from the information terminal, making the conversation smoother. Furthermore, the server stores the user's profile information and conversation history, and the generative AI model receives feedback based on this information and re-learns, improving the quality of the conversation.

[1301] "Elderly" refers to people who are relatively old and require physical or cognitive assistance.

[1302] An "information terminal" is a device operated by a user that has input, output, and communication functions. Examples include smartphones and smart speakers.

[1303] "Speech recognition" refers to the technology of analyzing voice data and converting it into text data.

[1304] "Text data" refers to text information converted from voice data using voice recognition technology.

[1305] "Communications network" refers to the network infrastructure for data communication between information terminals and servers, including the Internet.

[1306] "Server" refers to the computer system that hosts the generative AI model and emotion engine, receives data sent by users, analyzes it, and responds.

[1307] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-sounding sentences based on input data.

[1308] "Emotion engine" refers to technology that analyzes voice data and recognizes and evaluates the user's emotional state.

[1309] "Speech synthesis" refers to the technology of generating voice data based on text data.

[1310] "Profile Information" refers to data that records personal information and characteristics of a user.

[1311] "Conversation history" refers to data that records the contents of past conversations that a user has had with the system.

[1312] "Relearning" refers to the training process used to improve the performance of generative AI models and emotion engines based on stored data.

[1313] "Response" refers to the reply that the generative AI model gives in response to input from the user.

[1314] "Audio data" refers to data that represents audio in digital form.

[1315] This invention is a system that realizes natural conversations using information terminals used by elderly people and generative AI models. This invention can provide responses that take into account the user's emotions, enabling more human-like communication.

[1316] System Configuration

[1317] The system consists of the following main components: information terminals, servers, and communication networks.

[1318] Information terminal

[1319] An information terminal is a device operated by a user, and includes smartphones, smart speakers, etc. The terminal is equipped with a microphone and speaker, captures the user's voice, and plays a response voice using voice synthesis.

[1320] server

[1321] The server hosts the generative AI model and emotion engine, which play a central role in receiving voice and text data sent by users and generating appropriate responses. The generative AI model uses a natural language generation algorithm such as GPT-3. The emotion engine recognizes the user's emotional state from the voice data using, for example, an emotion analysis API.

[1322] communication network

[1323] A communication network enables data communication between information terminals and servers, and typically the Internet is used.

[1324] Specific examples of processing

[1325] The specific operation of the system will now be described.

[1326] 1. Start a conversation

[1327] User

[1328] The user speaks to the information terminal, for example, saying, "I'm very happy today."

[1329] Terminal

[1330] The device uses a microphone to capture voice data, converts it into text data using a voice recognition engine, and then transmits the text data and voice data to a server via a communication network.

[1331] 2. Emotion recognition

[1332] server

[1333] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state. For example, in this case, the emotional state of "happy" is recognized.

[1334] 3. Generating a Response

[1335] server

[1336] The server inputs the emotion engine's evaluation results and text data into the generative AI model to generate an appropriate response, such as "That's great! What did you do today?"

[1337] 4. Providing a Response

[1338] Terminal

[1339] The terminal inputs the response text sent from the server into a speech synthesis engine, converts it into voice data, and plays it back to the user through the speaker.

[1340] Feedback and Improvements

[1341] server

[1342] The server stores all conversation data and emotion evaluation data in logs, which are used to retrain the generative AI model and emotion engine, allowing the system to gradually develop more natural and emotionally relevant responses, improving the user experience.

[1343] Prompt Sentence Examples

[1344] Here is an example prompt:

[1345] "When a user inputs words, the voice data and emotion data are sent to the server. The server uses an emotion engine to recognize the emotion, and based on the results, a generative AI model generates an appropriate response. The generated text is sent to the device and played back to the user by the speech synthesis engine."

[1346] This system allows elderly people to enjoy natural conversations that take their emotions into consideration, and is expected to be highly effective when used in hospitals and nursing homes.

[1347] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1348] Step 1:

[1349] User initiated conversation

[1350] Input: User's spoken utterance

[1351] Specific action: The user speaks to the information terminal. For example, he says, "Hello, I'm a little tired today."

[1352] Output: User's voice data

[1353] Step 2:

[1354] Device capture and conversion of audio data

[1355] Input: User's voice data

[1356] What it does: The device uses the built-in microphone to capture the user's voice and converts it into text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1357] Output: Converted text data and original audio data

[1358] Step 3:

[1359] Sending data from the device to the server

[1360] Input: Text data, audio data

[1361] Specific operation: The terminal transmits the converted text data and the original voice data to a server via a communication network (e.g., the Internet).

[1362] Output: Text and audio data sent to the server

[1363] Step 4:

[1364] Emotion recognition by server

[1365] Input: Transmitted audio data

[1366] Specific operation: The server inputs the received voice data into an emotion engine (e.g., Microsoft Azure Emotion API) to analyze and evaluate the user's emotional state. In this case, the emotional state of "tired" is recognized from the voice data.

[1367] Output: Emotional state data (e.g., "Tired")

[1368] Step 5:

[1369] Server response generation

[1370] Input: Text data, emotional state data

[1371] Specific operation: The server inputs the emotional state data and text data obtained from the emotion engine into a generative AI model (e.g., GPT-3). The generative AI model generates an appropriate response based on this data. For example, it generates a response such as, "Hello, I recommend you take a short break. Then, think about what you want to do next."

[1372] Output: The generated response text

[1373] Step 6:

[1374] Sending response data from the server to the terminal

[1375] Input: Generated response text

[1376] Specific operation: The server sends the generated response text to the terminal.

[1377] Output: Response text sent to the terminal

[1378] Step 7:

[1379] Voice generation and playback of responses by the terminal

[1380] Input: Response text data

[1381] Specific operation: The device inputs the received response text into a speech synthesis engine (e.g., Amazon Polly) to convert the text into voice data, which the device then plays back to the user through the speaker.

[1382] Output: A spoken response to be played to the user (e.g., "Hello, I suggest you take a break and think about what you want to do next.")

[1383] Step 8:

[1384] Feedback and Data Storage

[1385] Input: Conversation data, emotion evaluation data

[1386] Specific operation: The server stores all conversation data and emotion evaluation data in a log, which is used to retrain the generative AI model and emotion engine.

[1387] Output: Updated training data

[1388] Through the above steps, users can enjoy natural conversations that take emotions into consideration.

[1389] (Application example 2)

[1390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1391] To enable elderly people to enjoy more natural and emotionally appropriate conversations on information devices, simply generating responses based on text data is not enough. Conventional systems are unable to generate responses that take the user's emotional state into account, and are therefore unable to provide satisfying interactions for users. Furthermore, particularly when serving customers in brick-and-mortar stores, staff are required to grasp the customer's emotional state in real time and provide services accordingly. Therefore, a more accurate conversation system that also takes the user's emotions into account is needed.

[1392] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1393] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for using an emotion recognition engine to perform emotion analysis on the voice data from the user, means for adjusting the response of the generative AI model based on the emotion analysis results, and means for visually displaying the response. This enables natural and satisfying conversations that correspond to the user's emotions, and makes it possible to provide flexible customer service based on the emotional state of customers, particularly in physical stores.

[1394] 1. An "information terminal" is a device operated by a user that has the ability to capture voice and convert it into text data.

[1395] 2. "Speech recognition" is a technology that converts a user's voice data into text data.

[1396] 3. "Communications network" means the infrastructure for transmitting and receiving data between information terminals and servers, including the Internet.

[1397] 4. "Server" means a computing system that hosts the generative AI model and emotion recognition engine and manages user conversation data and emotion data.

[1398] 5. A "generative AI model" is an algorithm that generates appropriate responses based on input text data.

[1399] 6. An "emotion recognition engine" is a technology that analyzes a user's voice data and evaluates their emotional state.

[1400] 7. “Sentiment analysis” is the process of identifying a user’s emotional state based on audio data.

[1401] 8. "Visually displaying" means presenting the results of an analysis or generated response to the user as text or graphics on a screen.

[1402] 9. "Speech synthesis" is a technology that converts text data into voice data.

[1403] 10. "Brick and Mortar Store" means a store that exists in a physical location and offers goods and services.

[1404] 11. "Customer service" refers to the service and interaction provided by staff to customers in physical stores.

[1405] This invention is a conversation system using an information terminal and a server for use by elderly people, which can provide natural conversation according to the user's emotions. This system is configured using an information terminal, a server, and a communication network. Specific embodiments for implementing this invention are described in detail below.

[1406] System configuration

[1407] server

[1408] The server hosts the generative AI model and emotion recognition engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[1409] The emotion recognition engine analyzes the received voice data to assess the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[1410] The server also stores user profile information and conversation history in a database, which is used to periodically retrain the generative AI model.

[1411] Terminal

[1412] A device is a user-operated device, such as smart glasses, a smartphone, or a smart speaker. A device has a microphone and a speaker to capture and play back the user's voice.

[1413] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[1414] Furthermore, the terminal can visually display the user's emotional state transmitted from the server.

[1415] user

[1416] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[1417] System Operation

[1418] 1. Start a conversation

[1419] The user speaks to the information terminal, for example, saying, "I want to know more about this product."

[1420] The device captures the voice with a microphone and converts the voice into text using voice recognition, which is then sent to a server via a communications network.

[1421] 2. Emotion recognition

[1422] The server inputs the received voice data into an emotion recognition engine to evaluate the user's emotional state (e.g., interested, happy, etc.).

[1423] The emotion recognition engine evaluates the emotional state and returns the results to the server.

[1424] 3. Generating a Response

[1425] Based on the evaluation results from the emotion recognition engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[1426] For example, if a user says, "I'd like to know more about this product," the response might be, "I'd like to tell you more about this product. Is there anything in particular you'd like to know?"

[1427] 4. Providing a Response

[1428] The server sends the generated text data to the terminal, which then converts the received text data into voice data using a speech synthesis engine and plays it back to the user.

[1429] The device can also visually display the evaluation results from the emotion recognition engine to the user.

[1430] Specific examples

[1431] Specific examples of conversations

[1432] 1. A user says, "I'd like to know more about this product."

[1433] 2. The device converts the voice into text and sends the text data, "I want to know more about this product," and the voice data to the server.

[1434] 3. The server inputs the voice data into an emotion recognition engine and evaluates the emotional state as "interested."

[1435] 4. The server uses the generative AI model to generate a response that says, "We'll explain this product in detail. Is there anything in particular you'd like to know?"

[1436] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[1437] Prompt Sentence Examples

[1438] The user's sentiment is 'interested'. Generate a sentiment-based response to the following text: "I'd like to know more about this product"

[1439] In this way, natural and meaningful conversations that take the user's emotions into account are realized.

[1440] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1441] Step 1:

[1442] A user speaks to an information terminal, for example, saying, "I'd like to know more about this product." The input is the user's voice, which is captured via a microphone. The output is raw voice data.

[1443] Step 2:

[1444] The device inputs the captured voice data into a voice recognition engine, which converts the voice data into text data. The voice recognition engine processes the voice data and outputs text data. Here, the input is raw voice data, and the output is text data.

[1445] Step 3:

[1446] The terminal transmits the converted text data and voice data to the server via the communication network. The input of this step is the text data and voice data, and the output is the data transmitted to the server.

[1447] Step 4:

[1448] The server inputs the received voice data into the emotion recognition engine to evaluate the user's emotional state. In this step, the emotion recognition engine analyzes the voice data to identify the emotional state and outputs the result. The input is the voice data, and the output is the evaluation result of the emotional state.

[1449] Step 5:

[1450] The server inputs the text data into the generative AI model along with the evaluation results from the emotion recognition engine. The generative AI model generates an optimal response based on the text data and the emotion evaluation results. The input for this step is the text data and the emotion evaluation results, and the output is the generated response text.

[1451] Step 6:

[1452] The server inputs the generated text data into a speech synthesis engine and converts it into voice data. The speech synthesis engine processes the text data and outputs voice data. The input is the response text data, and the output is the generated voice data.

[1453] Step 7:

[1454] The device plays the audio data sent from the server and provides it to the user. The device plays this audio data through a speaker so that the user can hear it. The device also visually displays the evaluation results from the emotion recognition engine. The input here is the generated audio data and the emotion evaluation result, and the output is the audio played to the user and the displayed emotional state.

[1455] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1456] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1457] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1458] [Fourth embodiment]

[1459] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1460] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1461] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1462] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1463] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1464] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1465] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1466] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1467] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1468] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1469] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1470] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1471] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1472] This invention provides a system that uses a generative AI model to enable elderly people to enjoy natural conversations. This system is composed of an information terminal, a server, and a communication network.

[1473] System configuration

[1474] server

[1475] The server hosts the generative AI model and manages conversation data with the user. The server receives text data sent by the user and generates appropriate responses using the generative AI model.

[1476] The server also stores user profile information (such as name, age, and hobbies) in a database, records conversation history, and periodically retrains the generative AI model to improve response accuracy.

[1477] Terminal

[1478] A device is a user-operated device, such as a smartphone or smart speaker, that has a microphone and speaker to capture and play back the user's voice.

[1479] The device has a voice recognition function and converts the user's voice data into text data. This text data is sent to the server, and the response received from the server is played back using a voice synthesis function.

[1480] user

[1481] The users are elderly people who enjoy conversations by speaking directly into the information terminal. Users can use the system in a natural manner without any special operations.

[1482] System Operation

[1483] 1. Start a conversation

[1484] The user speaks to the information terminal, asking questions such as, "Hello, what's in the news today?"

[1485] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the text data to a server via a communications network.

[1486] 2. Generating a Response

[1487] The server logs the received text data and inputs it into the generative AI model, which then analyzes the text data and generates an appropriate response.

[1488] The server transmits the generated response to the terminal as text data.

[1489] 3. Providing a Response

[1490] The device converts the received text data into voice data using a speech synthesis engine, which is then played back to the user through the speaker.

[1491] For example, if the response from the server is "Today, major newspapers are reporting the following news: XXX, XXX, XXX," this will be provided as audio.

[1492] Specific examples

[1493] Specific examples of conversations

[1494] 1. The user says, "Good morning, what's on my schedule today?"

[1495] 2. The device converts the voice into text and sends the text data, "Good morning, what are your plans for today?" to the server.

[1496] 3. The server uses the generative AI model to generate a response: "Good morning, we have plans to go for a walk at 10 o'clock today."

[1497] 4. The device converts the received response into audio and plays it back to the user.

[1498] Feedback and Improvements

[1499] 1. The server stores all conversation data in a log and uses it as training data for the generative AI model.

[1500] 2. The generative AI model undergoes periodic retraining, gradually achieving more natural and richer conversations.

[1501] The system of this invention allows elderly people to enjoy daily conversations, which is expected to promote brain activity and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[1502] The processing flow will be explained below.

[1503] Step 1:

[1504] The user speaks into the terminal.

[1505] For example, say, "Hello, what can I do for you today?"

[1506] Step 2:

[1507] The device captures the audio.

[1508] A microphone collects the user's voice.

[1509] Step 3:

[1510] The device converts the speech to text.

[1511] The voice recognition engine converts the voice data into text data.

[1512] The converted text becomes "Hello, what can I do for you today?"

[1513] Step 4:

[1514] The terminal transmits the text data to the server.

[1515] A POST request is sent to the server's API via a communication network.

[1516] The request includes the converted text data.

[1517] Step 5:

[1518] The server receives the text data.

[1519] The server receives the request and extracts the text data.

[1520] Step 6:

[1521] The server logs the text data.

[1522] The received text data is saved in a database and recorded as a log.

[1523] Step 7:

[1524] The server inputs text data into the generative AI model.

[1525] A generative AI model analyzes the text data "Hello, what should I do today?"

[1526] Step 8:

[1527] A generative AI model generates the appropriate response.

[1528] The model uses internal data and algorithms to generate a response like, "Hello, how about enjoying some reading or a short walk today?"

[1529] Step 9:

[1530] The server sends the generated response to the terminal.

[1531] The generated text data is sent as an API response to be returned to the device.

[1532] Step 10:

[1533] The terminal receives the response text.

[1534] Receive the response to the request and extract the text data.

[1535] Step 11:

[1536] The device converts the received text data into voice.

[1537] The speech synthesis engine converts the text "Hello, how about enjoying some reading or a short walk today?" into speech data.

[1538] Step 12:

[1539] The terminal plays the audio to the user.

[1540] The audio data is played through the speaker to provide a response to the user.

[1541] Example 1

[1542] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1543] There is a demand for systems that allow elderly people to enjoy natural conversations in their daily lives. However, many current systems have low accuracy in speech recognition and response generation, making them difficult for elderly people to use. Furthermore, continuous learning is necessary for generative AI models to maintain a certain level of response quality, but this is often not achieved. Furthermore, conversation data is not managed, and logs are not kept to generate appropriate responses. This makes it difficult for elderly people to enjoy comfortable communication.

[1544] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1545] In this invention, the server includes means for converting voice data into text data by performing voice recognition, means for transmitting the text data to the server via a communication network, means for generating a response to the text data using a generative AI model, means for converting the generated response into voice data by voice synthesis and providing the voice data to the user from the communication device, and means for recording the response in a log and periodically retraining the generative AI model. This allows elderly people to enjoy natural conversations and also enables improvement in the accuracy of the generative AI model and maintenance of response quality.

[1546] The term "elderly" generally refers to older people, especially those aged 65 and over.

[1547] A "communication device" is a device for inputting and outputting voice, and includes, for example, smartphones and smart speakers.

[1548] "Speech recognition" refers to the technology that converts a user's speech into text data.

[1549] "Text data" refers to character data converted using voice recognition technology.

[1550] "Communication network" refers to the network infrastructure for data communication, and specifically includes the Internet.

[1551] "Computer" refers to a device that has server functionality and processes and stores data.

[1552] "Generative AI model" refers to an artificial intelligence model that generates appropriate responses to input text data.

[1553] "Response" refers to the reply generated by the generative AI model in response to text data.

[1554] "Speech synthesis" refers to the technology of converting text data into voice data.

[1555] "Voice data" refers to digital data of voice generated by voice synthesis technology.

[1556] "Log" refers to data that records the system's operation history and conversation data exchanged.

[1557] "Relearning" refers to the learning process of improving the accuracy of a generative AI model based on previously logged data.

[1558] The present invention provides a system that allows elderly people to enjoy natural conversations. This system is configured using communication devices, a server, and a communication network. The following describes how this system is configured and how it functions.

[1559] System Overview

[1560] server

[1561] The server plays a central role in hosting the generative AI model and managing conversation data with users. The server has the following functions:

[1562] 1. Receiving and logging

[1563] The server receives the text data sent from the device and records it in a log, which includes the conversation history and user profile information (such as name, age, and hobbies).

[1564] 2. Response generation using a generative AI model

[1565] The received text data is input into a generative AI model to generate an appropriate response. The generative AI model can be, for example, a generative AI model.

[1566] 3. Sending the Response

[1567] The server transmits the generated response to the terminal as text data.

[1568] 4. Periodic Retraining

[1569] The server collects all conversation data and periodically retrains the generative AI model based on this data, improving the accuracy and naturalness of responses.

[1570] Terminal

[1571] A terminal is a device that is directly operated by a user and has the following functions:

[1572] 1. Voice Recognition

[1573] The microphone captures the user's speech into the device and converts it into text data using a speech recognition engine, such as voice recognition software.

[1574] 2. Texting

[1575] The converted text data is transmitted to a server via a communication network.

[1576] 3. Speech Synthesis

[1577] The text data received from the server is converted into voice data using a speech synthesis engine, which uses speech synthesis software.

[1578] 4. Audio playback

[1579] The converted audio data is played back to the user through a speaker.

[1580] user

[1581] The users are elderly people who can use the system in a natural manner without requiring any special technical knowledge or operation.

[1582] Specific examples

[1583] Specific examples of conversations

[1584] 1. User utterances

[1585] The user says, "Good morning, how's the weather today?"

[1586] 2. Voice recognition on the device

[1587] The device converts the voice into text and sends the text data, such as "Good morning, how's the weather today?" to the server.

[1588] 3. Server Response Generation

[1589] The server uses the generative AI model to generate a response: "Good morning, it's sunny today."

[1590] 4. Device speech synthesis

[1591] The device converts the received response into speech and plays it to the user: "Good morning, it's sunny today."

[1592] Prompt Sentence Examples

[1593] Here is an example of inputting the following prompt sentence into the generative AI model:

[1594] User input: "Good morning, how's the weather today?"

[1595] Sample prompt: "The user is asking, 'Good morning, how's the weather today?' Generate an appropriate response."

[1596] The system of this invention is expected to enable elderly people to enjoy natural conversations on a daily basis, promote brain activity, and contribute to the prevention of dementia. This system will also reduce the burden on staff at hospitals and nursing homes and improve communication with residents.

[1597] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1598] Step 1:

[1599] Voice input

[1600] A user speaks into a communication device. For example, they say something like, "Hello, what should I do today?" The input data is the user's voice. The output is the voice data captured by the communication device. Specifically, the user speaks into the microphone of the communication device, and the voice is recorded as digital voice data.

[1601] Step 2:

[1602] Voice Recognition

[1603] The device converts the captured voice data into text data using a voice recognition engine. For example, general voice recognition software is used as the voice recognition engine. The input data is voice data, and the output is text data converted by voice recognition. Specifically, the voice data is converted into the text data "Hello, what should I do today?"

[1604] Step 3:

[1605] Text sending

[1606] The terminal transmits the converted text data to the server via a communication network. The input data is the text data obtained by speech recognition, and the output is the status of the text data being transmitted to the server. Specifically, the text data is securely transmitted to the server via the Internet.

[1607] Step 4:

[1608] Response Generation

[1609] The server records the received text data in a log and inputs it into the generative AI model. The generative AI model analyzes the text data and generates an appropriate response. The input data is the received text data, and the output is the response text data generated by the generative AI model. Specifically, from the input "Hello, what should I do today?", a response such as "I recommend reading today" is generated.

[1610] Step 5:

[1611] Response Send

[1612] The server sends the generated response to the terminal as text data. The input data is the generated response text data, and the output is the status of the response text data being sent to the terminal. Specifically, the response text "I recommend reading today" is sent to the terminal.

[1613] Step 6:

[1614] Speech synthesis

[1615] The device converts the received response text data into voice data using a voice synthesis engine. For example, general voice synthesis software is used as the voice synthesis engine. The input data is the response text data, and the output is voice data generated by voice synthesis. Specifically, voice data is generated from the text "I recommend reading today."

[1616] Step 7:

[1617] Audio Output

[1618] The device plays the audio data to the user through the speaker. The input data is the audio data generated by speech synthesis, and the output is the audio that is played back to the user. Specifically, the speaker plays the audio "We recommend you read today," and the user can hear it.

[1619] The above are the specific processing steps of the program of this system.

[1620] (Application example 1)

[1621] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1622] Elderly people often rely on their family members or nursing home staff to ensure their safety and respond to emergencies. However, if these support services are insufficient, the elderly are at greater risk of facing emergencies. Elderly people may also feel lonely, which is becoming a social problem. Therefore, there is a need for a system that allows elderly people to easily call for help using their voice and receive prompt, appropriate assistance.

[1623] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1624] In this invention, the server includes means for generating appropriate suggestions for emergency responses when an elderly person utters an emergency word requesting help, means for providing the appropriate suggestions to the user at the information terminal, and means for notifying emergency contacts based on the emergency response suggestions. This enables a prompt and appropriate response when an elderly person faces an emergency, ensuring the safety of the elderly and reducing their sense of loneliness.

[1625] "Elderly people" refers to people who are over a certain age as defined by the Elderly People Act, and who often require special assistance in daily life.

[1626] An "information terminal" is a device operated by a user, including smartphones and smart speakers.

[1627] "Speech recognition" refers to the process by which an information device captures a user's voice and converts that voice data into text data.

[1628] "Generative AI model" refers to an algorithm or system that uses AI technology to analyze text data and generate natural-sounding conversational responses.

[1629] "Emergency response" refers to a series of actions and procedures to provide appropriate and prompt assistance to elderly people when they face an emergency.

[1630] "Appropriate Recommendations" refer to recommendations that direct appropriate and effective actions or responses to be taken in a particular situation.

[1631] "Communication network" refers to a network system for exchanging data between information terminals and servers, and includes the Internet.

[1632] "Speech synthesis" refers to the technology of converting text data into audio data and playing that audio back to the user.

[1633] "Emergency contacts" refers to pre-registered contacts, such as family members or care staff, who should be contacted in the event of an emergency.

[1634] "Safety checks" refer to regularly checking on the current status of elderly people and asking questions and making checks to ensure their safety.

[1635] "Means for generating a response" refers to the function or method by which the server uses a generative AI model to generate a response to the received text data.

[1636] "Voice data" refers to data generated using a voice synthesis engine for provision to the user by an information terminal.

[1637] This invention relates to a system that checks the safety of elderly people and responds to emergencies using information terminals, servers, and communication networks used by elderly people. The system aims to utilize generative AI models to have natural conversations with elderly people and provide appropriate responses in emergencies.

[1638] System configuration

[1639] server

[1640] The server hosts a generative AI model that generates responses based on the text data sent by the user. The generative AI model consists of an algorithm that analyzes the user's input text and generates the optimal response.

[1641] The server includes a natural language generation model such as GPT-2 as a means to generate responses, as well as an algorithm for detecting urgent words.

[1642] Information terminal

[1643] An information terminal is a device operated by a user, such as a smartphone or a smart speaker. An information terminal is equipped with a microphone and a speaker to capture and play back the user's voice.

[1644] The device is equipped with a voice recognition function that converts the user's speech into text, which is then sent to a server via a communications network.

[1645] communication network

[1646] Data communication between the information terminal and the server is carried out over the Internet, which allows for real-time communication.

[1647] System Operation

[1648] 1. Regular safety checks

[1649] The device periodically asks the elderly person questions to check their safety, such as "How are you?" and "Is there anything I can help you with?" This allows the device to periodically check the elderly person's condition.

[1650] The user responds to these questions verbally, which is converted into text by the device and sent to the server.

[1651] 2. Conversational Response Generation

[1652] The server analyzes the received text data and generates a response using a generative AI model, which is then sent to the device as text.

[1653] The device converts the received response into voice data using a voice synthesis engine and plays it back to the user.

[1654] 3. Emergency Response

[1655] When a user utters an emergency word such as "help," the device detects it and sends it to the server, which then generates appropriate suggestions for emergency response, such as "Should we call an ambulance?", and sends them to the device.

[1656] If necessary, the system will also notify pre-registered emergency contacts via email, SMS, or other means.

[1657] Program processing explanation

[1658] Speech recognition: Using the speech_recognition library, the information terminal analyzes the user's voice and converts it into text data.

[1659] Generative AI model: Implement a GPT-2 model using the transformers library to generate appropriate responses from text data received on the server.

[1660] Speech synthesis: Uses the pyttsx3 library to convert the generated text response into audio data and play it back to the user on the device.

[1661] Emergency response: Uses algorithms to detect specific emergency words, generates appropriate suggestions in an emergency, and notifies emergency contacts as needed.

[1662] Specific examples

[1663] Regular safety checks:

[1664] The device periodically asks, "How are you?", and the user responds, "I'm fine." This response is converted into text and analyzed by the server to generate follow-up questions such as, "That's great. Is there anything I can help you with today?"

[1665] Example prompt sentence:

[1666] Senior citizen: "What's the news today?"

[1667] Prompt: "Today's news"

[1668] The generative AI model responds, "Today's news is that XX was reported in major newspapers."

[1669] This allows elderly people to receive emergency response in real time while enjoying natural conversation.

[1670] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1671] Step 1:

[1672] The information terminal uses a microphone to capture the user's voice. The captured voice data is input. The voice data is converted into text data using a speech recognition engine (speech_recognition library). The converted text data is output.

[1673] Step 2:

[1674] The information terminal transmits the converted text data to a server via a communication network (Internet). At this time, data calculation is performed in which the text data is input and transmitted to the server using a communication protocol.

[1675] Step 3:

[1676] The server inputs the received text data into a generative AI model (GPT-2) and generates an appropriate response. The generative AI model performs text analysis and generates a response based on the prompt. This response is output. For example, in response to the prompt "Today's news is...", the response generated is "Today's news is that XX was reported in major newspapers."

[1677] Step 4:

[1678] The server transmits the generated response as text data to the information terminal. The text data is input and then processed by transmitting it to the information terminal via a communication network.

[1679] Step 5:

[1680] The information terminal converts the received text data into voice data using a speech synthesis engine (pyttsx3 library). Text data is input and voice data is output. The response converted into voice data is played back to the user through the speaker.

[1681] Step 6:

[1682] The device periodically asks the user safety confirmation questions and converts the responses to those questions into text data using a speech recognition engine. For example, the question "How are you?" is treated as a prompt, and the user responds "I'm fine." This response is then converted into text data.

[1683] Step 7:

[1684] When a user utters an emergency word (e.g., "Help"), the device detects the data and sends it to the server. The server generates appropriate suggestions for emergency response (e.g., "Should I call an ambulance?") and sends them to the device. The device then converts the generated suggestions into audio data and plays them back to the user.

[1685] Step 8:

[1686] If necessary, the server notifies pre-registered emergency contacts by entering their information and sending a notification using a communication protocol.

[1687] This will enable elderly people to enjoy natural conversation while receiving emergency response in real time, ensuring their safety.

[1688] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1689] This invention provides a system that can provide more natural and emotional responses by combining a conversation system using a generative AI model with an emotion engine that recognizes the user's emotions. The entire system is composed of information terminals, a server, and a communication network.

[1690] System configuration

[1691] server

[1692] The server hosts the generative AI model and emotion engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[1693] The emotion engine analyzes the received voice data to recognize and evaluate the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[1694] The server also stores user profile information and conversation history in a database for periodic retraining of the generative AI model.

[1695] Terminal

[1696] A device is a device operated by a user, such as a smartphone or a smart speaker. A device has a microphone and a speaker and can capture and play back the user's voice.

[1697] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[1698] Furthermore, the terminal can display the user's emotional state transmitted from the server.

[1699] user

[1700] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[1701] System Operation

[1702] 1. Start a conversation

[1703] The user speaks to the information terminal, for example, saying, "Hello, I'm a little tired today."

[1704] The device captures voice with a microphone and converts the voice into text using a speech recognition engine, which then transmits the converted text and voice data to a server via a communications network.

[1705] 2. Emotion recognition

[1706] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state (e.g., tired, fatigued tone).

[1707] The emotion engine evaluates the emotional state and returns the result to the server.

[1708] 3. Generating a Response

[1709] Based on the evaluation results from the emotion engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[1710] For example, if the user says, "Hello, I'm a little tired today," the response might be, "Hello, I suggest you take a little rest and then think about what you want to do next."

[1711] 4. Providing a Response

[1712] The server transmits the generated text data to the terminal, which converts the received text data into voice data using a voice synthesis engine and plays it back to the user.

[1713] The terminal can also visually display the evaluation results by the emotion engine to the user.

[1714] Specific examples

[1715] Specific examples of conversations

[1716] 1. The user says, "I'm very happy today."

[1717] 2. The device converts the voice into text and sends the text data, "I'm so happy today," and the voice data to the server.

[1718] 3. The server inputs the voice data into the emotion engine and evaluates the emotional state as "happy."

[1719] 4. The server uses the generative AI model to generate a response: "That's great! What did you do today?"

[1720] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[1721] Feedback and Improvements

[1722] 1. The server stores all conversation data and emotion evaluation data in logs and uses them as training data for the generative AI model and emotion engine.

[1723] 2. The generative AI model and emotion engine are periodically retrained to gradually achieve more natural and emotionally relevant responses.

[1724] The system of this invention allows elderly people to enjoy daily conversations that take their emotions into consideration, which is expected to promote brain activation and contribute to the prevention of dementia. Furthermore, introducing this system into hospitals and nursing homes can reduce the burden on staff and improve communication with residents.

[1725] The processing flow will be explained below.

[1726] Step 1:

[1727] The user speaks into the terminal.

[1728] For example, say, "Hello, I'm very happy today."

[1729] Step 2:

[1730] The device captures the audio.

[1731] The device's microphone collects the user's voice.

[1732] Step 3:

[1733] The device converts the speech to text.

[1734] The voice recognition engine converts the voice data into text data.

[1735] The converted text becomes "Hello, I'm very happy today."

[1736] Step 4:

[1737] The terminal transmits text data and voice data to the server.

[1738] A POST request is sent to the server's API via a communication network.

[1739] The request includes the converted text data and audio data.

[1740] Step 5:

[1741] The server receives the text data and the voice data.

[1742] The server receives the request and extracts this data.

[1743] Step 6:

[1744] The server inputs the voice data into the emotion engine.

[1745] An emotion engine analyzes the voice data and assesses the user's emotional state.

[1746] For example, the emotion engine recognizes the emotional state "happy."

[1747] Step 7:

[1748] The server inputs the emotion engine's evaluation results into the generative AI model.

[1749] The text data "Hello, I'm very happy today" and the emotional information "happy" are input into the generative AI model.

[1750] Step 8:

[1751] A generative AI model generates the appropriate response.

[1752] The model uses internal data and algorithms to generate a response: "That's great! What did you do today?"

[1753] Step 9:

[1754] The server sends the generated response to the terminal.

[1755] Send an API response to return the generated text data to the device.

[1756] Step 10:

[1757] The terminal receives the response text.

[1758] Receive the response to the request and extract the text data.

[1759] Step 11:

[1760] The device converts the received text data into voice.

[1761] The speech synthesis engine converts the text "That's great! What did you do today?" into speech data.

[1762] Step 12:

[1763] The terminal plays the audio to the user.

[1764] The audio data is played through the speaker to provide a response to the user.

[1765] Step 13:

[1766] The terminal displays the emotion evaluation result to the user.

[1767] For example, the screen might display "Emotion: Happy."

[1768] Step 14:

[1769] The server stores the conversation data and emotion evaluation data in a log.

[1770] All conversations and sentiment assessment details are recorded in a database.

[1771] Step 15:

[1772] The server periodically retrains the generative AI model and emotion engine.

[1773] The accuracy of the model is improved using collected conversational and emotional data.

[1774] Example 2

[1775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] In conversation systems using information terminals used by elderly people, conventional systems have had the problem of not taking the user's emotions into sufficient consideration when responding, making the conversation feel unnatural. Furthermore, because the system does not reflect the user's emotional state, the conversation may not be very satisfying. Furthermore, these systems do not fully utilize the user's profile information or conversation history, which means that they are unable to provide responses based on the user's past conversations and their emotions at the time.

[1777] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1778] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for recognizing and evaluating the user's emotional state from the received voice data using an emotion engine, and means for adjusting the response based on the user's emotional state. This enables a natural and satisfying response that reflects the user's emotions. Furthermore, the generated response is converted into voice data using voice synthesis and provided to the user from the information terminal, making the conversation smoother. Furthermore, the server stores the user's profile information and conversation history, and the generative AI model receives feedback based on this information and re-learns, improving the quality of the conversation.

[1779] "Elderly" refers to people who are relatively old and require physical or cognitive assistance.

[1780] An "information terminal" is a device operated by a user that has input, output, and communication functions. Examples include smartphones and smart speakers.

[1781] "Speech recognition" refers to the technology of analyzing voice data and converting it into text data.

[1782] "Text data" refers to text information converted from voice data using voice recognition technology.

[1783] "Communications network" refers to the network infrastructure for data communication between information terminals and servers, including the Internet.

[1784] "Server" refers to the computer system that hosts the generative AI model and emotion engine, receives data sent by users, analyzes it, and responds.

[1785] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-sounding sentences based on input data.

[1786] "Emotion engine" refers to technology that analyzes voice data and recognizes and evaluates the user's emotional state.

[1787] "Speech synthesis" refers to the technology of generating voice data based on text data.

[1788] "Profile Information" refers to data that records personal information and characteristics of a user.

[1789] "Conversation history" refers to data that records the contents of past conversations that a user has had with the system.

[1790] "Relearning" refers to the training process used to improve the performance of generative AI models and emotion engines based on stored data.

[1791] "Response" refers to the reply that the generative AI model gives in response to input from the user.

[1792] "Audio data" refers to data that represents audio in digital form.

[1793] This invention is a system that realizes natural conversations using information terminals used by elderly people and generative AI models. This invention can provide responses that take into account the user's emotions, enabling more human-like communication.

[1794] System Configuration

[1795] The system consists of the following main components: information terminals, servers, and communication networks.

[1796] Information terminal

[1797] An information terminal is a device operated by a user, and includes smartphones, smart speakers, etc. The terminal is equipped with a microphone and speaker, captures the user's voice, and plays a response voice using voice synthesis.

[1798] server

[1799] The server hosts the generative AI model and emotion engine, which play a central role in receiving voice and text data sent by users and generating appropriate responses. The generative AI model uses a natural language generation algorithm such as GPT-3. The emotion engine recognizes the user's emotional state from the voice data using, for example, an emotion analysis API.

[1800] communication network

[1801] A communication network enables data communication between information terminals and servers, and typically the Internet is used.

[1802] Specific examples of processing

[1803] The specific operation of the system will now be described.

[1804] 1. Start a conversation

[1805] User

[1806] The user speaks to the information terminal, for example, saying, "I'm very happy today."

[1807] Terminal

[1808] The device uses a microphone to capture voice data, converts it into text data using a voice recognition engine, and then transmits the text data and voice data to a server via a communication network.

[1809] 2. Emotion recognition

[1810] server

[1811] The server inputs the received voice data into an emotion engine to evaluate the user's emotional state. For example, in this case, the emotional state of "happy" is recognized.

[1812] 3. Generating a Response

[1813] server

[1814] The server inputs the emotion engine's evaluation results and text data into the generative AI model to generate an appropriate response, such as "That's great! What did you do today?"

[1815] 4. Providing a Response

[1816] Terminal

[1817] The terminal inputs the response text sent from the server into a speech synthesis engine, converts it into voice data, and plays it back to the user through the speaker.

[1818] Feedback and Improvements

[1819] server

[1820] The server stores all conversation data and emotion evaluation data in logs, which are used to retrain the generative AI model and emotion engine, allowing the system to gradually develop more natural and emotionally relevant responses, improving the user experience.

[1821] Prompt Sentence Examples

[1822] Here is an example prompt:

[1823] "When a user inputs words, the voice data and emotion data are sent to the server. The server uses an emotion engine to recognize the emotion, and based on the results, a generative AI model generates an appropriate response. The generated text is sent to the device and played back to the user by the speech synthesis engine."

[1824] This system allows elderly people to enjoy natural conversations that take their emotions into consideration, and is expected to be highly effective when used in hospitals and nursing homes.

[1825] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1826] Step 1:

[1827] User initiated conversation

[1828] Input: User's spoken utterance

[1829] Specific action: The user speaks to the information terminal. For example, he says, "Hello, I'm a little tired today."

[1830] Output: User's voice data

[1831] Step 2:

[1832] Device capture and conversion of audio data

[1833] Input: User's voice data

[1834] What it does: The device uses the built-in microphone to capture the user's voice and converts it into text using a speech recognition engine (e.g., Google Speech-to-Text API).

[1835] Output: Converted text data and original audio data

[1836] Step 3:

[1837] Sending data from the device to the server

[1838] Input: Text data, audio data

[1839] Specific operation: The terminal transmits the converted text data and the original voice data to a server via a communication network (e.g., the Internet).

[1840] Output: Text and audio data sent to the server

[1841] Step 4:

[1842] Emotion recognition by server

[1843] Input: Transmitted audio data

[1844] Specific operation: The server inputs the received voice data into an emotion engine (e.g., Microsoft Azure Emotion API) to analyze and evaluate the user's emotional state. In this case, the emotional state of "tired" is recognized from the voice data.

[1845] Output: Emotional state data (e.g., "Tired")

[1846] Step 5:

[1847] Server response generation

[1848] Input: Text data, emotional state data

[1849] Specific operation: The server inputs the emotional state data and text data obtained from the emotion engine into a generative AI model (e.g., GPT-3). The generative AI model generates an appropriate response based on this data. For example, it generates a response such as, "Hello, I recommend you take a short break. Then, think about what you want to do next."

[1850] Output: The generated response text

[1851] Step 6:

[1852] Sending response data from the server to the terminal

[1853] Input: Generated response text

[1854] Specific operation: The server sends the generated response text to the terminal.

[1855] Output: Response text sent to the terminal

[1856] Step 7:

[1857] Voice generation and playback of responses by the terminal

[1858] Input: Response text data

[1859] Specific operation: The device inputs the received response text into a speech synthesis engine (e.g., Amazon Polly) to convert the text into voice data, which the device then plays back to the user through the speaker.

[1860] Output: A spoken response to be played to the user (e.g., "Hello, I suggest you take a break and think about what you want to do next.")

[1861] Step 8:

[1862] Feedback and Data Storage

[1863] Input: Conversation data, emotion evaluation data

[1864] Specific operation: The server stores all conversation data and emotion evaluation data in a log, which is used to retrain the generative AI model and emotion engine.

[1865] Output: Updated training data

[1866] Through the above steps, users can enjoy natural conversations that take emotions into consideration.

[1867] (Application example 2)

[1868] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] To enable elderly people to enjoy more natural and emotionally appropriate conversations on information devices, simply generating responses based on text data is not enough. Conventional systems are unable to generate responses that take the user's emotional state into account, and are therefore unable to provide satisfying interactions for users. Furthermore, particularly when serving customers in brick-and-mortar stores, staff are required to grasp the customer's emotional state in real time and provide services accordingly. Therefore, a more accurate conversation system that also takes the user's emotions into account is needed.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1871] In this invention, the server includes means for generating a response to the text data using a generative AI model, means for using an emotion recognition engine to perform emotion analysis on the voice data from the user, means for adjusting the response of the generative AI model based on the emotion analysis results, and means for visually displaying the response. This enables natural and satisfying conversations that correspond to the user's emotions, and makes it possible to provide flexible customer service based on the emotional state of customers, particularly in physical stores.

[1872] 1. An "information terminal" is a device operated by a user that has the ability to capture voice and convert it into text data.

[1873] 2. "Speech recognition" is a technology that converts a user's voice data into text data.

[1874] 3. "Communications network" means the infrastructure for transmitting and receiving data between information terminals and servers, including the Internet.

[1875] 4. "Server" means a computing system that hosts the generative AI model and emotion recognition engine and manages user conversation data and emotion data.

[1876] 5. A "generative AI model" is an algorithm that generates appropriate responses based on input text data.

[1877] 6. An "emotion recognition engine" is a technology that analyzes a user's voice data and evaluates their emotional state.

[1878] 7. “Sentiment analysis” is the process of identifying a user’s emotional state based on audio data.

[1879] 8. "Visually displaying" means presenting the results of an analysis or generated response to the user as text or graphics on a screen.

[1880] 9. "Speech synthesis" is a technology that converts text data into voice data.

[1881] 10. "Brick and Mortar Store" means a store that exists in a physical location and offers goods and services.

[1882] 11. "Customer service" refers to the service and interaction provided by staff to customers in physical stores.

[1883] This invention is a conversation system using an information terminal and a server for use by elderly people, which can provide natural conversation according to the user's emotions. This system is configured using an information terminal, a server, and a communication network. Specific embodiments for implementing this invention are described in detail below.

[1884] System configuration

[1885] server

[1886] The server hosts the generative AI model and emotion recognition engine, manages conversation data and emotion data with the user, receives text data and voice data sent by the user, and generates appropriate responses using the generative AI model.

[1887] The emotion recognition engine analyzes the received voice data to assess the user's emotional state, which is then used to adjust the content and tone of the generative AI model's responses.

[1888] The server also stores user profile information and conversation history in a database, which is used to periodically retrain the generative AI model.

[1889] Terminal

[1890] A device is a user-operated device, such as smart glasses, a smartphone, or a smart speaker. A device has a microphone and a speaker to capture and play back the user's voice.

[1891] The device has a voice recognition function and converts the user's voice data into text data. This text data and voice data are sent to the server, and the response received from the server is played back using a voice synthesis function.

[1892] Furthermore, the terminal can visually display the user's emotional state transmitted from the server.

[1893] user

[1894] The users are elderly people who enjoy conversations by speaking directly into the information terminal. The users can use the system in a natural manner without any special operations. The system also takes into account the users' emotional state, enabling more meaningful conversations.

[1895] System Operation

[1896] 1. Start a conversation

[1897] The user speaks to the information terminal, for example, saying, "I want to know more about this product."

[1898] The device captures the voice with a microphone and converts the voice into text using voice recognition, which is then sent to a server via a communications network.

[1899] 2. Emotion recognition

[1900] The server inputs the received voice data into an emotion recognition engine to evaluate the user's emotional state (e.g., interested, happy, etc.).

[1901] The emotion recognition engine evaluates the emotional state and returns the results to the server.

[1902] 3. Generating a Response

[1903] Based on the evaluation results from the emotion recognition engine, the server inputs text data and emotional information into the generative AI model, which analyzes this data and generates an appropriate response by adjusting the response content and tone.

[1904] For example, if a user says, "I'd like to know more about this product," the response might be, "I'd like to tell you more about this product. Is there anything in particular you'd like to know?"

[1905] 4. Providing a Response

[1906] The server sends the generated text data to the terminal, which then converts the received text data into voice data using a speech synthesis engine and plays it back to the user.

[1907] The device can also visually display the evaluation results from the emotion recognition engine to the user.

[1908] Specific examples

[1909] Specific examples of conversations

[1910] 1. A user says, "I'd like to know more about this product."

[1911] 2. The device converts the voice into text and sends the text data, "I want to know more about this product," and the voice data to the server.

[1912] 3. The server inputs the voice data into an emotion recognition engine and evaluates the emotional state as "interested."

[1913] 4. The server uses the generative AI model to generate a response that says, "We'll explain this product in detail. Is there anything in particular you'd like to know?"

[1914] 5. The device uses a speech synthesis engine to convert the response into voice data and play it back to the user.

[1915] Prompt Sentence Examples

[1916] The user's sentiment is 'interested'. Generate a sentiment-based response to the following text: "I'd like to know more about this product"

[1917] In this way, natural and meaningful conversations that take the user's emotions into account are realized.

[1918] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1919] Step 1:

[1920] A user speaks to an information terminal, for example, saying, "I'd like to know more about this product." The input is the user's voice, which is captured via a microphone. The output is raw voice data.

[1921] Step 2:

[1922] The device inputs the captured voice data into a voice recognition engine, which converts the voice data into text data. The voice recognition engine processes the voice data and outputs text data. Here, the input is raw voice data, and the output is text data.

[1923] Step 3:

[1924] The terminal transmits the converted text data and voice data to the server via the communication network. The input of this step is the text data and voice data, and the output is the data transmitted to the server.

[1925] Step 4:

[1926] The server inputs the received voice data into the emotion recognition engine to evaluate the user's emotional state. In this step, the emotion recognition engine analyzes the voice data to identify the emotional state and outputs the result. The input is the voice data, and the output is the evaluation result of the emotional state.

[1927] Step 5:

[1928] The server inputs the text data into the generative AI model along with the evaluation results from the emotion recognition engine. The generative AI model generates an optimal response based on the text data and the emotion evaluation results. The input for this step is the text data and the emotion evaluation results, and the output is the generated response text.

[1929] Step 6:

[1930] The server inputs the generated text data into a speech synthesis engine and converts it into voice data. The speech synthesis engine processes the text data and outputs voice data. The input is the response text data, and the output is the generated voice data.

[1931] Step 7:

[1932] The device plays the audio data sent from the server and provides it to the user. The device plays this audio data through a speaker so that the user can hear it. The device also visually displays the evaluation results from the emotion recognition engine. The input here is the generated audio data and the emotion evaluation result, and the output is the audio played to the user and the displayed emotional state.

[1933] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1934] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1935] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1936] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1937] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1938] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1939] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1940] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1941] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1942] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1943] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1944] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1945] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1946] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1947] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1948] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1949] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1950] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1951] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1952] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1953] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1954] The following is further disclosed regarding the above embodiment.

[1955] (Claim 1)

[1956] Information terminals used by the elderly,

[1957] means for causing the information terminal to perform voice recognition and converting voice data into text data;

[1958] means for transmitting the text data to a server via a communication network;

[1959] A means for generating a response to the text data using a generative AI model in the server;

[1960] means for transmitting the response to the information terminal.

[1961] (Claim 2)

[1962] converting the response generated by the server into voice data by voice synthesis;

[1963] 2. The system according to claim 1, further comprising means for providing said voice data to a user from said information terminal.

[1964] (Claim 3)

[1965] 2. The system according to claim 1, wherein the communication network between the information terminal and the server includes the Internet.

[1966] (Claim 4)

[1967] The system of claim 1 , wherein the information terminal comprises a smartphone or a smart speaker.

[1968] (Claim 5)

[1969] The server uses the user's conversation data to train the generative AI model;

[1970] 10. The system of claim 1, including means for periodically retraining said generative AI model.

[1971] "Example 1"

[1972] (Claim 1)

[1973] Communication devices used by the elderly and

[1974] means for causing the communication device to perform voice recognition and converting voice data into text data;

[1975] means for transmitting the text data to a computer via a communications network;

[1976] means for generating a response to the text data using a generative AI model on the computer;

[1977] means for transmitting the response to the communication device;

[1978] A means to log the responses and periodically retrain the generative AI model

[1979] A system including:

[1980] (Claim 2)

[1981] converting the computer-generated response into voice data using voice synthesis;

[1982] 2. The system of claim 1, further comprising means for providing said audio data to a user from said communication device.

[1983] (Claim 3)

[1984] 2. The system of claim 1, wherein the communication network between the communication device and the computer includes the Internet.

[1985] "Application Example 1"

[1986] (Claim 1)

[1987] Information terminals used by the elderly,

[1988] means for causing the information terminal to perform voice recognition and converting voice data into text data;

[1989] means for transmitting the text data to a server via a communication network;

[1990] A means for generating a response to the text data using a generative AI model in the server;

[1991] means for transmitting the response to the information terminal;

[1992] A means for generating appropriate suggestions for emergency response when an elderly person utters an emergency word for help;

[1993] means for providing the appropriate suggestion to the user at the information terminal;

[1994] and notifying an emergency contact based on said emergency response proposal.

[1995] A system including:

[1996] (Claim 2)

[1997] converting the response generated by the server into voice data by voice synthesis;

[1998] means for providing the voice data to a user from the information terminal;

[1999] Regularly ask elderly people questions to confirm their safety,

[2000] 10. The system of claim 1, further comprising means for analyzing responses to said safety questions and suggesting necessary actions.

[2001] (Claim 3)

[2002] 2. The system according to claim 1, wherein the communication network between the information terminal and the server includes the Internet.

[2003] "Example 2: Combining Emotion Engines"

[2004] (Claim 1)

[2005] Information terminals used by the elderly,

[2006] means for causing the information terminal to perform voice recognition and converting voice data into text data;

[2007] means for transmitting the text data and voice data to a server via a communication network;

[2008] A means for generating a response to the text data using a generative AI model in the server;

[2009] A means for recognizing and evaluating the user's emotional state from the received voice data using an emotion engine in the server;

[2010] means for adjusting said response based on the emotional state of the user;

[2011] means for transmitting the adjusted response to the information terminal;

[2012] The system includes means for displaying the emotional state on the information terminal.

[2013] (Claim 2)

[2014] 2. The system according to claim 1, further comprising: means for converting the response generated by said server into voice data by voice synthesis and providing said voice data to the user from said information terminal; and means for displaying an evaluation result from the emotion engine on said information terminal.

[2015] (Claim 3)

[2016] 2. The system according to claim 1, wherein the communication network between the information terminal and the server includes the Internet.

[2017] "Application example 2 when combining emotion engines"

[2018] (Claim 1)

[2019] Information terminals used by the elderly,

[2020] means for causing the information terminal to perform voice recognition and converting voice data into text data;

[2021] means for transmitting the text data to a server via a communication network;

[2022] A means for generating a response to the text data using a generative AI model in the server;

[2023] means for transmitting the response to the information terminal;

[2024] a means for using an emotion recognition engine to perform emotion analysis on voice data from a user;

[2025] a means for adjusting the response of the generative AI model based on the sentiment analysis results; and

[2026] The system includes means for visually displaying said response.

[2027] (Claim 2)

[2028] converting the response generated by the server into voice data by voice synthesis;

[2029] 2. The system according to claim 1, further comprising means for providing said voice data to a user from said information terminal.

[2030] (Claim 3)

[2031] 2. The system according to claim 1, wherein the communication network between the information terminal and the server includes the Internet. [Explanation of symbols]

[2032] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Information terminals used by the elderly, means for causing the information terminal to perform voice recognition and converting voice data into text data; means for transmitting the text data to a server via a communication network; A means for generating a response to the text data using a generative AI model in the server; means for transmitting the response to the information terminal.

2. converting the response generated by the server into voice data by voice synthesis; 2. The system according to claim 1, further comprising means for providing said voice data to a user from said information terminal.

3. 2. The system according to claim 1, wherein the communication network between the information terminal and the server includes the Internet.

4. The system according to claim 1 , wherein the information terminal comprises a smartphone or a smart speaker.

5. The server uses the user's conversation data to train the generative AI model; 10. The system of claim 1, further comprising means for periodically retraining said generative AI model.

Images