System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a request from a server for performing safety confirmation and outputting a question for confirming a health condition to a user by voice; means for converting a voice response of the user from voice to text data; means for analyzing the converted text data and transmitting the text data to the server; and means for analyzing the text data by the server and transmitting a notification to an emergency contact when an abnormality is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, as the number of elderly people living alone increases, welfare workers are incurring significant costs in checking their safety, understanding their living conditions, and managing their health. Elderly people also often feel lonely, requiring psychological care. While efficient and continuous follow-up is necessary to resolve these issues, this increases the burden on welfare workers. Therefore, the present invention aims to provide a system that supports the daily lives of elderly people living alone while simultaneously reducing the workload of welfare workers. [Means for solving the problem]

[0005] The present invention relates to a system including: a means for receiving a request from a server for safety confirmation and outputting a question to the user by voice to confirm their health status; a means for converting the user's voice response from voice to text data; a means for analyzing the converted text data and transmitting it to the server; and a means for the server to analyze the text data and, if an abnormality is detected, transmit a notification to an emergency contact. The system also includes a means for selecting a topic for chatting with the user based on a request from the server; a means for initiating a voice chat based on the selected topic; and a means for analyzing the user's voice response to provide appropriate information or continue the conversation. The system also includes a schedule management means in the server for managing the user's medication schedule and meal plan; a means for notifying the user by voice of medication and meal times based on notifications from the server; a means for receiving the user's response and transmitting the response to the server; and a means for the server to update records based on the user's behavioral history. The present invention improves the efficiency of support for the daily lives of elderly people and significantly reduces the burden on welfare staff.

[0006] A "server" is a computer system that processes, stores, and manages data and has the ability to communicate with other devices and users over a network.

[0007] The term "terminal" refers to an electronic device that can be directly operated or used by a user, and in the present invention refers mainly to a speaker device that inputs and outputs audio.

[0008] The term "user" refers to an entity that uses this system to input information or receive provided information, and in the present invention refers primarily to elderly single people.

[0009] "Safety confirmation" refers to procedures and operations for confirming the health status and safety of a user, and includes the process of receiving a user's response to questions set at a specific time.

[0010] A "request" refers to an instruction or command sent from a server to a terminal to perform a specific action.

[0011] "Audio output" refers to the operation of a terminal to provide information to a user by voice, including playing a voice message using a speaker.

[0012] "Voice response" refers to a response or reply given by a user to a terminal using voice.

[0013] "Converting speech to text data" refers to the process by which a device recognizes a user's speech and converts it into a text format that a computer can process.

[0014] "Analysis" refers to a process of examining acquired data in detail to understand its meaning, and in the present invention refers to analyzing text data converted from speech.

[0015] "Notification" refers to information that a server sends to a terminal or other device that acts as a warning or instruction.

[0016] "Chat" refers to a free conversation between a user and a terminal about everyday topics, and includes conversational exchanges based on a specific topic.

[0017] "Topics" refer to the content or topics discussed during casual conversation, and are selected based on the user's interests and past conversation history.

[0018] A "medication schedule" refers to a plan or schedule that specifies when and how often a user should take medication.

[0019] "Meal plan" refers to a plan or schedule that specifies what meals a user should eat and when.

[0020] "Schedule management" refers to a method or system by which a server records, stores, and notifies medication schedules and meal plans.

[0021] "Behavioral history" refers to a record of a user's past actions and operations, and is data that is saved and managed by a server. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention relates to a smart speaker system equipped with AI to support the daily lives of elderly people living alone. Specific program processing and operations are described below.

[0044] Safety confirmation function

[0045] Send request

[0046] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, if the server's scheduler is set to be triggered at 9:00 AM, it generates an API request at that time and sends it to the smart speaker.

[0047] Voice question output

[0048] Device (smart speaker): When the smart speaker receives a request from the server, it outputs a voice message to the user asking, "How are you feeling today?" This is achieved by playing back a voice message generated using a natural language processing (NLP) engine after receiving the request.

[0049] Receiving and sending user responses

[0050] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response, converts it into text, analyzes it, and then sends it to the server.

[0051] Anomaly detection and notification

[0052] Server: The server analyzes the received text data and sends a notification to the emergency contact if it determines that the user's condition is abnormal. For example, if the server receives a response saying "I'm not feeling well," it will execute a function to contact the registered emergency contact via SMS or email.

[0053] Chat function and lifestyle support information provided

[0054] Choosing a topic and starting a conversation

[0055] Device: A smart speaker can automatically start a conversation with the user based on specific keywords or the time of day. For example, in the morning, it might say, "Today's weather is sunny. It's perfect for a walk."

[0056] Receiving user responses and providing information

[0057] User: The user can freely talk and the smart speaker will respond. For example, if the user asks, "What should I do today?", the smart speaker will suggest, "Yesterday you said you wanted to do the laundry. How about reading today?"

[0058] Tracking medication and dietary status and receiving push notifications

[0059] Scheduling and Notifications

[0060] Server: The server manages the user's medication schedule and meal plan and sends notifications to the user at appropriate times. For example, a notification stating "It's time to take your medicine" is sent to a smart speaker at 12 o'clock every day.

[0061] Emitting notifications and receiving responses

[0062] Device: The smart speaker notifies the user of the notification from the server as a voice message. When the user responds with "I just drank it," the smart speaker sends the information to the server.

[0063] Breaking records

[0064] Server: The server receives the user's response and records the information in a database, allowing it to keep a record of medication and dietary habits and to review and analyze the data as needed.

[0065] Providing administrative and daily information

[0066] Acquiring and outputting information

[0067] Server: The server periodically sends important government notifications and daily information to the smart speaker. For example, at the beginning of each month, it sends "This month's event information" to the smart speaker.

[0068] Terminal: The received information is periodically announced to the user through audio, such as "This month's local events are as follows."

[0069] In this way, the smart speaker system equipped with generative AI of the present invention supports the daily lives of the elderly, reduces feelings of loneliness, and supports health management, thereby reducing the workload of welfare workers.

[0070] The processing flow will be explained below.

[0071] Safety confirmation function

[0072] Step 1:

[0073] The server sends a safety check request to the smart speaker at a fixed time every day. Specifically, the scheduler on the server generates a safety check request at 9:00 a.m. and sends an API request to the smart speaker.

[0074] Step 2:

[0075] The device (smart speaker) receives a request from the server and asks the user aloud, "How are you feeling today?" After receiving the request, it uses a natural language processing (NLP) engine to output a predefined message from the speaker using text-to-speech (TTS).

[0076] Step 3:

[0077] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The voice is input via the microphone in the smart speaker.

[0078] Step 4:

[0079] The device converts the user's voice response from speech to text data. The voice input is converted to text using a Speech-to-Text (STT) engine.

[0080] Step 5:

[0081] The device analyzes the converted text data and sends the results to the server, which then sends the results to the server via an API call.

[0082] Step 6:

[0083] The server analyzes the text data from the user and sends a notification to emergency contacts if an abnormality is detected. Based on the anomaly detection algorithm, an SMS or email will be sent as needed.

[0084] Chat function and lifestyle support information provided

[0085] Step 1:

[0086] The device sets a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler calls the chat function at a specific time.

[0087] Step 2:

[0088] When the device is triggered, it selects a topic from the user's past conversation log or a preset topic list, analyzes the conversation data stored in the storage, and selects new topics and appropriate life support information.

[0089] Step 3:

[0090] The device will start a voice conversation based on the selected topic. For example, it will generate a message like, "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[0091] Step 4:

[0092] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[0093] Step 5:

[0094] The device analyzes the user's voice response and provides appropriate information or continues the conversation. For example, in response to the question, "What should I do today?", the device will make a suggestion such as, "You said you would do the laundry yesterday. How about reading?"

[0095] Tracking medication and dietary status and receiving push notifications

[0096] Step 1:

[0097] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[0098] Step 2:

[0099] The server generates a notification at a set time and sends it to the smart speaker. For example, a push notification request is sent to the smart speaker via API at 12:00 every day.

[0100] Step 3:

[0101] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[0102] Step 4:

[0103] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[0104] Step 5:

[0105] The device converts the voice response into text and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[0106] Step 6:

[0107] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] To support the lives of elderly people living alone, it is important to regularly check on their safety and record their health status. However, it is difficult for elderly people to do this themselves, and they often feel lonely. Furthermore, managing and notifying these people places an increased burden on welfare workers and their families. To solve these issues, there is a need for a system that automates safety checks and health management, and allows elderly people to access this information on a daily basis.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes: means for receiving a request from the server and outputting a question to the user by voice to confirm the user's health condition;

[0113] means for converting a user's voice response from voice to text data;

[0114] means for analyzing the converted text data and transmitting it to a server;

[0115] A server analyzes the text data, and if an abnormality is detected, sends a notification to an emergency contact;

[0116] A means for managing a user's medication schedule and meal plan on a server;

[0117] a means for notifying the user by voice of medication and meal times based on notifications from the server;

[0118] means for receiving a user response and transmitting the response to a server;

[0119] A means for the server to update the record based on the user's behavior history;

[0120] A means for selecting topics for chatting with a user using a generative AI model;

[0121] means for initiating an audio chat based on the selected topic;

[0122] A means for analyzing the user's voice response and providing appropriate information or continuing the conversation;

[0123] A server acquires administrative and daily information and transmits it to the user by voice.

[0124] This will automate the process of checking the safety of elderly people living alone, managing their health, and providing daily information, making it possible to support their daily lives.

[0125] A "server" is a computer system that sends, receives, and processes data over a network.

[0126] A "request" is a request from a server to a terminal for a specific operation or provision of information.

[0127] "User" refers to a person who uses this system, such as an elderly person living alone.

[0128] "Audio" refers to a signal that converts sound waveforms into digital data for processing.

[0129] "Text data" is data that is generated by analyzing speech and expressing it as a string of characters.

[0130] "Abnormal" refers to a state or problem that is different from normal in the user's health condition, behavior, etc.

[0131] "Emergency contact" refers to the person or organization to be notified when an abnormality is detected.

[0132] A "medication schedule" is a plan that governs when a user should take which medications.

[0133] A "meal plan" is a plan that manages the times and contents of meals that a user eats each day.

[0134] "Behavioral history" is recorded data such as a user's past actions and responses.

[0135] A "generative AI model" is an artificial intelligence algorithm that generates new information based on data.

[0136] A "prompt sentence" is an instruction or input sentence that a generative AI model uses to generate new information.

[0137] "Chat" refers to everyday conversations with users, and is communication without a specific purpose.

[0138] "Administrative information" refers to important notices and information provided by the government or local governments.

[0139] "Daily information" refers to information related to the user's daily life.

[0140] The present invention relates to a smart speaker system equipped with AI generation to support the daily lives of elderly people living alone. The specific configuration and operation are described below.

[0141] Safety confirmation function

[0142] Server Roles

[0143] The server sends a request to check the safety of elderly single people to the smart speaker at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 a.m. This is done using the Python scheduler library. The server generates an API request and sends it to the smart speaker to start checking the user's safety.

[0144] The role of the device (smart speaker)

[0145] When the smart speaker receives a request from the server, it asks the user a question by voice, such as, "How are you feeling today?" The NLP engine of Amazon Alexa or Google Assistant is used to output this question by voice.

[0146] User Roles

[0147] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. Amazon Transcribe and Google Speech-to-Text are used for the conversion. The converted text data is sent to the server.

[0148] Server analysis and notifications

[0149] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email.

[0150] Chat function and lifestyle support information provided

[0151] Select a topic

[0152] The smart speaker uses a generative AI model (such as OpenAI GPT-3) to select topics for conversation with the user based on specific keywords and the time of day. For example, in the morning, a topic such as "Today's weather is sunny. It's perfect for a walk" is provided.

[0153] Keeping the conversation going

[0154] If a user asks, "What should I do today?", the smart speaker can suggest, "Yesterday you said you wanted to do the laundry. How about reading today?" The conversation continues naturally based on generative AI models.

[0155] Tracking medication and dietary status and receiving push notifications

[0156] Schedule management

[0157] The server manages the user's medication schedule and meal plan, for example, by using the Google Calendar API to send a notification to the smart speaker every day at 12 o'clock saying, "It's time to take your medicine."

[0158] Receiving responses and updating records

[0159] If the user responds, "I just took it," the information is sent to the server via the smart speaker. The server records this response and updates the database based on the user's behavioral history, allowing for analysis of future medication and meal plans.

[0160] Providing administrative and daily information

[0161] Acquisition and provision of information

[0162] The server periodically obtains administrative and daily information and provides it to the user via voice. For example, at the beginning of each month, it obtains "This month's local events" and communicates them to the user via a smart speaker. This information is obtained using various APIs and web scraping technologies.

[0163] Providing concrete examples and prompts

[0164] Example 1: Safety confirmation

[0165] The server sends a safety check request at 9:00 AM, and the smart speaker outputs the question, "How are you feeling today?" The user responds, "I'm fine," and this is sent to the server. An example of the prompt sentence in this case is as follows:

[0166] "Generate a morning health check prompt for the elderly: 'How are you feeling today?'"

[0167] Example 2: Medication notification

[0168] The server sends a notification to the smart speaker every day at 12 o'clock stating "It's time to take your medicine," and the user responds "I just took it."

[0169] "Generate a daily medication reminder for the elderly: 'It's time to take your medication.'"

[0170] In this way, the smart speaker system equipped with generative AI of the present invention is expected to support the daily lives of the elderly, reduce feelings of loneliness, and support health management, thereby reducing the burden on welfare staff and their families.

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Safety confirmation function

[0173] Step 1: Server sends request

[0174] The server sends a safety confirmation request to the smart speaker every day at 9:00 AM. This process is triggered by the server's scheduler and uses the Python scheduler library. After the request is generated, it is sent to the smart speaker as an HTTP POST request. The input is the schedule setting and the timing of the request transmission, and the output is the request transmission to the smart speaker.

[0175] Specific operation: The server's scheduler detects that it is 9:00 AM, generates an API request, and sends it to the smart speaker.

[0176] Step 2: Output voice questions on the device

[0177] When the device (smart speaker) receives a request from the server, it outputs the question "How are you feeling today?" This process is performed by converting text to speech using the NLP engine of Amazon Alexa or Google Assistant. The input is the request received from the server, and the output is the spoken question to the user.

[0178] Specific operation: After the smart speaker receives the request, it will output the voice message, "How are you feeling today?"

[0179] Step 3: User's voice response

[0180] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. This process uses Amazon Transcribe and Google Speech-to-Text. The input is the user's voice response, and the output is text data.

[0181] Specific operation: The user responds "I'm fine," and the voice is converted into text data.

[0182] Step 4: Sending text data to the server

[0183] The smart speaker analyzes the converted text data and sends the results to the server. The input is the converted text data, and the output is sending the text data to the server.

[0184] Specific operation: Text data is sent to the server.

[0185] Step 5: Server analysis and notification

[0186] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email. The input is the user's text data and the output is a notification to the emergency contact.

[0187] Specific operation: The server detects an anomaly and sends an SMS using the Twilio API.

[0188] Chat function and lifestyle support information provided

[0189] Step 1: Select a topic

[0190] The server uses a generative AI model (OpenAI GPT-3) to select topics for chat with the user based on specific keywords and time of day. The input is the time of day and keywords, and the output is the generated topic.

[0191] Specific behavior: In the morning, generate topics such as "Good morning. Today's weather is sunny. It's perfect for a walk."

[0192] Step 2: Start chatting by voice

[0193] The device will start a voice chat based on the selected topic. The input is the generated topic, and the output is the start of a conversation with the user.

[0194] Specific behavior: The smart speaker will speak to the user based on the selected topic.

[0195] Step 3: User voice response and information provision

[0196] The user can freely converse, and the smart speaker responds. A generative AI model is used to provide appropriate information and continue the conversation. The input is the user's voice response, and the output is a response based on the generative AI model.

[0197] What it does: When a user asks, "What should I do today?" the smart speaker suggests, "Yesterday you said you wanted to do laundry. How about reading today?"

[0198] Tracking medication and dietary status and receiving push notifications

[0199] Step 1: Schedule Management

[0200] The server manages the user's medication schedule and meal plan. The input is the user's medication and meal schedule, and the output is the managed schedule information.

[0201] Specific operation: The server manages the schedule using the Google Calendar API.

[0202] Step 2: Sending notifications

[0203] The server sends notifications to the smart speaker based on medication and meal times. The input is schedule information and the output is notifications to the smart speaker.

[0204] Specific operation: Every day at 12 o'clock, a notification stating "It's time to take your medicine" is sent to the smart speaker.

[0205] Step 3: User responses and recording

[0206] The user responds to the smart speaker, and the response is sent to the server. The input is the user's response, and the output is the record update.

[0207] Specific operation: The user responds with "I just drank it," and the information is sent to the server and the record is updated.

[0208] Providing administrative and daily information

[0209] Step 1: Get the information

[0210] The server periodically receives important notices and daily information from the government. The input is a request to obtain government information, and the output is the obtained information.

[0211] Specific operation: The server obtains information using APIs or web scraping technology.

[0212] Step 2: Provide information

[0213] The acquired administrative information and everyday information is communicated to the user via a smart speaker. The input is the acquired information, and the output is the information provided to the user.

[0214] Specific operation: At the beginning of the month, play something like "This month's local events are as follows."

[0215] Through the above processing steps, the system enables safety confirmation for elderly single people, health management, chat support, medication and diet management, and the provision of government information.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] The lives of elderly people living alone are fraught with many physical and psychological anxieties. In particular, there are problems with the lack of means to respond quickly in emergencies and the difficulty of managing their health and confirming their safety in their daily lives. Security issues also remain a concern. A system that can resolve these issues and improve the quality of life for elderly people living alone is needed.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes means for receiving an inquiry from an information processing device to check the safety of the user and outputting a voice inquiry to the user to check their health condition, means for converting the user's voice response from voice to symbol data, means for analyzing the converted symbol data and sending it to the information processing device, means for the information processing device to analyze the symbol data and send a notification to an emergency contact if an abnormality is detected, means for monitoring the user's living environment using voice recognition technology, means for acquiring sensor information and sending a voice notification if an abnormality is detected, and means for installing an emergency button and sending a notification to an emergency contact when the button is used. This enables safety and health management for elderly people living alone and rapid response in emergencies.

[0221] "Safety confirmation" is a procedure to check the health and safety of elderly people living alone.

[0222] An "information processing device" is an electronic device for inputting, processing, and outputting data.

[0223] An "inquiry" is a question or request made to obtain some information.

[0224] "User" refers to a person who uses a system or device.

[0225] "Health status" means the physical and mental state of an individual.

[0226] "Audio output" is the process of generating sound and outputting it from a speaker or the like.

[0227] "Symbolic data" is a series of symbols that represent information such as speech or text.

[0228] "Analysis" is the process of analyzing given data or information and understanding its meaning and structure.

[0229] "Emergency contacts" are pre-defined contacts to be contacted in case of an emergency.

[0230] A "notification" is a message or signal that informs others of specific information.

[0231] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[0232] "Living environment" refers to the entire environment in which an individual spends their daily life.

[0233] "Sensor information" refers to data and measurements collected by sensors.

[0234] "Abnormal" refers to a state or phenomenon that is different from the normal.

[0235] An "emergency button" is a button that can be pressed in an emergency to trigger a specified action.

[0236] In today's society where many elderly people live alone, safety and health management are extremely important issues. This invention provides a system that supports the lives of elderly people by using a smart speaker equipped with a generative AI model.

[0237] To implement the invention, the following components are required:

[0238] 1. Safety confirmation function

[0239] The server sends a request to check the user's safety at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 AM, at which time it generates an API request and sends it to the smart speaker.

[0240] When the device (smart speaker) receives a request from the server, it outputs a voice message to the user asking, "Hello. How are you feeling today?" Google Text-to-Speech (gTTS) is used for voice synthesis.

[0241] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response through its microphone, converts the voice into text data using the SpeechRecognition library, and sends it to the server.

[0242] The server analyzes the received text data and, if an abnormality is detected, sends an email or SMS to an emergency contact.

[0243] 2.Security monitoring function

[0244] The information processing device (server) receives data from sensors attached to doors and windows. If an abnormality is detected, it notifies the user by voice, saying, "The front door has opened. Is it safe?"

[0245] If the user does not respond or responds negatively, the server sends a notification to the emergency contact.

[0246] 3. Emergency response function

[0247] The device (smart speaker) is equipped with an emergency button that the user can press in the event of an emergency. When the emergency button is pressed, a notification is immediately sent to the server, which then immediately sends a notification to pre-registered emergency contacts.

[0248] Examples of implementations of these features include:

[0249] Scenario: At 9am, a smart speaker asks, "Hello. How are you feeling today?" If the user does not reply "I'm fine," it automatically notifies emergency contacts that "No response."

[0250] Example prompt: "I want to create a voice interface to check the safety of elderly people living alone. I want it to check their safety every morning and notify emergency contacts if there is an abnormality. Please give me some concrete code examples."

[0251] This will ensure the safety and health management of elderly people living alone, as well as quick response in emergencies, and improve their quality of life.

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] Server: The server sends a request to check the user's safety at a fixed time every day. The server's scheduler is set to be triggered at 9:00 AM. When the scheduler reaches the specified time, it generates an API request and sends it to the smart speaker. The input is the scheduler's trigger time, and the output is the API request.

[0255] Step 2:

[0256] Device (Smart Speaker): Upon receiving a request from the server, the smart speaker outputs a voice message to the user asking, "Hello, how are you feeling today?" In this process, it uses Google Text-to-Speech (gTTS) to convert the text message into speech. The input is the API request from the server, and the output is the voice message.

[0257] Step 3:

[0258] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The input is the voice question from the smart speaker, and the output is the user's voice response.

[0259] Step 4:

[0260] Device (smart speaker): The smart speaker receives the user's voice response through the microphone and converts the voice into text data using the SpeechRecognition library. The converted text data is sent to the server. The input is the user's voice response, and the output is text data.

[0261] Step 5:

[0262] Server: The server analyzes the received text data and sends an email or SMS to the emergency contact if an abnormality is detected. The analysis includes the user's response and determines whether there is an abnormality. The input is text data, and the output is a notification to the emergency contact.

[0263] Step 6:

[0264] Server: The server periodically receives data from the sensors. If the sensors attached to the doors or windows detect an abnormality, the server acquires and analyzes the data. The input is the data from the sensors, and the output is the analysis result on whether or not there is an abnormality.

[0265] Step 7:

[0266] Terminal (smart speaker): If an abnormality is detected based on sensor information, the smart speaker will notify the user by voice, saying, "The front door has opened. Is it safe?" The input is the abnormality notification from the sensor, and the output is the voice notification.

[0267] Step 8:

[0268] User: The user responds verbally, saying "Yes, I'm safe" or "No, I need help." The input is the voice notification from the smart speaker, and the output is the user's voice response.

[0269] Step 9:

[0270] Device (smart speaker): Receives the user's response again, converts the speech into text using the SpeechRecognition library, and analyzes it. The input is the user's voice response, and the output is text data.

[0271] Step 10:

[0272] Server: Analyzes the user's voice response and sends a notification to emergency contacts if an abnormality is detected. The input is text data, and the output is a notification to emergency contacts.

[0273] Step 11:

[0274] Terminal (Smart Speaker): The smart speaker is equipped with an emergency button. When a user presses the emergency button, a notification is sent to the server immediately. The input is the trigger of the emergency button, and the output is a notification to the server.

[0275] Step 12:

[0276] Server: When the server receives a notification that the emergency button has been pressed, it immediately sends a notification to the pre-registered emergency contacts. The input is a notification that the emergency button has been pressed, and the output is a notification to the emergency contacts.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] This invention relates to a smart speaker system equipped with a generative AI to support the daily lives of elderly people living alone, and in particular, to a system that combines an emotion engine to analyze the user's emotions and respond appropriately. Specific program processing and operation are described below.

[0279] Safety confirmation function and emotion recognition function

[0280] Sending requests and outputting voice questions

[0281] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, when the server's scheduler is triggered at 9:00 a.m., it generates an API request and sends it to the smart speaker.

[0282] Device (smart speaker): When the smart speaker receives the request, it will ask the user aloud the question "How are you feeling today?", using a natural language processing (NLP) engine to play a predefined message via text-to-speech (TTS).

[0283] Receiving, converting, and parsing user responses

[0284] User: The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone inside the smart speaker picks up the voice.

[0285] Device: The smart speaker converts the user's voice response from speech to text using a STT (Speech-to-Text) engine. The converted text data is analyzed and the results are sent to the server.

[0286] Emotion Recognition and Anomaly Detection

[0287] Emotion engine: Smart speakers are equipped with an emotion engine that recognizes emotions from the user's voice. The emotion engine analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state (e.g., sadness, joy, anger, etc.).

[0288] Server: The server analyzes the received text data and emotion data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and the emotion recognition detects "sad," it sends a notification to the emergency contact.

[0289] Chat and emotion recognition

[0290] Topic selection and conversation start

[0291] Device: The smart speaker automatically starts chatting with the user based on specific keywords or the time of day. A timer or scheduler is set to trigger the chat function at a specific time. It provides topics such as, "Today's weather is sunny. It's perfect for a walk."

[0292] Receiving, analyzing, and providing information about user responses

[0293] User: The user can freely talk and the smart speaker will receive the response, for example, "What should I do today?"

[0294] Device: The smart speaker analyzes the user's voice response, and the emotion engine recognizes the user's emotions. If the user asks a question in a "sad voice," it will provide an appropriate encouraging message. For example, it will suggest, "You said you would do the laundry yesterday. How about reading a book?"

[0295] Tracking medication and dietary status and receiving push notifications

[0296] Scheduling and Notifications

[0297] Server: The server manages the user's medication schedule and meal plan and sends appropriate notifications to the user at specified times. For example, it generates a notification saying "It's time to take your medicine" at 12 o'clock every day and sends it to a smart speaker.

[0298] Device: The smart speaker conveys notifications from the server to the user as voice messages.

[0299] User responses and record updates

[0300] User: The user responds by saying "I just drank it." This speech is received by the smart speaker.

[0301] Device: The smart speaker converts voice into text, analyzes emotions, and sends it to the server.

[0302] Server: The server updates the user's behavioral history based on the received response data and emotion data.

[0303] Providing administrative and daily information

[0304] Acquisition and provision of information

[0305] Server: The server periodically sends important government notifications and useful information to the smart speaker. For example, it provides "This month's local event information" at the beginning of each month.

[0306] Device: The smart speaker will then relay the information to the user aloud, playing something like, "This month's events are as follows."

[0307] By combining this generative AI-equipped smart speaker system with an emotion engine, it is possible to provide more advanced user follow-up and personalized support. As a result, it is expected to significantly contribute to improving the quality of life of the elderly and reducing their sense of loneliness. It will also significantly reduce the workload of welfare workers.

[0308] The processing flow will be explained below.

[0309] Safety confirmation function and emotion recognition function

[0310] Step 1:

[0311] The server sends a safety check request to the smart speaker at a fixed time every day. The server's scheduler is triggered at 9:00 AM, generates a safety check request, and sends it to the smart speaker as an API request.

[0312] Step 2:

[0313] The device (smart speaker) receives a request from the server and outputs the question "How are you feeling today?" to the user by voice. A natural language processing (NLP) engine is used to play back a predefined message using text-to-speech (TTS).

[0314] Step 3:

[0315] The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone in the smart speaker picks up the user's voice.

[0316] Step 4:

[0317] The device uses a Speech-to-Text (STT) engine to convert the user's voice response from speech to text, then analyzes the converted text data and sends it to the server.

[0318] Step 5:

[0319] The device uses an emotion engine to recognize emotions from the user's responses. It analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state.

[0320] Step 6:

[0321] The server analyzes the received text data and emotional data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and emotion recognition detects "sad," it sends a notification to emergency contacts.

[0322] Chat and emotion recognition

[0323] Step 1:

[0324] The device will set a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler will call the chat function at a specific time.

[0325] Step 2:

[0326] The device selects topics from the user's past conversation logs or a preset topic list, analyzes the conversation data stored in storage, and selects new topics and appropriate lifestyle support information.

[0327] Step 3:

[0328] The device will start a voice chat based on the selected topic. It will generate the message "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[0329] Step 4:

[0330] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[0331] Step 5:

[0332] The device analyzes the user's voice response, and the emotion engine recognizes the user's emotions. For example, if the user asks a question in a sad voice, the device generates an appropriate encouraging message based on that emotional data, suggesting, "You said you would do the laundry yesterday. How about reading a book?"

[0333] Tracking medication and dietary status and receiving push notifications

[0334] Step 1:

[0335] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[0336] Step 2:

[0337] The server generates a notification at a set time and sends it to the smart speaker. A push notification request is sent to the smart speaker via API at 12:00 every day.

[0338] Step 3:

[0339] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[0340] Step 4:

[0341] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[0342] Step 5:

[0343] The device converts the voice response into text, analyzes emotions, and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[0344] Step 6:

[0345] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[0346] Example 2

[0347] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] In modern society, the increase in the number of elderly people living alone is becoming a major social problem. When elderly people living alone at home do not adequately manage their health or communicate with others on a daily basis, they experience feelings of loneliness and health risks. Furthermore, conventional healthcare systems have difficulty grasping the user's emotional state, which can prevent them from providing appropriate support. Furthermore, elderly people tend to neglect managing their medication and diet, creating a need for a means to properly manage their behavioral history. It is necessary to develop a system that can solve these issues and improve the quality of life for the elderly.

[0349] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0350] In this invention, the server includes means for receiving a request from the server for safety confirmation and outputting a question to the user by voice to confirm the user's health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for identifying emotions from the user's voice using an emotion recognition engine, means for sending the identified emotion data to the server, and means for the server to evaluate the user's condition based on the emotion data and the text data and send a notification to an emergency contact if an abnormality is detected. This makes it possible to grasp not only the user's health condition but also their emotional state and to detect abnormalities early.

[0351] "Safety check" is the act of checking the health and safety of residents.

[0352] A "server" is a computer system for storing, processing, and communicating data.

[0353] A "means for receiving a request" is a mechanism for receiving requests from the outside.

[0354] The "means for outputting questions by voice" refers to a device or system that uses voice synthesis technology to reproduce text information as voice.

[0355] A "means for converting a voice response from voice to text data" is a device or system that uses voice recognition technology to convert a voice signal into a corresponding text format.

[0356] The "means for analyzing the converted text data and transmitting it to the server" refers to the process for understanding and evaluating the text data and the communication means for transmitting that data to the server.

[0357] An "emotion recognition engine" is an algorithm or software that analyzes tone, pitch, speed, etc. from audio data to identify emotional states.

[0358] The "means for transmitting emotion data to the server" refers to a communication means for transferring the data obtained by emotion recognition to the server.

[0359] "Means for sending a notification to an emergency contact when an abnormality is detected" refers to a mechanism for sending a notification to a registered emergency contact when the system detects an abnormality in the user's behavior.

[0360] A "topic selection means" is a system or device that determines the topic of conversation with a user based on a pre-set algorithm.

[0361] The "means for starting a voice chat" is a system or device for starting a conversation based on a selected topic using voice synthesis technology.

[0362] The "means for providing information or continuing a conversation" refers to a system or device for returning appropriate information in response to a user's response or for continuing a conversation.

[0363] The "schedule management means in the server for managing medication schedules and meal plans" is a system for managing the times when a user takes medicine and the times when they eat, and saving these as a schedule on the server.

[0364] The "means for notifying the user of medication and meal times by voice" refers to a system or device that uses voice synthesis technology to notify the user of medication and meal times by voice.

[0365] The "means for receiving a response from the user and transmitting the response to the server" refers to a communication means for taking in a reply from the user and transmitting it as data to the server.

[0366] A "means for updating behavioral history" is a system or device that records data about a user's behavior and modifies or adds to that history information based on new data.

[0367] This invention relates to a smart speaker system that utilizes generative AI technology to support the daily lives of elderly people living alone. Specifically, it combines an emotion recognition engine to analyze the user's emotions and respond appropriately. The details are described below.

[0368] System Configuration

[0369] This system consists of a server, a smart speaker (hereafter referred to as the "terminal"), and an elderly person. The server stores, processes, and communicates data, while the terminal is a device that enables dialogue with the user and voice responses.

[0370] Hardware and software used

[0371] Server: The server is used to perform functions such as safety confirmation, data analysis, anomaly detection, emergency notification, etc. Examples include Linux-based servers and cloud services (AWS, Google Cloud, etc.).

[0372] Device (smart speaker): Equipped with a natural language processing (NLP) engine, text-to-speech (TTS), speech recognition (STT), and emotion recognition engine. Specifically, it uses Google Dialogflow, Google's STT engine, IBM Watson's Tone Analyzer, etc.

[0373] Operation overview

[0374] 1. Safety confirmation

[0375] Every morning at 9:00, the server uses a scheduler to send a safety confirmation request to the device. The device receives this request and asks the user verbally, "How are you feeling today?" The user's response is received by the device's microphone, and a speech recognition engine converts it into text. The converted text data is also sent to an emotion recognition engine, and is sent to the server along with emotion data. The server analyzes this data and sends a notification to emergency contacts as necessary.

[0376] 2. Chat function

[0377] The device will begin chatting with the user based on a specific time or trigger keyword. A topic selection algorithm will provide a topic through voice, such as, "Today's weather is sunny. Perfect for a walk." Once the user's response is received, the voice recognition engine again converts it into text data, and the emotion recognition engine identifies emotions. By providing appropriate information and continuing the conversation, the device can alleviate the user's sense of loneliness.

[0378] 3. Understanding medication and dietary status

[0379] The server manages the user's medication schedule and meal plan, and sends voice notifications through the device at designated times. An example message is "It's time to take your medicine." The user's responses are received by the device, converted into text, and sentiment analysis is performed before being sent to the server. The server updates the user's behavioral history based on this data.

[0380] 4. Providing administrative and daily information

[0381] The server periodically sends important notifications from the government and other useful information to the device. For example, the device will provide information about local events this month. The device will then communicate this information to the user via voice.

[0382] Examples of specific examples and prompts

[0383] Example 1: Safety confirmation

[0384] The server generates an API request at 9:00 a.m. and sends a safety confirmation message to the smart speaker.

[0385] The device asks the user aloud, "How are you feeling today?"

[0386] Example prompt: "Generate an API request to check the health of an elderly person at 9 AM."

[0387] Example 2: Emotion recognition and anomaly detection

[0388] The server analyzes the user's text data and emotional data, and if the user replies "I'm not feeling well" and the emotion is determined to be "sad," it sends a notification to the emergency contact.

[0389] Example prompt: "Design a system that analyzes a user's emotional data and physical responses, and notifies emergency contacts if there are any abnormalities."

[0390] Example 3: Chat function

[0391] At 2 p.m., the device triggers the chat function and offers the topic, "Today's weather is sunny. Perfect for a walk."

[0392] Example prompt: "Design a system that triggers a chat feature at a specific time and provides topics about today's weather."

[0393] By combining this generative AI-equipped smart speaker system with an emotion engine, it will improve the quality of life of the elderly and reduce their sense of loneliness. It will also reduce the workload of welfare workers and enable more personalized support.

[0394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0395] Step 1:

[0396] Server sends safety confirmation request

[0397] The server uses a scheduler to send a safety confirmation request to the device every morning at 9:00. In this process, when a trigger is received from the scheduler, an API request is generated and sent to the smart speaker.

[0398] Input: Trigger event from scheduler.

[0399] Processing: Generating and sending API requests.

[0400] Output: API request sent successfully.

[0401] Action: scheduler.trigger('9:00', sendHealthCheckRequest) is called, a request is generated and sent to the smart speaker.

[0402] Step 2:

[0403] Voice question output by terminal

[0404] The device (smart speaker) receives the request from the server and outputs the question "How are you feeling today?" via Google Dialogflow.

[0405] Input: The API request from the server.

[0406] Processing: Converts text to speech using a Text-to-Speech (TTS) engine.

[0407] Output: A spoken question to the user.

[0408] Behavior: The askHealthStatus() function is called, and TTS.synthesize("How are you feeling today?") is executed.

[0409] Step 3:

[0410] Receiving a user's voice response

[0411] The user responds verbally, and the smart speaker's microphone captures the audio, with examples of responses like "I'm fine" or "I'm not feeling well."

[0412] Input: The user's spoken response.

[0413] Processing: Receiving audio.

[0414] Output: Received audio data.

[0415] Operation: The microphone is always in standby mode, and microphone.capture() captures audio data.

[0416] Step 4:

[0417] Device-based speech-to-text conversion

[0418] Smart speakers use Google's Speech-to-Text (STT) engine to convert the user's voice into text data.

[0419] Input: Received audio data.

[0420] Processing: Converts audio data into text data using the STT engine.

[0421] Output: The converted text data.

[0422] What happens: The convertSpeechToText(audioData) function is called to convert the speech to text.

[0423] Step 5:

[0424] Device-based emotion recognition

[0425] The text data is sent to an emotion recognition engine (e.g., IBM Watson's Tone Analyzer) to analyze the user's emotions.

[0426] Input: The converted text data.

[0427] Processing: Uses an emotion recognition engine to analyze tone, pitch, speed, etc. to identify emotions.

[0428] Output: Emotion data.

[0429] Operation: The analyzeEmotion(textData) function is called to generate emotion data.

[0430] Step 6:

[0431] Analysis of text data and emotion data by the server

[0432] The server receives and analyzes the text data and emotional data sent from the smart speaker.

[0433] Input: Text data and emotion data.

[0434] Processing: Based on the results of the data analysis, the user's health status is evaluated.

[0435] Output: User state assessment results.

[0436] Operation: The evaluateUserStatus(textData, emotionData) function is called to determine whether there are any abnormalities.

[0437] Step 7:

[0438] Server-based anomaly detection and notification to emergency contacts

[0439] If the server detects an abnormality, it will send a notification via email or SMS to the designated emergency contact.

[0440] Input: User's condition assessment results.

[0441] Processing: Generating and sending notifications when an anomaly is detected.

[0442] Output: Notification sent to emergency contacts.

[0443] Action: The sendEmergencyNotification(contactInfo, userStatus) function is called to send the emergency notification.

[0444] (Application example 2)

[0445] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] In the daily lives of elderly people living alone, there is a need to grasp changes in their physical condition and emotions in real time, and to respond appropriately and contact them in emergencies. In particular, a system that can respond quickly when unexpected physical or mental illness occurs is needed. Another challenge is to provide a system that can manage medication schedules and meal plans, as well as provide encouragement and suggestions according to the user's emotional state.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0448] In this invention, the server includes means for receiving a request to check safety and outputting a question to the user by voice to check their health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for the server to analyze the text data and send a notification to an emergency contact if an abnormality is detected, emotion analysis means for recognizing emotions from the user's voice, and means for evaluating the state of the malfunction based on the analyzed emotion data and sending an additional notification to the emergency contact as necessary. This makes it possible to grasp and respond to not only the user's physical condition but also their emotional state in real time.

[0449] "Safety check" is an action taken to periodically check the health status and safety of a user.

[0450] A "server" is a computer system for sending, receiving, and processing data.

[0451] A "request" is a message from a server requesting specific information or an operation.

[0452] "Users" refer to people such as elderly people who use this system.

[0453] "Health status confirmation questions" are questions asked to find out the user's physical condition and mood.

[0454] "Audio output means" refers to a technology that converts text information into audio and conveys it to the user.

[0455] "Means for converting voice into text data" refers to a technology that converts voice information into text information using voice recognition technology.

[0456] "Means for analyzing text data" refers to technology for analyzing converted character information.

[0457] "Emotion analysis means" is a technology that estimates a user's emotions from voice and text information.

[0458] "Emergency contacts" are people or organizations to whom notifications should be sent in the event of an emergency.

[0459] The "means for evaluating a user's state of failure based on emotional data" is a technology that analyzes the user's emotional information and evaluates the user's current state.

[0460] An "additional notification" is a further message sent in addition to a notification already sent.

[0461] The "means for selecting a topic for chatting" is a technique for selecting a topic for conversation with a user.

[0462] The "means for starting a voice chat" is a technique for starting a voice conversation based on a selected topic.

[0463] "Means for continuing a conversation" refers to a technique for continuing a conversation depending on the user's reaction and situation.

[0464] A "medication schedule" is a plan that manages the types and times of medication a user takes.

[0465] A "meal plan" is a plan that manages the menu and times of meals that a user will eat.

[0466] "Schedule management tools" are technologies for recording and managing medication and meal plans.

[0467] "A means for notifying the user of medication and meal times by voice" refers to a technology that notifies the user of scheduled medication and meal times by voice.

[0468] "Behavioral history" is a record of the actions and reactions of a user.

[0469] "Means for updating behavioral history" refers to technology that updates behavioral records based on collected data.

[0470] The present invention relates to a security assistant system for supporting the daily lives of elderly people living alone, and in particular to a system that combines an emotion engine to analyze the user's emotions and take appropriate action. The following describes the processing and operation of the system program.

[0471] System Configuration

[0472] Hardware and Software

[0473] Server: Generates safety confirmation requests, analyzes data, and manages emergency notifications. An API server using Python is expected.

[0474] Terminal (smart speaker, smartphone, etc.): A device that acts as an interface with the user and is equipped with speech recognition (Speech-to-Text), speech synthesis (Text-to-Speech), emotion analysis engine (EmotionRecognizer), etc.

[0475] Emotion analysis engine: Extracts emotions from the user's voice and provides appropriate responses. The library used is expected to be EmotionRecognizer.

[0476] System Operation

[0477] Safety confirmation function

[0478] The server sends a safety confirmation request to the device at a fixed time every day. Triggered via an API, the question "How are you feeling today?" is output to the smart speaker using text-to-speech (TTS).

[0479] When the user responds by saying something like "I'm fine" or "I'm not feeling well," the device converts the voice into text data using a speech-to-text (STT) engine.

[0480] The converted text data is converted into emotion data by an emotion analysis engine, and then the text data and emotion data are sent to the server.

[0481] The server analyzes the transmitted data and sends a notification to emergency contacts if an abnormality is detected.

[0482] Chat function

[0483] Based on a request from the server, the device selects a specific topic and starts chatting with the user. A timer or scheduler is set to trigger the chat function at a specific time.

[0484] Based on the selected topic, a message such as "Today's weather is sunny. It's perfect for a walk" is output in voice.

[0485] When a user asks, "What should I do today?", the device converts the voice to text and the emotion engine analyzes the emotion. If the user asks the question in a "sad voice," it will provide an appropriate encouraging message.

[0486] Understanding medication and dietary status

[0487] The server manages the user's medication schedule and meal plan, and generates notifications such as "It's time to take your medicine" at specified times and sends them to the terminal.

[0488] When the user responds verbally by saying "I just drank it," the response is received by the device, and the data converted into text is analyzed by the emotion analysis engine.

[0489] The converted and analyzed data is sent to the server and the behavioral history is updated.

[0490] Examples of specific examples and prompts

[0491] Specific examples

[0492] If a user responds, "I'm feeling unwell today," the emotion engine will recognize it as "anxiety," and the server will send a notification to emergency contacts. Also, upon hearing that the user is feeling unwell, the device will offer an encouraging message, asking, "Are you feeling stressed or anxious? Is there anything we can do to help?"

[0493] Prompt Sentence Examples

[0494] "Build a model that recognizes emotions from a user's speech. Given the following spoken text: 'I'm feeling bad today', recognize the emotion as 'anxiety'."

[0495] This system will enable users to respond quickly and appropriately when they feel unwell or emotionally unwell, which is expected to improve their quality of life. It will also make it easier to ensure the safety of elderly people by providing appropriate advice and emergency responses according to the user's situation.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The server generates a safety confirmation request at a fixed time every day and sends it to the device via API. At this time, the server's scheduler is triggered and a message is generated asking, "How are you feeling today?"

[0499] Input: Server scheduler trigger

[0500] Output: Generate and send a safety confirmation request

[0501] How it works: The server uses its internal clock to generate a request at the specified time and send it to the device.

[0502] Step 2:

[0503] Based on the received request, the device outputs a voice question to the user, such as "How are you feeling today?" This process uses a text-to-speech (TTS) engine.

[0504] Input: Safety confirmation request from the server

[0505] Output: Voice question output

[0506] How it works: The device receives the request and uses its speech synthesis engine to play a predefined message aloud.

[0507] Step 3:

[0508] The user responds by saying "I'm fine" or "I'm not feeling well," and the device's microphone picks up the voice.

[0509] Input: User's voice response

[0510] Output: Microphone audio capture

[0511] How it works: The device's microphone captures the user's voice data and passes it to a speech recognition engine.

[0512] Step 4:

[0513] The device's speech recognition engine (STT) converts the speech into text.

[0514] Input: User's voice data

[0515] Output: Conversion to text data

[0516] How it works: The device's speech recognition engine analyzes the voice data and converts it into corresponding text data.

[0517] Step 5:

[0518] The text data is passed to a sentiment analysis engine, which analyzes the tone, pitch, and rate of the audio data to determine the emotional state.

[0519] Input: Converted text data

[0520] Output: Conversion to emotion data

[0521] How it works: The sentiment analysis engine analyzes emotions based on text data and identifies emotional states.

[0522] Step 6:

[0523] The device transmits the analyzed text data and emotion data to the server.

[0524] Input: Text data and emotion data

[0525] Output: Send data to the server

[0526] Operation: The device sends data to the server using an API.

[0527] Step 7:

[0528] The server analyzes the data and sends a notification to emergency contacts if an abnormality is detected. The notification is triggered when the text data matches "I'm not feeling well" and the emotion matches "anxiety" or "sadness."

[0529] Input: Text data and emotion data

[0530] Output: Send emergency notification

[0531] How it works: The server analyzes the data and if the conditions are met, it will send a notification to emergency contacts via email or SMS.

[0532] Step 8:

[0533] Based on a request from the server, the device selects a specific topic and begins chatting by voice.

[0534] Input: Chat request from server

[0535] Output: Start voice chat

[0536] How it works: The device's topic selection engine determines the topic and outputs it aloud using a speech synthesis engine.

[0537] Step 9:

[0538] The user responds to the chat, their voice is converted into text, and sentiment analysis is performed.

[0539] Input: User's voice response (chat)

[0540] Output: Generate text data and emotion data

[0541] How it works: The device converts the user's speech into text, and an emotion analysis engine analyzes the emotion.

[0542] Step 10:

[0543] The server manages the user's medication schedule and meal plan and sends notifications to the device at specified times.

[0544] Input: Medication schedule and meal plan

[0545] Output: Generate and send medication and meal notifications

[0546] How it works: The server uses an internal schedule management system to send notifications to the device at the specified time.

[0547] Step 11:

[0548] The user responds by saying something like "I just drank it," and the device converts the speech into text for sentiment analysis, which is then sent to the server.

[0549] Input: User's voice response (medication, food)

[0550] Output: Sending text and emotion data

[0551] How it works: The device converts the user's voice data into text, performs sentiment analysis, and sends the data to the server.

[0552] Step 12:

[0553] The server analyzes the received data and updates the records based on the user's behavioral history.

[0554] Input: Text data and emotion data

[0555] Output: Update history

[0556] How it works: The server analyzes the data and updates the behavioral history database.

[0557] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0558] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0559] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0560] [Second embodiment]

[0561] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0562] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0563] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0564] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0565] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0566] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0567] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0568] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0569] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0570] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0571] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0572] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0573] The present invention relates to a smart speaker system equipped with AI to support the daily lives of elderly people living alone. Specific program processing and operations are described below.

[0574] Safety confirmation function

[0575] Send request

[0576] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, if the server's scheduler is set to be triggered at 9:00 AM, it generates an API request at that time and sends it to the smart speaker.

[0577] Voice question output

[0578] Device (smart speaker): When the smart speaker receives a request from the server, it outputs a voice message to the user asking, "How are you feeling today?" This is achieved by playing back a voice message generated using a natural language processing (NLP) engine after receiving the request.

[0579] Receiving and sending user responses

[0580] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response, converts it into text, analyzes it, and then sends it to the server.

[0581] Anomaly detection and notification

[0582] Server: The server analyzes the received text data and sends a notification to the emergency contact if it determines that the user's condition is abnormal. For example, if the server receives a response saying "I'm not feeling well," it will execute a function to contact the registered emergency contact via SMS or email.

[0583] Chat function and lifestyle support information provided

[0584] Choosing a topic and starting a conversation

[0585] Device: A smart speaker can automatically start a conversation with the user based on specific keywords or the time of day. For example, in the morning, it might say, "Today's weather is sunny. It's perfect for a walk."

[0586] Receiving user responses and providing information

[0587] User: The user can freely talk and the smart speaker will respond. For example, if the user asks, "What should I do today?", the smart speaker will suggest, "Yesterday you said you wanted to do the laundry. How about reading today?"

[0588] Tracking medication and dietary status and receiving push notifications

[0589] Scheduling and Notifications

[0590] Server: The server manages the user's medication schedule and meal plan and sends notifications to the user at appropriate times. For example, a notification stating "It's time to take your medicine" is sent to a smart speaker at 12 o'clock every day.

[0591] Emitting notifications and receiving responses

[0592] Device: The smart speaker notifies the user of the notification from the server as a voice message. When the user responds with "I just drank it," the smart speaker sends the information to the server.

[0593] Breaking records

[0594] Server: The server receives the user's response and records the information in a database, allowing it to keep a record of medication and dietary habits and to review and analyze the data as needed.

[0595] Providing administrative and daily information

[0596] Acquiring and outputting information

[0597] Server: The server periodically sends important government notifications and daily information to the smart speaker. For example, at the beginning of each month, it sends "This month's event information" to the smart speaker.

[0598] Terminal: The received information is periodically announced to the user through audio, such as "This month's local events are as follows."

[0599] In this way, the smart speaker system equipped with generative AI of the present invention supports the daily lives of the elderly, reduces feelings of loneliness, and supports health management, thereby reducing the workload of welfare workers.

[0600] The processing flow will be explained below.

[0601] Safety confirmation function

[0602] Step 1:

[0603] The server sends a safety check request to the smart speaker at a fixed time every day. Specifically, the scheduler on the server generates a safety check request at 9:00 a.m. and sends an API request to the smart speaker.

[0604] Step 2:

[0605] The device (smart speaker) receives a request from the server and asks the user aloud, "How are you feeling today?" After receiving the request, it uses a natural language processing (NLP) engine to output a predefined message from the speaker using text-to-speech (TTS).

[0606] Step 3:

[0607] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The voice is input via the microphone in the smart speaker.

[0608] Step 4:

[0609] The device converts the user's voice response from speech to text data. The voice input is converted to text using a Speech-to-Text (STT) engine.

[0610] Step 5:

[0611] The device analyzes the converted text data and sends the results to the server, which then sends the results to the server via an API call.

[0612] Step 6:

[0613] The server analyzes the text data from the user and sends a notification to emergency contacts if an abnormality is detected. Based on the anomaly detection algorithm, an SMS or email will be sent as needed.

[0614] Chat function and lifestyle support information provided

[0615] Step 1:

[0616] The device sets a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler calls the chat function at a specific time.

[0617] Step 2:

[0618] When the device is triggered, it selects a topic from the user's past conversation log or a preset topic list, analyzes the conversation data stored in the storage, and selects new topics and appropriate life support information.

[0619] Step 3:

[0620] The device will start a voice conversation based on the selected topic. For example, it will generate a message like, "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[0621] Step 4:

[0622] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[0623] Step 5:

[0624] The device analyzes the user's voice response and provides appropriate information or continues the conversation. For example, in response to the question, "What should I do today?", the device will make a suggestion such as, "You said you would do the laundry yesterday. How about reading?"

[0625] Tracking medication and dietary status and receiving push notifications

[0626] Step 1:

[0627] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[0628] Step 2:

[0629] The server generates a notification at a set time and sends it to the smart speaker. For example, a push notification request is sent to the smart speaker via API at 12:00 every day.

[0630] Step 3:

[0631] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[0632] Step 4:

[0633] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[0634] Step 5:

[0635] The device converts the voice response into text and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[0636] Step 6:

[0637] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[0638] Example 1

[0639] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] To support the lives of elderly people living alone, it is important to regularly check on their safety and record their health status. However, it is difficult for elderly people to do this themselves, and they often feel lonely. Furthermore, managing and notifying these people places an increased burden on welfare workers and their families. To solve these issues, there is a need for a system that automates safety checks and health management, and allows elderly people to access this information on a daily basis.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0642] In this invention, the server includes: means for receiving a request from the server and outputting a question to the user by voice to confirm the user's health condition;

[0643] means for converting a user's voice response from voice to text data;

[0644] means for analyzing the converted text data and transmitting it to a server;

[0645] A server analyzes the text data, and if an abnormality is detected, sends a notification to an emergency contact;

[0646] A means for managing a user's medication schedule and meal plan on a server;

[0647] a means for notifying the user by voice of medication and meal times based on notifications from the server;

[0648] means for receiving a user response and transmitting the response to a server;

[0649] A means for the server to update the record based on the user's behavior history;

[0650] A means for selecting topics for chatting with a user using a generative AI model;

[0651] means for initiating an audio chat based on the selected topic;

[0652] A means for analyzing the user's voice response and providing appropriate information or continuing the conversation;

[0653] A server acquires administrative and daily information and transmits it to the user by voice.

[0654] This will automate the process of checking the safety of elderly people living alone, managing their health, and providing daily information, making it possible to support their daily lives.

[0655] A "server" is a computer system that sends, receives, and processes data over a network.

[0656] A "request" is a request from a server to a terminal for a specific operation or provision of information.

[0657] "User" refers to a person who uses this system, such as an elderly person living alone.

[0658] "Audio" refers to a signal that converts sound waveforms into digital data for processing.

[0659] "Text data" is data that is generated by analyzing speech and expressing it as a string of characters.

[0660] "Abnormal" refers to a state or problem that is different from normal in the user's health condition, behavior, etc.

[0661] "Emergency contact" refers to the person or organization to be notified when an abnormality is detected.

[0662] A "medication schedule" is a plan that governs when a user should take which medications.

[0663] A "meal plan" is a plan that manages the times and contents of meals that a user eats each day.

[0664] "Behavioral history" is recorded data such as a user's past actions and responses.

[0665] A "generative AI model" is an artificial intelligence algorithm that generates new information based on data.

[0666] A "prompt sentence" is an instruction or input sentence that a generative AI model uses to generate new information.

[0667] "Chat" refers to everyday conversations with users, and is communication without a specific purpose.

[0668] "Administrative information" refers to important notices and information provided by the government or local governments.

[0669] "Daily information" refers to information related to the user's daily life.

[0670] The present invention relates to a smart speaker system equipped with AI generation to support the daily lives of elderly people living alone. The specific configuration and operation are described below.

[0671] Safety confirmation function

[0672] Server Roles

[0673] The server sends a request to check the safety of elderly single people to the smart speaker at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 a.m. This is done using the Python scheduler library. The server generates an API request and sends it to the smart speaker to start checking the user's safety.

[0674] The role of the device (smart speaker)

[0675] When the smart speaker receives a request from the server, it asks the user a question by voice, such as, "How are you feeling today?" The NLP engine of Amazon Alexa or Google Assistant is used to output this question by voice.

[0676] User Roles

[0677] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. Amazon Transcribe and Google Speech-to-Text are used for the conversion. The converted text data is sent to the server.

[0678] Server analysis and notifications

[0679] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email.

[0680] Chat function and lifestyle support information provided

[0681] Select a topic

[0682] The smart speaker uses a generative AI model (such as OpenAI GPT-3) to select topics for conversation with the user based on specific keywords and the time of day. For example, in the morning, a topic such as "Today's weather is sunny. It's perfect for a walk" is provided.

[0683] Keeping the conversation going

[0684] If a user asks, "What should I do today?", the smart speaker can suggest, "Yesterday you said you wanted to do the laundry. How about reading today?" The conversation continues naturally based on generative AI models.

[0685] Tracking medication and dietary status and receiving push notifications

[0686] Schedule management

[0687] The server manages the user's medication schedule and meal plan, for example, by using the Google Calendar API to send a notification to the smart speaker every day at 12 o'clock saying, "It's time to take your medicine."

[0688] Receiving responses and updating records

[0689] If the user responds, "I just took it," the information is sent to the server via the smart speaker. The server records this response and updates the database based on the user's behavioral history, allowing for analysis of future medication and meal plans.

[0690] Providing administrative and daily information

[0691] Acquisition and provision of information

[0692] The server periodically obtains administrative and daily information and provides it to the user via voice. For example, at the beginning of each month, it obtains "This month's local events" and communicates them to the user via a smart speaker. This information is obtained using various APIs and web scraping technologies.

[0693] Providing concrete examples and prompts

[0694] Example 1: Safety confirmation

[0695] The server sends a safety check request at 9:00 AM, and the smart speaker outputs the question, "How are you feeling today?" The user responds, "I'm fine," and this is sent to the server. An example of the prompt sentence in this case is as follows:

[0696] "Generate a morning health check prompt for the elderly: 'How are you feeling today?'"

[0697] Example 2: Medication notification

[0698] The server sends a notification to the smart speaker every day at 12 o'clock stating "It's time to take your medicine," and the user responds "I just took it."

[0699] "Generate a daily medication reminder for the elderly: 'It's time to take your medication.'"

[0700] In this way, the smart speaker system equipped with generative AI of the present invention is expected to support the daily lives of the elderly, reduce feelings of loneliness, and support health management, thereby reducing the burden on welfare staff and their families.

[0701] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0702] Safety confirmation function

[0703] Step 1: Server sends request

[0704] The server sends a safety confirmation request to the smart speaker every day at 9:00 AM. This process is triggered by the server's scheduler and uses the Python scheduler library. After the request is generated, it is sent to the smart speaker as an HTTP POST request. The input is the schedule setting and the timing of the request transmission, and the output is the request transmission to the smart speaker.

[0705] Specific operation: The server's scheduler detects that it is 9:00 AM, generates an API request, and sends it to the smart speaker.

[0706] Step 2: Output voice questions on the device

[0707] When the device (smart speaker) receives a request from the server, it outputs the question "How are you feeling today?" This process is performed by converting text to speech using the NLP engine of Amazon Alexa or Google Assistant. The input is the request received from the server, and the output is the spoken question to the user.

[0708] Specific operation: After the smart speaker receives the request, it will output the voice message, "How are you feeling today?"

[0709] Step 3: User's voice response

[0710] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. This process uses Amazon Transcribe and Google Speech-to-Text. The input is the user's voice response, and the output is text data.

[0711] Specific operation: The user responds "I'm fine," and the voice is converted into text data.

[0712] Step 4: Sending text data to the server

[0713] The smart speaker analyzes the converted text data and sends the results to the server. The input is the converted text data, and the output is sending the text data to the server.

[0714] Specific operation: Text data is sent to the server.

[0715] Step 5: Server analysis and notification

[0716] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email. The input is the user's text data and the output is a notification to the emergency contact.

[0717] Specific operation: The server detects an anomaly and sends an SMS using the Twilio API.

[0718] Chat function and lifestyle support information provided

[0719] Step 1: Select a topic

[0720] The server uses a generative AI model (OpenAI GPT-3) to select topics for chat with the user based on specific keywords and time of day. The input is the time of day and keywords, and the output is the generated topic.

[0721] Specific behavior: In the morning, generate topics such as "Good morning. Today's weather is sunny. It's perfect for a walk."

[0722] Step 2: Start chatting by voice

[0723] The device will start a voice chat based on the selected topic. The input is the generated topic, and the output is the start of a conversation with the user.

[0724] Specific behavior: The smart speaker will speak to the user based on the selected topic.

[0725] Step 3: User voice response and information provision

[0726] The user can freely converse, and the smart speaker responds. A generative AI model is used to provide appropriate information and continue the conversation. The input is the user's voice response, and the output is a response based on the generative AI model.

[0727] What it does: When a user asks, "What should I do today?" the smart speaker suggests, "Yesterday you said you wanted to do laundry. How about reading today?"

[0728] Tracking medication and dietary status and receiving push notifications

[0729] Step 1: Schedule Management

[0730] The server manages the user's medication schedule and meal plan. The input is the user's medication and meal schedule, and the output is the managed schedule information.

[0731] Specific operation: The server manages the schedule using the Google Calendar API.

[0732] Step 2: Sending notifications

[0733] The server sends notifications to the smart speaker based on medication and meal times. The input is schedule information and the output is notifications to the smart speaker.

[0734] Specific operation: Every day at 12 o'clock, a notification stating "It's time to take your medicine" is sent to the smart speaker.

[0735] Step 3: User responses and recording

[0736] The user responds to the smart speaker, and the response is sent to the server. The input is the user's response, and the output is the record update.

[0737] Specific operation: The user responds with "I just drank it," and the information is sent to the server and the record is updated.

[0738] Providing administrative and daily information

[0739] Step 1: Get the information

[0740] The server periodically receives important notices and daily information from the government. The input is a request to obtain government information, and the output is the obtained information.

[0741] Specific operation: The server obtains information using APIs or web scraping technology.

[0742] Step 2: Provide information

[0743] The acquired administrative information and everyday information is communicated to the user via a smart speaker. The input is the acquired information, and the output is the information provided to the user.

[0744] Specific operation: At the beginning of the month, play something like "This month's local events are as follows."

[0745] Through the above processing steps, the system enables safety confirmation for elderly single people, health management, chat support, medication and diet management, and the provision of government information.

[0746] (Application example 1)

[0747] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0748] The lives of elderly people living alone are fraught with many physical and psychological anxieties. In particular, there are problems with the lack of means to respond quickly in emergencies and the difficulty of managing their health and confirming their safety in their daily lives. Security issues also remain a concern. A system that can resolve these issues and improve the quality of life for elderly people living alone is needed.

[0749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0750] In this invention, the server includes means for receiving an inquiry from an information processing device to check the safety of the user and outputting a voice inquiry to the user to check their health condition, means for converting the user's voice response from voice to symbol data, means for analyzing the converted symbol data and sending it to the information processing device, means for the information processing device to analyze the symbol data and send a notification to an emergency contact if an abnormality is detected, means for monitoring the user's living environment using voice recognition technology, means for acquiring sensor information and sending a voice notification if an abnormality is detected, and means for installing an emergency button and sending a notification to an emergency contact when the button is used. This enables safety and health management for elderly people living alone and rapid response in emergencies.

[0751] "Safety confirmation" is a procedure to check the health and safety of elderly people living alone.

[0752] An "information processing device" is an electronic device for inputting, processing, and outputting data.

[0753] An "inquiry" is a question or request made to obtain some information.

[0754] "User" refers to a person who uses a system or device.

[0755] "Health status" means the physical and mental state of an individual.

[0756] "Audio output" is the process of generating sound and outputting it from a speaker or the like.

[0757] "Symbolic data" is a series of symbols that represent information such as speech or text.

[0758] "Analysis" is the process of analyzing given data or information and understanding its meaning and structure.

[0759] "Emergency contacts" are pre-defined contacts to be contacted in case of an emergency.

[0760] A "notification" is a message or signal that informs others of specific information.

[0761] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[0762] "Living environment" refers to the entire environment in which an individual spends their daily life.

[0763] "Sensor information" refers to data and measurements collected by sensors.

[0764] "Abnormal" refers to a state or phenomenon that is different from the normal.

[0765] An "emergency button" is a button that can be pressed in an emergency to trigger a specified action.

[0766] In today's society where many elderly people live alone, safety and health management are extremely important issues. This invention provides a system that supports the lives of elderly people by using a smart speaker equipped with a generative AI model.

[0767] To implement the invention, the following components are required:

[0768] 1. Safety confirmation function

[0769] The server sends a request to check the user's safety at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 AM, at which time it generates an API request and sends it to the smart speaker.

[0770] When the device (smart speaker) receives a request from the server, it outputs a voice message to the user asking, "Hello. How are you feeling today?" Google Text-to-Speech (gTTS) is used for voice synthesis.

[0771] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response through its microphone, converts the voice into text data using the SpeechRecognition library, and sends it to the server.

[0772] The server analyzes the received text data and, if an abnormality is detected, sends an email or SMS to an emergency contact.

[0773] 2.Security monitoring function

[0774] The information processing device (server) receives data from sensors attached to doors and windows. If an abnormality is detected, it notifies the user by voice, saying, "The front door has opened. Is it safe?"

[0775] If the user does not respond or responds negatively, the server sends a notification to the emergency contact.

[0776] 3. Emergency response function

[0777] The device (smart speaker) is equipped with an emergency button that the user can press in the event of an emergency. When the emergency button is pressed, a notification is immediately sent to the server, which then immediately sends a notification to pre-registered emergency contacts.

[0778] Examples of implementations of these features include:

[0779] Scenario: At 9am, a smart speaker asks, "Hello. How are you feeling today?" If the user does not reply "I'm fine," it automatically notifies emergency contacts that "No response."

[0780] Example prompt: "I want to create a voice interface to check the safety of elderly people living alone. I want it to check their safety every morning and notify emergency contacts if there is an abnormality. Please give me some concrete code examples."

[0781] This will ensure the safety and health management of elderly people living alone, as well as quick response in emergencies, and improve their quality of life.

[0782] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0783] Step 1:

[0784] Server: The server sends a request to check the user's safety at a fixed time every day. The server's scheduler is set to be triggered at 9:00 AM. When the scheduler reaches the specified time, it generates an API request and sends it to the smart speaker. The input is the scheduler's trigger time, and the output is the API request.

[0785] Step 2:

[0786] Device (Smart Speaker): Upon receiving a request from the server, the smart speaker outputs a voice message to the user asking, "Hello, how are you feeling today?" In this process, it uses Google Text-to-Speech (gTTS) to convert the text message into speech. The input is the API request from the server, and the output is the voice message.

[0787] Step 3:

[0788] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The input is the voice question from the smart speaker, and the output is the user's voice response.

[0789] Step 4:

[0790] Device (smart speaker): The smart speaker receives the user's voice response through the microphone and converts the voice into text data using the SpeechRecognition library. The converted text data is sent to the server. The input is the user's voice response, and the output is text data.

[0791] Step 5:

[0792] Server: The server analyzes the received text data and sends an email or SMS to the emergency contact if an abnormality is detected. The analysis includes the user's response and determines whether there is an abnormality. The input is text data, and the output is a notification to the emergency contact.

[0793] Step 6:

[0794] Server: The server periodically receives data from the sensors. If the sensors attached to the doors or windows detect an abnormality, the server acquires and analyzes the data. The input is the data from the sensors, and the output is the analysis result on whether or not there is an abnormality.

[0795] Step 7:

[0796] Terminal (smart speaker): If an abnormality is detected based on sensor information, the smart speaker will notify the user by voice, saying, "The front door has opened. Is it safe?" The input is the abnormality notification from the sensor, and the output is the voice notification.

[0797] Step 8:

[0798] User: The user responds verbally, saying "Yes, I'm safe" or "No, I need help." The input is the voice notification from the smart speaker, and the output is the user's voice response.

[0799] Step 9:

[0800] Device (smart speaker): Receives the user's response again, converts the speech into text using the SpeechRecognition library, and analyzes it. The input is the user's voice response, and the output is text data.

[0801] Step 10:

[0802] Server: Analyzes the user's voice response and sends a notification to emergency contacts if an abnormality is detected. The input is text data, and the output is a notification to emergency contacts.

[0803] Step 11:

[0804] Terminal (Smart Speaker): The smart speaker is equipped with an emergency button. When a user presses the emergency button, a notification is sent to the server immediately. The input is the trigger of the emergency button, and the output is a notification to the server.

[0805] Step 12:

[0806] Server: When the server receives a notification that the emergency button has been pressed, it immediately sends a notification to the pre-registered emergency contacts. The input is a notification that the emergency button has been pressed, and the output is a notification to the emergency contacts.

[0807] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0808] This invention relates to a smart speaker system equipped with a generative AI to support the daily lives of elderly people living alone, and in particular, to a system that combines an emotion engine to analyze the user's emotions and respond appropriately. Specific program processing and operation are described below.

[0809] Safety confirmation function and emotion recognition function

[0810] Sending requests and outputting voice questions

[0811] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, when the server's scheduler is triggered at 9:00 a.m., it generates an API request and sends it to the smart speaker.

[0812] Device (smart speaker): When the smart speaker receives the request, it will ask the user aloud the question "How are you feeling today?", using a natural language processing (NLP) engine to play a predefined message via text-to-speech (TTS).

[0813] Receiving, converting, and parsing user responses

[0814] User: The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone inside the smart speaker picks up the voice.

[0815] Device: The smart speaker converts the user's voice response from speech to text using a STT (Speech-to-Text) engine. The converted text data is analyzed and the results are sent to the server.

[0816] Emotion Recognition and Anomaly Detection

[0817] Emotion engine: Smart speakers are equipped with an emotion engine that recognizes emotions from the user's voice. The emotion engine analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state (e.g., sadness, joy, anger, etc.).

[0818] Server: The server analyzes the received text data and emotion data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and the emotion recognition detects "sad," it sends a notification to the emergency contact.

[0819] Chat and emotion recognition

[0820] Topic selection and conversation start

[0821] Device: The smart speaker automatically starts chatting with the user based on specific keywords or the time of day. A timer or scheduler is set to trigger the chat function at a specific time. It provides topics such as, "Today's weather is sunny. It's perfect for a walk."

[0822] Receiving, analyzing, and providing information about user responses

[0823] User: The user can freely talk and the smart speaker will receive the response, for example, "What should I do today?"

[0824] Device: The smart speaker analyzes the user's voice response, and the emotion engine recognizes the user's emotions. If the user asks a question in a "sad voice," it will provide an appropriate encouraging message. For example, it will suggest, "You said you would do the laundry yesterday. How about reading a book?"

[0825] Tracking medication and dietary status and receiving push notifications

[0826] Scheduling and Notifications

[0827] Server: The server manages the user's medication schedule and meal plan and sends appropriate notifications to the user at specified times. For example, it generates a notification saying "It's time to take your medicine" at 12 o'clock every day and sends it to a smart speaker.

[0828] Device: The smart speaker conveys notifications from the server to the user as voice messages.

[0829] User responses and record updates

[0830] User: The user responds by saying "I just drank it." This speech is received by the smart speaker.

[0831] Device: The smart speaker converts voice into text, analyzes emotions, and sends it to the server.

[0832] Server: The server updates the user's behavioral history based on the received response data and emotion data.

[0833] Providing administrative and daily information

[0834] Acquisition and provision of information

[0835] Server: The server periodically sends important government notifications and useful information to the smart speaker. For example, it provides "This month's local event information" at the beginning of each month.

[0836] Device: The smart speaker will then relay the information to the user aloud, playing something like, "This month's events are as follows."

[0837] By combining this generative AI-equipped smart speaker system with an emotion engine, it is possible to provide more advanced user follow-up and personalized support. As a result, it is expected to significantly contribute to improving the quality of life of the elderly and reducing their sense of loneliness. It will also significantly reduce the workload of welfare workers.

[0838] The processing flow will be explained below.

[0839] Safety confirmation function and emotion recognition function

[0840] Step 1:

[0841] The server sends a safety check request to the smart speaker at a fixed time every day. The server's scheduler is triggered at 9:00 AM, generates a safety check request, and sends it to the smart speaker as an API request.

[0842] Step 2:

[0843] The device (smart speaker) receives a request from the server and outputs the question "How are you feeling today?" to the user by voice. A natural language processing (NLP) engine is used to play back a predefined message using text-to-speech (TTS).

[0844] Step 3:

[0845] The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone in the smart speaker picks up the user's voice.

[0846] Step 4:

[0847] The device uses a Speech-to-Text (STT) engine to convert the user's voice response from speech to text, then analyzes the converted text data and sends it to the server.

[0848] Step 5:

[0849] The device uses an emotion engine to recognize emotions from the user's responses. It analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state.

[0850] Step 6:

[0851] The server analyzes the received text data and emotional data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and emotion recognition detects "sad," it sends a notification to emergency contacts.

[0852] Chat and emotion recognition

[0853] Step 1:

[0854] The device will set a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler will call the chat function at a specific time.

[0855] Step 2:

[0856] The device selects topics from the user's past conversation logs or a preset topic list, analyzes the conversation data stored in storage, and selects new topics and appropriate lifestyle support information.

[0857] Step 3:

[0858] The device will start a voice chat based on the selected topic. It will generate the message "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[0859] Step 4:

[0860] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[0861] Step 5:

[0862] The device analyzes the user's voice response, and the emotion engine recognizes the user's emotions. For example, if the user asks a question in a sad voice, the device generates an appropriate encouraging message based on that emotional data, suggesting, "You said you would do the laundry yesterday. How about reading a book?"

[0863] Tracking medication and dietary status and receiving push notifications

[0864] Step 1:

[0865] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[0866] Step 2:

[0867] The server generates a notification at a set time and sends it to the smart speaker. A push notification request is sent to the smart speaker via API at 12:00 every day.

[0868] Step 3:

[0869] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[0870] Step 4:

[0871] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[0872] Step 5:

[0873] The device converts the voice response into text, analyzes emotions, and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[0874] Step 6:

[0875] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[0876] Example 2

[0877] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] In modern society, the increase in the number of elderly people living alone is becoming a major social problem. When elderly people living alone at home do not adequately manage their health or communicate with others on a daily basis, they experience feelings of loneliness and health risks. Furthermore, conventional healthcare systems have difficulty grasping the user's emotional state, which can prevent them from providing appropriate support. Furthermore, elderly people tend to neglect managing their medication and diet, creating a need for a means to properly manage their behavioral history. It is necessary to develop a system that can solve these issues and improve the quality of life for the elderly.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0880] In this invention, the server includes means for receiving a request from the server for safety confirmation and outputting a question to the user by voice to confirm the user's health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for identifying emotions from the user's voice using an emotion recognition engine, means for sending the identified emotion data to the server, and means for the server to evaluate the user's condition based on the emotion data and the text data and send a notification to an emergency contact if an abnormality is detected. This makes it possible to grasp not only the user's health condition but also their emotional state and to detect abnormalities early.

[0881] "Safety check" is the act of checking the health and safety of residents.

[0882] A "server" is a computer system for storing, processing, and communicating data.

[0883] A "means for receiving a request" is a mechanism for receiving requests from the outside.

[0884] The "means for outputting questions by voice" refers to a device or system that uses voice synthesis technology to reproduce text information as voice.

[0885] A "means for converting a voice response from voice to text data" is a device or system that uses voice recognition technology to convert a voice signal into a corresponding text format.

[0886] The "means for analyzing the converted text data and transmitting it to the server" refers to the process for understanding and evaluating the text data and the communication means for transmitting that data to the server.

[0887] An "emotion recognition engine" is an algorithm or software that analyzes tone, pitch, speed, etc. from audio data to identify emotional states.

[0888] The "means for transmitting emotion data to the server" refers to a communication means for transferring the data obtained by emotion recognition to the server.

[0889] "Means for sending a notification to an emergency contact when an abnormality is detected" refers to a mechanism for sending a notification to a registered emergency contact when the system detects an abnormality in the user's behavior.

[0890] A "topic selection means" is a system or device that determines the topic of conversation with a user based on a pre-set algorithm.

[0891] The "means for starting a voice chat" is a system or device for starting a conversation based on a selected topic using voice synthesis technology.

[0892] The "means for providing information or continuing a conversation" refers to a system or device for returning appropriate information in response to a user's response or for continuing a conversation.

[0893] The "schedule management means in the server for managing medication schedules and meal plans" is a system for managing the times when a user takes medicine and the times when they eat, and saving these as a schedule on the server.

[0894] The "means for notifying the user of medication and meal times by voice" refers to a system or device that uses voice synthesis technology to notify the user of medication and meal times by voice.

[0895] The "means for receiving a response from the user and transmitting the response to the server" refers to a communication means for taking in a reply from the user and transmitting it as data to the server.

[0896] A "means for updating behavioral history" is a system or device that records data about a user's behavior and modifies or adds to that history information based on new data.

[0897] This invention relates to a smart speaker system that utilizes generative AI technology to support the daily lives of elderly people living alone. Specifically, it combines an emotion recognition engine to analyze the user's emotions and respond appropriately. The details are described below.

[0898] System Configuration

[0899] This system consists of a server, a smart speaker (hereafter referred to as the "terminal"), and an elderly person. The server stores, processes, and communicates data, while the terminal is a device that enables dialogue with the user and voice responses.

[0900] Hardware and software used

[0901] Server: The server is used to perform functions such as safety confirmation, data analysis, anomaly detection, emergency notification, etc. Examples include Linux-based servers and cloud services (AWS, Google Cloud, etc.).

[0902] Device (smart speaker): Equipped with a natural language processing (NLP) engine, text-to-speech (TTS), speech recognition (STT), and emotion recognition engine. Specifically, it uses Google Dialogflow, Google's STT engine, IBM Watson's Tone Analyzer, etc.

[0903] Operation overview

[0904] 1. Safety confirmation

[0905] Every morning at 9:00, the server uses a scheduler to send a safety confirmation request to the device. The device receives this request and asks the user verbally, "How are you feeling today?" The user's response is received by the device's microphone, and a speech recognition engine converts it into text. The converted text data is also sent to an emotion recognition engine, and is sent to the server along with emotion data. The server analyzes this data and sends a notification to emergency contacts as necessary.

[0906] 2. Chat function

[0907] The device will begin chatting with the user based on a specific time or trigger keyword. A topic selection algorithm will provide a topic through voice, such as, "Today's weather is sunny. Perfect for a walk." Once the user's response is received, the voice recognition engine again converts it into text data, and the emotion recognition engine identifies emotions. By providing appropriate information and continuing the conversation, the device can alleviate the user's sense of loneliness.

[0908] 3. Understanding medication and dietary status

[0909] The server manages the user's medication schedule and meal plan, and sends voice notifications through the device at designated times. An example message is "It's time to take your medicine." The user's responses are received by the device, converted into text, and sentiment analysis is performed before being sent to the server. The server updates the user's behavioral history based on this data.

[0910] 4. Providing administrative and daily information

[0911] The server periodically sends important notifications from the government and other useful information to the device. For example, the device will provide information about local events this month. The device will then communicate this information to the user via voice.

[0912] Examples of specific examples and prompts

[0913] Example 1: Safety confirmation

[0914] The server generates an API request at 9:00 a.m. and sends a safety confirmation message to the smart speaker.

[0915] The device asks the user aloud, "How are you feeling today?"

[0916] Example prompt: "Generate an API request to check the health of an elderly person at 9 AM."

[0917] Example 2: Emotion recognition and anomaly detection

[0918] The server analyzes the user's text data and emotional data, and if the user replies "I'm not feeling well" and the emotion is determined to be "sad," it sends a notification to the emergency contact.

[0919] Example prompt: "Design a system that analyzes a user's emotional data and physical responses, and notifies emergency contacts if there are any abnormalities."

[0920] Example 3: Chat function

[0921] At 2 p.m., the device triggers the chat function and offers the topic, "Today's weather is sunny. Perfect for a walk."

[0922] Example prompt: "Design a system that triggers a chat feature at a specific time and provides topics about today's weather."

[0923] By combining this generative AI-equipped smart speaker system with an emotion engine, it will improve the quality of life of the elderly and reduce their sense of loneliness. It will also reduce the workload of welfare workers and enable more personalized support.

[0924] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0925] Step 1:

[0926] Server sends safety confirmation request

[0927] The server uses a scheduler to send a safety confirmation request to the device every morning at 9:00. In this process, when a trigger is received from the scheduler, an API request is generated and sent to the smart speaker.

[0928] Input: Trigger event from scheduler.

[0929] Processing: Generating and sending API requests.

[0930] Output: API request sent successfully.

[0931] Action: scheduler.trigger('9:00', sendHealthCheckRequest) is called, a request is generated and sent to the smart speaker.

[0932] Step 2:

[0933] Voice question output by terminal

[0934] The device (smart speaker) receives the request from the server and outputs the question "How are you feeling today?" via Google Dialogflow.

[0935] Input: The API request from the server.

[0936] Processing: Converts text to speech using a Text-to-Speech (TTS) engine.

[0937] Output: A spoken question to the user.

[0938] Behavior: The askHealthStatus() function is called, and TTS.synthesize("How are you feeling today?") is executed.

[0939] Step 3:

[0940] Receiving a user's voice response

[0941] The user responds verbally, and the smart speaker's microphone captures the audio, with examples of responses like "I'm fine" or "I'm not feeling well."

[0942] Input: The user's spoken response.

[0943] Processing: Receiving audio.

[0944] Output: Received audio data.

[0945] Operation: The microphone is always in standby mode, and microphone.capture() captures audio data.

[0946] Step 4:

[0947] Device-based speech-to-text conversion

[0948] Smart speakers use Google's Speech-to-Text (STT) engine to convert the user's voice into text data.

[0949] Input: Received audio data.

[0950] Processing: Converts audio data into text data using the STT engine.

[0951] Output: The converted text data.

[0952] What happens: The convertSpeechToText(audioData) function is called to convert the speech to text.

[0953] Step 5:

[0954] Device-based emotion recognition

[0955] The text data is sent to an emotion recognition engine (e.g., IBM Watson's Tone Analyzer) to analyze the user's emotions.

[0956] Input: The converted text data.

[0957] Processing: Uses an emotion recognition engine to analyze tone, pitch, speed, etc. to identify emotions.

[0958] Output: Emotion data.

[0959] Operation: The analyzeEmotion(textData) function is called to generate emotion data.

[0960] Step 6:

[0961] Analysis of text data and emotion data by the server

[0962] The server receives and analyzes the text data and emotional data sent from the smart speaker.

[0963] Input: Text data and emotion data.

[0964] Processing: Based on the results of the data analysis, the user's health status is evaluated.

[0965] Output: User state assessment results.

[0966] Operation: The evaluateUserStatus(textData, emotionData) function is called to determine whether there are any abnormalities.

[0967] Step 7:

[0968] Server-based anomaly detection and notification to emergency contacts

[0969] If the server detects an abnormality, it will send a notification via email or SMS to the designated emergency contact.

[0970] Input: User's condition assessment results.

[0971] Processing: Generating and sending notifications when an anomaly is detected.

[0972] Output: Notification sent to emergency contacts.

[0973] Action: The sendEmergencyNotification(contactInfo, userStatus) function is called to send the emergency notification.

[0974] (Application example 2)

[0975] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0976] In the daily lives of elderly people living alone, there is a need to grasp changes in their physical condition and emotions in real time, and to respond appropriately and contact them in emergencies. In particular, a system that can respond quickly when unexpected physical or mental illness occurs is needed. Another challenge is to provide a system that can manage medication schedules and meal plans, as well as provide encouragement and suggestions according to the user's emotional state.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0978] In this invention, the server includes means for receiving a request to check safety and outputting a question to the user by voice to check their health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for the server to analyze the text data and send a notification to an emergency contact if an abnormality is detected, emotion analysis means for recognizing emotions from the user's voice, and means for evaluating the state of the malfunction based on the analyzed emotion data and sending an additional notification to the emergency contact as necessary. This makes it possible to grasp and respond to not only the user's physical condition but also their emotional state in real time.

[0979] "Safety check" is an action taken to periodically check the health status and safety of a user.

[0980] A "server" is a computer system for sending, receiving, and processing data.

[0981] A "request" is a message from a server requesting specific information or an operation.

[0982] "Users" refer to people such as elderly people who use this system.

[0983] "Health status confirmation questions" are questions asked to find out the user's physical condition and mood.

[0984] "Audio output means" refers to a technology that converts text information into audio and conveys it to the user.

[0985] "Means for converting voice into text data" refers to a technology that converts voice information into text information using voice recognition technology.

[0986] "Means for analyzing text data" refers to technology for analyzing converted character information.

[0987] "Emotion analysis means" is a technology that estimates a user's emotions from voice and text information.

[0988] "Emergency contacts" are people or organizations to whom notifications should be sent in the event of an emergency.

[0989] The "means for evaluating a user's state of failure based on emotional data" is a technology that analyzes the user's emotional information and evaluates the user's current state.

[0990] An "additional notification" is a further message sent in addition to a notification already sent.

[0991] The "means for selecting a topic for chatting" is a technique for selecting a topic for conversation with a user.

[0992] The "means for starting a voice chat" is a technique for starting a voice conversation based on a selected topic.

[0993] "Means for continuing a conversation" refers to a technique for continuing a conversation depending on the user's reaction and situation.

[0994] A "medication schedule" is a plan that manages the types and times of medication a user takes.

[0995] A "meal plan" is a plan that manages the menu and times of meals that a user will eat.

[0996] "Schedule management tools" are technologies for recording and managing medication and meal plans.

[0997] "A means for notifying the user of medication and meal times by voice" refers to a technology that notifies the user of scheduled medication and meal times by voice.

[0998] "Behavioral history" is a record of the actions and reactions of a user.

[0999] "Means for updating behavioral history" refers to technology that updates behavioral records based on collected data.

[1000] The present invention relates to a security assistant system for supporting the daily lives of elderly people living alone, and in particular to a system that combines an emotion engine to analyze the user's emotions and take appropriate action. The following describes the processing and operation of the system program.

[1001] System Configuration

[1002] Hardware and Software

[1003] Server: Generates safety confirmation requests, analyzes data, and manages emergency notifications. An API server using Python is expected.

[1004] Terminal (smart speaker, smartphone, etc.): A device that acts as an interface with the user and is equipped with speech recognition (Speech-to-Text), speech synthesis (Text-to-Speech), emotion analysis engine (EmotionRecognizer), etc.

[1005] Emotion analysis engine: Extracts emotions from the user's voice and provides appropriate responses. The library used is expected to be EmotionRecognizer.

[1006] System Operation

[1007] Safety confirmation function

[1008] The server sends a safety confirmation request to the device at a fixed time every day. Triggered via an API, the question "How are you feeling today?" is output to the smart speaker using text-to-speech (TTS).

[1009] When the user responds by saying something like "I'm fine" or "I'm not feeling well," the device converts the voice into text data using a speech-to-text (STT) engine.

[1010] The converted text data is converted into emotion data by an emotion analysis engine, and then the text data and emotion data are sent to the server.

[1011] The server analyzes the transmitted data and sends a notification to emergency contacts if an abnormality is detected.

[1012] Chat function

[1013] Based on a request from the server, the device selects a specific topic and starts chatting with the user. A timer or scheduler is set to trigger the chat function at a specific time.

[1014] Based on the selected topic, a message such as "Today's weather is sunny. It's perfect for a walk" is output in voice.

[1015] When a user asks, "What should I do today?", the device converts the voice to text and the emotion engine analyzes the emotion. If the user asks the question in a "sad voice," it will provide an appropriate encouraging message.

[1016] Understanding medication and dietary status

[1017] The server manages the user's medication schedule and meal plan, and generates notifications such as "It's time to take your medicine" at specified times and sends them to the terminal.

[1018] When the user responds verbally by saying "I just drank it," the response is received by the device, and the data converted into text is analyzed by the emotion analysis engine.

[1019] The converted and analyzed data is sent to the server and the behavioral history is updated.

[1020] Examples of specific examples and prompts

[1021] Specific examples

[1022] If a user responds, "I'm feeling unwell today," the emotion engine will recognize it as "anxiety," and the server will send a notification to emergency contacts. Also, upon hearing that the user is feeling unwell, the device will offer an encouraging message, asking, "Are you feeling stressed or anxious? Is there anything we can do to help?"

[1023] Prompt Sentence Examples

[1024] "Build a model that recognizes emotions from a user's speech. Given the following spoken text: 'I'm feeling bad today', recognize the emotion as 'anxiety'."

[1025] This system will enable users to respond quickly and appropriately when they feel unwell or emotionally unwell, which is expected to improve their quality of life. It will also make it easier to ensure the safety of elderly people by providing appropriate advice and emergency responses according to the user's situation.

[1026] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1027] Step 1:

[1028] The server generates a safety confirmation request at a fixed time every day and sends it to the device via API. At this time, the server's scheduler is triggered and a message is generated asking, "How are you feeling today?"

[1029] Input: Server scheduler trigger

[1030] Output: Generate and send a safety confirmation request

[1031] How it works: The server uses its internal clock to generate a request at the specified time and send it to the device.

[1032] Step 2:

[1033] Based on the received request, the device outputs a voice question to the user, such as "How are you feeling today?" This process uses a text-to-speech (TTS) engine.

[1034] Input: Safety confirmation request from the server

[1035] Output: Voice question output

[1036] How it works: The device receives the request and uses its speech synthesis engine to play a predefined message aloud.

[1037] Step 3:

[1038] The user responds by saying "I'm fine" or "I'm not feeling well," and the device's microphone picks up the voice.

[1039] Input: User's voice response

[1040] Output: Microphone audio capture

[1041] How it works: The device's microphone captures the user's voice data and passes it to a speech recognition engine.

[1042] Step 4:

[1043] The device's speech recognition engine (STT) converts the speech into text.

[1044] Input: User's voice data

[1045] Output: Conversion to text data

[1046] How it works: The device's speech recognition engine analyzes the voice data and converts it into corresponding text data.

[1047] Step 5:

[1048] The text data is passed to a sentiment analysis engine, which analyzes the tone, pitch, and rate of the audio data to determine the emotional state.

[1049] Input: Converted text data

[1050] Output: Conversion to emotion data

[1051] How it works: The sentiment analysis engine analyzes emotions based on text data and identifies emotional states.

[1052] Step 6:

[1053] The device transmits the analyzed text data and emotion data to the server.

[1054] Input: Text data and emotion data

[1055] Output: Send data to the server

[1056] Operation: The device sends data to the server using an API.

[1057] Step 7:

[1058] The server analyzes the data and sends a notification to emergency contacts if an abnormality is detected. The notification is triggered when the text data matches "I'm not feeling well" and the emotion matches "anxiety" or "sadness."

[1059] Input: Text data and emotion data

[1060] Output: Send emergency notification

[1061] How it works: The server analyzes the data and if the conditions are met, it will send a notification to emergency contacts via email or SMS.

[1062] Step 8:

[1063] Based on a request from the server, the device selects a specific topic and begins chatting by voice.

[1064] Input: Chat request from server

[1065] Output: Start voice chat

[1066] How it works: The device's topic selection engine determines the topic and outputs it aloud using a speech synthesis engine.

[1067] Step 9:

[1068] The user responds to the chat, their voice is converted into text, and sentiment analysis is performed.

[1069] Input: User's voice response (chat)

[1070] Output: Generate text data and emotion data

[1071] How it works: The device converts the user's speech into text, and an emotion analysis engine analyzes the emotion.

[1072] Step 10:

[1073] The server manages the user's medication schedule and meal plan and sends notifications to the device at specified times.

[1074] Input: Medication schedule and meal plan

[1075] Output: Generate and send medication and meal notifications

[1076] How it works: The server uses an internal schedule management system to send notifications to the device at the specified time.

[1077] Step 11:

[1078] The user responds by saying something like "I just drank it," and the device converts the speech into text for sentiment analysis, which is then sent to the server.

[1079] Input: User's voice response (medication, food)

[1080] Output: Sending text and emotion data

[1081] How it works: The device converts the user's voice data into text, performs sentiment analysis, and sends the data to the server.

[1082] Step 12:

[1083] The server analyzes the received data and updates the records based on the user's behavioral history.

[1084] Input: Text data and emotion data

[1085] Output: Update history

[1086] How it works: The server analyzes the data and updates the behavioral history database.

[1087] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1088] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1089] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1090] [Third embodiment]

[1091] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1092] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1093] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1094] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1095] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1096] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1097] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1098] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1099] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1100] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1101] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1102] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1103] The present invention relates to a smart speaker system equipped with AI to support the daily lives of elderly people living alone. Specific program processing and operations are described below.

[1104] Safety confirmation function

[1105] Send request

[1106] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, if the server's scheduler is set to be triggered at 9:00 AM, it generates an API request at that time and sends it to the smart speaker.

[1107] Voice question output

[1108] Device (smart speaker): When the smart speaker receives a request from the server, it outputs a voice message to the user asking, "How are you feeling today?" This is achieved by playing back a voice message generated using a natural language processing (NLP) engine after receiving the request.

[1109] Receiving and sending user responses

[1110] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response, converts it into text, analyzes it, and then sends it to the server.

[1111] Anomaly detection and notification

[1112] Server: The server analyzes the received text data and sends a notification to the emergency contact if it determines that the user's condition is abnormal. For example, if the server receives a response saying "I'm not feeling well," it will execute a function to contact the registered emergency contact via SMS or email.

[1113] Chat function and lifestyle support information provided

[1114] Choosing a topic and starting a conversation

[1115] Device: A smart speaker can automatically start a conversation with the user based on specific keywords or the time of day. For example, in the morning, it might say, "Today's weather is sunny. It's perfect for a walk."

[1116] Receiving user responses and providing information

[1117] User: The user can freely talk and the smart speaker will respond. For example, if the user asks, "What should I do today?", the smart speaker will suggest, "Yesterday you said you wanted to do the laundry. How about reading today?"

[1118] Tracking medication and dietary status and receiving push notifications

[1119] Scheduling and Notifications

[1120] Server: The server manages the user's medication schedule and meal plan and sends notifications to the user at appropriate times. For example, a notification stating "It's time to take your medicine" is sent to a smart speaker at 12 o'clock every day.

[1121] Emitting notifications and receiving responses

[1122] Device: The smart speaker notifies the user of the notification from the server as a voice message. When the user responds with "I just drank it," the smart speaker sends the information to the server.

[1123] Breaking records

[1124] Server: The server receives the user's response and records the information in a database, allowing it to keep a record of medication and dietary habits and to review and analyze the data as needed.

[1125] Providing administrative and daily information

[1126] Acquiring and outputting information

[1127] Server: The server periodically sends important government notifications and daily information to the smart speaker. For example, at the beginning of each month, it sends "This month's event information" to the smart speaker.

[1128] Terminal: The received information is periodically announced to the user through audio, such as "This month's local events are as follows."

[1129] In this way, the smart speaker system equipped with generative AI of the present invention supports the daily lives of the elderly, reduces feelings of loneliness, and supports health management, thereby reducing the workload of welfare workers.

[1130] The processing flow will be explained below.

[1131] Safety confirmation function

[1132] Step 1:

[1133] The server sends a safety check request to the smart speaker at a fixed time every day. Specifically, the scheduler on the server generates a safety check request at 9:00 a.m. and sends an API request to the smart speaker.

[1134] Step 2:

[1135] The device (smart speaker) receives a request from the server and asks the user aloud, "How are you feeling today?" After receiving the request, it uses a natural language processing (NLP) engine to output a predefined message from the speaker using text-to-speech (TTS).

[1136] Step 3:

[1137] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The voice is input via the microphone in the smart speaker.

[1138] Step 4:

[1139] The device converts the user's voice response from speech to text data. The voice input is converted to text using a Speech-to-Text (STT) engine.

[1140] Step 5:

[1141] The device analyzes the converted text data and sends the results to the server, which then sends the results to the server via an API call.

[1142] Step 6:

[1143] The server analyzes the text data from the user and sends a notification to emergency contacts if an abnormality is detected. Based on the anomaly detection algorithm, an SMS or email will be sent as needed.

[1144] Chat function and lifestyle support information provided

[1145] Step 1:

[1146] The device sets a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler calls the chat function at a specific time.

[1147] Step 2:

[1148] When the device is triggered, it selects a topic from the user's past conversation log or a preset topic list, analyzes the conversation data stored in the storage, and selects new topics and appropriate life support information.

[1149] Step 3:

[1150] The device will start a voice conversation based on the selected topic. For example, it will generate a message like, "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[1151] Step 4:

[1152] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[1153] Step 5:

[1154] The device analyzes the user's voice response and provides appropriate information or continues the conversation. For example, in response to the question, "What should I do today?", the device will make a suggestion such as, "You said you would do the laundry yesterday. How about reading?"

[1155] Tracking medication and dietary status and receiving push notifications

[1156] Step 1:

[1157] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[1158] Step 2:

[1159] The server generates a notification at a set time and sends it to the smart speaker. For example, a push notification request is sent to the smart speaker via API at 12:00 every day.

[1160] Step 3:

[1161] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[1162] Step 4:

[1163] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[1164] Step 5:

[1165] The device converts the voice response into text and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[1166] Step 6:

[1167] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[1168] Example 1

[1169] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1170] To support the lives of elderly people living alone, it is important to regularly check on their safety and record their health status. However, it is difficult for elderly people to do this themselves, and they often feel lonely. Furthermore, managing and notifying these people places an increased burden on welfare workers and their families. To solve these issues, there is a need for a system that automates safety checks and health management, and allows elderly people to access this information on a daily basis.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1172] In this invention, the server includes: means for receiving a request from the server and outputting a question to the user by voice to confirm the user's health condition;

[1173] means for converting a user's voice response from voice to text data;

[1174] means for analyzing the converted text data and transmitting it to a server;

[1175] A server analyzes the text data, and if an abnormality is detected, sends a notification to an emergency contact;

[1176] A means for managing a user's medication schedule and meal plan on a server;

[1177] a means for notifying the user by voice of medication and meal times based on notifications from the server;

[1178] means for receiving a user response and transmitting the response to a server;

[1179] A means for the server to update the record based on the user's behavior history;

[1180] A means for selecting topics for chatting with a user using a generative AI model;

[1181] means for initiating an audio chat based on the selected topic;

[1182] A means for analyzing the user's voice response and providing appropriate information or continuing the conversation;

[1183] A server acquires administrative and daily information and transmits it to the user by voice.

[1184] This will automate the process of checking the safety of elderly people living alone, managing their health, and providing daily information, making it possible to support their daily lives.

[1185] A "server" is a computer system that sends, receives, and processes data over a network.

[1186] A "request" is a request from a server to a terminal for a specific operation or provision of information.

[1187] "User" refers to a person who uses this system, such as an elderly person living alone.

[1188] "Audio" refers to a signal that converts sound waveforms into digital data for processing.

[1189] "Text data" is data that is generated by analyzing speech and expressing it as a string of characters.

[1190] "Abnormal" refers to a state or problem that is different from normal in the user's health condition, behavior, etc.

[1191] "Emergency contact" refers to the person or organization to be notified when an abnormality is detected.

[1192] A "medication schedule" is a plan that governs when a user should take which medications.

[1193] A "meal plan" is a plan that manages the times and contents of meals that a user eats each day.

[1194] "Behavioral history" is recorded data such as a user's past actions and responses.

[1195] A "generative AI model" is an artificial intelligence algorithm that generates new information based on data.

[1196] A "prompt sentence" is an instruction or input sentence that a generative AI model uses to generate new information.

[1197] "Chat" refers to everyday conversations with users, and is communication without a specific purpose.

[1198] "Administrative information" refers to important notices and information provided by the government or local governments.

[1199] "Daily information" refers to information related to the user's daily life.

[1200] The present invention relates to a smart speaker system equipped with AI generation to support the daily lives of elderly people living alone. The specific configuration and operation are described below.

[1201] Safety confirmation function

[1202] Server Roles

[1203] The server sends a request to check the safety of elderly single people to the smart speaker at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 a.m. This is done using the Python scheduler library. The server generates an API request and sends it to the smart speaker to start checking the user's safety.

[1204] The role of the device (smart speaker)

[1205] When the smart speaker receives a request from the server, it asks the user a question by voice, such as, "How are you feeling today?" The NLP engine of Amazon Alexa or Google Assistant is used to output this question by voice.

[1206] User Roles

[1207] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. Amazon Transcribe and Google Speech-to-Text are used for the conversion. The converted text data is sent to the server.

[1208] Server analysis and notifications

[1209] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email.

[1210] Chat function and lifestyle support information provided

[1211] Select a topic

[1212] The smart speaker uses a generative AI model (such as OpenAI GPT-3) to select topics for conversation with the user based on specific keywords and the time of day. For example, in the morning, a topic such as "Today's weather is sunny. It's perfect for a walk" is provided.

[1213] Keeping the conversation going

[1214] If a user asks, "What should I do today?", the smart speaker can suggest, "Yesterday you said you wanted to do the laundry. How about reading today?" The conversation continues naturally based on generative AI models.

[1215] Tracking medication and dietary status and receiving push notifications

[1216] Schedule management

[1217] The server manages the user's medication schedule and meal plan, for example, by using the Google Calendar API to send a notification to the smart speaker every day at 12 o'clock saying, "It's time to take your medicine."

[1218] Receiving responses and updating records

[1219] If the user responds, "I just took it," the information is sent to the server via the smart speaker. The server records this response and updates the database based on the user's behavioral history, allowing for analysis of future medication and meal plans.

[1220] Providing administrative and daily information

[1221] Acquisition and provision of information

[1222] The server periodically obtains administrative and daily information and provides it to the user via voice. For example, at the beginning of each month, it obtains "This month's local events" and communicates them to the user via a smart speaker. This information is obtained using various APIs and web scraping technologies.

[1223] Providing concrete examples and prompts

[1224] Example 1: Safety confirmation

[1225] The server sends a safety check request at 9:00 AM, and the smart speaker outputs the question, "How are you feeling today?" The user responds, "I'm fine," and this is sent to the server. An example of the prompt sentence in this case is as follows:

[1226] "Generate a morning health check prompt for the elderly: 'How are you feeling today?'"

[1227] Example 2: Medication notification

[1228] The server sends a notification to the smart speaker every day at 12 o'clock stating "It's time to take your medicine," and the user responds "I just took it."

[1229] "Generate a daily medication reminder for the elderly: 'It's time to take your medication.'"

[1230] In this way, the smart speaker system equipped with generative AI of the present invention is expected to support the daily lives of the elderly, reduce feelings of loneliness, and support health management, thereby reducing the burden on welfare staff and their families.

[1231] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1232] Safety confirmation function

[1233] Step 1: Server sends request

[1234] The server sends a safety confirmation request to the smart speaker every day at 9:00 AM. This process is triggered by the server's scheduler and uses the Python scheduler library. After the request is generated, it is sent to the smart speaker as an HTTP POST request. The input is the schedule setting and the timing of the request transmission, and the output is the request transmission to the smart speaker.

[1235] Specific operation: The server's scheduler detects that it is 9:00 AM, generates an API request, and sends it to the smart speaker.

[1236] Step 2: Output voice questions on the device

[1237] When the device (smart speaker) receives a request from the server, it outputs the question "How are you feeling today?" This process is performed by converting text to speech using the NLP engine of Amazon Alexa or Google Assistant. The input is the request received from the server, and the output is the spoken question to the user.

[1238] Specific operation: After the smart speaker receives the request, it will output the voice message, "How are you feeling today?"

[1239] Step 3: User's voice response

[1240] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. This process uses Amazon Transcribe and Google Speech-to-Text. The input is the user's voice response, and the output is text data.

[1241] Specific operation: The user responds "I'm fine," and the voice is converted into text data.

[1242] Step 4: Sending text data to the server

[1243] The smart speaker analyzes the converted text data and sends the results to the server. The input is the converted text data, and the output is sending the text data to the server.

[1244] Specific operation: Text data is sent to the server.

[1245] Step 5: Server analysis and notification

[1246] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email. The input is the user's text data and the output is a notification to the emergency contact.

[1247] Specific operation: The server detects an anomaly and sends an SMS using the Twilio API.

[1248] Chat function and lifestyle support information provided

[1249] Step 1: Select a topic

[1250] The server uses a generative AI model (OpenAI GPT-3) to select topics for chat with the user based on specific keywords and time of day. The input is the time of day and keywords, and the output is the generated topic.

[1251] Specific behavior: In the morning, generate topics such as "Good morning. Today's weather is sunny. It's perfect for a walk."

[1252] Step 2: Start chatting by voice

[1253] The device will start a voice chat based on the selected topic. The input is the generated topic, and the output is the start of a conversation with the user.

[1254] Specific behavior: The smart speaker will speak to the user based on the selected topic.

[1255] Step 3: User voice response and information provision

[1256] The user can freely converse, and the smart speaker responds. A generative AI model is used to provide appropriate information and continue the conversation. The input is the user's voice response, and the output is a response based on the generative AI model.

[1257] What it does: When a user asks, "What should I do today?" the smart speaker suggests, "Yesterday you said you wanted to do laundry. How about reading today?"

[1258] Tracking medication and dietary status and receiving push notifications

[1259] Step 1: Schedule Management

[1260] The server manages the user's medication schedule and meal plan. The input is the user's medication and meal schedule, and the output is the managed schedule information.

[1261] Specific operation: The server manages the schedule using the Google Calendar API.

[1262] Step 2: Sending notifications

[1263] The server sends notifications to the smart speaker based on medication and meal times. The input is schedule information and the output is notifications to the smart speaker.

[1264] Specific operation: Every day at 12 o'clock, a notification stating "It's time to take your medicine" is sent to the smart speaker.

[1265] Step 3: User responses and recording

[1266] The user responds to the smart speaker, and the response is sent to the server. The input is the user's response, and the output is the record update.

[1267] Specific operation: The user responds with "I just drank it," and the information is sent to the server and the record is updated.

[1268] Providing administrative and daily information

[1269] Step 1: Get the information

[1270] The server periodically receives important notices and daily information from the government. The input is a request to obtain government information, and the output is the obtained information.

[1271] Specific operation: The server obtains information using APIs or web scraping technology.

[1272] Step 2: Provide information

[1273] The acquired administrative information and everyday information is communicated to the user via a smart speaker. The input is the acquired information, and the output is the information provided to the user.

[1274] Specific operation: At the beginning of the month, play something like "This month's local events are as follows."

[1275] Through the above processing steps, the system enables safety confirmation for elderly single people, health management, chat support, medication and diet management, and the provision of government information.

[1276] (Application example 1)

[1277] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1278] The lives of elderly people living alone are fraught with many physical and psychological anxieties. In particular, there are problems with the lack of means to respond quickly in emergencies and the difficulty of managing their health and confirming their safety in their daily lives. Security issues also remain a concern. A system that can resolve these issues and improve the quality of life for elderly people living alone is needed.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1280] In this invention, the server includes means for receiving an inquiry from an information processing device to check the safety of the user and outputting a voice inquiry to the user to check their health condition, means for converting the user's voice response from voice to symbol data, means for analyzing the converted symbol data and sending it to the information processing device, means for the information processing device to analyze the symbol data and send a notification to an emergency contact if an abnormality is detected, means for monitoring the user's living environment using voice recognition technology, means for acquiring sensor information and sending a voice notification if an abnormality is detected, and means for installing an emergency button and sending a notification to an emergency contact when the button is used. This enables safety and health management for elderly people living alone and rapid response in emergencies.

[1281] "Safety confirmation" is a procedure to check the health and safety of elderly people living alone.

[1282] An "information processing device" is an electronic device for inputting, processing, and outputting data.

[1283] An "inquiry" is a question or request made to obtain some information.

[1284] "User" refers to a person who uses a system or device.

[1285] "Health status" means the physical and mental state of an individual.

[1286] "Audio output" is the process of generating sound and outputting it from a speaker or the like.

[1287] "Symbolic data" is a series of symbols that represent information such as speech or text.

[1288] "Analysis" is the process of analyzing given data or information and understanding its meaning and structure.

[1289] "Emergency contacts" are pre-defined contacts to be contacted in case of an emergency.

[1290] A "notification" is a message or signal that informs others of specific information.

[1291] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[1292] "Living environment" refers to the entire environment in which an individual spends their daily life.

[1293] "Sensor information" refers to data and measurements collected by sensors.

[1294] "Abnormal" refers to a state or phenomenon that is different from the normal.

[1295] An "emergency button" is a button that can be pressed in an emergency to trigger a specified action.

[1296] In today's society where many elderly people live alone, safety and health management are extremely important issues. This invention provides a system that supports the lives of elderly people by using a smart speaker equipped with a generative AI model.

[1297] To implement the invention, the following components are required:

[1298] 1. Safety confirmation function

[1299] The server sends a request to check the user's safety at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 AM, at which time it generates an API request and sends it to the smart speaker.

[1300] When the device (smart speaker) receives a request from the server, it outputs a voice message to the user asking, "Hello. How are you feeling today?" Google Text-to-Speech (gTTS) is used for voice synthesis.

[1301] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response through its microphone, converts the voice into text data using the SpeechRecognition library, and sends it to the server.

[1302] The server analyzes the received text data and, if an abnormality is detected, sends an email or SMS to an emergency contact.

[1303] 2.Security monitoring function

[1304] The information processing device (server) receives data from sensors attached to doors and windows. If an abnormality is detected, it notifies the user by voice, saying, "The front door has opened. Is it safe?"

[1305] If the user does not respond or responds negatively, the server sends a notification to the emergency contact.

[1306] 3. Emergency response function

[1307] The device (smart speaker) is equipped with an emergency button that the user can press in the event of an emergency. When the emergency button is pressed, a notification is immediately sent to the server, which then immediately sends a notification to pre-registered emergency contacts.

[1308] Examples of implementations of these features include:

[1309] Scenario: At 9am, a smart speaker asks, "Hello. How are you feeling today?" If the user does not reply "I'm fine," it automatically notifies emergency contacts that "No response."

[1310] Example prompt: "I want to create a voice interface to check the safety of elderly people living alone. I want it to check their safety every morning and notify emergency contacts if there is an abnormality. Please give me some concrete code examples."

[1311] This will ensure the safety and health management of elderly people living alone, as well as quick response in emergencies, and improve their quality of life.

[1312] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1313] Step 1:

[1314] Server: The server sends a request to check the user's safety at a fixed time every day. The server's scheduler is set to be triggered at 9:00 AM. When the scheduler reaches the specified time, it generates an API request and sends it to the smart speaker. The input is the scheduler's trigger time, and the output is the API request.

[1315] Step 2:

[1316] Device (Smart Speaker): Upon receiving a request from the server, the smart speaker outputs a voice message to the user asking, "Hello, how are you feeling today?" In this process, it uses Google Text-to-Speech (gTTS) to convert the text message into speech. The input is the API request from the server, and the output is the voice message.

[1317] Step 3:

[1318] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The input is the voice question from the smart speaker, and the output is the user's voice response.

[1319] Step 4:

[1320] Device (smart speaker): The smart speaker receives the user's voice response through the microphone and converts the voice into text data using the SpeechRecognition library. The converted text data is sent to the server. The input is the user's voice response, and the output is text data.

[1321] Step 5:

[1322] Server: The server analyzes the received text data and sends an email or SMS to the emergency contact if an abnormality is detected. The analysis includes the user's response and determines whether there is an abnormality. The input is text data, and the output is a notification to the emergency contact.

[1323] Step 6:

[1324] Server: The server periodically receives data from the sensors. If the sensors attached to the doors or windows detect an abnormality, the server acquires and analyzes the data. The input is the data from the sensors, and the output is the analysis result on whether or not there is an abnormality.

[1325] Step 7:

[1326] Terminal (smart speaker): If an abnormality is detected based on sensor information, the smart speaker will notify the user by voice, saying, "The front door has opened. Is it safe?" The input is the abnormality notification from the sensor, and the output is the voice notification.

[1327] Step 8:

[1328] User: The user responds verbally, saying "Yes, I'm safe" or "No, I need help." The input is the voice notification from the smart speaker, and the output is the user's voice response.

[1329] Step 9:

[1330] Device (smart speaker): Receives the user's response again, converts the speech into text using the SpeechRecognition library, and analyzes it. The input is the user's voice response, and the output is text data.

[1331] Step 10:

[1332] Server: Analyzes the user's voice response and sends a notification to emergency contacts if an abnormality is detected. The input is text data, and the output is a notification to emergency contacts.

[1333] Step 11:

[1334] Terminal (Smart Speaker): The smart speaker is equipped with an emergency button. When a user presses the emergency button, a notification is sent to the server immediately. The input is the trigger of the emergency button, and the output is a notification to the server.

[1335] Step 12:

[1336] Server: When the server receives a notification that the emergency button has been pressed, it immediately sends a notification to the pre-registered emergency contacts. The input is a notification that the emergency button has been pressed, and the output is a notification to the emergency contacts.

[1337] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1338] This invention relates to a smart speaker system equipped with a generative AI to support the daily lives of elderly people living alone, and in particular, to a system that combines an emotion engine to analyze the user's emotions and respond appropriately. Specific program processing and operation are described below.

[1339] Safety confirmation function and emotion recognition function

[1340] Sending requests and outputting voice questions

[1341] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, when the server's scheduler is triggered at 9:00 a.m., it generates an API request and sends it to the smart speaker.

[1342] Device (smart speaker): When the smart speaker receives the request, it will ask the user aloud the question "How are you feeling today?", using a natural language processing (NLP) engine to play a predefined message via text-to-speech (TTS).

[1343] Receiving, converting, and parsing user responses

[1344] User: The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone inside the smart speaker picks up the voice.

[1345] Device: The smart speaker converts the user's voice response from speech to text using a STT (Speech-to-Text) engine. The converted text data is analyzed and the results are sent to the server.

[1346] Emotion Recognition and Anomaly Detection

[1347] Emotion engine: Smart speakers are equipped with an emotion engine that recognizes emotions from the user's voice. The emotion engine analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state (e.g., sadness, joy, anger, etc.).

[1348] Server: The server analyzes the received text data and emotion data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and the emotion recognition detects "sad," it sends a notification to the emergency contact.

[1349] Chat and emotion recognition

[1350] Topic selection and conversation start

[1351] Device: The smart speaker automatically starts chatting with the user based on specific keywords or the time of day. A timer or scheduler is set to trigger the chat function at a specific time. It provides topics such as, "Today's weather is sunny. It's perfect for a walk."

[1352] Receiving, analyzing, and providing information about user responses

[1353] User: The user can freely talk and the smart speaker will receive the response, for example, "What should I do today?"

[1354] Device: The smart speaker analyzes the user's voice response, and the emotion engine recognizes the user's emotions. If the user asks a question in a "sad voice," it will provide an appropriate encouraging message. For example, it will suggest, "You said you would do the laundry yesterday. How about reading a book?"

[1355] Tracking medication and dietary status and receiving push notifications

[1356] Scheduling and Notifications

[1357] Server: The server manages the user's medication schedule and meal plan and sends appropriate notifications to the user at specified times. For example, it generates a notification saying "It's time to take your medicine" at 12 o'clock every day and sends it to a smart speaker.

[1358] Device: The smart speaker conveys notifications from the server to the user as voice messages.

[1359] User responses and record updates

[1360] User: The user responds by saying "I just drank it." This speech is received by the smart speaker.

[1361] Device: The smart speaker converts voice into text, analyzes emotions, and sends it to the server.

[1362] Server: The server updates the user's behavioral history based on the received response data and emotion data.

[1363] Providing administrative and daily information

[1364] Acquisition and provision of information

[1365] Server: The server periodically sends important government notifications and useful information to the smart speaker. For example, it provides "This month's local event information" at the beginning of each month.

[1366] Device: The smart speaker will then relay the information to the user aloud, playing something like, "This month's events are as follows."

[1367] By combining this generative AI-equipped smart speaker system with an emotion engine, it is possible to provide more advanced user follow-up and personalized support. As a result, it is expected to significantly contribute to improving the quality of life of the elderly and reducing their sense of loneliness. It will also significantly reduce the workload of welfare workers.

[1368] The processing flow will be explained below.

[1369] Safety confirmation function and emotion recognition function

[1370] Step 1:

[1371] The server sends a safety check request to the smart speaker at a fixed time every day. The server's scheduler is triggered at 9:00 AM, generates a safety check request, and sends it to the smart speaker as an API request.

[1372] Step 2:

[1373] The device (smart speaker) receives a request from the server and outputs the question "How are you feeling today?" to the user by voice. A natural language processing (NLP) engine is used to play back a predefined message using text-to-speech (TTS).

[1374] Step 3:

[1375] The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone in the smart speaker picks up the user's voice.

[1376] Step 4:

[1377] The device uses a Speech-to-Text (STT) engine to convert the user's voice response from speech to text, then analyzes the converted text data and sends it to the server.

[1378] Step 5:

[1379] The device uses an emotion engine to recognize emotions from the user's responses. It analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state.

[1380] Step 6:

[1381] The server analyzes the received text data and emotional data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and emotion recognition detects "sad," it sends a notification to emergency contacts.

[1382] Chat and emotion recognition

[1383] Step 1:

[1384] The device will set a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler will call the chat function at a specific time.

[1385] Step 2:

[1386] The device selects topics from the user's past conversation logs or a preset topic list, analyzes the conversation data stored in storage, and selects new topics and appropriate lifestyle support information.

[1387] Step 3:

[1388] The device will start a voice chat based on the selected topic. It will generate the message "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[1389] Step 4:

[1390] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[1391] Step 5:

[1392] The device analyzes the user's voice response, and the emotion engine recognizes the user's emotions. For example, if the user asks a question in a sad voice, the device generates an appropriate encouraging message based on that emotional data, suggesting, "You said you would do the laundry yesterday. How about reading a book?"

[1393] Tracking medication and dietary status and receiving push notifications

[1394] Step 1:

[1395] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[1396] Step 2:

[1397] The server generates a notification at a set time and sends it to the smart speaker. A push notification request is sent to the smart speaker via API at 12:00 every day.

[1398] Step 3:

[1399] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[1400] Step 4:

[1401] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[1402] Step 5:

[1403] The device converts the voice response into text, analyzes emotions, and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[1404] Step 6:

[1405] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[1406] Example 2

[1407] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1408] In modern society, the increase in the number of elderly people living alone is becoming a major social problem. When elderly people living alone at home do not adequately manage their health or communicate with others on a daily basis, they experience feelings of loneliness and health risks. Furthermore, conventional healthcare systems have difficulty grasping the user's emotional state, which can prevent them from providing appropriate support. Furthermore, elderly people tend to neglect managing their medication and diet, creating a need for a means to properly manage their behavioral history. It is necessary to develop a system that can solve these issues and improve the quality of life for the elderly.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1410] In this invention, the server includes means for receiving a request from the server for safety confirmation and outputting a question to the user by voice to confirm the user's health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for identifying emotions from the user's voice using an emotion recognition engine, means for sending the identified emotion data to the server, and means for the server to evaluate the user's condition based on the emotion data and the text data and send a notification to an emergency contact if an abnormality is detected. This makes it possible to grasp not only the user's health condition but also their emotional state and to detect abnormalities early.

[1411] "Safety check" is the act of checking the health and safety of residents.

[1412] A "server" is a computer system for storing, processing, and communicating data.

[1413] A "means for receiving a request" is a mechanism for receiving requests from the outside.

[1414] The "means for outputting questions by voice" refers to a device or system that uses voice synthesis technology to reproduce text information as voice.

[1415] A "means for converting a voice response from voice to text data" is a device or system that uses voice recognition technology to convert a voice signal into a corresponding text format.

[1416] The "means for analyzing the converted text data and transmitting it to the server" refers to the process for understanding and evaluating the text data and the communication means for transmitting that data to the server.

[1417] An "emotion recognition engine" is an algorithm or software that analyzes tone, pitch, speed, etc. from audio data to identify emotional states.

[1418] The "means for transmitting emotion data to the server" refers to a communication means for transferring the data obtained by emotion recognition to the server.

[1419] "Means for sending a notification to an emergency contact when an abnormality is detected" refers to a mechanism for sending a notification to a registered emergency contact when the system detects an abnormality in the user's behavior.

[1420] A "topic selection means" is a system or device that determines the topic of conversation with a user based on a pre-set algorithm.

[1421] The "means for starting a voice chat" is a system or device for starting a conversation based on a selected topic using voice synthesis technology.

[1422] The "means for providing information or continuing a conversation" refers to a system or device for returning appropriate information in response to a user's response or for continuing a conversation.

[1423] The "schedule management means in the server for managing medication schedules and meal plans" is a system for managing the times when a user takes medicine and the times when they eat, and saving these as a schedule on the server.

[1424] The "means for notifying the user of medication and meal times by voice" refers to a system or device that uses voice synthesis technology to notify the user of medication and meal times by voice.

[1425] The "means for receiving a response from the user and transmitting the response to the server" refers to a communication means for taking in a reply from the user and transmitting it as data to the server.

[1426] A "means for updating behavioral history" is a system or device that records data about a user's behavior and modifies or adds to that history information based on new data.

[1427] This invention relates to a smart speaker system that utilizes generative AI technology to support the daily lives of elderly people living alone. Specifically, it combines an emotion recognition engine to analyze the user's emotions and respond appropriately. The details are described below.

[1428] System Configuration

[1429] This system consists of a server, a smart speaker (hereafter referred to as the "terminal"), and an elderly person. The server stores, processes, and communicates data, while the terminal is a device that enables dialogue with the user and voice responses.

[1430] Hardware and software used

[1431] Server: The server is used to perform functions such as safety confirmation, data analysis, anomaly detection, emergency notification, etc. Examples include Linux-based servers and cloud services (AWS, Google Cloud, etc.).

[1432] Device (smart speaker): Equipped with a natural language processing (NLP) engine, text-to-speech (TTS), speech recognition (STT), and emotion recognition engine. Specifically, it uses Google Dialogflow, Google's STT engine, IBM Watson's Tone Analyzer, etc.

[1433] Operation overview

[1434] 1. Safety confirmation

[1435] Every morning at 9:00, the server uses a scheduler to send a safety confirmation request to the device. The device receives this request and asks the user verbally, "How are you feeling today?" The user's response is received by the device's microphone, and a speech recognition engine converts it into text. The converted text data is also sent to an emotion recognition engine, and is sent to the server along with emotion data. The server analyzes this data and sends a notification to emergency contacts as necessary.

[1436] 2. Chat function

[1437] The device will begin chatting with the user based on a specific time or trigger keyword. A topic selection algorithm will provide a topic through voice, such as, "Today's weather is sunny. Perfect for a walk." Once the user's response is received, the voice recognition engine again converts it into text data, and the emotion recognition engine identifies emotions. By providing appropriate information and continuing the conversation, the device can alleviate the user's sense of loneliness.

[1438] 3. Understanding medication and dietary status

[1439] The server manages the user's medication schedule and meal plan, and sends voice notifications through the device at designated times. An example message is "It's time to take your medicine." The user's responses are received by the device, converted into text, and sentiment analysis is performed before being sent to the server. The server updates the user's behavioral history based on this data.

[1440] 4. Providing administrative and daily information

[1441] The server periodically sends important notifications from the government and other useful information to the device. For example, the device will provide information about local events this month. The device will then communicate this information to the user via voice.

[1442] Examples of specific examples and prompts

[1443] Example 1: Safety confirmation

[1444] The server generates an API request at 9:00 a.m. and sends a safety confirmation message to the smart speaker.

[1445] The device asks the user aloud, "How are you feeling today?"

[1446] Example prompt: "Generate an API request to check the health of an elderly person at 9 AM."

[1447] Example 2: Emotion recognition and anomaly detection

[1448] The server analyzes the user's text data and emotional data, and if the user replies "I'm not feeling well" and the emotion is determined to be "sad," it sends a notification to the emergency contact.

[1449] Example prompt: "Design a system that analyzes a user's emotional data and physical responses, and notifies emergency contacts if there are any abnormalities."

[1450] Example 3: Chat function

[1451] At 2 p.m., the device triggers the chat function and offers the topic, "Today's weather is sunny. Perfect for a walk."

[1452] Example prompt: "Design a system that triggers a chat feature at a specific time and provides topics about today's weather."

[1453] By combining this generative AI-equipped smart speaker system with an emotion engine, it will improve the quality of life of the elderly and reduce their sense of loneliness. It will also reduce the workload of welfare workers and enable more personalized support.

[1454] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1455] Step 1:

[1456] Server sends safety confirmation request

[1457] The server uses a scheduler to send a safety confirmation request to the device every morning at 9:00. In this process, when a trigger is received from the scheduler, an API request is generated and sent to the smart speaker.

[1458] Input: Trigger event from scheduler.

[1459] Processing: Generating and sending API requests.

[1460] Output: API request sent successfully.

[1461] Action: scheduler.trigger('9:00', sendHealthCheckRequest) is called, a request is generated and sent to the smart speaker.

[1462] Step 2:

[1463] Voice question output by terminal

[1464] The device (smart speaker) receives the request from the server and outputs the question "How are you feeling today?" via Google Dialogflow.

[1465] Input: The API request from the server.

[1466] Processing: Converts text to speech using a Text-to-Speech (TTS) engine.

[1467] Output: A spoken question to the user.

[1468] Behavior: The askHealthStatus() function is called, and TTS.synthesize("How are you feeling today?") is executed.

[1469] Step 3:

[1470] Receiving a user's voice response

[1471] The user responds verbally, and the smart speaker's microphone captures the audio, with examples of responses like "I'm fine" or "I'm not feeling well."

[1472] Input: The user's spoken response.

[1473] Processing: Receiving audio.

[1474] Output: Received audio data.

[1475] Operation: The microphone is always in standby mode, and microphone.capture() captures audio data.

[1476] Step 4:

[1477] Device-based speech-to-text conversion

[1478] Smart speakers use Google's Speech-to-Text (STT) engine to convert the user's voice into text data.

[1479] Input: Received audio data.

[1480] Processing: Converts audio data into text data using the STT engine.

[1481] Output: The converted text data.

[1482] What happens: The convertSpeechToText(audioData) function is called to convert the speech to text.

[1483] Step 5:

[1484] Device-based emotion recognition

[1485] The text data is sent to an emotion recognition engine (e.g., IBM Watson's Tone Analyzer) to analyze the user's emotions.

[1486] Input: The converted text data.

[1487] Processing: Uses an emotion recognition engine to analyze tone, pitch, speed, etc. to identify emotions.

[1488] Output: Emotion data.

[1489] Operation: The analyzeEmotion(textData) function is called to generate emotion data.

[1490] Step 6:

[1491] Analysis of text data and emotion data by the server

[1492] The server receives and analyzes the text data and emotional data sent from the smart speaker.

[1493] Input: Text data and emotion data.

[1494] Processing: Based on the results of the data analysis, the user's health status is evaluated.

[1495] Output: User state assessment results.

[1496] Operation: The evaluateUserStatus(textData, emotionData) function is called to determine whether there are any abnormalities.

[1497] Step 7:

[1498] Server-based anomaly detection and notification to emergency contacts

[1499] If the server detects an abnormality, it will send a notification via email or SMS to the designated emergency contact.

[1500] Input: User's condition assessment results.

[1501] Processing: Generating and sending notifications when an anomaly is detected.

[1502] Output: Notification sent to emergency contacts.

[1503] Action: The sendEmergencyNotification(contactInfo, userStatus) function is called to send the emergency notification.

[1504] (Application example 2)

[1505] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1506] In the daily lives of elderly people living alone, there is a need to grasp changes in their physical condition and emotions in real time, and to respond appropriately and contact them in emergencies. In particular, a system that can respond quickly when unexpected physical or mental illness occurs is needed. Another challenge is to provide a system that can manage medication schedules and meal plans, as well as provide encouragement and suggestions according to the user's emotional state.

[1507] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1508] In this invention, the server includes means for receiving a request to check safety and outputting a question to the user by voice to check their health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for the server to analyze the text data and send a notification to an emergency contact if an abnormality is detected, emotion analysis means for recognizing emotions from the user's voice, and means for evaluating the state of the malfunction based on the analyzed emotion data and sending an additional notification to the emergency contact as necessary. This makes it possible to grasp and respond to not only the user's physical condition but also their emotional state in real time.

[1509] "Safety check" is an action taken to periodically check the health status and safety of a user.

[1510] A "server" is a computer system for sending, receiving, and processing data.

[1511] A "request" is a message from a server requesting specific information or an operation.

[1512] "Users" refer to people such as elderly people who use this system.

[1513] "Health status confirmation questions" are questions asked to find out the user's physical condition and mood.

[1514] "Audio output means" refers to a technology that converts text information into audio and conveys it to the user.

[1515] "Means for converting voice into text data" refers to a technology that converts voice information into text information using voice recognition technology.

[1516] "Means for analyzing text data" refers to technology for analyzing converted character information.

[1517] "Emotion analysis means" is a technology that estimates a user's emotions from voice and text information.

[1518] "Emergency contacts" are people or organizations to whom notifications should be sent in the event of an emergency.

[1519] The "means for evaluating a user's state of failure based on emotional data" is a technology that analyzes the user's emotional information and evaluates the user's current state.

[1520] An "additional notification" is a further message sent in addition to a notification already sent.

[1521] The "means for selecting a topic for chatting" is a technique for selecting a topic for conversation with a user.

[1522] The "means for starting a voice chat" is a technique for starting a voice conversation based on a selected topic.

[1523] "Means for continuing a conversation" refers to a technique for continuing a conversation depending on the user's reaction and situation.

[1524] A "medication schedule" is a plan that manages the types and times of medication a user takes.

[1525] A "meal plan" is a plan that manages the menu and times of meals that a user will eat.

[1526] "Schedule management tools" are technologies for recording and managing medication and meal plans.

[1527] "A means for notifying the user of medication and meal times by voice" refers to a technology that notifies the user of scheduled medication and meal times by voice.

[1528] "Behavioral history" is a record of the actions and reactions of a user.

[1529] "Means for updating behavioral history" refers to technology that updates behavioral records based on collected data.

[1530] The present invention relates to a security assistant system for supporting the daily lives of elderly people living alone, and in particular to a system that combines an emotion engine to analyze the user's emotions and take appropriate action. The following describes the processing and operation of the system program.

[1531] System Configuration

[1532] Hardware and Software

[1533] Server: Generates safety confirmation requests, analyzes data, and manages emergency notifications. An API server using Python is expected.

[1534] Terminal (smart speaker, smartphone, etc.): A device that acts as an interface with the user and is equipped with speech recognition (Speech-to-Text), speech synthesis (Text-to-Speech), emotion analysis engine (EmotionRecognizer), etc.

[1535] Emotion analysis engine: Extracts emotions from the user's voice and provides appropriate responses. The library used is expected to be EmotionRecognizer.

[1536] System Operation

[1537] Safety confirmation function

[1538] The server sends a safety confirmation request to the device at a fixed time every day. Triggered via an API, the question "How are you feeling today?" is output to the smart speaker using text-to-speech (TTS).

[1539] When the user responds by saying something like "I'm fine" or "I'm not feeling well," the device converts the voice into text data using a speech-to-text (STT) engine.

[1540] The converted text data is converted into emotion data by an emotion analysis engine, and then the text data and emotion data are sent to the server.

[1541] The server analyzes the transmitted data and sends a notification to emergency contacts if an abnormality is detected.

[1542] Chat function

[1543] Based on a request from the server, the device selects a specific topic and starts chatting with the user. A timer or scheduler is set to trigger the chat function at a specific time.

[1544] Based on the selected topic, a message such as "Today's weather is sunny. It's perfect for a walk" is output in voice.

[1545] When a user asks, "What should I do today?", the device converts the voice to text and the emotion engine analyzes the emotion. If the user asks the question in a "sad voice," it will provide an appropriate encouraging message.

[1546] Understanding medication and dietary status

[1547] The server manages the user's medication schedule and meal plan, and generates notifications such as "It's time to take your medicine" at specified times and sends them to the terminal.

[1548] When the user responds verbally by saying "I just drank it," the response is received by the device, and the data converted into text is analyzed by the emotion analysis engine.

[1549] The converted and analyzed data is sent to the server and the behavioral history is updated.

[1550] Examples of specific examples and prompts

[1551] Specific examples

[1552] If a user responds, "I'm feeling unwell today," the emotion engine will recognize it as "anxiety," and the server will send a notification to emergency contacts. Also, upon hearing that the user is feeling unwell, the device will offer an encouraging message, asking, "Are you feeling stressed or anxious? Is there anything we can do to help?"

[1553] Prompt Sentence Examples

[1554] "Build a model that recognizes emotions from a user's speech. Given the following spoken text: 'I'm feeling bad today', recognize the emotion as 'anxiety'."

[1555] This system will enable users to respond quickly and appropriately when they feel unwell or emotionally unwell, which is expected to improve their quality of life. It will also make it easier to ensure the safety of elderly people by providing appropriate advice and emergency responses according to the user's situation.

[1556] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1557] Step 1:

[1558] The server generates a safety confirmation request at a fixed time every day and sends it to the device via API. At this time, the server's scheduler is triggered and a message is generated asking, "How are you feeling today?"

[1559] Input: Server scheduler trigger

[1560] Output: Generate and send a safety confirmation request

[1561] How it works: The server uses its internal clock to generate a request at the specified time and send it to the device.

[1562] Step 2:

[1563] Based on the received request, the device outputs a voice question to the user, such as "How are you feeling today?" This process uses a text-to-speech (TTS) engine.

[1564] Input: Safety confirmation request from the server

[1565] Output: Voice question output

[1566] How it works: The device receives the request and uses its speech synthesis engine to play a predefined message aloud.

[1567] Step 3:

[1568] The user responds by saying "I'm fine" or "I'm not feeling well," and the device's microphone picks up the voice.

[1569] Input: User's voice response

[1570] Output: Microphone audio capture

[1571] How it works: The device's microphone captures the user's voice data and passes it to a speech recognition engine.

[1572] Step 4:

[1573] The device's speech recognition engine (STT) converts the speech into text.

[1574] Input: User's voice data

[1575] Output: Conversion to text data

[1576] How it works: The device's speech recognition engine analyzes the voice data and converts it into corresponding text data.

[1577] Step 5:

[1578] The text data is passed to a sentiment analysis engine, which analyzes the tone, pitch, and rate of the audio data to determine the emotional state.

[1579] Input: Converted text data

[1580] Output: Conversion to emotion data

[1581] How it works: The sentiment analysis engine analyzes emotions based on text data and identifies emotional states.

[1582] Step 6:

[1583] The device transmits the analyzed text data and emotion data to the server.

[1584] Input: Text data and emotion data

[1585] Output: Send data to the server

[1586] Operation: The device sends data to the server using an API.

[1587] Step 7:

[1588] The server analyzes the data and sends a notification to emergency contacts if an abnormality is detected. The notification is triggered when the text data matches "I'm not feeling well" and the emotion matches "anxiety" or "sadness."

[1589] Input: Text data and emotion data

[1590] Output: Send emergency notification

[1591] How it works: The server analyzes the data and if the conditions are met, it will send a notification to emergency contacts via email or SMS.

[1592] Step 8:

[1593] Based on a request from the server, the device selects a specific topic and begins chatting by voice.

[1594] Input: Chat request from server

[1595] Output: Start voice chat

[1596] How it works: The device's topic selection engine determines the topic and outputs it aloud using a speech synthesis engine.

[1597] Step 9:

[1598] The user responds to the chat, their voice is converted into text, and sentiment analysis is performed.

[1599] Input: User's voice response (chat)

[1600] Output: Generate text data and emotion data

[1601] How it works: The device converts the user's speech into text, and an emotion analysis engine analyzes the emotion.

[1602] Step 10:

[1603] The server manages the user's medication schedule and meal plan and sends notifications to the device at specified times.

[1604] Input: Medication schedule and meal plan

[1605] Output: Generate and send medication and meal notifications

[1606] How it works: The server uses an internal schedule management system to send notifications to the device at the specified time.

[1607] Step 11:

[1608] The user responds by saying something like "I just drank it," and the device converts the speech into text for sentiment analysis, which is then sent to the server.

[1609] Input: User's voice response (medication, food)

[1610] Output: Sending text and emotion data

[1611] How it works: The device converts the user's voice data into text, performs sentiment analysis, and sends the data to the server.

[1612] Step 12:

[1613] The server analyzes the received data and updates the records based on the user's behavioral history.

[1614] Input: Text data and emotion data

[1615] Output: Update history

[1616] How it works: The server analyzes the data and updates the behavioral history database.

[1617] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1618] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1619] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1620] [Fourth embodiment]

[1621] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1622] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1623] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1625] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1627] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1628] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1629] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1630] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1631] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1632] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1633] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1634] The present invention relates to a smart speaker system equipped with AI to support the daily lives of elderly people living alone. Specific program processing and operations are described below.

[1635] Safety confirmation function

[1636] Send request

[1637] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, if the server's scheduler is set to be triggered at 9:00 AM, it generates an API request at that time and sends it to the smart speaker.

[1638] Voice question output

[1639] Device (smart speaker): When the smart speaker receives a request from the server, it outputs a voice message to the user asking, "How are you feeling today?" This is achieved by playing back a voice message generated using a natural language processing (NLP) engine after receiving the request.

[1640] Receiving and sending user responses

[1641] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response, converts it into text, analyzes it, and then sends it to the server.

[1642] Anomaly detection and notification

[1643] Server: The server analyzes the received text data and sends a notification to the emergency contact if it determines that the user's condition is abnormal. For example, if the server receives a response saying "I'm not feeling well," it will execute a function to contact the registered emergency contact via SMS or email.

[1644] Chat function and lifestyle support information provided

[1645] Choosing a topic and starting a conversation

[1646] Device: A smart speaker can automatically start a conversation with the user based on specific keywords or the time of day. For example, in the morning, it might say, "Today's weather is sunny. It's perfect for a walk."

[1647] Receiving user responses and providing information

[1648] User: The user can freely talk and the smart speaker will respond. For example, if the user asks, "What should I do today?", the smart speaker will suggest, "Yesterday you said you wanted to do the laundry. How about reading today?"

[1649] Tracking medication and dietary status and receiving push notifications

[1650] Scheduling and Notifications

[1651] Server: The server manages the user's medication schedule and meal plan and sends notifications to the user at appropriate times. For example, a notification stating "It's time to take your medicine" is sent to a smart speaker at 12 o'clock every day.

[1652] Emitting notifications and receiving responses

[1653] Device: The smart speaker notifies the user of the notification from the server as a voice message. When the user responds with "I just drank it," the smart speaker sends the information to the server.

[1654] Breaking records

[1655] Server: The server receives the user's response and records the information in a database, allowing it to keep a record of medication and dietary habits and to review and analyze the data as needed.

[1656] Providing administrative and daily information

[1657] Acquiring and outputting information

[1658] Server: The server periodically sends important government notifications and daily information to the smart speaker. For example, at the beginning of each month, it sends "This month's event information" to the smart speaker.

[1659] Terminal: The received information is periodically announced to the user through audio, such as "This month's local events are as follows."

[1660] In this way, the smart speaker system equipped with generative AI of the present invention supports the daily lives of the elderly, reduces feelings of loneliness, and supports health management, thereby reducing the workload of welfare workers.

[1661] The processing flow will be explained below.

[1662] Safety confirmation function

[1663] Step 1:

[1664] The server sends a safety check request to the smart speaker at a fixed time every day. Specifically, the scheduler on the server generates a safety check request at 9:00 a.m. and sends an API request to the smart speaker.

[1665] Step 2:

[1666] The device (smart speaker) receives a request from the server and asks the user aloud, "How are you feeling today?" After receiving the request, it uses a natural language processing (NLP) engine to output a predefined message from the speaker using text-to-speech (TTS).

[1667] Step 3:

[1668] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The voice is input via the microphone in the smart speaker.

[1669] Step 4:

[1670] The device converts the user's voice response from speech to text data. The voice input is converted to text using a Speech-to-Text (STT) engine.

[1671] Step 5:

[1672] The device analyzes the converted text data and sends the results to the server, which then sends the results to the server via an API call.

[1673] Step 6:

[1674] The server analyzes the text data from the user and sends a notification to emergency contacts if an abnormality is detected. Based on the anomaly detection algorithm, an SMS or email will be sent as needed.

[1675] Chat function and lifestyle support information provided

[1676] Step 1:

[1677] The device sets a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler calls the chat function at a specific time.

[1678] Step 2:

[1679] When the device is triggered, it selects a topic from the user's past conversation log or a preset topic list, analyzes the conversation data stored in the storage, and selects new topics and appropriate life support information.

[1680] Step 3:

[1681] The device will start a voice conversation based on the selected topic. For example, it will generate a message like, "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[1682] Step 4:

[1683] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[1684] Step 5:

[1685] The device analyzes the user's voice response and provides appropriate information or continues the conversation. For example, in response to the question, "What should I do today?", the device will make a suggestion such as, "You said you would do the laundry yesterday. How about reading?"

[1686] Tracking medication and dietary status and receiving push notifications

[1687] Step 1:

[1688] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[1689] Step 2:

[1690] The server generates a notification at a set time and sends it to the smart speaker. For example, a push notification request is sent to the smart speaker via API at 12:00 every day.

[1691] Step 3:

[1692] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[1693] Step 4:

[1694] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[1695] Step 5:

[1696] The device converts the voice response into text and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[1697] Step 6:

[1698] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[1699] Example 1

[1700] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1701] To support the lives of elderly people living alone, it is important to regularly check on their safety and record their health status. However, it is difficult for elderly people to do this themselves, and they often feel lonely. Furthermore, managing and notifying these people places an increased burden on welfare workers and their families. To solve these issues, there is a need for a system that automates safety checks and health management, and allows elderly people to access this information on a daily basis.

[1702] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1703] In this invention, the server includes: means for receiving a request from the server and outputting a question to the user by voice to confirm the user's health condition;

[1704] means for converting a user's voice response from voice to text data;

[1705] means for analyzing the converted text data and transmitting it to a server;

[1706] A server analyzes the text data, and if an abnormality is detected, sends a notification to an emergency contact;

[1707] A means for managing a user's medication schedule and meal plan on a server;

[1708] a means for notifying the user by voice of medication and meal times based on notifications from the server;

[1709] means for receiving a user response and transmitting the response to a server;

[1710] A means for the server to update the record based on the user's behavior history;

[1711] A means for selecting topics for chatting with a user using a generative AI model;

[1712] means for initiating an audio chat based on the selected topic;

[1713] A means for analyzing the user's voice response and providing appropriate information or continuing the conversation;

[1714] A server acquires administrative and daily information and transmits it to the user by voice.

[1715] This will automate the process of checking the safety of elderly people living alone, managing their health, and providing daily information, making it possible to support their daily lives.

[1716] A "server" is a computer system that sends, receives, and processes data over a network.

[1717] A "request" is a request from a server to a terminal for a specific operation or provision of information.

[1718] "User" refers to a person who uses this system, such as an elderly person living alone.

[1719] "Audio" refers to a signal that converts sound waveforms into digital data for processing.

[1720] "Text data" is data that is generated by analyzing speech and expressing it as a string of characters.

[1721] "Abnormal" refers to a state or problem that is different from normal in the user's health condition, behavior, etc.

[1722] "Emergency contact" refers to the person or organization to be notified when an abnormality is detected.

[1723] A "medication schedule" is a plan that governs when a user should take which medications.

[1724] A "meal plan" is a plan that manages the times and contents of meals that a user eats each day.

[1725] "Behavioral history" is recorded data such as a user's past actions and responses.

[1726] A "generative AI model" is an artificial intelligence algorithm that generates new information based on data.

[1727] A "prompt sentence" is an instruction or input sentence that a generative AI model uses to generate new information.

[1728] "Chat" refers to everyday conversations with users, and is communication without a specific purpose.

[1729] "Administrative information" refers to important notices and information provided by the government or local governments.

[1730] "Daily information" refers to information related to the user's daily life.

[1731] The present invention relates to a smart speaker system equipped with AI generation to support the daily lives of elderly people living alone. The specific configuration and operation are described below.

[1732] Safety confirmation function

[1733] Server Roles

[1734] The server sends a request to check the safety of elderly single people to the smart speaker at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 a.m. This is done using the Python scheduler library. The server generates an API request and sends it to the smart speaker to start checking the user's safety.

[1735] The role of the device (smart speaker)

[1736] When the smart speaker receives a request from the server, it asks the user a question by voice, such as, "How are you feeling today?" The NLP engine of Amazon Alexa or Google Assistant is used to output this question by voice.

[1737] User Roles

[1738] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. Amazon Transcribe and Google Speech-to-Text are used for the conversion. The converted text data is sent to the server.

[1739] Server analysis and notifications

[1740] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email.

[1741] Chat function and lifestyle support information provided

[1742] Select a topic

[1743] The smart speaker uses a generative AI model (such as OpenAI GPT-3) to select topics for conversation with the user based on specific keywords and the time of day. For example, in the morning, a topic such as "Today's weather is sunny. It's perfect for a walk" is provided.

[1744] Keeping the conversation going

[1745] If a user asks, "What should I do today?", the smart speaker can suggest, "Yesterday you said you wanted to do the laundry. How about reading today?" The conversation continues naturally based on generative AI models.

[1746] Tracking medication and dietary status and receiving push notifications

[1747] Schedule management

[1748] The server manages the user's medication schedule and meal plan, for example, by using the Google Calendar API to send a notification to the smart speaker every day at 12 o'clock saying, "It's time to take your medicine."

[1749] Receiving responses and updating records

[1750] If the user responds, "I just took it," the information is sent to the server via the smart speaker. The server records this response and updates the database based on the user's behavioral history, allowing for analysis of future medication and meal plans.

[1751] Providing administrative and daily information

[1752] Acquisition and provision of information

[1753] The server periodically obtains administrative and daily information and provides it to the user via voice. For example, at the beginning of each month, it obtains "This month's local events" and communicates them to the user via a smart speaker. This information is obtained using various APIs and web scraping technologies.

[1754] Providing concrete examples and prompts

[1755] Example 1: Safety confirmation

[1756] The server sends a safety check request at 9:00 AM, and the smart speaker outputs the question, "How are you feeling today?" The user responds, "I'm fine," and this is sent to the server. An example of the prompt sentence in this case is as follows:

[1757] "Generate a morning health check prompt for the elderly: 'How are you feeling today?'"

[1758] Example 2: Medication notification

[1759] The server sends a notification to the smart speaker every day at 12 o'clock stating "It's time to take your medicine," and the user responds "I just took it."

[1760] "Generate a daily medication reminder for the elderly: 'It's time to take your medication.'"

[1761] In this way, the smart speaker system equipped with generative AI of the present invention is expected to support the daily lives of the elderly, reduce feelings of loneliness, and support health management, thereby reducing the burden on welfare staff and their families.

[1762] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1763] Safety confirmation function

[1764] Step 1: Server sends request

[1765] The server sends a safety confirmation request to the smart speaker every day at 9:00 AM. This process is triggered by the server's scheduler and uses the Python scheduler library. After the request is generated, it is sent to the smart speaker as an HTTP POST request. The input is the schedule setting and the timing of the request transmission, and the output is the request transmission to the smart speaker.

[1766] Specific operation: The server's scheduler detects that it is 9:00 AM, generates an API request, and sends it to the smart speaker.

[1767] Step 2: Output voice questions on the device

[1768] When the device (smart speaker) receives a request from the server, it outputs the question "How are you feeling today?" This process is performed by converting text to speech using the NLP engine of Amazon Alexa or Google Assistant. The input is the request received from the server, and the output is the spoken question to the user.

[1769] Specific operation: After the smart speaker receives the request, it will output the voice message, "How are you feeling today?"

[1770] Step 3: User's voice response

[1771] The user responds to the smart speaker by saying "I'm fine" or "I'm not feeling well." The smart speaker receives the speech and converts it into text data. This process uses Amazon Transcribe and Google Speech-to-Text. The input is the user's voice response, and the output is text data.

[1772] Specific operation: The user responds "I'm fine," and the voice is converted into text data.

[1773] Step 4: Sending text data to the server

[1774] The smart speaker analyzes the converted text data and sends the results to the server. The input is the converted text data, and the output is sending the text data to the server.

[1775] Specific operation: Text data is sent to the server.

[1776] Step 5: Server analysis and notification

[1777] The server analyzes the received text data and sends a notification to the emergency contact if an abnormality is detected. For example, if a response saying "I'm not feeling well" is received, the server uses the Twilio API to notify the emergency contact via SMS or email. The input is the user's text data and the output is a notification to the emergency contact.

[1778] Specific operation: The server detects an anomaly and sends an SMS using the Twilio API.

[1779] Chat function and lifestyle support information provided

[1780] Step 1: Select a topic

[1781] The server uses a generative AI model (OpenAI GPT-3) to select topics for chat with the user based on specific keywords and time of day. The input is the time of day and keywords, and the output is the generated topic.

[1782] Specific behavior: In the morning, generate topics such as "Good morning. Today's weather is sunny. It's perfect for a walk."

[1783] Step 2: Start chatting by voice

[1784] The device will start a voice chat based on the selected topic. The input is the generated topic, and the output is the start of a conversation with the user.

[1785] Specific behavior: The smart speaker will speak to the user based on the selected topic.

[1786] Step 3: User voice response and information provision

[1787] The user can freely converse, and the smart speaker responds. A generative AI model is used to provide appropriate information and continue the conversation. The input is the user's voice response, and the output is a response based on the generative AI model.

[1788] What it does: When a user asks, "What should I do today?" the smart speaker suggests, "Yesterday you said you wanted to do laundry. How about reading today?"

[1789] Tracking medication and dietary status and receiving push notifications

[1790] Step 1: Schedule Management

[1791] The server manages the user's medication schedule and meal plan. The input is the user's medication and meal schedule, and the output is the managed schedule information.

[1792] Specific operation: The server manages the schedule using the Google Calendar API.

[1793] Step 2: Sending notifications

[1794] The server sends notifications to the smart speaker based on medication and meal times. The input is schedule information and the output is notifications to the smart speaker.

[1795] Specific operation: Every day at 12 o'clock, a notification stating "It's time to take your medicine" is sent to the smart speaker.

[1796] Step 3: User responses and recording

[1797] The user responds to the smart speaker, and the response is sent to the server. The input is the user's response, and the output is the record update.

[1798] Specific operation: The user responds with "I just drank it," and the information is sent to the server and the record is updated.

[1799] Providing administrative and daily information

[1800] Step 1: Get the information

[1801] The server periodically receives important notices and daily information from the government. The input is a request to obtain government information, and the output is the obtained information.

[1802] Specific operation: The server obtains information using APIs or web scraping technology.

[1803] Step 2: Provide information

[1804] The acquired administrative information and everyday information is communicated to the user via a smart speaker. The input is the acquired information, and the output is the information provided to the user.

[1805] Specific operation: At the beginning of the month, play something like "This month's local events are as follows."

[1806] Through the above processing steps, the system enables safety confirmation for elderly single people, health management, chat support, medication and diet management, and the provision of government information.

[1807] (Application example 1)

[1808] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1809] The lives of elderly people living alone are fraught with many physical and psychological anxieties. In particular, there are problems with the lack of means to respond quickly in emergencies and the difficulty of managing their health and confirming their safety in their daily lives. Security issues also remain a concern. A system that can resolve these issues and improve the quality of life for elderly people living alone is needed.

[1810] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1811] In this invention, the server includes means for receiving an inquiry from an information processing device to check the safety of the user and outputting a voice inquiry to the user to check their health condition, means for converting the user's voice response from voice to symbol data, means for analyzing the converted symbol data and sending it to the information processing device, means for the information processing device to analyze the symbol data and send a notification to an emergency contact if an abnormality is detected, means for monitoring the user's living environment using voice recognition technology, means for acquiring sensor information and sending a voice notification if an abnormality is detected, and means for installing an emergency button and sending a notification to an emergency contact when the button is used. This enables safety and health management for elderly people living alone and rapid response in emergencies.

[1812] "Safety confirmation" is a procedure to check the health and safety of elderly people living alone.

[1813] An "information processing device" is an electronic device for inputting, processing, and outputting data.

[1814] An "inquiry" is a question or request made to obtain some information.

[1815] "User" refers to a person who uses a system or device.

[1816] "Health status" means the physical and mental state of an individual.

[1817] "Audio output" is the process of generating sound and outputting it from a speaker or the like.

[1818] "Symbolic data" is a series of symbols that represent information such as speech or text.

[1819] "Analysis" is the process of analyzing given data or information and understanding its meaning and structure.

[1820] "Emergency contacts" are pre-defined contacts to be contacted in case of an emergency.

[1821] A "notification" is a message or signal that informs others of specific information.

[1822] "Voice recognition technology" is a technology that analyzes voice and converts it into text data.

[1823] "Living environment" refers to the entire environment in which an individual spends their daily life.

[1824] "Sensor information" refers to data and measurements collected by sensors.

[1825] "Abnormal" refers to a state or phenomenon that is different from the normal.

[1826] An "emergency button" is a button that can be pressed in an emergency to trigger a specified action.

[1827] In today's society where many elderly people live alone, safety and health management are extremely important issues. This invention provides a system that supports the lives of elderly people by using a smart speaker equipped with a generative AI model.

[1828] To implement the invention, the following components are required:

[1829] 1. Safety confirmation function

[1830] The server sends a request to check the user's safety at a fixed time every day. For example, the server's scheduler is set to be triggered at 9:00 AM, at which time it generates an API request and sends it to the smart speaker.

[1831] When the device (smart speaker) receives a request from the server, it outputs a voice message to the user asking, "Hello. How are you feeling today?" Google Text-to-Speech (gTTS) is used for voice synthesis.

[1832] The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The smart speaker receives the voice response through its microphone, converts the voice into text data using the SpeechRecognition library, and sends it to the server.

[1833] The server analyzes the received text data and, if an abnormality is detected, sends an email or SMS to an emergency contact.

[1834] 2.Security monitoring function

[1835] The information processing device (server) receives data from sensors attached to doors and windows. If an abnormality is detected, it notifies the user by voice, saying, "The front door has opened. Is it safe?"

[1836] If the user does not respond or responds negatively, the server sends a notification to the emergency contact.

[1837] 3. Emergency response function

[1838] The device (smart speaker) is equipped with an emergency button that the user can press in the event of an emergency. When the emergency button is pressed, a notification is immediately sent to the server, which then immediately sends a notification to pre-registered emergency contacts.

[1839] Examples of implementations of these features include:

[1840] Scenario: At 9am, a smart speaker asks, "Hello. How are you feeling today?" If the user does not reply "I'm fine," it automatically notifies emergency contacts that "No response."

[1841] Example prompt: "I want to create a voice interface to check the safety of elderly people living alone. I want it to check their safety every morning and notify emergency contacts if there is an abnormality. Please give me some concrete code examples."

[1842] This will ensure the safety and health management of elderly people living alone, as well as quick response in emergencies, and improve their quality of life.

[1843] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1844] Step 1:

[1845] Server: The server sends a request to check the user's safety at a fixed time every day. The server's scheduler is set to be triggered at 9:00 AM. When the scheduler reaches the specified time, it generates an API request and sends it to the smart speaker. The input is the scheduler's trigger time, and the output is the API request.

[1846] Step 2:

[1847] Device (Smart Speaker): Upon receiving a request from the server, the smart speaker outputs a voice message to the user asking, "Hello, how are you feeling today?" In this process, it uses Google Text-to-Speech (gTTS) to convert the text message into speech. The input is the API request from the server, and the output is the voice message.

[1848] Step 3:

[1849] User: The user responds to the smart speaker by saying something like "I'm fine" or "I'm not feeling well." The input is the voice question from the smart speaker, and the output is the user's voice response.

[1850] Step 4:

[1851] Device (smart speaker): The smart speaker receives the user's voice response through the microphone and converts the voice into text data using the SpeechRecognition library. The converted text data is sent to the server. The input is the user's voice response, and the output is text data.

[1852] Step 5:

[1853] Server: The server analyzes the received text data and sends an email or SMS to the emergency contact if an abnormality is detected. The analysis includes the user's response and determines whether there is an abnormality. The input is text data, and the output is a notification to the emergency contact.

[1854] Step 6:

[1855] Server: The server periodically receives data from the sensors. If the sensors attached to the doors or windows detect an abnormality, the server acquires and analyzes the data. The input is the data from the sensors, and the output is the analysis result on whether or not there is an abnormality.

[1856] Step 7:

[1857] Terminal (smart speaker): If an abnormality is detected based on sensor information, the smart speaker will notify the user by voice, saying, "The front door has opened. Is it safe?" The input is the abnormality notification from the sensor, and the output is the voice notification.

[1858] Step 8:

[1859] User: The user responds verbally, saying "Yes, I'm safe" or "No, I need help." The input is the voice notification from the smart speaker, and the output is the user's voice response.

[1860] Step 9:

[1861] Device (smart speaker): Receives the user's response again, converts the speech into text using the SpeechRecognition library, and analyzes it. The input is the user's voice response, and the output is text data.

[1862] Step 10:

[1863] Server: Analyzes the user's voice response and sends a notification to emergency contacts if an abnormality is detected. The input is text data, and the output is a notification to emergency contacts.

[1864] Step 11:

[1865] Terminal (Smart Speaker): The smart speaker is equipped with an emergency button. When a user presses the emergency button, a notification is sent to the server immediately. The input is the trigger of the emergency button, and the output is a notification to the server.

[1866] Step 12:

[1867] Server: When the server receives a notification that the emergency button has been pressed, it immediately sends a notification to the pre-registered emergency contacts. The input is a notification that the emergency button has been pressed, and the output is a notification to the emergency contacts.

[1868] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1869] This invention relates to a smart speaker system equipped with a generative AI to support the daily lives of elderly people living alone, and in particular, to a system that combines an emotion engine to analyze the user's emotions and respond appropriately. Specific program processing and operation are described below.

[1870] Safety confirmation function and emotion recognition function

[1871] Sending requests and outputting voice questions

[1872] Server: The server sends a safety check request to the smart speaker at a fixed time every day. For example, when the server's scheduler is triggered at 9:00 a.m., it generates an API request and sends it to the smart speaker.

[1873] Device (smart speaker): When the smart speaker receives the request, it will ask the user aloud the question "How are you feeling today?", using a natural language processing (NLP) engine to play a predefined message via text-to-speech (TTS).

[1874] Receiving, converting, and parsing user responses

[1875] User: The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone inside the smart speaker picks up the voice.

[1876] Device: The smart speaker converts the user's voice response from speech to text using a STT (Speech-to-Text) engine. The converted text data is analyzed and the results are sent to the server.

[1877] Emotion Recognition and Anomaly Detection

[1878] Emotion engine: Smart speakers are equipped with an emotion engine that recognizes emotions from the user's voice. The emotion engine analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state (e.g., sadness, joy, anger, etc.).

[1879] Server: The server analyzes the received text data and emotion data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and the emotion recognition detects "sad," it sends a notification to the emergency contact.

[1880] Chat and emotion recognition

[1881] Topic selection and conversation start

[1882] Device: The smart speaker automatically starts chatting with the user based on specific keywords or the time of day. A timer or scheduler is set to trigger the chat function at a specific time. It provides topics such as, "Today's weather is sunny. It's perfect for a walk."

[1883] Receiving, analyzing, and providing information about user responses

[1884] User: The user can freely talk and the smart speaker will receive the response, for example, "What should I do today?"

[1885] Device: The smart speaker analyzes the user's voice response, and the emotion engine recognizes the user's emotions. If the user asks a question in a "sad voice," it will provide an appropriate encouraging message. For example, it will suggest, "You said you would do the laundry yesterday. How about reading a book?"

[1886] Tracking medication and dietary status and receiving push notifications

[1887] Scheduling and Notifications

[1888] Server: The server manages the user's medication schedule and meal plan and sends appropriate notifications to the user at specified times. For example, it generates a notification saying "It's time to take your medicine" at 12 o'clock every day and sends it to a smart speaker.

[1889] Device: The smart speaker conveys notifications from the server to the user as voice messages.

[1890] User responses and record updates

[1891] User: The user responds by saying "I just drank it." This speech is received by the smart speaker.

[1892] Device: The smart speaker converts voice into text, analyzes emotions, and sends it to the server.

[1893] Server: The server updates the user's behavioral history based on the received response data and emotion data.

[1894] Providing administrative and daily information

[1895] Acquisition and provision of information

[1896] Server: The server periodically sends important government notifications and useful information to the smart speaker. For example, it provides "This month's local event information" at the beginning of each month.

[1897] Device: The smart speaker will then relay the information to the user aloud, playing something like, "This month's events are as follows."

[1898] By combining this generative AI-equipped smart speaker system with an emotion engine, it is possible to provide more advanced user follow-up and personalized support. As a result, it is expected to significantly contribute to improving the quality of life of the elderly and reducing their sense of loneliness. It will also significantly reduce the workload of welfare workers.

[1899] The processing flow will be explained below.

[1900] Safety confirmation function and emotion recognition function

[1901] Step 1:

[1902] The server sends a safety check request to the smart speaker at a fixed time every day. The server's scheduler is triggered at 9:00 AM, generates a safety check request, and sends it to the smart speaker as an API request.

[1903] Step 2:

[1904] The device (smart speaker) receives a request from the server and outputs the question "How are you feeling today?" to the user by voice. A natural language processing (NLP) engine is used to play back a predefined message using text-to-speech (TTS).

[1905] Step 3:

[1906] The user responds by saying something like "I'm fine" or "I'm not feeling well." The microphone in the smart speaker picks up the user's voice.

[1907] Step 4:

[1908] The device uses a Speech-to-Text (STT) engine to convert the user's voice response from speech to text, then analyzes the converted text data and sends it to the server.

[1909] Step 5:

[1910] The device uses an emotion engine to recognize emotions from the user's responses. It analyzes the tone, pitch, and speed of the voice data to determine the user's emotional state.

[1911] Step 6:

[1912] The server analyzes the received text data and emotional data, and if it determines that the user's condition is abnormal, for example, if the user responds "I'm not feeling well" and emotion recognition detects "sad," it sends a notification to emergency contacts.

[1913] Chat and emotion recognition

[1914] Step 1:

[1915] The device will set a chat trigger at the start of the day or at a suitable time, and an internal timer or scheduler will call the chat function at a specific time.

[1916] Step 2:

[1917] The device selects topics from the user's past conversation logs or a preset topic list, analyzes the conversation data stored in storage, and selects new topics and appropriate lifestyle support information.

[1918] Step 3:

[1919] The device will start a voice chat based on the selected topic. It will generate the message "Today's weather is sunny. It's perfect for a walk," and play it back through text-to-speech (TTS).

[1920] Step 4:

[1921] The user can freely converse with the device, asking questions and responding to topics by voice, which is then input into the smart speaker.

[1922] Step 5:

[1923] The device analyzes the user's voice response, and the emotion engine recognizes the user's emotions. For example, if the user asks a question in a sad voice, the device generates an appropriate encouraging message based on that emotional data, suggesting, "You said you would do the laundry yesterday. How about reading a book?"

[1924] Tracking medication and dietary status and receiving push notifications

[1925] Step 1:

[1926] The server manages the user's medication schedule and meal plan. The schedule management system sets and saves each user's medication and meal plan in the database.

[1927] Step 2:

[1928] The server generates a notification at a set time and sends it to the smart speaker. A push notification request is sent to the smart speaker via API at 12:00 every day.

[1929] Step 3:

[1930] The device outputs the notification sent from the server as voice, using voice synthesis to play messages such as "It's almost time to take your medicine."

[1931] Step 4:

[1932] The user responds verbally by saying, "I just drank it." The user's response is communicated to the smart speaker using voice input.

[1933] Step 5:

[1934] The device converts the voice response into text, analyzes emotions, and sends it to the server. The voice is converted into text by the STT engine, and the analysis results are sent to the server via API.

[1935] Step 6:

[1936] The server records the user's response in a database, and medication and dietary history is updated based on the received data.

[1937] Example 2

[1938] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1939] In modern society, the increase in the number of elderly people living alone is becoming a major social problem. When elderly people living alone at home do not adequately manage their health or communicate with others on a daily basis, they experience feelings of loneliness and health risks. Furthermore, conventional healthcare systems have difficulty grasping the user's emotional state, which can prevent them from providing appropriate support. Furthermore, elderly people tend to neglect managing their medication and diet, creating a need for a means to properly manage their behavioral history. It is necessary to develop a system that can solve these issues and improve the quality of life for the elderly.

[1940] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1941] In this invention, the server includes means for receiving a request from the server for safety confirmation and outputting a question to the user by voice to confirm the user's health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for identifying emotions from the user's voice using an emotion recognition engine, means for sending the identified emotion data to the server, and means for the server to evaluate the user's condition based on the emotion data and the text data and send a notification to an emergency contact if an abnormality is detected. This makes it possible to grasp not only the user's health condition but also their emotional state and to detect abnormalities early.

[1942] "Safety check" is the act of checking the health and safety of residents.

[1943] A "server" is a computer system for storing, processing, and communicating data.

[1944] A "means for receiving a request" is a mechanism for receiving requests from the outside.

[1945] The "means for outputting questions by voice" refers to a device or system that uses voice synthesis technology to reproduce text information as voice.

[1946] A "means for converting a voice response from voice to text data" is a device or system that uses voice recognition technology to convert a voice signal into a corresponding text format.

[1947] The "means for analyzing the converted text data and transmitting it to the server" refers to the process for understanding and evaluating the text data and the communication means for transmitting that data to the server.

[1948] An "emotion recognition engine" is an algorithm or software that analyzes tone, pitch, speed, etc. from audio data to identify emotional states.

[1949] The "means for transmitting emotion data to the server" refers to a communication means for transferring the data obtained by emotion recognition to the server.

[1950] "Means for sending a notification to an emergency contact when an abnormality is detected" refers to a mechanism for sending a notification to a registered emergency contact when the system detects an abnormality in the user's behavior.

[1951] A "topic selection means" is a system or device that determines the topic of conversation with a user based on a pre-set algorithm.

[1952] The "means for starting a voice chat" is a system or device for starting a conversation based on a selected topic using voice synthesis technology.

[1953] The "means for providing information or continuing a conversation" refers to a system or device for returning appropriate information in response to a user's response or for continuing a conversation.

[1954] The "schedule management means in the server for managing medication schedules and meal plans" is a system for managing the times when a user takes medicine and the times when they eat, and saving these as a schedule on the server.

[1955] The "means for notifying the user of medication and meal times by voice" refers to a system or device that uses voice synthesis technology to notify the user of medication and meal times by voice.

[1956] The "means for receiving a response from the user and transmitting the response to the server" refers to a communication means for taking in a reply from the user and transmitting it as data to the server.

[1957] A "means for updating behavioral history" is a system or device that records data about a user's behavior and modifies or adds to that history information based on new data.

[1958] This invention relates to a smart speaker system that utilizes generative AI technology to support the daily lives of elderly people living alone. Specifically, it combines an emotion recognition engine to analyze the user's emotions and respond appropriately. The details are described below.

[1959] System Configuration

[1960] This system consists of a server, a smart speaker (hereafter referred to as the "terminal"), and an elderly person. The server stores, processes, and communicates data, while the terminal is a device that enables dialogue with the user and voice responses.

[1961] Hardware and software used

[1962] Server: The server is used to perform functions such as safety confirmation, data analysis, anomaly detection, emergency notification, etc. Examples include Linux-based servers and cloud services (AWS, Google Cloud, etc.).

[1963] Device (smart speaker): Equipped with a natural language processing (NLP) engine, text-to-speech (TTS), speech recognition (STT), and emotion recognition engine. Specifically, it uses Google Dialogflow, Google's STT engine, IBM Watson's Tone Analyzer, etc.

[1964] Operation overview

[1965] 1. Safety confirmation

[1966] Every morning at 9:00, the server uses a scheduler to send a safety confirmation request to the device. The device receives this request and asks the user verbally, "How are you feeling today?" The user's response is received by the device's microphone, and a speech recognition engine converts it into text. The converted text data is also sent to an emotion recognition engine, and is sent to the server along with emotion data. The server analyzes this data and sends a notification to emergency contacts as necessary.

[1967] 2. Chat function

[1968] The device will begin chatting with the user based on a specific time or trigger keyword. A topic selection algorithm will provide a topic through voice, such as, "Today's weather is sunny. Perfect for a walk." Once the user's response is received, the voice recognition engine again converts it into text data, and the emotion recognition engine identifies emotions. By providing appropriate information and continuing the conversation, the device can alleviate the user's sense of loneliness.

[1969] 3. Understanding medication and dietary status

[1970] The server manages the user's medication schedule and meal plan, and sends voice notifications through the device at designated times. An example message is "It's time to take your medicine." The user's responses are received by the device, converted into text, and sentiment analysis is performed before being sent to the server. The server updates the user's behavioral history based on this data.

[1971] 4. Providing administrative and daily information

[1972] The server periodically sends important notifications from the government and other useful information to the device. For example, the device will provide information about local events this month. The device will then communicate this information to the user via voice.

[1973] Examples of specific examples and prompts

[1974] Example 1: Safety confirmation

[1975] The server generates an API request at 9:00 a.m. and sends a safety confirmation message to the smart speaker.

[1976] The device asks the user aloud, "How are you feeling today?"

[1977] Example prompt: "Generate an API request to check the health of an elderly person at 9 AM."

[1978] Example 2: Emotion recognition and anomaly detection

[1979] The server analyzes the user's text data and emotional data, and if the user replies "I'm not feeling well" and the emotion is determined to be "sad," it sends a notification to the emergency contact.

[1980] Example prompt: "Design a system that analyzes a user's emotional data and physical responses, and notifies emergency contacts if there are any abnormalities."

[1981] Example 3: Chat function

[1982] At 2 p.m., the device triggers the chat function and offers the topic, "Today's weather is sunny. Perfect for a walk."

[1983] Example prompt: "Design a system that triggers a chat feature at a specific time and provides topics about today's weather."

[1984] By combining this generative AI-equipped smart speaker system with an emotion engine, it will improve the quality of life of the elderly and reduce their sense of loneliness. It will also reduce the workload of welfare workers and enable more personalized support.

[1985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1986] Step 1:

[1987] Server sends safety confirmation request

[1988] The server uses a scheduler to send a safety confirmation request to the device every morning at 9:00. In this process, when a trigger is received from the scheduler, an API request is generated and sent to the smart speaker.

[1989] Input: Trigger event from scheduler.

[1990] Processing: Generating and sending API requests.

[1991] Output: API request sent successfully.

[1992] Action: scheduler.trigger('9:00', sendHealthCheckRequest) is called, a request is generated and sent to the smart speaker.

[1993] Step 2:

[1994] Voice question output by terminal

[1995] The device (smart speaker) receives the request from the server and outputs the question "How are you feeling today?" via Google Dialogflow.

[1996] Input: The API request from the server.

[1997] Processing: Converts text to speech using a Text-to-Speech (TTS) engine.

[1998] Output: A spoken question to the user.

[1999] Behavior: The askHealthStatus() function is called, and TTS.synthesize("How are you feeling today?") is executed.

[2000] Step 3:

[2001] Receiving a user's voice response

[2002] The user responds verbally, and the smart speaker's microphone captures the audio, with examples of responses like "I'm fine" or "I'm not feeling well."

[2003] Input: The user's spoken response.

[2004] Processing: Receiving audio.

[2005] Output: Received audio data.

[2006] Operation: The microphone is always in standby mode, and microphone.capture() captures audio data.

[2007] Step 4:

[2008] Device-based speech-to-text conversion

[2009] Smart speakers use Google's Speech-to-Text (STT) engine to convert the user's voice into text data.

[2010] Input: Received audio data.

[2011] Processing: Converts audio data into text data using the STT engine.

[2012] Output: The converted text data.

[2013] What happens: The convertSpeechToText(audioData) function is called to convert the speech to text.

[2014] Step 5:

[2015] Device-based emotion recognition

[2016] The text data is sent to an emotion recognition engine (e.g., IBM Watson's Tone Analyzer) to analyze the user's emotions.

[2017] Input: The converted text data.

[2018] Processing: Uses an emotion recognition engine to analyze tone, pitch, speed, etc. to identify emotions.

[2019] Output: Emotion data.

[2020] Operation: The analyzeEmotion(textData) function is called to generate emotion data.

[2021] Step 6:

[2022] Analysis of text data and emotion data by the server

[2023] The server receives and analyzes the text data and emotional data sent from the smart speaker.

[2024] Input: Text data and emotion data.

[2025] Processing: Based on the results of the data analysis, the user's health status is evaluated.

[2026] Output: User state assessment results.

[2027] Operation: The evaluateUserStatus(textData, emotionData) function is called to determine whether there are any abnormalities.

[2028] Step 7:

[2029] Server-based anomaly detection and notification to emergency contacts

[2030] If the server detects an abnormality, it will send a notification via email or SMS to the designated emergency contact.

[2031] Input: User's condition assessment results.

[2032] Processing: Generating and sending notifications when an anomaly is detected.

[2033] Output: Notification sent to emergency contacts.

[2034] Action: The sendEmergencyNotification(contactInfo, userStatus) function is called to send the emergency notification.

[2035] (Application example 2)

[2036] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2037] In the daily lives of elderly people living alone, there is a need to grasp changes in their physical condition and emotions in real time, and to respond appropriately and contact them in emergencies. In particular, a system that can respond quickly when unexpected physical or mental illness occurs is needed. Another challenge is to provide a system that can manage medication schedules and meal plans, as well as provide encouragement and suggestions according to the user's emotional state.

[2038] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2039] In this invention, the server includes means for receiving a request to check safety and outputting a question to the user by voice to check their health condition, means for converting the user's voice response from voice to text data, means for analyzing the converted text data and sending it to the server, means for the server to analyze the text data and send a notification to an emergency contact if an abnormality is detected, emotion analysis means for recognizing emotions from the user's voice, and means for evaluating the state of the malfunction based on the analyzed emotion data and sending an additional notification to the emergency contact as necessary. This makes it possible to grasp and respond to not only the user's physical condition but also their emotional state in real time.

[2040] "Safety check" is an action taken to periodically check the health status and safety of a user.

[2041] A "server" is a computer system for sending, receiving, and processing data.

[2042] A "request" is a message from a server requesting specific information or an operation.

[2043] "Users" refer to people such as elderly people who use this system.

[2044] "Health status confirmation questions" are questions asked to find out the user's physical condition and mood.

[2045] "Audio output means" refers to a technology that converts text information into audio and conveys it to the user.

[2046] "Means for converting voice into text data" refers to a technology that converts voice information into text information using voice recognition technology.

[2047] "Means for analyzing text data" refers to technology for analyzing converted character information.

[2048] "Emotion analysis means" is a technology that estimates a user's emotions from voice and text information.

[2049] "Emergency contacts" are people or organizations to whom notifications should be sent in the event of an emergency.

[2050] The "means for evaluating a user's state of failure based on emotional data" is a technology that analyzes the user's emotional information and evaluates the user's current state.

[2051] An "additional notification" is a further message sent in addition to a notification already sent.

[2052] The "means for selecting a topic for chatting" is a technique for selecting a topic for conversation with a user.

[2053] The "means for starting a voice chat" is a technique for starting a voice conversation based on a selected topic.

[2054] "Means for continuing a conversation" refers to a technique for continuing a conversation depending on the user's reaction and situation.

[2055] A "medication schedule" is a plan that manages the types and times of medication a user takes.

[2056] A "meal plan" is a plan that manages the menu and times of meals that a user will eat.

[2057] "Schedule management tools" are technologies for recording and managing medication and meal plans.

[2058] "A means for notifying the user of medication and meal times by voice" refers to a technology that notifies the user of scheduled medication and meal times by voice.

[2059] "Behavioral history" is a record of the actions and reactions of a user.

[2060] "Means for updating behavioral history" refers to technology that updates behavioral records based on collected data.

[2061] The present invention relates to a security assistant system for supporting the daily lives of elderly people living alone, and in particular to a system that combines an emotion engine to analyze the user's emotions and take appropriate action. The following describes the processing and operation of the system program.

[2062] System Configuration

[2063] Hardware and Software

[2064] Server: Generates safety confirmation requests, analyzes data, and manages emergency notifications. An API server using Python is expected.

[2065] Terminal (smart speaker, smartphone, etc.): A device that acts as an interface with the user and is equipped with speech recognition (Speech-to-Text), speech synthesis (Text-to-Speech), emotion analysis engine (EmotionRecognizer), etc.

[2066] Emotion analysis engine: Extracts emotions from the user's voice and provides appropriate responses. The library used is expected to be EmotionRecognizer.

[2067] System Operation

[2068] Safety confirmation function

[2069] The server sends a safety confirmation request to the device at a fixed time every day. Triggered via an API, the question "How are you feeling today?" is output to the smart speaker using text-to-speech (TTS).

[2070] When the user responds by saying something like "I'm fine" or "I'm not feeling well," the device converts the voice into text data using a speech-to-text (STT) engine.

[2071] The converted text data is converted into emotion data by an emotion analysis engine, and then the text data and emotion data are sent to the server.

[2072] The server analyzes the transmitted data and sends a notification to emergency contacts if an abnormality is detected.

[2073] Chat function

[2074] Based on a request from the server, the device selects a specific topic and starts chatting with the user. A timer or scheduler is set to trigger the chat function at a specific time.

[2075] Based on the selected topic, a message such as "Today's weather is sunny. It's perfect for a walk" is output in voice.

[2076] When a user asks, "What should I do today?", the device converts the voice to text and the emotion engine analyzes the emotion. If the user asks the question in a "sad voice," it will provide an appropriate encouraging message.

[2077] Understanding medication and dietary status

[2078] ...

Claims

1. a means for receiving a request from a server for safety confirmation and outputting a question to the user by voice to confirm the user's health condition; means for converting a user's voice response from voice to text data; means for analyzing the converted text data and transmitting it to a server; A server analyzes the text data, and if an abnormality is detected, sends a notification to an emergency contact; A system including:

2. means for selecting a topic for chatting with the user based on a request from the server; means for initiating an audio chat based on the selected topic; A means for analyzing the user's voice response and providing appropriate information or continuing the conversation; The system of claim 1 further comprising:

3. a schedule management means in the server for managing the user's medication schedule and meal plan; a means for notifying the user by voice of medication and meal times based on notifications from the server; means for receiving a user response and transmitting the response to a server; A means for the server to update the record based on the user's behavior history; The system of claim 1 further comprising:

Images