system

Abstract

Provide a system. The method includes: means for periodically initiating a conversation with an elderly person device; means for transmitting audio data of the conversation from the senior citizen device; a means for converting the audio data into text; a means for summarizing text data; a means for storing the summarized data in a database; The system includes means for sending a notification of the summarized data to a user device.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, it is common for elderly people and their families to live apart, making it difficult for family members to keep track of the elderly person's daily life and health status. This situation is particularly pronounced when the elderly person lives alone, which can lead to feelings of loneliness and increased health risks. Furthermore, because the elderly person's life cycle differs from that of their family, it is often difficult to communicate regularly. This leads to a lack of ways to keep track of the elderly person's condition, and opportunities to provide better support are lost. [Means for solving the problem]

[0005] The present invention provides a system in which an AI avatar initiates a conversation with an elderly person at regular intervals. This system includes a means for initiating a conversation by specifying a time for the elderly person device, a means for transmitting the conversation's voice data from the elderly person device to a server, a means for converting the voice data to text on the server, a means for summarizing the text data, a means for storing the summarized data in a database, and a means for sending a notification of the summarized data to a user device. This configuration allows family members to check the elderly person's daily condition at any time, thereby reducing the elderly person's sense of loneliness and improving their health management. Specifically, the elderly person device, such as a smartwatch or smartphone, periodically converses with the AI ​​avatar, and the content of the conversation is sent to a server where it is analyzed and summarized by a generative AI. The summarized data is then sent to the user device, allowing family members to understand the elderly person's condition.

[0006] "Elderly devices" refers to portable devices such as smartwatches and smartphones used by seniors.

[0007] "Server" refers to the central control device that processes and stores data sent from the Senior Device and transmits data to the User Device as needed.

[0008] "User devices" refer to devices such as smartphones and PCs used by elderly people's family members and others to check information.

[0009] "AI avatar" refers to a virtual character that uses artificial intelligence to converse with the elderly.

[0010] "Audio Data" refers to audio recordings of conversations collected from Senior Devices.

[0011] "Generative AI" refers to artificial intelligence technology that converts voice data into text data.

[0012] "Text data" refers to character information generated by analyzing voice data.

[0013] A "summary" refers to information that is concisely summarized by extracting important information and keywords from text data.

[0014] "Database" refers to an electronic storage device for storing abstract data and other information.

[0015] "Notification" refers to a message that informs a user device of the generation of summary data or other important information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] System Overview

[0038] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: elderly people's devices (smartwatches and smartphones), a server, and user devices (smartphones and PCs).

[0039] Specific examples of system programs and processing

[0040] 1. The server sends a conversation start trigger

[0041] The server sends a trigger to start a conversation to the elderly device at a set time every day. For example, if the trigger is set to 9:00 a.m. and 6:00 p.m., the server uses a scheduler function to send an HTTP request to the specified device.

[0042] Specific examples

[0043] The server sends a trigger to start a conversation at 9 a.m. A notification is sent to the elderly person's device, activating the AI ​​avatar.

[0044] 2. Initiating and recording conversations with senior devices

[0045] Based on the received trigger, the AI ​​avatar app on the elderly person's device launches and speaks to the user (elderly person), saying, "Good morning. What are your plans for today?" The device then records the elderly person's response.

[0046] Specific examples

[0047] The elderly person responds, "I plan to go to the hospital today." This conversation is recorded by the elderly device.

[0048] 3. Sending audio data to the server

[0049] After the recording is complete, the audio data will be automatically sent to the server, and the device will upload the audio data to the server in the appropriate format.

[0050] Specific examples

[0051] After recording is complete, the smartphone sends the audio file to the server, which receives and stores it.

[0052] 4. Converting Audio Data to Text

[0053] The server converts the received voice data into text data using generative AI, which is the processing part of the speech recognition engine.

[0054] Specific examples

[0055] The voice data "I plan to go to the hospital today" is converted into the text "I plan to go to the hospital today."

[0056] 5. Summarizing Text Data

[0057] The converted text data is further summarized by a generative AI on the server, which extracts important information and generates a short summary.

[0058] Specific examples

[0059] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0060] 6. Database storage of summary data

[0061] The summarized data is stored in a database, which allows for easy retrieval and reference later.

[0062] Specific examples

[0063] The summarized "hospital visit schedule" is stored in a database.

[0064] 7. Notifications to User Devices

[0065] Once saved, the server notifies the user device that the summary data has been generated, either via email or app push notification.

[0066] Specific examples

[0067] The server notifies the user of the generated summary data, and the user's smartphone receives a notification that "an elderly person is scheduled to go to the hospital today."

[0068] 8. User review of summary data

[0069] Users receive notifications and can access summary data via a dedicated app or web portal, along with detailed text and audio data.

[0070] Specific examples

[0071] The child opens their smartphone and checks a summary of the "hospital appointment" in a dedicated app, and if necessary, can further review the original audio data and detailed text.

[0072] As described above, this system enables regular and efficient communication between the elderly and their families, allowing family members to quickly understand the elderly's daily condition and provide the necessary support.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The server uses the scheduler function to send a conversation start trigger to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[0076] Step 2:

[0077] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[0078] Step 3:

[0079] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[0080] Step 4:

[0081] The device sends the recorded audio data to the server, where it is uploaded to the server in the appropriate format (e.g., MP3 or WAV file).

[0082] Step 5:

[0083] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[0084] Step 6:

[0085] The server summarizes the generated text data. At this stage, the generative AI extracts important information and keywords from the text data and creates a compact summary.

[0086] Step 7:

[0087] The server stores the abstract data in a database, where the abstract is registered as a database entry and stored for future search and retrieval.

[0088] Step 8:

[0089] The server notifies the user device that the summary data has been generated and saved, either via email or app push notification.

[0090] Step 9:

[0091] Users receive notifications and can check summary data via a dedicated app or web portal. By viewing the summary data, users can understand the elderly person's daily condition and, if necessary, can also view detailed text and audio data.

[0092] This is the process flow of the entire system, which enables effective communication between the elderly and their families.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Family members who live far away from their elderly loved ones face the challenge of regular and effective communication. In particular, there are few ways to keep track of the elderly's health status and schedules in real time, which can delay emergency response. Furthermore, if the elderly person does not provide information voluntarily, family members have no way of knowing their elderly loved one's condition.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes: means for sending a trigger to start a conversation to the elderly device at a specified time; means for automatically sending recorded conversation audio data from the elderly device to the server; means for converting the audio data into text data using a generative AI model; means for summarizing the text data using the generative AI model; means for storing the summarized data in a relational database; means for notifying the user device of the completion of generation of the summarized data; and means for the user to check the summarized data through a dedicated app or web portal. This enables regular and efficient communication between the elderly and their family members. Family members can quickly understand the elderly's daily condition and provide necessary support.

[0098] "Senior device" refers to an electronic device used daily by seniors, including a smartwatch or smartphone.

[0099] A "trigger" refers to a signal or instruction to initiate a specific action, in this case sent from the server to the senior device to encourage the start of a conversation.

[0100] "Audio Data" means conversations recorded on an Elderly Device and stored in electronic format.

[0101] A "generative AI model" refers to an artificial intelligence model that uses machine learning and is used to automatically perform specific tasks (such as speech recognition and text summarization).

[0102] "Text data" refers to written information converted from voice data using a generative AI model.

[0103] A "summary" is a short summary created by analyzing text data and extracting only the important information and content.

[0104] A "relational database" is a database system that stores data in a structured tabular format, and allows data to be searched and manipulated using query languages ​​such as SQL.

[0105] "Notification" refers to a message or alert that notifies the user of a particular event or the result of a process, in this case notifying the user that summary data generation has been completed.

[0106] A "dedicated app" refers to a software application developed to provide specific functions or services, which users use on their smartphones or PCs.

[0107] "Web portal" refers to a website entrance that allows users to access specific information or services over the Internet, and is accessed by a user using a browser.

[0108] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: an elderly person's device (such as a smartwatch or smartphone), a server, and a user device (such as a smartphone or PC).

[0109] System Components and Functions

[0110] 1. Elderly devices:

[0111] These are electronic devices used by seniors in their daily lives, including smartwatches and smartphones, such as the Apple Watch, Samsung Galaxy Watch, iPhone, and Android smartphones.

[0112] The device receives a trigger from the server and launches an AI avatar app (e.g., a custom skill for Google Assistant or Amazon Alexa).

[0113] 2. Server:

[0114] The server is configured with Linux (registered trademark) servers and has the following functions:

[0115] A trigger is sent to the elderly device to start a conversation at a set time.

[0116] Receives voice data sent from the elderly device and stores it in an appropriate format.

[0117] The voice data is converted into text data using a generative AI model (e.g., Google Cloud Speech-to-Text API, IBM Watson® Speech to Text, etc.).

[0118] Summarize text data using generative AI models (e.g., OpenAI's GPT series).

[0119] The summarized data is stored in a relational database (e.g., MySQL (registered trademark), PostgreSQL, etc.).

[0120] The completion of generation of the summary data is notified to the user device.

[0121] 3. User Device:

[0122] Family members of elderly people can access information through a dedicated app or web portal on their smartphones or PCs, and notification services use Firebase and AWS (registered trademark) SNS.

[0123] Specific operation of the system

[0124] 1. Send a trigger to start a conversation:

[0125] The server uses a scheduler function such as crontab to send a trigger to start a conversation to the elderly device every morning at 9:00 and evening at 6:00. The trigger is sent using an HTTP request.

[0126] 2. Start and record the conversation:

[0127] Based on a trigger received from the server, the elderly person's device launches the AI ​​avatar app and asks, "Good morning. What are your plans for today?" If the elderly person replies, "I plan to go to the hospital today," the conversation is recorded.

[0128] 3. Sending audio data to the server:

[0129] Once the recording is complete, the senior's device uploads the audio file to a server using HTTPS, and the data is sent along with an authentication token for security reasons.

[0130] 4. Audio to text conversion:

[0131] The server converts the voice data into text using a generative AI model. For example, a speech saying "I plan to go to the hospital today" is converted directly into text.

[0132] 5. Summarizing text data:

[0133] Using the generative AI model, the server analyzes the text data, extracts key information, and generates a summary. For example, the text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0134] 6. Summary Data Storage and Notification:

[0135] The summarized data is stored in a relational database and immediately sent to the user's device, where they can receive notifications and view detailed text and audio data via a dedicated app or web portal.

[0136] Specific examples

[0137] (Example prompt) "What are your plans for today?" The elderly person responds, "I plan to go to the hospital today," and the AI ​​model summarizes this as "I plan to go to the hospital."

[0138] The child will receive a summary of their planned hospital visit notification in a dedicated app, and can view detailed text and audio data.

[0139] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1:

[0142] The server uses the scheduler function to send a conversation start trigger to the elderly device at the set times of 9:00 a.m. and 6:00 p.m. every day. For this purpose, the server generates an HTTP request and sends it to the elderly device. This process ensures that the elderly device receives a notification at the specified time.

[0143] Specifically, a script executed at a fixed time by the server's crontab generates and sends an HTTP request. The input is the time set by the scheduler, and the output is a trigger sent to the elderly device.

[0144] Step 2:

[0145] The elderly device launches the AI ​​avatar app based on the received trigger. At this time, the AI ​​avatar speaks to the elderly person, saying, "Good morning. What are your plans for today?" The elderly person responds, and the conversation is recorded by the elderly device. The input is the trigger from the server, and the output is the elderly person's voice data.

[0146] Specifically, when the elderly care device receives a trigger, the AI ​​avatar app automatically launches and speaks a pre-set prompt to the elderly. When the elderly responds, the voice data is recorded on the device.

[0147] Step 3:

[0148] After recording is completed, the elderly device sends the audio data to the server. This transmission uses HTTPS, and the audio data is uploaded to the server along with an authentication token. The input is the recorded audio data, and the output is the uploaded audio file to the server.

[0149] Specifically, the device temporarily stores the recorded audio data and then uploads the audio file to a server using the HTTPS protocol, using an authentication token for security during the upload process.

[0150] Step 4:

[0151] The server converts the received voice data into text data using a generative AI model (for example, Google Cloud Speech-to-Text API or IBM Watson Speech to Text). This process converts the voice data into a format that can be read as text. The input is voice data, and the output is the converted text data.

[0152] Specifically, the server sends the voice data to the AI ​​model and receives the resulting converted text data. The voice data, "I plan to go to the hospital today," is converted directly into text.

[0153] Step 5:

[0154] The server uses a generative AI model (such as OpenAI's GPT series) to summarize the text data, extracting important information and generating a short summary. The input is the converted text data, and the output is the summarized text data.

[0155] Specifically, the server sends text data to the AI ​​model and receives the summarized text data. For example, the text data "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0156] Step 6:

[0157] The server stores the summarized data in a relational database, which structures the data and makes it easier to search and reference later. The input is the summarized text data, and the output is storage in the database.

[0158] Specifically, the server uses an SQL query to store the summary data in a relational database. For example, the summary data "plan to go to the hospital" is stored as a specific entry in the database.

[0159] Step 7:

[0160] Once the server has finished storing the data, it uses a notification service (e.g., Firebase, AWS SNS, etc.) to send a push notification to the user device. The notification contains the summary data. The input is the summary data stored in the database, and the output is the notification sent to the user device.

[0161] Specifically, the server calls the notification service and sends a push notification to the user device stating, "An elderly person is scheduled to go to the hospital today."

[0162] Step 8:

[0163] Users receive notifications and can view summary data via a dedicated app or web portal, which also provides detailed text and audio data. The input is a notification to the user device, and the output is the user reviewing the data.

[0164] Specifically, the user opens the dedicated smartphone app and checks the summary data, such as "I plan to go to the hospital." If necessary, they can also read the original audio data and detailed text.

[0165] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0166] (Application example 1)

[0167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] Elderly people face difficulties in communicating regularly and effectively with their families who live far away. There is also a lack of methods to quickly check on the safety and well-being of elderly people. Furthermore, there is a need for a method to quickly detect when an elderly person may be in an emergency and take appropriate action.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0170] In this invention, the server includes a means for periodically initiating a conversation with the elderly person's device, a means for converting voice data into text, and a means for extracting specific keywords from the voice data and detecting abnormalities, thereby enabling effective monitoring of the elderly person's condition and immediate response in the event of an emergency.

[0171] "Senior device" refers to an electronic device designed for use by seniors, such as a smartwatch or smartphone.

[0172] The "conversation initiation means" is a mechanism for periodically initiating a conversation with the elderly person device, and includes a function for sending a trigger at a specific timing.

[0173] The "voice data transmission means" is a mechanism for transmitting voice data of conversation acquired from the elderly person device to the server.

[0174] The "voice data text conversion means" is a function for converting transmitted voice data into text data, and uses a voice recognition engine.

[0175] The "text data summarizing means" is a function for summarizing the generated text data and extracting important information.

[0176] The "database storage means" is a mechanism for storing summarized data in a database so that it can be easily searched and referenced later.

[0177] The "notification sending means" is a function for notifying the user device of summarized data, and includes email and push notification.

[0178] The "specific keyword extraction means" is a function for extracting predetermined keywords from the voice data and detecting abnormalities based on the contents of the keywords.

[0179] The "anomaly detection means" is a mechanism for detecting anomalies based on extracted keywords, and is used to determine an emergency situation.

[0180] The "emergency contact notification means" is a function for sending a notification to an emergency contact when an abnormality is detected.

[0181] System Overview

[0182] This invention relates to a system that enables elderly people and their families living far away to communicate regularly and effectively. The system mainly comprises an elderly person device, a server, and a user device.

[0183] Hardware and software used

[0184] Elderly devices: Using a smartwatch or smartphone, which allows seniors easy access and regular notifications.

[0185] Server: We use a high-performance computer server to process and store audio data.

[0186] User devices: Provide an interface for family members to check on the daily activities of their elderly relatives using smartphones or PCs.

[0187] Software: Uses Google's Speech-to-Text API as the speech recognition engine, smtplib for sending emails, Python's schedule library for scheduling, and a generative AI model (e.g., OpenAI's GPT) for text processing.

[0188] System Operation

[0189] 1. Start regular conversations

[0190] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation. When the elderly device receives this trigger, the AI ​​avatar starts the interaction and speaks to the elderly.

[0191] 2. Collection and transmission of voice data

[0192] The elderly person's device records the conversation and sends the audio data to the server, where it is uploaded in an appropriate format.

[0193] 3. Speech to text conversion and summarization

[0194] The server uses Google's Speech-to-Text API to convert the received audio data into text data, and then uses a generative AI model to summarize the text data.

[0195] 4. Anomaly Detection

[0196] The server extracts specific keywords from the converted text data and detects anomalies. For example, if keywords such as "help" or "in trouble" are included, the server will determine that there is an anomaly.

[0197] 5. Notification to emergency contacts

[0198] If an anomaly is detected, the server will notify emergency contacts via email using smtplib.

[0199] 6. Notification and confirmation to user devices

[0200] The summarized data is stored in a database and then sent to the user's device via push notification or email. Family members can receive these notifications on their device and check the elderly person's condition.

[0201] Specific examples

[0202] The elderly person's device receives a conversation trigger at 9:00 a.m., asking, "Good morning. What are your plans for today?" The elderly person replies, "I plan to go to the hospital today." This conversation is recorded and sent to a server as an audio file. The server converts the audio file into text data saying, "I plan to go to the hospital today," and then uses generative AI to summarize it as "I plan to go to the hospital." The summary data is stored in a database, and a notification is sent to the user device saying, "The elderly person plans to go to the hospital today." The user checks the notification on their smartphone and plays the original audio data or detailed text data as needed.

[0203] Sample prompt sentence

[0204] "Convert voice data to text and notify me if there is an anomaly."

[0205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0206] Step 1:

[0207] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation to begin. The input here is the scheduled time, and the server checks that time and sends the trigger. The server outputs an HTTP request. Specifically, the server sends an HTTP request to the elderly device saying, "Please start a conversation."

[0208] Step 2:

[0209] The elderly device activates an AI avatar based on the received trigger. The AI ​​avatar speaks to the elderly person, saying, "Good morning. What do you have planned for today?" The input here is an HTTP request sent from the server, and the device outputs an audio message prompting the elderly person to respond. Specifically, the audio message is played from the speaker on the elderly device.

[0210] Step 3:

[0211] In response to the AI ​​avatar's question, the elderly responds, "I plan to go to the hospital today." The elderly device records this conversation. The input for this step is the elderly's speech, and the output of the device is a recorded audio file. Specifically, the microphone inside the device picks up the speech and saves it as an audio file.

[0212] Step 4:

[0213] After the recording is completed, the elderly device sends the audio data to the server. The input is the audio file, and the server outputs the received audio data. Specifically, the device converts the audio file into an appropriate format and uploads it to the server.

[0214] Step 5:

[0215] The server converts the received audio data into text data using Google's Speech-to-Text API. The input is an audio file and the output is text data. Specifically, the server sends the audio file to the API and receives the returned text data.

[0216] Step 6:

[0217] The server summarizes the converted text data using a generative AI model. The input is the converted text data, and the output is the summarized text. Specifically, the server invokes a generative AI model (e.g., GPT) to generate a summary.

[0218] Step 7:

[0219] The server extracts specific keywords from the summarized text and detects anomalies. The input is the summarized text, and the output is the anomaly detection result (e.g., whether the keyword was found or not). Specifically, the server compares the keyword list with the summarized text to determine whether an anomaly exists.

[0220] Step 8:

[0221] If an abnormality is detected, the server will send a notification to the emergency contact. The input is the result of the abnormality detection, and the output is an emergency notification email. Specifically, the server uses smtplib to send an email to the emergency contact.

[0222] Step 9:

[0223] The summarized data is stored in a database. The input is the summarized text and the output is a database item. In concrete terms, the server writes the summarized text to the database.

[0224] Step 10:

[0225] Once the data has been saved to the database, the server sends the summary data to the user's device. The input is the summary data, and the output is a push notification or email. Specifically, the server sends a push notification to the user's smartphone or sends an email.

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] System Overview

[0228] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides more detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[0229] Specific examples of system programs and processing

[0230] 1. The server sends a conversation start trigger

[0231] The server uses a scheduler function to send a conversation start trigger to the elderly device at a set time every day, for example, by sending an HTTP request at 9:00 a.m. and 6:00 p.m.

[0232] Specific examples

[0233] The server sends a trigger to start a conversation at 9 a.m. This notifies the elderly person's device and activates the AI ​​avatar.

[0234] 2. Initiating and recording conversations with senior devices

[0235] Based on the received trigger, an AI avatar app on the senior's device is launched. The app speaks to the senior, asking, "Good morning. What do you have planned for today?" The device automatically turns on the microphone to record the senior's response and continues recording until the response is complete.

[0236] Specific examples

[0237] The elderly person responds, "I'm going to the hospital today." The conversation is recorded.

[0238] 3. Sending voice data to the server and emotion engine

[0239] After recording is complete, the audio data is automatically sent from the device to the server, which then sends it to the emotion engine. The server receives the audio data in the appropriate format and passes it to the generative AI for processing.

[0240] Specific examples

[0241] Once the recording is complete, the smartphone sends the audio file to the server and emotion engine, which receives it and begins analyzing it.

[0242] 4. Speech data text conversion and sentiment analysis

[0243] The server converts the voice data into text using generative AI. At the same time, the emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This consists of a processing part using a voice recognition engine and an emotion recognition algorithm.

[0244] Specific examples

[0245] From the speech data "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated, and the emotion engine recognizes emotional states such as "I'm worried."

[0246] 5. Text Summarization and Sentiment Integration

[0247] The generated text data is summarized. Emotional data recognized by the emotion engine is also processed at the same time and added to the final summary data. The generative AI extracts important information and keywords from the text data and creates a concise summary.

[0248] Specific examples

[0249] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and the emotional state of "I am worried" is integrated.

[0250] 6. Database storage of summary data and emotion data

[0251] The summarized text data and recognized emotion data are stored in a database, allowing for easy search and reference of the information in the future.

[0252] Specific examples

[0253] The summary "I plan to go to the hospital" and the emotion "I'm worried" are stored in a database.

[0254] 7. Notifications to User Devices

[0255] Once the saving is complete, the server notifies the user device that the summary data and emotion data have been generated, either via email or a push notification in the app.

[0256] Specific examples

[0257] The server notifies the user of the generated summary data and emotion data. The user receives a notification on their smartphone saying, "An elderly person is planning to go to the hospital today and is worried."

[0258] 8. User confirmation of summary data and emotion data

[0259] Users will receive notifications and can check summary and emotional data through a dedicated app or web portal, allowing them to understand the elderly person's daily condition and emotional state, and to view detailed text and voice data as needed.

[0260] Specific examples

[0261] The child opens their smartphone and checks the summary data and emotional state of "I plan to go to the hospital" and "I'm worried" on a dedicated app. If necessary, the original audio data and detailed text can also be checked.

[0262] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[0263] The processing flow will be explained below.

[0264] Step 1:

[0265] The server uses a scheduler function to send a trigger to start a conversation to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[0266] Step 2:

[0267] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[0268] Step 3:

[0269] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[0270] Step 4:

[0271] The device sends the recorded audio data to the server and emotion engine. At this stage, the audio data is uploaded to the server and emotion engine in an appropriate format (e.g., MP3 or WAV file).

[0272] Step 5:

[0273] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[0274] Step 6:

[0275] The emotion engine analyzes the voice data and recognizes the emotions of the elderly. The emotion recognition algorithm analyzes the tone, speed, and pauses of the voice to identify the emotional state (e.g., happy, sad, angry, etc.).

[0276] Step 7:

[0277] The server summarizes the generated text data. The generative AI extracts important information and keywords from the text data and creates a concise summary. At the same time, the emotion engine adds emotional data recognized by the summary.

[0278] Step 8:

[0279] The server stores the summary data and emotional data in a database. The summary sentence and emotional state are registered as database entries, and the information is stored in a form that allows for easy search and retrieval in the future.

[0280] Step 9:

[0281] The server notifies the user device that the summary data and emotion data have been generated and saved. Notifications are sent via email or app push notifications.

[0282] Step 10:

[0283] Users receive notifications and can view summary and emotion data via a dedicated app or web portal. Users can view summary data and see detailed text, audio, and emotional states.

[0284] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[0285] Example 2

[0286] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0287] When elderly people live alone, it is difficult for their families to keep track of their situation and emotional state. As a result, elderly people may not receive appropriate support and care, increasing the risk of feeling lonely and anxious. Furthermore, conventional communication systems simply transmit voice data, and lack the ability to analyze their emotional state or provide detailed summary information, making them insufficient for their families.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0289] In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for performing emotion analysis, means for summarizing the text data, means for storing the summarized data and emotion data in a database, and means for transmitting a notification of the summarized data and emotion data to the user device. This enables family members to appropriately understand the situation and emotional state of the elderly person and to quickly provide appropriate support.

[0290] "Senior Device" means a personal digital assistant or mobile electronic device used by an elderly person.

[0291] "Means for starting a conversation" refers to a function that periodically sends a trigger to start a conversation to the elderly person's device.

[0292] "Means for transmitting voice data" refers to the function of transmitting recorded voice data from the elderly person device to the server and the emotion analysis engine.

[0293] "Means for converting voice data into text" refers to a function that converts voice data into text information using a generative model.

[0294] "Means for performing emotion analysis" refers to the function of recognizing the emotional state of elderly people from voice data using an emotion analysis engine.

[0295] "Means for summarizing text data" refers to a function for concisely summarizing text data that has been converted using a generative model.

[0296] "Means for storing in a database" refers to a function for recording summarized text data and emotion data in a database.

[0297] "Means for sending notifications" refers to the ability to send emails or push notifications to notify the user device that data generation has been completed.

[0298] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[0299] server

[0300] The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. This trigger is sent using an HTTP request and arrives at the elderly device at the specified time. For example, you can use Crontab or Windows Task Scheduler to set it to send the trigger at 9:00 a.m. and 6:00 p.m.

[0301] Terminals (elderly devices)

[0302] When the elderly person's device receives a conversation start trigger, the AI ​​avatar app automatically launches. The app speaks to the elderly person, saying, "Good morning. What are your plans for today?" When the elderly person begins to respond, the device's microphone automatically turns on and continues recording until the response is completed. The recorded audio data is saved as an audio file.

[0303] Sending audio data

[0304] Once the recording is complete, the elderly device sends the audio data to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and simultaneously sending the same audio data to the emotion engine.

[0305] Analysis of voice data and conversion to text

[0306] The server converts the received voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson). In parallel, an emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This process is performed by a speech recognition engine and emotion recognition algorithm (e.g., Azure® Cognitive Services Emotion API).

[0307] Summarization and emotional data integration

[0308] The generated text data is analyzed on the server to extract important information and create a summary. Emotional data obtained from an emotion engine is also integrated into the summary. This is done using a generative AI model (e.g., GPT-4 (registered trademark)).

[0309] Data Retention and Notification

[0310] The summarized text data and emotion data are stored in a database (e.g., MySQL, MongoDB). This allows for easy search and reference of the data later. Once the data has been saved, the server sends a notification to the user's device. This notification can be via email or push notification and includes the summary data and emotion data.

[0311] Verification by user device

[0312] Users receive notifications and can view summary and emotion data through a dedicated app or web portal, as well as the original audio data and detailed text if needed.

[0313] Specific examples

[0314] One morning, the server sends a conversation start trigger at a set time, and the elderly person's smartwatch receives a notification, launching the AI ​​avatar app. The app asks, "Good morning. What are your plans for today?" and the elderly person replies, "I plan to go to the hospital today." This response is recorded, and the voice data is sent to the server and emotion engine. The server converts the voice data into text, and the emotion engine analyzes the emotional state as "I'm worried." The generated text and emotion data are integrated and summarized as "I'm planning to go to the hospital" and "I'm worried." This is stored in a database, and a notification is sent to the user's device. The user can check this information in a dedicated app and provide appropriate support for the elderly person's condition.

[0315] The above is a specific embodiment for carrying out the invention.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] Input: The server receives date and time information based on the scheduler settings.

[0319] Processing: The server sends a conversation start trigger to the elderly device based on the date and time information. It uses a scheduler function (e.g., Crontab, Windows Task Scheduler) to send an HTTP request at the specified time.

[0320] Output: A conversation start trigger is sent to the senior device.

[0321] Specific operation: The server sends an HTTP request (e.g. POST / trigger) at 9:00 AM, notifying the smartwatch that a conversation has started.

[0322] Step 2:

[0323] Input: The senior device receives a conversation start trigger.

[0324] Processing: The senior's device (smartwatch or smartphone) receives a notification and automatically launches the AI ​​avatar app, which asks, "Good morning. What are your plans for today?"

[0325] Output: A voice response from the elderly person is obtained.

[0326] Specific operation: Based on the trigger received by the smartwatch, the app will launch and say, "Good morning. What are your plans for today?"

[0327] Step 3:

[0328] Input: Elderly person answers verbally to the AI ​​avatar.

[0329] Processing: The microphone on the senior's device is automatically turned on to record the voice response. The recording is saved as an audio file (e.g., .wav format) on the device.

[0330] Output: You will get the recorded audio file.

[0331] Specific actions: The elderly person answers, "I plan to go to the hospital today," and the audio is recorded.

[0332] Step 4:

[0333] Input: The recorded audio file is saved on the senior's device.

[0334] Processing: Once the elderly device finishes recording, it sends the audio file to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and emotion engine.

[0335] Output: The audio file is sent to the server and emotion engine.

[0336] Specific operation: The smartphone sends the audio file to the server and emotion engine via an HTTP POST request (e.g., POST / upload).

[0337] Step 5:

[0338] Input: The server receives the audio file.

[0339] Processing: The server converts the voice data into text using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson), and an emotion engine analyzes the voice data to recognize the emotional state. This is done using a speech recognition engine and emotion recognition algorithms (e.g., Azure Cognitive Services Emotion API).

[0340] Output: The converted text data and emotion data are obtained.

[0341] Specific operation: The server inputs the audio file into the generative AI model, and at the same time obtains the text "I plan to go to the hospital today," the emotion engine analyzes it as "I'm worried."

[0342] Step 6:

[0343] Input: Acquired text data and sentiment data.

[0344] Processing: The server summarizes the text data and integrates it with sentiment data. A generative AI model (e.g., GPT-4) is used to concisely summarize the text, extract key information, and create a summary along with sentiment data.

[0345] Output: Summarized text data and integrated sentiment data.

[0346] Specific operation: The generative AI model summarizes the text data "I plan to go to the hospital today" into "I plan to go to the hospital" and integrates the emotion "I am worried."

[0347] Step 7:

[0348] Input: Summarized text data and sentiment data.

[0349] Processing: The server stores these data in a database. It executes queries to insert summary data and emotion data into the database (e.g., MySQL, MongoDB).

[0350] Output: The saved data is recorded in a database.

[0351] Specific operation: The server executes an SQL statement to insert the summary data "Planning to go to the hospital" and the emotion data "Worried" into the database.

[0352] Step 8:

[0353] Input: Summary data and sentiment data stored in a database.

[0354] Processing: The server notifies the user device that the save is complete. This notification can be sent via email or push notification.

[0355] Output: A notification is sent to the user device.

[0356] Specific operation: The server sends a push notification (e.g., Firebase Cloud Messaging) to the user's smartphone, informing them that "an elderly person is scheduled to go to the hospital today and is worried."

[0357] Step 9:

[0358] Input: Notifications received on the user device.

[0359] Processing: Users receive notifications and can review summary and emotion data through a dedicated app or web portal. They can also view the original audio data and detailed text if needed.

[0360] Output: The user checks the summary data and emotion data.

[0361] Specific operation: The child opens the dedicated app and checks the summary data "Plan to go to the hospital" and the emotion data "I'm worried." If necessary, the original audio data and detailed text are played and displayed.

[0362] (Application example 2)

[0363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] There is a problem that families who live far away from elderly people lack the means of communication to quickly and accurately grasp the elderly's living situation and emotional state. With conventional methods, it is difficult for family members to contact them regularly and to understand the elderly's emotional state. As a result, the elderly may feel lonely or may not receive appropriate support.

[0365] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for summarizing the text data, means for storing the summarized data in a database, means for transmitting a notification of the summarized data to the user device, and means for performing emotion analysis of the voice data. This enables family members to quickly and in detail understand the living situation and emotional state of the elderly person.

[0366] "Senior devices" are portable electronic devices used daily by seniors, including smartwatches and smartphones.

[0367] A "server" is a computer system that communicates with other devices over a network and processes, stores, and manages data.

[0368] "Emotion analysis" is a technology that analyzes voice data, text data, etc. to estimate and classify the speaker's emotional state.

[0369] A "user device" is a portable or stationary electronic device used by a family member, caregiver, or other person, including a smartphone or computer.

[0370] "Generative AI" is a type of artificial intelligence technology that includes algorithms for natural language processing and speech recognition, among other things.

[0371] The present invention comprises an elderly device used by the elderly in their daily lives, a user device used by family members living far away, and a server system that links these devices.

[0372] System configuration

[0373] Elderly device: A smartwatch or smartphone is used to send a trigger to start a conversation to the elderly at a set time and record the response voice.

[0374] User device: A smartphone or computer used by a family member or caregiver to receive notifications and view detailed data.

[0375] Server: A computer system that processes and stores data and runs programs for sentiment analysis and text conversion. Major software includes web frameworks such as Django and Flask, Google Cloud Speech-to-Text API, and TextBlob.

[0376] Processing flow

[0377] 1. Periodic transmission of conversation triggers: The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. For example, the server sends an HTTP request at 9:00 a.m. or 6:00 p.m.

[0378] Example: At 9:00 a.m., a trigger is sent from the server, causing an elderly person's smartphone to ask, "Good morning. What are your plans for today?"

[0379] 2. Conversation recording: The elderly device receives a trigger, and the AI ​​avatar app launches and speaks to the elderly. After the elderly responds, the audio is recorded and sent to the server.

[0380] Example: An elderly person replies, "I'm going to the hospital today." This is recorded and sent to the server.

[0381] 3. Audio data processing: The server converts the received audio data into text using a generative AI model, and then passes the same audio data to an emotion analysis engine to analyze the elderly person's emotional state.

[0382] Example: From the speech "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated and the emotional state "I am worried" is analyzed.

[0383] 4. Data Summarization: The generated text data is summarized and the emotional state is integrated. Software such as TextBlob is used to extract important information and keywords and compile them into a concise summary.

[0384] Example: The text "I'm going to the hospital today" is summarized as "I'm going to the hospital" and the emotion "I'm worried" is integrated.

[0385] 5. Notification to user device: The text data and emotion data stored in the database are sent to the user's device via email or app push notification.

[0386] Example: The server notifies the generated data, and a family member receives a notification on their smartphone saying, "An elderly person is scheduled to go to the hospital today, and they are worried."

[0387] 6. User review: Users can review summary data and emotion data through a dedicated app or web portal. They can also review detailed text and original audio data if necessary.

[0388] Example: A child opens their smartphone and checks summary data and emotional state such as "I'm going to the hospital" and "I'm worried" in a dedicated app.

[0389] Examples and prompts

[0390] Example: If an elderly person answers, "I plan to go to the hospital today," the text "I plan to go to the hospital" and the emotion data "I'm worried" are generated.

[0391] Example prompt sentence:

[0392] Example prompts for text conversion and summarization:

[0393] Summarize the following sentence: "I'm going to the hospital today."

[0394] Sentiment analysis prompt example:

[0395] Analyze the sentiment of the following sentence: "I'm going to the hospital today."

[0396] With the above configuration, the present invention realizes an effective solution for notifying family members of the elderly's living situation and emotional state in detail and quickly, and for providing appropriate support.

[0397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0398] Step 1:

[0399] The server uses a scheduler function to periodically send a conversation start trigger to the elderly device at a set time. Here, the server's input is the scheduler setting, and its output is a conversation start trigger (HTTP request). This trigger starts a conversation on the elderly device.

[0400] Step 2:

[0401] Based on the received conversation start trigger, the elderly device launches the AI ​​avatar app and speaks to the elderly, saying, "Good morning. What are your plans for today?" Here, the input of the elderly device is the conversation start trigger, and its output is the start of voice recording and the subsequent voice data (the elderly person's response).

[0402] Step 3:

[0403] The elderly device sends the recorded voice data to the server, which receives the voice data and treats it as input. The output is a trigger to store the voice data and start the voice recognition process.

[0404] Step 4:

[0405] The server converts the voice data into text data using a generative AI model. The input in this process is voice data, and the output is the corresponding text data. In a specific example, the text "I plan to go to the hospital today" is generated from the voice "I plan to go to the hospital today."

[0406] Step 5:

[0407] The server then sends the text data to an emotion analysis engine to analyze the emotional state. The input is text data, and the output is the analyzed emotional state (e.g., "worried"). This allows the elderly person's emotional state to be estimated.

[0408] Step 6:

[0409] The text data is summarized and integrated with the emotional state. To do this, the server sends the text data to the summarization engine as input. The output is the integrated summary data. In a specific example, "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and integrated with the emotion "I'm worried."

[0410] Step 7:

[0411] The server stores this text data and emotion data in a database. The input is the integrated summary data, and the output is the results stored in the database. The stored data can be searched and analyzed later.

[0412] Step 8:

[0413] After the storage is completed, the server sends a notification of the summary data and emotion data to the user device, where the server's input is the stored data and its output is a notification message (push notification or email).

[0414] Step 9:

[0415] The user device displays the received notification, and the family member can check the summary data and emotional state through a dedicated app or web portal. The input here is the notification message, and the output is the displayed summary data and emotional state. In a specific example, the family member can open the smartphone app and check the data for "Plan to go to the hospital" and "Worried."

[0416] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0417] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0418] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0419] [Second embodiment]

[0420] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0421] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0422] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0423] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0424] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0425] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0426] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0427] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0430] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0431] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0432] System Overview

[0433] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: elderly people's devices (smartwatches and smartphones), a server, and user devices (smartphones and PCs).

[0434] Specific examples of system programs and processing

[0435] 1. The server sends a conversation start trigger

[0436] The server sends a trigger to start a conversation to the elderly device at a set time every day. For example, if the trigger is set to 9:00 a.m. and 6:00 p.m., the server uses a scheduler function to send an HTTP request to the specified device.

[0437] Specific examples

[0438] The server sends a trigger to start a conversation at 9 a.m. A notification is sent to the elderly person's device, activating the AI ​​avatar.

[0439] 2. Initiating and recording conversations with senior devices

[0440] Based on the received trigger, the AI ​​avatar app on the elderly person's device launches and speaks to the user (elderly person), saying, "Good morning. What are your plans for today?" The device then records the elderly person's response.

[0441] Specific examples

[0442] The elderly person responds, "I plan to go to the hospital today." This conversation is recorded by the elderly device.

[0443] 3. Sending audio data to the server

[0444] After the recording is complete, the audio data will be automatically sent to the server, and the device will upload the audio data to the server in the appropriate format.

[0445] Specific examples

[0446] After recording is complete, the smartphone sends the audio file to the server, which receives and stores it.

[0447] 4. Converting Audio Data to Text

[0448] The server converts the received voice data into text data using generative AI, which is the processing part of the speech recognition engine.

[0449] Specific examples

[0450] The voice data "I plan to go to the hospital today" is converted into the text "I plan to go to the hospital today."

[0451] 5. Summarizing Text Data

[0452] The converted text data is further summarized by a generative AI on the server, which extracts important information and generates a short summary.

[0453] Specific examples

[0454] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0455] 6. Database storage of summary data

[0456] The summarized data is stored in a database, which allows for easy retrieval and reference later.

[0457] Specific examples

[0458] The summarized "hospital visit schedule" is stored in a database.

[0459] 7. Notifications to User Devices

[0460] Once saved, the server notifies the user device that the summary data has been generated, either via email or app push notification.

[0461] Specific examples

[0462] The server notifies the user of the generated summary data, and the user's smartphone receives a notification that "an elderly person is scheduled to go to the hospital today."

[0463] 8. User review of summary data

[0464] Users receive notifications and can access summary data via a dedicated app or web portal, along with detailed text and audio data.

[0465] Specific examples

[0466] The child opens their smartphone and checks a summary of the "hospital appointment" in a dedicated app, and if necessary, can further review the original audio data and detailed text.

[0467] As described above, this system enables regular and efficient communication between the elderly and their families, allowing family members to quickly understand the elderly's daily condition and provide the necessary support.

[0468] The processing flow will be explained below.

[0469] Step 1:

[0470] The server uses the scheduler function to send a conversation start trigger to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[0471] Step 2:

[0472] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[0473] Step 3:

[0474] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[0475] Step 4:

[0476] The device sends the recorded audio data to the server, where it is uploaded to the server in the appropriate format (e.g., MP3 or WAV file).

[0477] Step 5:

[0478] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[0479] Step 6:

[0480] The server summarizes the generated text data. At this stage, the generative AI extracts important information and keywords from the text data and creates a compact summary.

[0481] Step 7:

[0482] The server stores the abstract data in a database, where the abstract is registered as a database entry and stored for future search and retrieval.

[0483] Step 8:

[0484] The server notifies the user device that the summary data has been generated and saved, either via email or app push notification.

[0485] Step 9:

[0486] Users receive notifications and can check summary data via a dedicated app or web portal. By viewing the summary data, users can understand the elderly person's daily condition and, if necessary, can also view detailed text and audio data.

[0487] This is the process flow of the entire system, which enables effective communication between the elderly and their families.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] Family members who live far away from their elderly loved ones face the challenge of regular and effective communication. In particular, there are few ways to keep track of the elderly's health status and schedules in real time, which can delay emergency response. Furthermore, if the elderly person does not provide information voluntarily, family members have no way of knowing their elderly loved one's condition.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes: means for sending a trigger to start a conversation to the elderly device at a specified time; means for automatically sending recorded conversation audio data from the elderly device to the server; means for converting the audio data into text data using a generative AI model; means for summarizing the text data using the generative AI model; means for storing the summarized data in a relational database; means for notifying the user device of the completion of generation of the summarized data; and means for the user to check the summarized data through a dedicated app or web portal. This enables regular and efficient communication between the elderly and their family members. Family members can quickly understand the elderly's daily condition and provide necessary support.

[0493] "Senior device" refers to an electronic device used daily by seniors, including a smartwatch or smartphone.

[0494] A "trigger" refers to a signal or instruction to initiate a specific action, in this case sent from the server to the senior device to encourage the start of a conversation.

[0495] "Audio Data" means conversations recorded on an Elderly Device and stored in electronic format.

[0496] A "generative AI model" refers to an artificial intelligence model that uses machine learning and is used to automatically perform specific tasks (such as speech recognition and text summarization).

[0497] "Text data" refers to written information converted from voice data using a generative AI model.

[0498] A "summary" is a short summary created by analyzing text data and extracting only the important information and content.

[0499] A "relational database" is a database system that stores data in a structured tabular format, and allows data to be searched and manipulated using query languages ​​such as SQL.

[0500] "Notification" refers to a message or alert that notifies the user of a particular event or the result of a process, in this case notifying the user that summary data generation has been completed.

[0501] A "dedicated app" refers to a software application developed to provide specific functions or services, which users use on their smartphones or PCs.

[0502] "Web portal" refers to a website entrance that allows users to access specific information or services over the Internet, and is accessed by a user using a browser.

[0503] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: an elderly person's device (such as a smartwatch or smartphone), a server, and a user device (such as a smartphone or PC).

[0504] System Components and Functions

[0505] 1. Elderly devices:

[0506] This refers to electronic devices used by seniors in their daily lives, including smartwatches and smartphones, such as the Apple Watch, Samsung Galaxy Watch, iPhone, and Android smartphones.

[0507] The device receives a trigger from the server and launches an AI avatar app (e.g., a custom skill for Google Assistant or Amazon Alexa).

[0508] 2. Server:

[0509] The server is composed of Linux servers and has the following functions:

[0510] A trigger is sent to the elderly device to start a conversation at a set time.

[0511] Receives voice data sent from the elderly device and stores it in an appropriate format.

[0512] Convert the voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text API, IBM Watson Speech to Text, etc.).

[0513] Summarize text data using generative AI models (e.g., OpenAI's GPT series).

[0514] Store the summarized data in a relational database (e.g., MySQL, PostgreSQL, etc.).

[0515] The completion of generation of the summary data is notified to the user device.

[0516] 3. User Device:

[0517] Family members of elderly people can access information through a dedicated app or web portal on their smartphones or PCs, and notification services use Firebase and AWS SNS.

[0518] Specific operation of the system

[0519] 1. Send a trigger to start a conversation:

[0520] The server uses a scheduler function such as crontab to send a trigger to start a conversation to the elderly device every morning at 9:00 and evening at 6:00. The trigger is sent using an HTTP request.

[0521] 2. Start and record the conversation:

[0522] Based on a trigger received from the server, the elderly person's device launches the AI ​​avatar app and asks, "Good morning. What are your plans for today?" If the elderly person replies, "I plan to go to the hospital today," the conversation is recorded.

[0523] 3. Sending audio data to the server:

[0524] Once the recording is complete, the senior's device uploads the audio file to a server using HTTPS, and the data is sent along with an authentication token for security reasons.

[0525] 4. Audio to text conversion:

[0526] The server converts the voice data into text using a generative AI model. For example, a speech saying "I plan to go to the hospital today" is converted directly into text.

[0527] 5. Summarizing text data:

[0528] Using the generative AI model, the server analyzes the text data, extracts key information, and generates a summary. For example, the text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0529] 6. Summary Data Storage and Notification:

[0530] The summarized data is stored in a relational database and immediately sent to the user's device, where they can receive notifications and view detailed text and audio data via a dedicated app or web portal.

[0531] Specific examples

[0532] (Example prompt) "What are your plans for today?" The elderly person responds, "I plan to go to the hospital today," and the AI ​​model summarizes this as "I plan to go to the hospital."

[0533] The child will receive a summary of their planned hospital visit notification in a dedicated app, and can view detailed text and audio data.

[0534] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0535] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0536] Step 1:

[0537] The server uses the scheduler function to send a conversation start trigger to the elderly device at the set times of 9:00 a.m. and 6:00 p.m. every day. For this purpose, the server generates an HTTP request and sends it to the elderly device. This process ensures that the elderly device receives a notification at the specified time.

[0538] Specifically, a script executed at a fixed time by the server's crontab generates and sends an HTTP request. The input is the time set by the scheduler, and the output is a trigger sent to the elderly device.

[0539] Step 2:

[0540] The elderly device launches the AI ​​avatar app based on the received trigger. At this time, the AI ​​avatar speaks to the elderly person, saying, "Good morning. What are your plans for today?" The elderly person responds, and the conversation is recorded by the elderly device. The input is the trigger from the server, and the output is the elderly person's voice data.

[0541] Specifically, when the elderly care device receives a trigger, the AI ​​avatar app automatically launches and speaks a pre-set prompt to the elderly. When the elderly responds, the voice data is recorded on the device.

[0542] Step 3:

[0543] After recording is completed, the elderly device sends the audio data to the server. This transmission uses HTTPS, and the audio data is uploaded to the server along with an authentication token. The input is the recorded audio data, and the output is the uploaded audio file to the server.

[0544] Specifically, the device temporarily stores the recorded audio data and then uploads the audio file to a server using the HTTPS protocol, using an authentication token for security during the upload process.

[0545] Step 4:

[0546] The server converts the received voice data into text data using a generative AI model (for example, Google Cloud Speech-to-Text API or IBM Watson Speech to Text). This process converts the voice data into a format that can be read as text. The input is voice data, and the output is the converted text data.

[0547] Specifically, the server sends the voice data to the AI ​​model and receives the resulting converted text data. The voice data, "I plan to go to the hospital today," is converted directly into text.

[0548] Step 5:

[0549] The server uses a generative AI model (such as OpenAI's GPT series) to summarize the text data, extracting important information and generating a short summary. The input is the converted text data, and the output is the summarized text data.

[0550] Specifically, the server sends text data to the AI ​​model and receives the summarized text data. For example, the text data "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0551] Step 6:

[0552] The server stores the summarized data in a relational database, which structures the data and makes it easier to search and reference later. The input is the summarized text data, and the output is storage in the database.

[0553] Specifically, the server uses an SQL query to store the summary data in a relational database. For example, the summary data "plan to go to the hospital" is stored as a specific entry in the database.

[0554] Step 7:

[0555] Once the server has finished storing the data, it uses a notification service (e.g., Firebase, AWS SNS, etc.) to send a push notification to the user device. The notification contains the summary data. The input is the summary data stored in the database, and the output is the notification sent to the user device.

[0556] Specifically, the server calls the notification service and sends a push notification to the user device stating, "An elderly person is scheduled to go to the hospital today."

[0557] Step 8:

[0558] Users receive notifications and can view summary data via a dedicated app or web portal, which also provides detailed text and audio data. The input is a notification to the user device, and the output is the user reviewing the data.

[0559] Specifically, the user opens the dedicated smartphone app and checks the summary data, such as "I plan to go to the hospital." If necessary, they can also read the original audio data and detailed text.

[0560] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0561] (Application example 1)

[0562] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0563] Elderly people face difficulties in communicating regularly and effectively with their families who live far away. There is also a lack of methods to quickly check on the safety and well-being of elderly people. Furthermore, there is a need for a method to quickly detect when an elderly person may be in an emergency and take appropriate action.

[0564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0565] In this invention, the server includes a means for periodically initiating a conversation with the elderly person's device, a means for converting voice data into text, and a means for extracting specific keywords from the voice data and detecting abnormalities, thereby enabling effective monitoring of the elderly person's condition and immediate response in the event of an emergency.

[0566] "Senior device" refers to an electronic device designed for use by seniors, such as a smartwatch or smartphone.

[0567] The "conversation initiation means" is a mechanism for periodically initiating a conversation with the elderly person device, and includes a function for sending a trigger at a specific timing.

[0568] The "voice data transmission means" is a mechanism for transmitting voice data of conversation acquired from the elderly person device to the server.

[0569] The "voice data text conversion means" is a function for converting transmitted voice data into text data, and uses a voice recognition engine.

[0570] The "text data summarizing means" is a function for summarizing the generated text data and extracting important information.

[0571] The "database storage means" is a mechanism for storing summarized data in a database so that it can be easily searched and referenced later.

[0572] The "notification sending means" is a function for notifying the user device of summarized data, and includes email and push notification.

[0573] The "specific keyword extraction means" is a function for extracting predetermined keywords from the voice data and detecting abnormalities based on the contents of the keywords.

[0574] The "anomaly detection means" is a mechanism for detecting anomalies based on extracted keywords, and is used to determine an emergency situation.

[0575] The "emergency contact notification means" is a function for sending a notification to an emergency contact when an abnormality is detected.

[0576] System Overview

[0577] This invention relates to a system that enables elderly people and their families living far away to communicate regularly and effectively. The system mainly comprises an elderly person device, a server, and a user device.

[0578] Hardware and software used

[0579] Elderly devices: Using a smartwatch or smartphone, which allows seniors easy access and regular notifications.

[0580] Server: We use a high-performance computer server to process and store audio data.

[0581] User devices: Provide an interface for family members to check on the daily activities of their elderly relatives using smartphones or PCs.

[0582] Software: Uses Google's Speech-to-Text API as the speech recognition engine, smtplib for sending emails, Python's schedule library for scheduling, and a generative AI model (e.g., OpenAI's GPT) for text processing.

[0583] System Operation

[0584] 1. Start regular conversations

[0585] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation. When the elderly device receives this trigger, the AI ​​avatar starts the interaction and speaks to the elderly.

[0586] 2. Collection and transmission of voice data

[0587] The elderly person's device records the conversation and sends the audio data to the server, where it is uploaded in an appropriate format.

[0588] 3. Speech to text conversion and summarization

[0589] The server uses Google's Speech-to-Text API to convert the received audio data into text data, and then uses a generative AI model to summarize the text data.

[0590] 4. Anomaly Detection

[0591] The server extracts specific keywords from the converted text data and detects anomalies. For example, if keywords such as "help" or "in trouble" are included, the server will determine that there is an anomaly.

[0592] 5. Notification to emergency contacts

[0593] If an anomaly is detected, the server will notify emergency contacts via email using smtplib.

[0594] 6. Notification and confirmation to user devices

[0595] The summarized data is stored in a database and then sent to the user's device via push notification or email. Family members can receive these notifications on their device and check the elderly person's condition.

[0596] Specific examples

[0597] The elderly person's device receives a conversation trigger at 9:00 a.m., asking, "Good morning. What are your plans for today?" The elderly person replies, "I plan to go to the hospital today." This conversation is recorded and sent to a server as an audio file. The server converts the audio file into text data saying, "I plan to go to the hospital today," and then uses generative AI to summarize it as "I plan to go to the hospital." The summary data is stored in a database, and a notification is sent to the user device saying, "The elderly person plans to go to the hospital today." The user checks the notification on their smartphone and plays the original audio data or detailed text data as needed.

[0598] Sample prompt sentence

[0599] "Convert voice data to text and notify me if there is an anomaly."

[0600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0601] Step 1:

[0602] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation to begin. The input here is the scheduled time, and the server checks that time and sends the trigger. The server outputs an HTTP request. Specifically, the server sends an HTTP request to the elderly device saying, "Please start a conversation."

[0603] Step 2:

[0604] The elderly device activates an AI avatar based on the received trigger. The AI ​​avatar speaks to the elderly person, saying, "Good morning. What do you have planned for today?" The input here is an HTTP request sent from the server, and the device outputs an audio message prompting the elderly person to respond. Specifically, the audio message is played from the speaker on the elderly device.

[0605] Step 3:

[0606] In response to the AI ​​avatar's question, the elderly responds, "I plan to go to the hospital today." The elderly device records this conversation. The input for this step is the elderly's speech, and the output of the device is a recorded audio file. Specifically, the microphone inside the device picks up the speech and saves it as an audio file.

[0607] Step 4:

[0608] After the recording is completed, the elderly device sends the audio data to the server. The input is the audio file, and the server outputs the received audio data. Specifically, the device converts the audio file into an appropriate format and uploads it to the server.

[0609] Step 5:

[0610] The server converts the received audio data into text data using Google's Speech-to-Text API. The input is an audio file and the output is text data. Specifically, the server sends the audio file to the API and receives the returned text data.

[0611] Step 6:

[0612] The server summarizes the converted text data using a generative AI model. The input is the converted text data, and the output is the summarized text. Specifically, the server invokes a generative AI model (e.g., GPT) to generate a summary.

[0613] Step 7:

[0614] The server extracts specific keywords from the summarized text and detects anomalies. The input is the summarized text, and the output is the anomaly detection result (e.g., whether the keyword was found or not). Specifically, the server compares the keyword list with the summarized text to determine whether an anomaly exists.

[0615] Step 8:

[0616] If an abnormality is detected, the server will send a notification to the emergency contact. The input is the result of the abnormality detection, and the output is an emergency notification email. Specifically, the server uses smtplib to send an email to the emergency contact.

[0617] Step 9:

[0618] The summarized data is stored in a database. The input is the summarized text and the output is a database item. In concrete terms, the server writes the summarized text to the database.

[0619] Step 10:

[0620] Once the data has been saved to the database, the server sends the summary data to the user's device. The input is the summary data, and the output is a push notification or email. Specifically, the server sends a push notification to the user's smartphone or sends an email.

[0621] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0622] System Overview

[0623] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides more detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[0624] Specific examples of system programs and processing

[0625] 1. The server sends a conversation start trigger

[0626] The server uses a scheduler function to send a conversation start trigger to the elderly device at a set time every day, for example, by sending an HTTP request at 9:00 a.m. and 6:00 p.m.

[0627] Specific examples

[0628] The server sends a trigger to start a conversation at 9 a.m. This notifies the elderly person's device and activates the AI ​​avatar.

[0629] 2. Initiating and recording conversations with senior devices

[0630] Based on the received trigger, an AI avatar app on the senior's device is launched. The app speaks to the senior, asking, "Good morning. What do you have planned for today?" The device automatically turns on the microphone to record the senior's response and continues recording until the response is complete.

[0631] Specific examples

[0632] The elderly person responds, "I'm going to the hospital today." The conversation is recorded.

[0633] 3. Sending voice data to the server and emotion engine

[0634] After recording is complete, the audio data is automatically sent from the device to the server, which then sends it to the emotion engine. The server receives the audio data in the appropriate format and passes it to the generative AI for processing.

[0635] Specific examples

[0636] Once the recording is complete, the smartphone sends the audio file to the server and emotion engine, which receives it and begins analyzing it.

[0637] 4. Speech data text conversion and sentiment analysis

[0638] The server converts the voice data into text using generative AI. At the same time, the emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This consists of a processing part using a voice recognition engine and an emotion recognition algorithm.

[0639] Specific examples

[0640] From the speech data "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated, and the emotion engine recognizes emotional states such as "I'm worried."

[0641] 5. Text Summarization and Sentiment Integration

[0642] The generated text data is summarized. Emotional data recognized by the emotion engine is also processed at the same time and added to the final summary data. The generative AI extracts important information and keywords from the text data and creates a concise summary.

[0643] Specific examples

[0644] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and the emotional state of "I am worried" is integrated.

[0645] 6. Database storage of summary data and emotion data

[0646] The summarized text data and recognized emotion data are stored in a database, allowing for easy search and reference of the information in the future.

[0647] Specific examples

[0648] The summary "I plan to go to the hospital" and the emotion "I'm worried" are stored in a database.

[0649] 7. Notifications to User Devices

[0650] Once the saving is complete, the server notifies the user device that the summary data and emotion data have been generated, either via email or a push notification in the app.

[0651] Specific examples

[0652] The server notifies the user of the generated summary data and emotion data. The user receives a notification on their smartphone saying, "An elderly person is planning to go to the hospital today and is worried."

[0653] 8. User confirmation of summary data and emotion data

[0654] Users will receive notifications and can check summary and emotional data through a dedicated app or web portal, allowing them to understand the elderly person's daily condition and emotional state, and to view detailed text and voice data as needed.

[0655] Specific examples

[0656] The child opens their smartphone and checks the summary data and emotional state of "I plan to go to the hospital" and "I'm worried" on a dedicated app. If necessary, the original audio data and detailed text can also be checked.

[0657] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[0658] The processing flow will be explained below.

[0659] Step 1:

[0660] The server uses a scheduler function to send a trigger to start a conversation to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[0661] Step 2:

[0662] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[0663] Step 3:

[0664] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[0665] Step 4:

[0666] The device sends the recorded audio data to the server and emotion engine. At this stage, the audio data is uploaded to the server and emotion engine in an appropriate format (e.g., MP3 or WAV file).

[0667] Step 5:

[0668] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[0669] Step 6:

[0670] The emotion engine analyzes the voice data and recognizes the emotions of the elderly. The emotion recognition algorithm analyzes the tone, speed, and pauses of the voice to identify the emotional state (e.g., happy, sad, angry, etc.).

[0671] Step 7:

[0672] The server summarizes the generated text data. The generative AI extracts important information and keywords from the text data and creates a concise summary. At the same time, the emotion engine adds emotional data recognized by the summary.

[0673] Step 8:

[0674] The server stores the summary data and emotional data in a database. The summary sentence and emotional state are registered as database entries, and the information is stored in a form that allows for easy search and retrieval in the future.

[0675] Step 9:

[0676] The server notifies the user device that the summary data and emotion data have been generated and saved. Notifications are sent via email or app push notifications.

[0677] Step 10:

[0678] Users receive notifications and can view summary and emotion data via a dedicated app or web portal. Users can view summary data and see detailed text, audio, and emotional states.

[0679] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[0680] Example 2

[0681] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] When elderly people live alone, it is difficult for their families to keep track of their situation and emotional state. As a result, elderly people may not receive appropriate support and care, increasing the risk of feeling lonely and anxious. Furthermore, conventional communication systems simply transmit voice data, and lack the ability to analyze their emotional state or provide detailed summary information, making them insufficient for their families.

[0683] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0684] In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for performing emotion analysis, means for summarizing the text data, means for storing the summarized data and emotion data in a database, and means for transmitting a notification of the summarized data and emotion data to the user device. This enables family members to appropriately understand the situation and emotional state of the elderly person and to quickly provide appropriate support.

[0685] "Senior Device" means a personal digital assistant or mobile electronic device used by an elderly person.

[0686] "Means for starting a conversation" refers to a function that periodically sends a trigger to start a conversation to the elderly person's device.

[0687] "Means for transmitting voice data" refers to the function of transmitting recorded voice data from the elderly person device to the server and the emotion analysis engine.

[0688] "Means for converting voice data into text" refers to a function that converts voice data into text information using a generative model.

[0689] "Means for performing emotion analysis" refers to the function of recognizing the emotional state of elderly people from voice data using an emotion analysis engine.

[0690] "Means for summarizing text data" refers to a function for concisely summarizing text data that has been converted using a generative model.

[0691] "Means for storing in a database" refers to a function for recording summarized text data and emotion data in a database.

[0692] "Means for sending notifications" refers to the ability to send emails or push notifications to notify the user device that data generation has been completed.

[0693] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[0694] server

[0695] The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. This trigger is sent using an HTTP request and arrives at the elderly device at the specified time. For example, you can use Crontab or Windows Task Scheduler to set it to send the trigger at 9:00 a.m. and 6:00 p.m.

[0696] Terminals (elderly devices)

[0697] When the elderly person's device receives a conversation start trigger, the AI ​​avatar app automatically launches. The app speaks to the elderly person, saying, "Good morning. What are your plans for today?" When the elderly person begins to respond, the device's microphone automatically turns on and continues recording until the response is completed. The recorded audio data is saved as an audio file.

[0698] Sending audio data

[0699] Once the recording is complete, the elderly device sends the audio data to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and simultaneously sending the same audio data to the emotion engine.

[0700] Analysis of voice data and conversion to text

[0701] The server converts the received voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson). In parallel, an emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This process is performed by a speech recognition engine and emotion recognition algorithm (e.g., Azure Cognitive Services Emotion API).

[0702] Summarization and emotional data integration

[0703] The generated text data is analyzed on the server to extract important information and create a summary, which is then integrated with emotion data obtained from an emotion engine using a generative AI model (e.g., GPT-4).

[0704] Data Retention and Notification

[0705] The summarized text data and emotion data are stored in a database (e.g., MySQL, MongoDB). This allows for easy search and reference of the data later. Once the data has been saved, the server sends a notification to the user's device. This notification can be via email or push notification and includes the summary data and emotion data.

[0706] Verification by user device

[0707] Users receive notifications and can view summary and emotion data through a dedicated app or web portal, as well as the original audio data and detailed text if needed.

[0708] Specific examples

[0709] One morning, the server sends a conversation start trigger at a set time, and the elderly person's smartwatch receives a notification, launching the AI ​​avatar app. The app asks, "Good morning. What are your plans for today?" and the elderly person replies, "I plan to go to the hospital today." This response is recorded, and the voice data is sent to the server and emotion engine. The server converts the voice data into text, and the emotion engine analyzes the emotional state as "I'm worried." The generated text and emotion data are integrated and summarized as "I'm planning to go to the hospital" and "I'm worried." This is stored in a database, and a notification is sent to the user's device. The user can check this information in a dedicated app and provide appropriate support for the elderly person's condition.

[0710] The above is a specific embodiment for carrying out the invention.

[0711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0712] Step 1:

[0713] Input: The server receives date and time information based on the scheduler settings.

[0714] Processing: The server sends a conversation start trigger to the elderly device based on the date and time information. It uses a scheduler function (e.g., Crontab, Windows Task Scheduler) to send an HTTP request at the specified time.

[0715] Output: A conversation start trigger is sent to the senior device.

[0716] Specific operation: The server sends an HTTP request (e.g. POST / trigger) at 9:00 AM, notifying the smartwatch that a conversation has started.

[0717] Step 2:

[0718] Input: The senior device receives a conversation start trigger.

[0719] Processing: The senior's device (smartwatch or smartphone) receives a notification and automatically launches the AI ​​avatar app, which asks, "Good morning. What are your plans for today?"

[0720] Output: A voice response from the elderly person is obtained.

[0721] Specific operation: Based on the trigger received by the smartwatch, the app will launch and say, "Good morning. What are your plans for today?"

[0722] Step 3:

[0723] Input: Elderly person answers verbally to the AI ​​avatar.

[0724] Processing: The microphone on the senior's device is automatically turned on to record the voice response. The recording is saved as an audio file (e.g., .wav format) on the device.

[0725] Output: You will get the recorded audio file.

[0726] Specific actions: The elderly person answers, "I plan to go to the hospital today," and the audio is recorded.

[0727] Step 4:

[0728] Input: The recorded audio file is saved on the senior's device.

[0729] Processing: Once the elderly device finishes recording, it sends the audio file to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and emotion engine.

[0730] Output: The audio file is sent to the server and emotion engine.

[0731] Specific operation: The smartphone sends the audio file to the server and emotion engine via an HTTP POST request (e.g., POST / upload).

[0732] Step 5:

[0733] Input: The server receives the audio file.

[0734] Processing: The server converts the voice data into text using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson), and an emotion engine analyzes the voice data to recognize the emotional state. This is done using a speech recognition engine and emotion recognition algorithms (e.g., Azure Cognitive Services Emotion API).

[0735] Output: The converted text data and emotion data are obtained.

[0736] Specific operation: The server inputs the audio file into the generative AI model, and at the same time obtains the text "I plan to go to the hospital today," the emotion engine analyzes it as "I'm worried."

[0737] Step 6:

[0738] Input: Acquired text data and sentiment data.

[0739] Processing: The server summarizes the text data and integrates it with sentiment data. A generative AI model (e.g., GPT-4) is used to concisely summarize the text, extract key information, and create a summary along with sentiment data.

[0740] Output: Summarized text data and integrated sentiment data.

[0741] Specific operation: The generative AI model summarizes the text data "I plan to go to the hospital today" into "I plan to go to the hospital" and integrates the emotion "I am worried."

[0742] Step 7:

[0743] Input: Summarized text data and sentiment data.

[0744] Processing: The server stores these data in a database. It executes queries to insert summary data and emotion data into the database (e.g., MySQL, MongoDB).

[0745] Output: The saved data is recorded in a database.

[0746] Specific operation: The server executes an SQL statement to insert the summary data "Planning to go to the hospital" and the emotion data "Worried" into the database.

[0747] Step 8:

[0748] Input: Summary data and sentiment data stored in a database.

[0749] Processing: The server notifies the user device that the save is complete. This notification can be sent via email or push notification.

[0750] Output: A notification is sent to the user device.

[0751] Specific operation: The server sends a push notification (e.g., Firebase Cloud Messaging) to the user's smartphone, informing them that "an elderly person is scheduled to go to the hospital today and is worried."

[0752] Step 9:

[0753] Input: Notifications received on the user device.

[0754] Processing: Users receive notifications and can review summary and emotion data through a dedicated app or web portal. They can also view the original audio data and detailed text if needed.

[0755] Output: The user checks the summary data and emotion data.

[0756] Specific operation: The child opens the dedicated app and checks the summary data "Plan to go to the hospital" and the emotion data "I'm worried." If necessary, the original audio data and detailed text are played and displayed.

[0757] (Application example 2)

[0758] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0759] There is a problem that families who live far away from elderly people lack the means of communication to quickly and accurately grasp the elderly's living situation and emotional state. With conventional methods, it is difficult for family members to contact them regularly and to understand the elderly's emotional state. As a result, the elderly may feel lonely or may not receive appropriate support.

[0760] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for summarizing the text data, means for storing the summarized data in a database, means for transmitting a notification of the summarized data to the user device, and means for performing emotion analysis of the voice data. This enables family members to quickly and in detail understand the living situation and emotional state of the elderly person.

[0761] "Senior devices" are portable electronic devices used daily by seniors, including smartwatches and smartphones.

[0762] A "server" is a computer system that communicates with other devices over a network and processes, stores, and manages data.

[0763] "Emotion analysis" is a technology that analyzes voice data, text data, etc. to estimate and classify the speaker's emotional state.

[0764] A "user device" is a portable or stationary electronic device used by a family member, caregiver, or other person, including a smartphone or computer.

[0765] "Generative AI" is a type of artificial intelligence technology that includes algorithms for natural language processing and speech recognition, among other things.

[0766] The present invention comprises an elderly device used by the elderly in their daily lives, a user device used by family members living far away, and a server system that links these devices.

[0767] System configuration

[0768] Elderly device: A smartwatch or smartphone is used to send a trigger to start a conversation to the elderly at a set time and record the response voice.

[0769] User device: A smartphone or computer used by a family member or caregiver to receive notifications and view detailed data.

[0770] Server: A computer system that processes and stores data and runs programs for sentiment analysis and text conversion. Major software includes web frameworks such as Django and Flask, Google Cloud Speech-to-Text API, and TextBlob.

[0771] Processing flow

[0772] 1. Periodic transmission of conversation triggers: The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. For example, the server sends an HTTP request at 9:00 a.m. or 6:00 p.m.

[0773] Example: At 9:00 a.m., a trigger is sent from the server, causing an elderly person's smartphone to ask, "Good morning. What are your plans for today?"

[0774] 2. Conversation recording: The elderly device receives a trigger, and the AI ​​avatar app launches and speaks to the elderly. After the elderly responds, the audio is recorded and sent to the server.

[0775] Example: An elderly person replies, "I'm going to the hospital today." This is recorded and sent to the server.

[0776] 3. Audio data processing: The server converts the received audio data into text using a generative AI model, and then passes the same audio data to an emotion analysis engine to analyze the elderly person's emotional state.

[0777] Example: From the speech "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated and the emotional state "I am worried" is analyzed.

[0778] 4. Data Summarization: The generated text data is summarized and the emotional state is integrated. Software such as TextBlob is used to extract important information and keywords and compile them into a concise summary.

[0779] Example: The text "I'm going to the hospital today" is summarized as "I'm going to the hospital" and the emotion "I'm worried" is integrated.

[0780] 5. Notification to user device: The text data and emotion data stored in the database are sent to the user's device via email or app push notification.

[0781] Example: The server notifies the generated data, and a family member receives a notification on their smartphone saying, "An elderly person is scheduled to go to the hospital today, and they are worried."

[0782] 6. User review: Users can review summary data and emotion data through a dedicated app or web portal. They can also review detailed text and original audio data if necessary.

[0783] Example: A child opens their smartphone and checks summary data and emotional state such as "I'm going to the hospital" and "I'm worried" in a dedicated app.

[0784] Examples and prompts

[0785] Example: If an elderly person answers, "I plan to go to the hospital today," the text "I plan to go to the hospital" and the emotion data "I'm worried" are generated.

[0786] Example prompt sentence:

[0787] Example prompts for text conversion and summarization:

[0788] Summarize the following sentence: "I'm going to the hospital today."

[0789] Sentiment analysis prompt example:

[0790] Analyze the sentiment of the following sentence: "I'm going to the hospital today."

[0791] With the above configuration, the present invention realizes an effective solution for notifying family members of the elderly's living situation and emotional state in detail and quickly, and for providing appropriate support.

[0792] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0793] Step 1:

[0794] The server uses a scheduler function to periodically send a conversation start trigger to the elderly device at a set time. Here, the server's input is the scheduler setting, and its output is a conversation start trigger (HTTP request). This trigger starts a conversation on the elderly device.

[0795] Step 2:

[0796] Based on the received conversation start trigger, the elderly device launches the AI ​​avatar app and speaks to the elderly, saying, "Good morning. What are your plans for today?" Here, the input of the elderly device is the conversation start trigger, and its output is the start of voice recording and the subsequent voice data (the elderly person's response).

[0797] Step 3:

[0798] The elderly device sends the recorded voice data to the server, which receives the voice data and treats it as input. The output is a trigger to store the voice data and start the voice recognition process.

[0799] Step 4:

[0800] The server converts the voice data into text data using a generative AI model. The input in this process is voice data, and the output is the corresponding text data. In a specific example, the text "I plan to go to the hospital today" is generated from the voice "I plan to go to the hospital today."

[0801] Step 5:

[0802] The server then sends the text data to an emotion analysis engine to analyze the emotional state. The input is text data, and the output is the analyzed emotional state (e.g., "worried"). This allows the elderly person's emotional state to be estimated.

[0803] Step 6:

[0804] The text data is summarized and integrated with the emotional state. To do this, the server sends the text data to the summarization engine as input. The output is the integrated summary data. In a specific example, "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and integrated with the emotion "I'm worried."

[0805] Step 7:

[0806] The server stores this text data and emotion data in a database. The input is the integrated summary data, and the output is the results stored in the database. The stored data can be searched and analyzed later.

[0807] Step 8:

[0808] After the storage is completed, the server sends a notification of the summary data and emotion data to the user device, where the server's input is the stored data and its output is a notification message (push notification or email).

[0809] Step 9:

[0810] The user device displays the received notification, and the family member can check the summary data and emotional state through a dedicated app or web portal. The input here is the notification message, and the output is the displayed summary data and emotional state. In a specific example, the family member can open the smartphone app and check the data for "Plan to go to the hospital" and "Worried."

[0811] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0813] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0814] [Third embodiment]

[0815] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0816] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0817] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0818] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0819] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0820] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0821] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0822] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0823] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0824] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0825] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0826] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0827] System Overview

[0828] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: elderly people's devices (smartwatches and smartphones), a server, and user devices (smartphones and PCs).

[0829] Specific examples of system programs and processing

[0830] 1. The server sends a conversation start trigger

[0831] The server sends a trigger to start a conversation to the elderly device at a set time every day. For example, if the trigger is set to 9:00 a.m. and 6:00 p.m., the server uses a scheduler function to send an HTTP request to the specified device.

[0832] Specific examples

[0833] The server sends a trigger to start a conversation at 9 a.m. A notification is sent to the elderly person's device, activating the AI ​​avatar.

[0834] 2. Initiating and recording conversations with senior devices

[0835] Based on the received trigger, the AI ​​avatar app on the elderly person's device launches and speaks to the user (elderly person), saying, "Good morning. What are your plans for today?" The device then records the elderly person's response.

[0836] Specific examples

[0837] The elderly person responds, "I plan to go to the hospital today." This conversation is recorded by the elderly device.

[0838] 3. Sending audio data to the server

[0839] After the recording is complete, the audio data will be automatically sent to the server, and the device will upload the audio data to the server in the appropriate format.

[0840] Specific examples

[0841] After recording is complete, the smartphone sends the audio file to the server, which receives and stores it.

[0842] 4. Converting Audio Data to Text

[0843] The server converts the received voice data into text data using generative AI, which is the processing part of the speech recognition engine.

[0844] Specific examples

[0845] The voice data "I plan to go to the hospital today" is converted into the text "I plan to go to the hospital today."

[0846] 5. Summarizing Text Data

[0847] The converted text data is further summarized by a generative AI on the server, which extracts important information and generates a short summary.

[0848] Specific examples

[0849] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0850] 6. Database storage of summary data

[0851] The summarized data is stored in a database, which allows for easy retrieval and reference later.

[0852] Specific examples

[0853] The summarized "hospital visit schedule" is stored in a database.

[0854] 7. Notifications to User Devices

[0855] Once saved, the server notifies the user device that the summary data has been generated, either via email or app push notification.

[0856] Specific examples

[0857] The server notifies the user of the generated summary data, and the user's smartphone receives a notification that "an elderly person is scheduled to go to the hospital today."

[0858] 8. User review of summary data

[0859] Users receive notifications and can access summary data via a dedicated app or web portal, along with detailed text and audio data.

[0860] Specific examples

[0861] The child opens their smartphone and checks a summary of the "hospital appointment" in a dedicated app, and if necessary, can further review the original audio data and detailed text.

[0862] As described above, this system enables regular and efficient communication between the elderly and their families, allowing family members to quickly understand the elderly's daily condition and provide the necessary support.

[0863] The processing flow will be explained below.

[0864] Step 1:

[0865] The server uses the scheduler function to send a conversation start trigger to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[0866] Step 2:

[0867] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[0868] Step 3:

[0869] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[0870] Step 4:

[0871] The device sends the recorded audio data to the server, where it is uploaded to the server in the appropriate format (e.g., MP3 or WAV file).

[0872] Step 5:

[0873] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[0874] Step 6:

[0875] The server summarizes the generated text data. At this stage, the generative AI extracts important information and keywords from the text data and creates a compact summary.

[0876] Step 7:

[0877] The server stores the abstract data in a database, where the abstract is registered as a database entry and stored for future search and retrieval.

[0878] Step 8:

[0879] The server notifies the user device that the summary data has been generated and saved, either via email or app push notification.

[0880] Step 9:

[0881] Users receive notifications and can check summary data via a dedicated app or web portal. By viewing the summary data, users can understand the elderly person's daily condition and, if necessary, can also view detailed text and audio data.

[0882] This is the process flow of the entire system, which enables effective communication between the elderly and their families.

[0883] Example 1

[0884] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0885] Family members who live far away from their elderly loved ones face the challenge of regular and effective communication. In particular, there are few ways to keep track of the elderly's health status and schedules in real time, which can delay emergency response. Furthermore, if the elderly person does not provide information voluntarily, family members have no way of knowing their elderly loved one's condition.

[0886] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0887] In this invention, the server includes: means for sending a trigger to start a conversation to the elderly device at a specified time; means for automatically sending recorded conversation audio data from the elderly device to the server; means for converting the audio data into text data using a generative AI model; means for summarizing the text data using the generative AI model; means for storing the summarized data in a relational database; means for notifying the user device of the completion of generation of the summarized data; and means for the user to check the summarized data through a dedicated app or web portal. This enables regular and efficient communication between the elderly and their family members. Family members can quickly understand the elderly's daily condition and provide necessary support.

[0888] "Senior device" refers to an electronic device used daily by seniors, including a smartwatch or smartphone.

[0889] A "trigger" refers to a signal or instruction to initiate a specific action, in this case sent from the server to the senior device to encourage the start of a conversation.

[0890] "Audio Data" means conversations recorded on an Elderly Device and stored in electronic format.

[0891] A "generative AI model" refers to an artificial intelligence model that uses machine learning and is used to automatically perform specific tasks (such as speech recognition and text summarization).

[0892] "Text data" refers to written information converted from voice data using a generative AI model.

[0893] A "summary" is a short summary created by analyzing text data and extracting only the important information and content.

[0894] A "relational database" is a database system that stores data in a structured tabular format, and allows data to be searched and manipulated using query languages ​​such as SQL.

[0895] "Notification" refers to a message or alert that notifies the user of a particular event or the result of a process, in this case notifying the user that summary data generation has been completed.

[0896] A "dedicated app" refers to a software application developed to provide specific functions or services, which users use on their smartphones or PCs.

[0897] "Web portal" refers to a website entrance that allows users to access specific information or services over the Internet, and is accessed by a user using a browser.

[0898] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: an elderly person's device (such as a smartwatch or smartphone), a server, and a user device (such as a smartphone or PC).

[0899] System Components and Functions

[0900] 1. Elderly devices:

[0901] This refers to electronic devices used by seniors in their daily lives, including smartwatches and smartphones, such as the Apple Watch, Samsung Galaxy Watch, iPhone, and Android smartphones.

[0902] The device receives a trigger from the server and launches an AI avatar app (e.g., a custom skill for Google Assistant or Amazon Alexa).

[0903] 2. Server:

[0904] The server is composed of Linux servers and has the following functions:

[0905] A trigger is sent to the elderly device to start a conversation at a set time.

[0906] Receives voice data sent from the elderly device and stores it in an appropriate format.

[0907] Convert the voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text API, IBM Watson Speech to Text, etc.).

[0908] Summarize text data using generative AI models (e.g., OpenAI's GPT series).

[0909] Store the summarized data in a relational database (e.g., MySQL, PostgreSQL, etc.).

[0910] The completion of generation of the summary data is notified to the user device.

[0911] 3. User Device:

[0912] Family members of elderly people can access information through a dedicated app or web portal on their smartphones or PCs, and notification services use Firebase and AWS SNS.

[0913] Specific operation of the system

[0914] 1. Send a trigger to start a conversation:

[0915] The server uses a scheduler function such as crontab to send a trigger to start a conversation to the elderly device every morning at 9:00 and evening at 6:00. The trigger is sent using an HTTP request.

[0916] 2. Start and record the conversation:

[0917] Based on a trigger received from the server, the elderly person's device launches the AI ​​avatar app and asks, "Good morning. What are your plans for today?" If the elderly person replies, "I plan to go to the hospital today," the conversation is recorded.

[0918] 3. Sending audio data to the server:

[0919] Once the recording is complete, the senior's device uploads the audio file to a server using HTTPS, and the data is sent along with an authentication token for security reasons.

[0920] 4. Audio to text conversion:

[0921] The server converts the voice data into text using a generative AI model. For example, a speech saying "I plan to go to the hospital today" is converted directly into text.

[0922] 5. Summarizing text data:

[0923] Using the generative AI model, the server analyzes the text data, extracts key information, and generates a summary. For example, the text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0924] 6. Summary Data Storage and Notification:

[0925] The summarized data is stored in a relational database and immediately sent to the user's device, where they can receive notifications and view detailed text and audio data via a dedicated app or web portal.

[0926] Specific examples

[0927] (Example prompt) "What are your plans for today?" The elderly person responds, "I plan to go to the hospital today," and the AI ​​model summarizes this as "I plan to go to the hospital."

[0928] The child will receive a summary of their planned hospital visit notification in a dedicated app, and can view detailed text and audio data.

[0929] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0930] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0931] Step 1:

[0932] The server uses the scheduler function to send a conversation start trigger to the elderly device at the set times of 9:00 a.m. and 6:00 p.m. every day. For this purpose, the server generates an HTTP request and sends it to the elderly device. This process ensures that the elderly device receives a notification at the specified time.

[0933] Specifically, a script executed at a fixed time by the server's crontab generates and sends an HTTP request. The input is the time set by the scheduler, and the output is a trigger sent to the elderly device.

[0934] Step 2:

[0935] The elderly device launches the AI ​​avatar app based on the received trigger. At this time, the AI ​​avatar speaks to the elderly person, saying, "Good morning. What are your plans for today?" The elderly person responds, and the conversation is recorded by the elderly device. The input is the trigger from the server, and the output is the elderly person's voice data.

[0936] Specifically, when the elderly care device receives a trigger, the AI ​​avatar app automatically launches and speaks a pre-set prompt to the elderly. When the elderly responds, the voice data is recorded on the device.

[0937] Step 3:

[0938] After recording is completed, the elderly device sends the audio data to the server. This transmission uses HTTPS, and the audio data is uploaded to the server along with an authentication token. The input is the recorded audio data, and the output is the uploaded audio file to the server.

[0939] Specifically, the device temporarily stores the recorded audio data and then uploads the audio file to a server using the HTTPS protocol, using an authentication token for security during the upload process.

[0940] Step 4:

[0941] The server converts the received voice data into text data using a generative AI model (for example, Google Cloud Speech-to-Text API or IBM Watson Speech to Text). This process converts the voice data into a format that can be read as text. The input is voice data, and the output is the converted text data.

[0942] Specifically, the server sends the voice data to the AI ​​model and receives the resulting converted text data. The voice data, "I plan to go to the hospital today," is converted directly into text.

[0943] Step 5:

[0944] The server uses a generative AI model (such as OpenAI's GPT series) to summarize the text data, extracting important information and generating a short summary. The input is the converted text data, and the output is the summarized text data.

[0945] Specifically, the server sends text data to the AI ​​model and receives the summarized text data. For example, the text data "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[0946] Step 6:

[0947] The server stores the summarized data in a relational database, which structures the data and makes it easier to search and reference later. The input is the summarized text data, and the output is storage in the database.

[0948] Specifically, the server uses an SQL query to store the summary data in a relational database. For example, the summary data "plan to go to the hospital" is stored as a specific entry in the database.

[0949] Step 7:

[0950] Once the server has finished storing the data, it uses a notification service (e.g., Firebase, AWS SNS, etc.) to send a push notification to the user device. The notification contains the summary data. The input is the summary data stored in the database, and the output is the notification sent to the user device.

[0951] Specifically, the server calls the notification service and sends a push notification to the user device stating, "An elderly person is scheduled to go to the hospital today."

[0952] Step 8:

[0953] Users receive notifications and can view summary data via a dedicated app or web portal, which also provides detailed text and audio data. The input is a notification to the user device, and the output is the user reviewing the data.

[0954] Specifically, the user opens the dedicated smartphone app and checks the summary data, such as "I plan to go to the hospital." If necessary, they can also read the original audio data and detailed text.

[0955] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[0956] (Application example 1)

[0957] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] Elderly people face difficulties in communicating regularly and effectively with their families who live far away. There is also a lack of methods to quickly check on the safety and well-being of elderly people. Furthermore, there is a need for a method to quickly detect when an elderly person may be in an emergency and take appropriate action.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0960] In this invention, the server includes a means for periodically initiating a conversation with the elderly person's device, a means for converting voice data into text, and a means for extracting specific keywords from the voice data and detecting abnormalities, thereby enabling effective monitoring of the elderly person's condition and immediate response in the event of an emergency.

[0961] "Senior device" refers to an electronic device designed for use by seniors, such as a smartwatch or smartphone.

[0962] The "conversation initiation means" is a mechanism for periodically initiating a conversation with the elderly person device, and includes a function for sending a trigger at a specific timing.

[0963] The "voice data transmission means" is a mechanism for transmitting voice data of conversation acquired from the elderly person device to the server.

[0964] The "voice data text conversion means" is a function for converting transmitted voice data into text data, and uses a voice recognition engine.

[0965] The "text data summarizing means" is a function for summarizing the generated text data and extracting important information.

[0966] The "database storage means" is a mechanism for storing summarized data in a database so that it can be easily searched and referenced later.

[0967] The "notification sending means" is a function for notifying the user device of summarized data, and includes email and push notification.

[0968] The "specific keyword extraction means" is a function for extracting predetermined keywords from the voice data and detecting abnormalities based on the contents of the keywords.

[0969] The "anomaly detection means" is a mechanism for detecting anomalies based on extracted keywords, and is used to determine an emergency situation.

[0970] The "emergency contact notification means" is a function for sending a notification to an emergency contact when an abnormality is detected.

[0971] System Overview

[0972] This invention relates to a system that enables elderly people and their families living far away to communicate regularly and effectively. The system mainly comprises an elderly person device, a server, and a user device.

[0973] Hardware and software used

[0974] Elderly devices: Using a smartwatch or smartphone, which allows seniors easy access and regular notifications.

[0975] Server: We use a high-performance computer server to process and store audio data.

[0976] User devices: Provide an interface for family members to check on the daily activities of their elderly relatives using smartphones or PCs.

[0977] Software: Uses Google's Speech-to-Text API as the speech recognition engine, smtplib for sending emails, Python's schedule library for scheduling, and a generative AI model (e.g., OpenAI's GPT) for text processing.

[0978] System Operation

[0979] 1. Start regular conversations

[0980] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation. When the elderly device receives this trigger, the AI ​​avatar starts the interaction and speaks to the elderly.

[0981] 2. Collection and transmission of voice data

[0982] The elderly person's device records the conversation and sends the audio data to the server, where it is uploaded in an appropriate format.

[0983] 3. Speech to text conversion and summarization

[0984] The server uses Google's Speech-to-Text API to convert the received audio data into text data, and then uses a generative AI model to summarize the text data.

[0985] 4. Anomaly Detection

[0986] The server extracts specific keywords from the converted text data and detects anomalies. For example, if keywords such as "help" or "in trouble" are included, the server will determine that there is an anomaly.

[0987] 5. Notification to emergency contacts

[0988] If an anomaly is detected, the server will notify emergency contacts via email using smtplib.

[0989] 6. Notification and confirmation to user devices

[0990] The summarized data is stored in a database and then sent to the user's device via push notification or email. Family members can receive these notifications on their device and check the elderly person's condition.

[0991] Specific examples

[0992] The elderly person's device receives a conversation trigger at 9:00 a.m., asking, "Good morning. What are your plans for today?" The elderly person replies, "I plan to go to the hospital today." This conversation is recorded and sent to a server as an audio file. The server converts the audio file into text data saying, "I plan to go to the hospital today," and then uses generative AI to summarize it as "I plan to go to the hospital." The summary data is stored in a database, and a notification is sent to the user device saying, "The elderly person plans to go to the hospital today." The user checks the notification on their smartphone and plays the original audio data or detailed text data as needed.

[0993] Sample prompt sentence

[0994] "Convert voice data to text and notify me if there is an anomaly."

[0995] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0996] Step 1:

[0997] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation to begin. The input here is the scheduled time, and the server checks that time and sends the trigger. The server outputs an HTTP request. Specifically, the server sends an HTTP request to the elderly device saying, "Please start a conversation."

[0998] Step 2:

[0999] The elderly device activates an AI avatar based on the received trigger. The AI ​​avatar speaks to the elderly person, saying, "Good morning. What do you have planned for today?" The input here is an HTTP request sent from the server, and the device outputs an audio message prompting the elderly person to respond. Specifically, the audio message is played from the speaker on the elderly device.

[1000] Step 3:

[1001] In response to the AI ​​avatar's question, the elderly responds, "I plan to go to the hospital today." The elderly device records this conversation. The input for this step is the elderly's speech, and the output of the device is a recorded audio file. Specifically, the microphone inside the device picks up the speech and saves it as an audio file.

[1002] Step 4:

[1003] After the recording is completed, the elderly device sends the audio data to the server. The input is the audio file, and the server outputs the received audio data. Specifically, the device converts the audio file into an appropriate format and uploads it to the server.

[1004] Step 5:

[1005] The server converts the received audio data into text data using Google's Speech-to-Text API. The input is an audio file and the output is text data. Specifically, the server sends the audio file to the API and receives the returned text data.

[1006] Step 6:

[1007] The server summarizes the converted text data using a generative AI model. The input is the converted text data, and the output is the summarized text. Specifically, the server invokes a generative AI model (e.g., GPT) to generate a summary.

[1008] Step 7:

[1009] The server extracts specific keywords from the summarized text and detects anomalies. The input is the summarized text, and the output is the anomaly detection result (e.g., whether the keyword was found or not). Specifically, the server compares the keyword list with the summarized text to determine whether an anomaly exists.

[1010] Step 8:

[1011] If an abnormality is detected, the server will send a notification to the emergency contact. The input is the result of the abnormality detection, and the output is an emergency notification email. Specifically, the server uses smtplib to send an email to the emergency contact.

[1012] Step 9:

[1013] The summarized data is stored in a database. The input is the summarized text and the output is a database item. In concrete terms, the server writes the summarized text to the database.

[1014] Step 10:

[1015] Once the data has been saved to the database, the server sends the summary data to the user's device. The input is the summary data, and the output is a push notification or email. Specifically, the server sends a push notification to the user's smartphone or sends an email.

[1016] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1017] System Overview

[1018] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides more detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[1019] Specific examples of system programs and processing

[1020] 1. The server sends a conversation start trigger

[1021] The server uses a scheduler function to send a conversation start trigger to the elderly device at a set time every day, for example, by sending an HTTP request at 9:00 a.m. and 6:00 p.m.

[1022] Specific examples

[1023] The server sends a trigger to start a conversation at 9 a.m. This notifies the elderly person's device and activates the AI ​​avatar.

[1024] 2. Initiating and recording conversations with senior devices

[1025] Based on the received trigger, an AI avatar app on the senior's device is launched. The app speaks to the senior, asking, "Good morning. What do you have planned for today?" The device automatically turns on the microphone to record the senior's response and continues recording until the response is complete.

[1026] Specific examples

[1027] The elderly person responds, "I'm going to the hospital today." The conversation is recorded.

[1028] 3. Sending voice data to the server and emotion engine

[1029] After recording is complete, the audio data is automatically sent from the device to the server, which then sends it to the emotion engine. The server receives the audio data in the appropriate format and passes it to the generative AI for processing.

[1030] Specific examples

[1031] Once the recording is complete, the smartphone sends the audio file to the server and emotion engine, which receives it and begins analyzing it.

[1032] 4. Speech data text conversion and sentiment analysis

[1033] The server converts the voice data into text using generative AI. At the same time, the emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This consists of a processing part using a voice recognition engine and an emotion recognition algorithm.

[1034] Specific examples

[1035] From the speech data "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated, and the emotion engine recognizes emotional states such as "I'm worried."

[1036] 5. Text Summarization and Sentiment Integration

[1037] The generated text data is summarized. Emotional data recognized by the emotion engine is also processed at the same time and added to the final summary data. The generative AI extracts important information and keywords from the text data and creates a concise summary.

[1038] Specific examples

[1039] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and the emotional state of "I am worried" is integrated.

[1040] 6. Database storage of summary data and emotion data

[1041] The summarized text data and recognized emotion data are stored in a database, allowing for easy search and reference of the information in the future.

[1042] Specific examples

[1043] The summary "I plan to go to the hospital" and the emotion "I'm worried" are stored in a database.

[1044] 7. Notifications to User Devices

[1045] Once the saving is complete, the server notifies the user device that the summary data and emotion data have been generated, either via email or a push notification in the app.

[1046] Specific examples

[1047] The server notifies the user of the generated summary data and emotion data. The user receives a notification on their smartphone saying, "An elderly person is planning to go to the hospital today and is worried."

[1048] 8. User confirmation of summary data and emotion data

[1049] Users will receive notifications and can check summary and emotional data through a dedicated app or web portal, allowing them to understand the elderly person's daily condition and emotional state, and to view detailed text and voice data as needed.

[1050] Specific examples

[1051] The child opens their smartphone and checks the summary data and emotional state of "I plan to go to the hospital" and "I'm worried" on a dedicated app. If necessary, the original audio data and detailed text can also be checked.

[1052] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[1053] The processing flow will be explained below.

[1054] Step 1:

[1055] The server uses a scheduler function to send a trigger to start a conversation to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[1056] Step 2:

[1057] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[1058] Step 3:

[1059] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[1060] Step 4:

[1061] The device sends the recorded audio data to the server and emotion engine. At this stage, the audio data is uploaded to the server and emotion engine in an appropriate format (e.g., MP3 or WAV file).

[1062] Step 5:

[1063] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[1064] Step 6:

[1065] The emotion engine analyzes the voice data and recognizes the emotions of the elderly. The emotion recognition algorithm analyzes the tone, speed, and pauses of the voice to identify the emotional state (e.g., happy, sad, angry, etc.).

[1066] Step 7:

[1067] The server summarizes the generated text data. The generative AI extracts important information and keywords from the text data and creates a concise summary. At the same time, the emotion engine adds emotional data recognized by the summary.

[1068] Step 8:

[1069] The server stores the summary data and emotional data in a database. The summary sentence and emotional state are registered as database entries, and the information is stored in a form that allows for easy search and retrieval in the future.

[1070] Step 9:

[1071] The server notifies the user device that the summary data and emotion data have been generated and saved. Notifications are sent via email or app push notifications.

[1072] Step 10:

[1073] Users receive notifications and can view summary and emotion data via a dedicated app or web portal. Users can view summary data and see detailed text, audio, and emotional states.

[1074] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[1075] Example 2

[1076] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] When elderly people live alone, it is difficult for their families to keep track of their situation and emotional state. As a result, elderly people may not receive appropriate support and care, increasing the risk of feeling lonely and anxious. Furthermore, conventional communication systems simply transmit voice data, and lack the ability to analyze their emotional state or provide detailed summary information, making them insufficient for their families.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1079] In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for performing emotion analysis, means for summarizing the text data, means for storing the summarized data and emotion data in a database, and means for transmitting a notification of the summarized data and emotion data to the user device. This enables family members to appropriately understand the situation and emotional state of the elderly person and to quickly provide appropriate support.

[1080] "Senior Device" means a personal digital assistant or mobile electronic device used by an elderly person.

[1081] "Means for starting a conversation" refers to a function that periodically sends a trigger to start a conversation to the elderly person's device.

[1082] "Means for transmitting voice data" refers to the function of transmitting recorded voice data from the elderly person device to the server and the emotion analysis engine.

[1083] "Means for converting voice data into text" refers to a function that converts voice data into text information using a generative model.

[1084] "Means for performing emotion analysis" refers to the function of recognizing the emotional state of elderly people from voice data using an emotion analysis engine.

[1085] "Means for summarizing text data" refers to a function for concisely summarizing text data that has been converted using a generative model.

[1086] "Means for storing in a database" refers to a function for recording summarized text data and emotion data in a database.

[1087] "Means for sending notifications" refers to the ability to send emails or push notifications to notify the user device that data generation has been completed.

[1088] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[1089] server

[1090] The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. This trigger is sent using an HTTP request and arrives at the elderly device at the specified time. For example, you can use Crontab or Windows Task Scheduler to set it to send the trigger at 9:00 a.m. and 6:00 p.m.

[1091] Terminals (elderly devices)

[1092] When the elderly person's device receives a conversation start trigger, the AI ​​avatar app automatically launches. The app speaks to the elderly person, saying, "Good morning. What are your plans for today?" When the elderly person begins to respond, the device's microphone automatically turns on and continues recording until the response is completed. The recorded audio data is saved as an audio file.

[1093] Sending audio data

[1094] Once the recording is complete, the elderly device sends the audio data to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and simultaneously sending the same audio data to the emotion engine.

[1095] Analysis of voice data and conversion to text

[1096] The server converts the received voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson). In parallel, an emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This process is performed by a speech recognition engine and emotion recognition algorithm (e.g., Azure Cognitive Services Emotion API).

[1097] Summarization and emotional data integration

[1098] The generated text data is analyzed on the server to extract important information and create a summary, which is then integrated with emotion data obtained from an emotion engine using a generative AI model (e.g., GPT-4).

[1099] Data Retention and Notification

[1100] The summarized text data and emotion data are stored in a database (e.g., MySQL, MongoDB). This allows for easy search and reference of the data later. Once the data has been saved, the server sends a notification to the user's device. This notification can be via email or push notification and includes the summary data and emotion data.

[1101] Verification by user device

[1102] Users receive notifications and can view summary and emotion data through a dedicated app or web portal, as well as the original audio data and detailed text if needed.

[1103] Specific examples

[1104] One morning, the server sends a conversation start trigger at a set time, and the elderly person's smartwatch receives a notification, launching the AI ​​avatar app. The app asks, "Good morning. What are your plans for today?" and the elderly person replies, "I plan to go to the hospital today." This response is recorded, and the voice data is sent to the server and emotion engine. The server converts the voice data into text, and the emotion engine analyzes the emotional state as "I'm worried." The generated text and emotion data are integrated and summarized as "I'm planning to go to the hospital" and "I'm worried." This is stored in a database, and a notification is sent to the user's device. The user can check this information in a dedicated app and provide appropriate support for the elderly person's condition.

[1105] The above is a specific embodiment for carrying out the invention.

[1106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] Input: The server receives date and time information based on the scheduler settings.

[1109] Processing: The server sends a conversation start trigger to the elderly device based on the date and time information. It uses a scheduler function (e.g., Crontab, Windows Task Scheduler) to send an HTTP request at the specified time.

[1110] Output: A conversation start trigger is sent to the senior device.

[1111] Specific operation: The server sends an HTTP request (e.g. POST / trigger) at 9:00 AM, notifying the smartwatch that a conversation has started.

[1112] Step 2:

[1113] Input: The senior device receives a conversation start trigger.

[1114] Processing: The senior's device (smartwatch or smartphone) receives a notification and automatically launches the AI ​​avatar app, which asks, "Good morning. What are your plans for today?"

[1115] Output: A voice response from the elderly person is obtained.

[1116] Specific operation: Based on the trigger received by the smartwatch, the app will launch and say, "Good morning. What are your plans for today?"

[1117] Step 3:

[1118] Input: Elderly person answers verbally to the AI ​​avatar.

[1119] Processing: The microphone on the senior's device is automatically turned on to record the voice response. The recording is saved as an audio file (e.g., .wav format) on the device.

[1120] Output: You will get the recorded audio file.

[1121] Specific actions: The elderly person answers, "I plan to go to the hospital today," and the audio is recorded.

[1122] Step 4:

[1123] Input: The recorded audio file is saved on the senior's device.

[1124] Processing: Once the elderly device finishes recording, it sends the audio file to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and emotion engine.

[1125] Output: The audio file is sent to the server and emotion engine.

[1126] Specific operation: The smartphone sends the audio file to the server and emotion engine via an HTTP POST request (e.g., POST / upload).

[1127] Step 5:

[1128] Input: The server receives the audio file.

[1129] Processing: The server converts the voice data into text using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson), and an emotion engine analyzes the voice data to recognize the emotional state. This is done using a speech recognition engine and emotion recognition algorithms (e.g., Azure Cognitive Services Emotion API).

[1130] Output: The converted text data and emotion data are obtained.

[1131] Specific operation: The server inputs the audio file into the generative AI model, and at the same time obtains the text "I plan to go to the hospital today," the emotion engine analyzes it as "I'm worried."

[1132] Step 6:

[1133] Input: Acquired text data and sentiment data.

[1134] Processing: The server summarizes the text data and integrates it with sentiment data. A generative AI model (e.g., GPT-4) is used to concisely summarize the text, extract key information, and create a summary along with sentiment data.

[1135] Output: Summarized text data and integrated sentiment data.

[1136] Specific operation: The generative AI model summarizes the text data "I plan to go to the hospital today" into "I plan to go to the hospital" and integrates the emotion "I am worried."

[1137] Step 7:

[1138] Input: Summarized text data and sentiment data.

[1139] Processing: The server stores these data in a database. It executes queries to insert summary data and emotion data into the database (e.g., MySQL, MongoDB).

[1140] Output: The saved data is recorded in a database.

[1141] Specific operation: The server executes an SQL statement to insert the summary data "Planning to go to the hospital" and the emotion data "Worried" into the database.

[1142] Step 8:

[1143] Input: Summary data and sentiment data stored in a database.

[1144] Processing: The server notifies the user device that the save is complete. This notification can be sent via email or push notification.

[1145] Output: A notification is sent to the user device.

[1146] Specific operation: The server sends a push notification (e.g., Firebase Cloud Messaging) to the user's smartphone, informing them that "an elderly person is scheduled to go to the hospital today and is worried."

[1147] Step 9:

[1148] Input: Notifications received on the user device.

[1149] Processing: Users receive notifications and can review summary and emotion data through a dedicated app or web portal. They can also view the original audio data and detailed text if needed.

[1150] Output: The user checks the summary data and emotion data.

[1151] Specific operation: The child opens the dedicated app and checks the summary data "Plan to go to the hospital" and the emotion data "I'm worried." If necessary, the original audio data and detailed text are played and displayed.

[1152] (Application example 2)

[1153] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] There is a problem that families who live far away from elderly people lack the means of communication to quickly and accurately grasp the elderly's living situation and emotional state. With conventional methods, it is difficult for family members to contact them regularly and to understand the elderly's emotional state. As a result, the elderly may feel lonely or may not receive appropriate support.

[1155] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for summarizing the text data, means for storing the summarized data in a database, means for transmitting a notification of the summarized data to the user device, and means for performing emotion analysis of the voice data. This enables family members to quickly and in detail understand the living situation and emotional state of the elderly person.

[1156] "Senior devices" are portable electronic devices used daily by seniors, including smartwatches and smartphones.

[1157] A "server" is a computer system that communicates with other devices over a network and processes, stores, and manages data.

[1158] "Emotion analysis" is a technology that analyzes voice data, text data, etc. to estimate and classify the speaker's emotional state.

[1159] A "user device" is a portable or stationary electronic device used by a family member, caregiver, or other person, including a smartphone or computer.

[1160] "Generative AI" is a type of artificial intelligence technology that includes algorithms for natural language processing and speech recognition, among other things.

[1161] The present invention comprises an elderly device used by the elderly in their daily lives, a user device used by family members living far away, and a server system that links these devices.

[1162] System configuration

[1163] Elderly device: A smartwatch or smartphone is used to send a trigger to start a conversation to the elderly at a set time and record the response voice.

[1164] User device: A smartphone or computer used by a family member or caregiver to receive notifications and view detailed data.

[1165] Server: A computer system that processes and stores data and runs programs for sentiment analysis and text conversion. Major software includes web frameworks such as Django and Flask, Google Cloud Speech-to-Text API, and TextBlob.

[1166] Processing flow

[1167] 1. Periodic transmission of conversation triggers: The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. For example, the server sends an HTTP request at 9:00 a.m. or 6:00 p.m.

[1168] Example: At 9:00 a.m., a trigger is sent from the server, causing an elderly person's smartphone to ask, "Good morning. What are your plans for today?"

[1169] 2. Conversation recording: The elderly device receives a trigger, and the AI ​​avatar app launches and speaks to the elderly. After the elderly responds, the audio is recorded and sent to the server.

[1170] Example: An elderly person replies, "I'm going to the hospital today." This is recorded and sent to the server.

[1171] 3. Audio data processing: The server converts the received audio data into text using a generative AI model, and then passes the same audio data to an emotion analysis engine to analyze the elderly person's emotional state.

[1172] Example: From the speech "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated and the emotional state "I am worried" is analyzed.

[1173] 4. Data Summarization: The generated text data is summarized and the emotional state is integrated. Software such as TextBlob is used to extract important information and keywords and compile them into a concise summary.

[1174] Example: The text "I'm going to the hospital today" is summarized as "I'm going to the hospital" and the emotion "I'm worried" is integrated.

[1175] 5. Notification to user device: The text data and emotion data stored in the database are sent to the user's device via email or app push notification.

[1176] Example: The server notifies the generated data, and a family member receives a notification on their smartphone saying, "An elderly person is scheduled to go to the hospital today, and they are worried."

[1177] 6. User review: Users can review summary data and emotion data through a dedicated app or web portal. They can also review detailed text and original audio data if necessary.

[1178] Example: A child opens their smartphone and checks summary data and emotional state such as "I'm going to the hospital" and "I'm worried" in a dedicated app.

[1179] Examples and prompts

[1180] Example: If an elderly person answers, "I plan to go to the hospital today," the text "I plan to go to the hospital" and the emotion data "I'm worried" are generated.

[1181] Example prompt sentence:

[1182] Example prompts for text conversion and summarization:

[1183] Summarize the following sentence: "I'm going to the hospital today."

[1184] Sentiment analysis prompt example:

[1185] Analyze the sentiment of the following sentence: "I'm going to the hospital today."

[1186] With the above configuration, the present invention realizes an effective solution for notifying family members of the elderly's living situation and emotional state in detail and quickly, and for providing appropriate support.

[1187] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1188] Step 1:

[1189] The server uses a scheduler function to periodically send a conversation start trigger to the elderly device at a set time. Here, the server's input is the scheduler setting, and its output is a conversation start trigger (HTTP request). This trigger starts a conversation on the elderly device.

[1190] Step 2:

[1191] Based on the received conversation start trigger, the elderly device launches the AI ​​avatar app and speaks to the elderly, saying, "Good morning. What are your plans for today?" Here, the input of the elderly device is the conversation start trigger, and its output is the start of voice recording and the subsequent voice data (the elderly person's response).

[1192] Step 3:

[1193] The elderly device sends the recorded voice data to the server, which receives the voice data and treats it as input. The output is a trigger to store the voice data and start the voice recognition process.

[1194] Step 4:

[1195] The server converts the voice data into text data using a generative AI model. The input in this process is voice data, and the output is the corresponding text data. In a specific example, the text "I plan to go to the hospital today" is generated from the voice "I plan to go to the hospital today."

[1196] Step 5:

[1197] The server then sends the text data to an emotion analysis engine to analyze the emotional state. The input is text data, and the output is the analyzed emotional state (e.g., "worried"). This allows the elderly person's emotional state to be estimated.

[1198] Step 6:

[1199] The text data is summarized and integrated with the emotional state. To do this, the server sends the text data to the summarization engine as input. The output is the integrated summary data. In a specific example, "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and integrated with the emotion "I'm worried."

[1200] Step 7:

[1201] The server stores this text data and emotion data in a database. The input is the integrated summary data, and the output is the results stored in the database. The stored data can be searched and analyzed later.

[1202] Step 8:

[1203] After the storage is completed, the server sends a notification of the summary data and emotion data to the user device, where the server's input is the stored data and its output is a notification message (push notification or email).

[1204] Step 9:

[1205] The user device displays the received notification, and the family member can check the summary data and emotional state through a dedicated app or web portal. The input here is the notification message, and the output is the displayed summary data and emotional state. In a specific example, the family member can open the smartphone app and check the data for "Plan to go to the hospital" and "Worried."

[1206] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1208] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1209] [Fourth embodiment]

[1210] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1211] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1212] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1213] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1214] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1216] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1217] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1218] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1219] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1220] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1221] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1222] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] System Overview

[1224] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: elderly people's devices (smartwatches and smartphones), a server, and user devices (smartphones and PCs).

[1225] Specific examples of system programs and processing

[1226] 1. The server sends a conversation start trigger

[1227] The server sends a trigger to start a conversation to the elderly device at a set time every day. For example, if the trigger is set to 9:00 a.m. and 6:00 p.m., the server uses a scheduler function to send an HTTP request to the specified device.

[1228] Specific examples

[1229] The server sends a trigger to start a conversation at 9 a.m. A notification is sent to the elderly person's device, activating the AI ​​avatar.

[1230] 2. Initiating and recording conversations with senior devices

[1231] Based on the received trigger, the AI ​​avatar app on the elderly person's device launches and speaks to the user (elderly person), saying, "Good morning. What are your plans for today?" The device then records the elderly person's response.

[1232] Specific examples

[1233] The elderly person responds, "I plan to go to the hospital today." This conversation is recorded by the elderly device.

[1234] 3. Sending audio data to the server

[1235] After the recording is complete, the audio data will be automatically sent to the server, and the device will upload the audio data to the server in the appropriate format.

[1236] Specific examples

[1237] After recording is complete, the smartphone sends the audio file to the server, which receives and stores it.

[1238] 4. Converting Audio Data to Text

[1239] The server converts the received voice data into text data using generative AI, which is the processing part of the speech recognition engine.

[1240] Specific examples

[1241] The voice data "I plan to go to the hospital today" is converted into the text "I plan to go to the hospital today."

[1242] 5. Summarizing Text Data

[1243] The converted text data is further summarized by a generative AI on the server, which extracts important information and generates a short summary.

[1244] Specific examples

[1245] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[1246] 6. Database storage of summary data

[1247] The summarized data is stored in a database, which allows for easy retrieval and reference later.

[1248] Specific examples

[1249] The summarized "hospital visit schedule" is stored in a database.

[1250] 7. Notifications to User Devices

[1251] Once saved, the server notifies the user device that the summary data has been generated, either via email or app push notification.

[1252] Specific examples

[1253] The server notifies the user of the generated summary data, and the user's smartphone receives a notification that "an elderly person is scheduled to go to the hospital today."

[1254] 8. User review of summary data

[1255] Users receive notifications and can access summary data via a dedicated app or web portal, along with detailed text and audio data.

[1256] Specific examples

[1257] The child opens their smartphone and checks a summary of the "hospital appointment" in a dedicated app, and if necessary, can further review the original audio data and detailed text.

[1258] As described above, this system enables regular and efficient communication between the elderly and their families, allowing family members to quickly understand the elderly's daily condition and provide the necessary support.

[1259] The processing flow will be explained below.

[1260] Step 1:

[1261] The server uses the scheduler function to send a conversation start trigger to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[1262] Step 2:

[1263] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[1264] Step 3:

[1265] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[1266] Step 4:

[1267] The device sends the recorded audio data to the server, where it is uploaded to the server in the appropriate format (e.g., MP3 or WAV file).

[1268] Step 5:

[1269] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[1270] Step 6:

[1271] The server summarizes the generated text data. At this stage, the generative AI extracts important information and keywords from the text data and creates a compact summary.

[1272] Step 7:

[1273] The server stores the abstract data in a database, where the abstract is registered as a database entry and stored for future search and retrieval.

[1274] Step 8:

[1275] The server notifies the user device that the summary data has been generated and saved, either via email or app push notification.

[1276] Step 9:

[1277] Users receive notifications and can check summary data via a dedicated app or web portal. By viewing the summary data, users can understand the elderly person's daily condition and, if necessary, can also view detailed text and audio data.

[1278] This is the process flow of the entire system, which enables effective communication between the elderly and their families.

[1279] Example 1

[1280] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1281] Family members who live far away from their elderly loved ones face the challenge of regular and effective communication. In particular, there are few ways to keep track of the elderly's health status and schedules in real time, which can delay emergency response. Furthermore, if the elderly person does not provide information voluntarily, family members have no way of knowing their elderly loved one's condition.

[1282] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1283] In this invention, the server includes: means for sending a trigger to start a conversation to the elderly device at a specified time; means for automatically sending recorded conversation audio data from the elderly device to the server; means for converting the audio data into text data using a generative AI model; means for summarizing the text data using the generative AI model; means for storing the summarized data in a relational database; means for notifying the user device of the completion of generation of the summarized data; and means for the user to check the summarized data through a dedicated app or web portal. This enables regular and efficient communication between the elderly and their family members. Family members can quickly understand the elderly's daily condition and provide necessary support.

[1284] "Senior device" refers to an electronic device used daily by seniors, including a smartwatch or smartphone.

[1285] A "trigger" refers to a signal or instruction to initiate a specific action, in this case sent from the server to the senior device to encourage the start of a conversation.

[1286] "Audio Data" means conversations recorded on an Elderly Device and stored in electronic format.

[1287] A "generative AI model" refers to an artificial intelligence model that uses machine learning and is used to automatically perform specific tasks (such as speech recognition and text summarization).

[1288] "Text data" refers to written information converted from voice data using a generative AI model.

[1289] A "summary" is a short summary created by analyzing text data and extracting only the important information and content.

[1290] A "relational database" is a database system that stores data in a structured tabular format, and allows data to be searched and manipulated using query languages ​​such as SQL.

[1291] "Notification" refers to a message or alert that notifies the user of a particular event or the result of a process, in this case notifying the user that summary data generation has been completed.

[1292] A "dedicated app" refers to a software application developed to provide specific functions or services, which users use on their smartphones or PCs.

[1293] "Web portal" refers to a website entrance that allows users to access specific information or services over the Internet, and is accessed by a user using a browser.

[1294] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. The system mainly consists of the following components: an elderly person's device (such as a smartwatch or smartphone), a server, and a user device (such as a smartphone or PC).

[1295] System Components and Functions

[1296] 1. Elderly devices:

[1297] This refers to electronic devices used by seniors in their daily lives, including smartwatches and smartphones, such as the Apple Watch, Samsung Galaxy Watch, iPhone, and Android smartphones.

[1298] The device receives a trigger from the server and launches an AI avatar app (e.g., a custom skill for Google Assistant or Amazon Alexa).

[1299] 2. Server:

[1300] The server is composed of Linux servers and has the following functions:

[1301] A trigger is sent to the elderly device to start a conversation at a set time.

[1302] Receives voice data sent from the elderly device and stores it in an appropriate format.

[1303] Convert the voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text API, IBM Watson Speech to Text, etc.).

[1304] Summarize text data using generative AI models (e.g., OpenAI's GPT series).

[1305] Store the summarized data in a relational database (e.g., MySQL, PostgreSQL, etc.).

[1306] The completion of generation of the summary data is notified to the user device.

[1307] 3. User Device:

[1308] Family members of elderly people can access information through a dedicated app or web portal on their smartphones or PCs, and notification services use Firebase and AWS SNS.

[1309] Specific operation of the system

[1310] 1. Send a trigger to start a conversation:

[1311] The server uses a scheduler function such as crontab to send a trigger to start a conversation to the elderly device every morning at 9:00 and evening at 6:00. The trigger is sent using an HTTP request.

[1312] 2. Start and record the conversation:

[1313] Based on a trigger received from the server, the elderly person's device launches the AI ​​avatar app and asks, "Good morning. What are your plans for today?" If the elderly person replies, "I plan to go to the hospital today," the conversation is recorded.

[1314] 3. Sending audio data to the server:

[1315] Once the recording is complete, the senior's device uploads the audio file to a server using HTTPS, and the data is sent along with an authentication token for security reasons.

[1316] 4. Audio to text conversion:

[1317] The server converts the voice data into text using a generative AI model. For example, a speech saying "I plan to go to the hospital today" is converted directly into text.

[1318] 5. Summarizing text data:

[1319] Using the generative AI model, the server analyzes the text data, extracts key information, and generates a summary. For example, the text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[1320] 6. Summary Data Storage and Notification:

[1321] The summarized data is stored in a relational database and immediately sent to the user's device, where they can receive notifications and view detailed text and audio data via a dedicated app or web portal.

[1322] Specific examples

[1323] (Example prompt) "What are your plans for today?" The elderly person responds, "I plan to go to the hospital today," and the AI ​​model summarizes this as "I plan to go to the hospital."

[1324] The child will receive a summary of their planned hospital visit notification in a dedicated app, and can view detailed text and audio data.

[1325] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[1326] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1327] Step 1:

[1328] The server uses the scheduler function to send a conversation start trigger to the elderly device at the set times of 9:00 a.m. and 6:00 p.m. every day. For this purpose, the server generates an HTTP request and sends it to the elderly device. This process ensures that the elderly device receives a notification at the specified time.

[1329] Specifically, a script executed at a fixed time by the server's crontab generates and sends an HTTP request. The input is the time set by the scheduler, and the output is a trigger sent to the elderly device.

[1330] Step 2:

[1331] The elderly device launches the AI ​​avatar app based on the received trigger. At this time, the AI ​​avatar speaks to the elderly person, saying, "Good morning. What are your plans for today?" The elderly person responds, and the conversation is recorded by the elderly device. The input is the trigger from the server, and the output is the elderly person's voice data.

[1332] Specifically, when the elderly care device receives a trigger, the AI ​​avatar app automatically launches and speaks a pre-set prompt to the elderly. When the elderly responds, the voice data is recorded on the device.

[1333] Step 3:

[1334] After recording is completed, the elderly device sends the audio data to the server. This transmission uses HTTPS, and the audio data is uploaded to the server along with an authentication token. The input is the recorded audio data, and the output is the uploaded audio file to the server.

[1335] Specifically, the device temporarily stores the recorded audio data and then uploads the audio file to a server using the HTTPS protocol, using an authentication token for security during the upload process.

[1336] Step 4:

[1337] The server converts the received voice data into text data using a generative AI model (for example, Google Cloud Speech-to-Text API or IBM Watson Speech to Text). This process converts the voice data into a format that can be read as text. The input is voice data, and the output is the converted text data.

[1338] Specifically, the server sends the voice data to the AI ​​model and receives the resulting converted text data. The voice data, "I plan to go to the hospital today," is converted directly into text.

[1339] Step 5:

[1340] The server uses a generative AI model (such as OpenAI's GPT series) to summarize the text data, extracting important information and generating a short summary. The input is the converted text data, and the output is the summarized text data.

[1341] Specifically, the server sends text data to the AI ​​model and receives the summarized text data. For example, the text data "I plan to go to the hospital today" is summarized as "I plan to go to the hospital."

[1342] Step 6:

[1343] The server stores the summarized data in a relational database, which structures the data and makes it easier to search and reference later. The input is the summarized text data, and the output is storage in the database.

[1344] Specifically, the server uses an SQL query to store the summary data in a relational database. For example, the summary data "plan to go to the hospital" is stored as a specific entry in the database.

[1345] Step 7:

[1346] Once the server has finished storing the data, it uses a notification service (e.g., Firebase, AWS SNS, etc.) to send a push notification to the user device. The notification contains the summary data. The input is the summary data stored in the database, and the output is the notification sent to the user device.

[1347] Specifically, the server calls the notification service and sends a push notification to the user device stating, "An elderly person is scheduled to go to the hospital today."

[1348] Step 8:

[1349] Users receive notifications and can view summary data via a dedicated app or web portal, which also provides detailed text and audio data. The input is a notification to the user device, and the output is the user reviewing the data.

[1350] Specifically, the user opens the dedicated smartphone app and checks the summary data, such as "I plan to go to the hospital." If necessary, they can also read the original audio data and detailed text.

[1351] As described above, this system enables regular and efficient communication between the elderly and their families, allowing the family to quickly understand the elderly's daily condition and provide the necessary support.

[1352] (Application example 1)

[1353] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1354] Elderly people face difficulties in communicating regularly and effectively with their families who live far away. There is also a lack of methods to quickly check on the safety and well-being of elderly people. Furthermore, there is a need for a method to quickly detect when an elderly person may be in an emergency and take appropriate action.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1356] In this invention, the server includes a means for periodically initiating a conversation with the elderly person's device, a means for converting voice data into text, and a means for extracting specific keywords from the voice data and detecting abnormalities, thereby enabling effective monitoring of the elderly person's condition and immediate response in the event of an emergency.

[1357] "Senior device" refers to an electronic device designed for use by seniors, such as a smartwatch or smartphone.

[1358] The "conversation initiation means" is a mechanism for periodically initiating a conversation with the elderly person device, and includes a function for sending a trigger at a specific timing.

[1359] The "voice data transmission means" is a mechanism for transmitting voice data of conversation acquired from the elderly person device to the server.

[1360] The "voice data text conversion means" is a function for converting transmitted voice data into text data, and uses a voice recognition engine.

[1361] The "text data summarizing means" is a function for summarizing the generated text data and extracting important information.

[1362] The "database storage means" is a mechanism for storing summarized data in a database so that it can be easily searched and referenced later.

[1363] The "notification sending means" is a function for notifying the user device of summarized data, and includes email and push notification.

[1364] The "specific keyword extraction means" is a function for extracting predetermined keywords from the voice data and detecting abnormalities based on the contents of the keywords.

[1365] The "anomaly detection means" is a mechanism for detecting anomalies based on extracted keywords, and is used to determine an emergency situation.

[1366] The "emergency contact notification means" is a function for sending a notification to an emergency contact when an abnormality is detected.

[1367] System Overview

[1368] This invention relates to a system that enables elderly people and their families living far away to communicate regularly and effectively. The system mainly comprises an elderly person device, a server, and a user device.

[1369] Hardware and software used

[1370] Elderly devices: Using a smartwatch or smartphone, which allows seniors easy access and regular notifications.

[1371] Server: We use a high-performance computer server to process and store audio data.

[1372] User devices: Provide an interface for family members to check on the daily activities of their elderly relatives using smartphones or PCs.

[1373] Software: Uses Google's Speech-to-Text API as the speech recognition engine, smtplib for sending emails, Python's schedule library for scheduling, and a generative AI model (e.g., OpenAI's GPT) for text processing.

[1374] System Operation

[1375] 1. Start regular conversations

[1376] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation. When the elderly device receives this trigger, the AI ​​avatar starts the interaction and speaks to the elderly.

[1377] 2. Collection and transmission of voice data

[1378] The elderly person's device records the conversation and sends the audio data to the server, where it is uploaded in an appropriate format.

[1379] 3. Speech to text conversion and summarization

[1380] The server uses Google's Speech-to-Text API to convert the received audio data into text data, and then uses a generative AI model to summarize the text data.

[1381] 4. Anomaly Detection

[1382] The server extracts specific keywords from the converted text data and detects anomalies. For example, if keywords such as "help" or "in trouble" are included, the server will determine that there is an anomaly.

[1383] 5. Notification to emergency contacts

[1384] If an anomaly is detected, the server will notify emergency contacts via email using smtplib.

[1385] 6. Notification and confirmation to user devices

[1386] The summarized data is stored in a database and then sent to the user's device via push notification or email. Family members can receive these notifications on their device and check the elderly person's condition.

[1387] Specific examples

[1388] The elderly person's device receives a conversation trigger at 9:00 a.m., asking, "Good morning. What are your plans for today?" The elderly person replies, "I plan to go to the hospital today." This conversation is recorded and sent to a server as an audio file. The server converts the audio file into text data saying, "I plan to go to the hospital today," and then uses generative AI to summarize it as "I plan to go to the hospital." The summary data is stored in a database, and a notification is sent to the user device saying, "The elderly person plans to go to the hospital today." The user checks the notification on their smartphone and plays the original audio data or detailed text data as needed.

[1389] Sample prompt sentence

[1390] "Convert voice data to text and notify me if there is an anomaly."

[1391] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1392] Step 1:

[1393] The server uses the schedule library to send an HTTP request to the elderly device at a specific time every day, triggering the conversation to begin. The input here is the scheduled time, and the server checks that time and sends the trigger. The server outputs an HTTP request. Specifically, the server sends an HTTP request to the elderly device saying, "Please start a conversation."

[1394] Step 2:

[1395] The elderly device activates an AI avatar based on the received trigger. The AI ​​avatar speaks to the elderly person, saying, "Good morning. What do you have planned for today?" The input here is an HTTP request sent from the server, and the device outputs an audio message prompting the elderly person to respond. Specifically, the audio message is played from the speaker on the elderly device.

[1396] Step 3:

[1397] In response to the AI ​​avatar's question, the elderly responds, "I plan to go to the hospital today." The elderly device records this conversation. The input for this step is the elderly's speech, and the output of the device is a recorded audio file. Specifically, the microphone inside the device picks up the speech and saves it as an audio file.

[1398] Step 4:

[1399] After the recording is completed, the elderly device sends the audio data to the server. The input is the audio file, and the server outputs the received audio data. Specifically, the device converts the audio file into an appropriate format and uploads it to the server.

[1400] Step 5:

[1401] The server converts the received audio data into text data using Google's Speech-to-Text API. The input is an audio file and the output is text data. Specifically, the server sends the audio file to the API and receives the returned text data.

[1402] Step 6:

[1403] The server summarizes the converted text data using a generative AI model. The input is the converted text data, and the output is the summarized text. Specifically, the server invokes a generative AI model (e.g., GPT) to generate a summary.

[1404] Step 7:

[1405] The server extracts specific keywords from the summarized text and detects anomalies. The input is the summarized text, and the output is the anomaly detection result (e.g., whether the keyword was found or not). Specifically, the server compares the keyword list with the summarized text to determine whether an anomaly exists.

[1406] Step 8:

[1407] If an abnormality is detected, the server will send a notification to the emergency contact. The input is the result of the abnormality detection, and the output is an emergency notification email. Specifically, the server uses smtplib to send an email to the emergency contact.

[1408] Step 9:

[1409] The summarized data is stored in a database. The input is the summarized text and the output is a database item. In concrete terms, the server writes the summarized text to the database.

[1410] Step 10:

[1411] Once the data has been saved to the database, the server sends the summary data to the user's device. The input is the summary data, and the output is a push notification or email. Specifically, the server sends a push notification to the user's smartphone or sends an email.

[1412] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1413] System Overview

[1414] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides more detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[1415] Specific examples of system programs and processing

[1416] 1. The server sends a conversation start trigger

[1417] The server uses a scheduler function to send a conversation start trigger to the elderly device at a set time every day, for example, by sending an HTTP request at 9:00 a.m. and 6:00 p.m.

[1418] Specific examples

[1419] The server sends a trigger to start a conversation at 9 a.m. This notifies the elderly person's device and activates the AI ​​avatar.

[1420] 2. Initiating and recording conversations with senior devices

[1421] Based on the received trigger, an AI avatar app on the senior's device is launched. The app speaks to the senior, asking, "Good morning. What do you have planned for today?" The device automatically turns on the microphone to record the senior's response and continues recording until the response is complete.

[1422] Specific examples

[1423] The elderly person responds, "I'm going to the hospital today." The conversation is recorded.

[1424] 3. Sending voice data to the server and emotion engine

[1425] After recording is complete, the audio data is automatically sent from the device to the server, which then sends it to the emotion engine. The server receives the audio data in the appropriate format and passes it to the generative AI for processing.

[1426] Specific examples

[1427] Once the recording is complete, the smartphone sends the audio file to the server and emotion engine, which receives it and begins analyzing it.

[1428] 4. Speech data text conversion and sentiment analysis

[1429] The server converts the voice data into text using generative AI. At the same time, the emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This consists of a processing part using a voice recognition engine and an emotion recognition algorithm.

[1430] Specific examples

[1431] From the speech data "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated, and the emotion engine recognizes emotional states such as "I'm worried."

[1432] 5. Text Summarization and Sentiment Integration

[1433] The generated text data is summarized. Emotional data recognized by the emotion engine is also processed at the same time and added to the final summary data. The generative AI extracts important information and keywords from the text data and creates a concise summary.

[1434] Specific examples

[1435] The text "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and the emotional state of "I am worried" is integrated.

[1436] 6. Database storage of summary data and emotion data

[1437] The summarized text data and recognized emotion data are stored in a database, allowing for easy search and reference of the information in the future.

[1438] Specific examples

[1439] The summary "I plan to go to the hospital" and the emotion "I'm worried" are stored in a database.

[1440] 7. Notifications to User Devices

[1441] Once the saving is complete, the server notifies the user device that the summary data and emotion data have been generated, either via email or a push notification in the app.

[1442] Specific examples

[1443] The server notifies the user of the generated summary data and emotion data. The user receives a notification on their smartphone saying, "An elderly person is planning to go to the hospital today and is worried."

[1444] 8. User confirmation of summary data and emotion data

[1445] Users will receive notifications and can check summary and emotional data through a dedicated app or web portal, allowing them to understand the elderly person's daily condition and emotional state, and to view detailed text and voice data as needed.

[1446] Specific examples

[1447] The child opens their smartphone and checks the summary data and emotional state of "I plan to go to the hospital" and "I'm worried" on a dedicated app. If necessary, the original audio data and detailed text can also be checked.

[1448] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[1449] The processing flow will be explained below.

[1450] Step 1:

[1451] The server uses a scheduler function to send a trigger to start a conversation to the elderly device at a fixed time every day. Specifically, it sends an HTTP request at the specified time to issue an instruction to the elderly device.

[1452] Step 2:

[1453] The device (elderly device) launches the AI ​​avatar app based on a trigger received from the server. The app then speaks to the elderly with a predefined message (e.g., "Good morning. What are your plans for today?").

[1454] Step 3:

[1455] The device will record the elderly person's voice response, and the app will automatically turn on the microphone and continue recording until the elderly person finishes answering.

[1456] Step 4:

[1457] The device sends the recorded audio data to the server and emotion engine. At this stage, the audio data is uploaded to the server and emotion engine in an appropriate format (e.g., MP3 or WAV file).

[1458] Step 5:

[1459] The server converts the received voice data into text data using generative AI. The speech recognition engine analyzes the voice data and generates the corresponding text.

[1460] Step 6:

[1461] The emotion engine analyzes the voice data and recognizes the emotions of the elderly. The emotion recognition algorithm analyzes the tone, speed, and pauses of the voice to identify the emotional state (e.g., happy, sad, angry, etc.).

[1462] Step 7:

[1463] The server summarizes the generated text data. The generative AI extracts important information and keywords from the text data and creates a concise summary. At the same time, the emotion engine adds emotional data recognized by the summary.

[1464] Step 8:

[1465] The server stores the summary data and emotional data in a database. The summary sentence and emotional state are registered as database entries, and the information is stored in a form that allows for easy search and retrieval in the future.

[1466] Step 9:

[1467] The server notifies the user device that the summary data and emotion data have been generated and saved. Notifications are sent via email or app push notifications.

[1468] Step 10:

[1469] Users receive notifications and can view summary and emotion data via a dedicated app or web portal. Users can view summary data and see detailed text, audio, and emotional states.

[1470] This is the overall processing flow of the system that combines the emotion engine. This process enables more detailed and effective communication between the elderly and their families. Family members can quickly understand the elderly's daily condition and emotional state, and provide more appropriate support.

[1471] Example 2

[1472] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] When elderly people live alone, it is difficult for their families to keep track of their situation and emotional state. As a result, elderly people may not receive appropriate support and care, increasing the risk of feeling lonely and anxious. Furthermore, conventional communication systems simply transmit voice data, and lack the ability to analyze their emotional state or provide detailed summary information, making them insufficient for their families.

[1474] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1475] In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for performing emotion analysis, means for summarizing the text data, means for storing the summarized data and emotion data in a database, and means for transmitting a notification of the summarized data and emotion data to the user device. This enables family members to appropriately understand the situation and emotional state of the elderly person and to quickly provide appropriate support.

[1476] "Senior Device" means a personal digital assistant or mobile electronic device used by an elderly person.

[1477] "Means for starting a conversation" refers to a function that periodically sends a trigger to start a conversation to the elderly person's device.

[1478] "Means for transmitting voice data" refers to the function of transmitting recorded voice data from the elderly person device to the server and the emotion analysis engine.

[1479] "Means for converting voice data into text" refers to a function that converts voice data into text information using a generative model.

[1480] "Means for performing emotion analysis" refers to the function of recognizing the emotional state of elderly people from voice data using an emotion analysis engine.

[1481] "Means for summarizing text data" refers to a function for concisely summarizing text data that has been converted using a generative model.

[1482] "Means for storing in a database" refers to a function for recording summarized text data and emotion data in a database.

[1483] "Means for sending notifications" refers to the ability to send emails or push notifications to notify the user device that data generation has been completed.

[1484] This invention relates to a system that enables elderly people and their families who live far away to communicate regularly and effectively. This system is unique in that it also recognizes the emotional state of the elderly person and provides detailed information to the family. The system consists of the following main components: an elderly person's device (smartwatch or smartphone), a server, an emotion engine, and a user device (smartphone or PC).

[1485] server

[1486] The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. This trigger is sent using an HTTP request and arrives at the elderly device at the specified time. For example, you can use Crontab or Windows Task Scheduler to set it to send the trigger at 9:00 a.m. and 6:00 p.m.

[1487] Terminals (elderly devices)

[1488] When the elderly person's device receives a conversation start trigger, the AI ​​avatar app automatically launches. The app speaks to the elderly person, saying, "Good morning. What are your plans for today?" When the elderly person begins to respond, the device's microphone automatically turns on and continues recording until the response is completed. The recorded audio data is saved as an audio file.

[1489] Sending audio data

[1490] Once the recording is complete, the elderly device sends the audio data to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and simultaneously sending the same audio data to the emotion engine.

[1491] Analysis of voice data and conversion to text

[1492] The server converts the received voice data into text data using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson). In parallel, an emotion engine analyzes the voice data and recognizes the elderly person's emotional state. This process is performed by a speech recognition engine and emotion recognition algorithm (e.g., Azure Cognitive Services Emotion API).

[1493] Summarization and emotional data integration

[1494] The generated text data is analyzed on the server to extract important information and create a summary, which is then integrated with emotion data obtained from an emotion engine using a generative AI model (e.g., GPT-4).

[1495] Data Retention and Notification

[1496] The summarized text data and emotion data are stored in a database (e.g., MySQL, MongoDB). This allows for easy search and reference of the data later. Once the data has been saved, the server sends a notification to the user's device. This notification can be via email or push notification and includes the summary data and emotion data.

[1497] Verification by user device

[1498] Users receive notifications and can view summary and emotion data through a dedicated app or web portal, as well as the original audio data and detailed text if needed.

[1499] Specific examples

[1500] One morning, the server sends a conversation start trigger at a set time, and the elderly person's smartwatch receives a notification, launching the AI ​​avatar app. The app asks, "Good morning. What are your plans for today?" and the elderly person replies, "I plan to go to the hospital today." This response is recorded, and the voice data is sent to the server and emotion engine. The server converts the voice data into text, and the emotion engine analyzes the emotional state as "I'm worried." The generated text and emotion data are integrated and summarized as "I'm planning to go to the hospital" and "I'm worried." This is stored in a database, and a notification is sent to the user's device. The user can check this information in a dedicated app and provide appropriate support for the elderly person's condition.

[1501] The above is a specific embodiment for carrying out the invention.

[1502] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1503] Step 1:

[1504] Input: The server receives date and time information based on the scheduler settings.

[1505] Processing: The server sends a conversation start trigger to the elderly device based on the date and time information. It uses a scheduler function (e.g., Crontab, Windows Task Scheduler) to send an HTTP request at the specified time.

[1506] Output: A conversation start trigger is sent to the senior device.

[1507] Specific operation: The server sends an HTTP request (e.g. POST / trigger) at 9:00 AM, notifying the smartwatch that a conversation has started.

[1508] Step 2:

[1509] Input: The senior device receives a conversation start trigger.

[1510] Processing: The senior's device (smartwatch or smartphone) receives a notification and automatically launches the AI ​​avatar app, which asks, "Good morning. What are your plans for today?"

[1511] Output: A voice response from the elderly person is obtained.

[1512] Specific operation: Based on the trigger received by the smartwatch, the app will launch and say, "Good morning. What are your plans for today?"

[1513] Step 3:

[1514] Input: Elderly person answers verbally to the AI ​​avatar.

[1515] Processing: The microphone on the senior's device is automatically turned on to record the voice response. The recording is saved as an audio file (e.g., .wav format) on the device.

[1516] Output: You will get the recorded audio file.

[1517] Specific actions: The elderly person answers, "I plan to go to the hospital today," and the audio is recorded.

[1518] Step 4:

[1519] Input: The recorded audio file is saved on the senior's device.

[1520] Processing: Once the elderly device finishes recording, it sends the audio file to the server and emotion engine using an HTTP POST request, uploading the audio file to the server and emotion engine.

[1521] Output: The audio file is sent to the server and emotion engine.

[1522] Specific operation: The smartphone sends the audio file to the server and emotion engine via an HTTP POST request (e.g., POST / upload).

[1523] Step 5:

[1524] Input: The server receives the audio file.

[1525] Processing: The server converts the voice data into text using a generative AI model (e.g., Google Cloud Speech-to-Text, IBM Watson), and an emotion engine analyzes the voice data to recognize the emotional state. This is done using a speech recognition engine and emotion recognition algorithms (e.g., Azure Cognitive Services Emotion API).

[1526] Output: The converted text data and emotion data are obtained.

[1527] Specific operation: The server inputs the audio file into the generative AI model, and at the same time obtains the text "I plan to go to the hospital today," the emotion engine analyzes it as "I'm worried."

[1528] Step 6:

[1529] Input: Acquired text data and sentiment data.

[1530] Processing: The server summarizes the text data and integrates it with sentiment data. A generative AI model (e.g., GPT-4) is used to concisely summarize the text, extract key information, and create a summary along with sentiment data.

[1531] Output: Summarized text data and integrated sentiment data.

[1532] Specific operation: The generative AI model summarizes the text data "I plan to go to the hospital today" into "I plan to go to the hospital" and integrates the emotion "I am worried."

[1533] Step 7:

[1534] Input: Summarized text data and sentiment data.

[1535] Processing: The server stores these data in a database. It executes queries to insert summary data and emotion data into the database (e.g., MySQL, MongoDB).

[1536] Output: The saved data is recorded in a database.

[1537] Specific operation: The server executes an SQL statement to insert the summary data "Planning to go to the hospital" and the emotion data "Worried" into the database.

[1538] Step 8:

[1539] Input: Summary data and sentiment data stored in a database.

[1540] Processing: The server notifies the user device that the save is complete. This notification can be sent via email or push notification.

[1541] Output: A notification is sent to the user device.

[1542] Specific operation: The server sends a push notification (e.g., Firebase Cloud Messaging) to the user's smartphone, informing them that "an elderly person is scheduled to go to the hospital today and is worried."

[1543] Step 9:

[1544] Input: Notifications received on the user device.

[1545] Processing: Users receive notifications and can review summary and emotion data through a dedicated app or web portal. They can also view the original audio data and detailed text if needed.

[1546] Output: The user checks the summary data and emotion data.

[1547] Specific operation: The child opens the dedicated app and checks the summary data "Plan to go to the hospital" and the emotion data "I'm worried." If necessary, the original audio data and detailed text are played and displayed.

[1548] (Application example 2)

[1549] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1550] There is a problem that families who live far away from elderly people lack the means of communication to quickly and accurately grasp the elderly's living situation and emotional state. With conventional methods, it is difficult for family members to contact them regularly and to understand the elderly's emotional state. As a result, the elderly may feel lonely or may not receive appropriate support.

[1551] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for periodically initiating a conversation with the elderly person device, means for transmitting voice data of the conversation from the elderly person device, means for converting the voice data into text, means for summarizing the text data, means for storing the summarized data in a database, means for transmitting a notification of the summarized data to the user device, and means for performing emotion analysis of the voice data. This enables family members to quickly and in detail understand the living situation and emotional state of the elderly person.

[1552] "Senior devices" are portable electronic devices used daily by seniors, including smartwatches and smartphones.

[1553] A "server" is a computer system that communicates with other devices over a network and processes, stores, and manages data.

[1554] "Emotion analysis" is a technology that analyzes voice data, text data, etc. to estimate and classify the speaker's emotional state.

[1555] A "user device" is a portable or stationary electronic device used by a family member, caregiver, or other person, including a smartphone or computer.

[1556] "Generative AI" is a type of artificial intelligence technology that includes algorithms for natural language processing and speech recognition, among other things.

[1557] The present invention comprises an elderly device used by the elderly in their daily lives, a user device used by family members living far away, and a server system that links these devices.

[1558] System configuration

[1559] Elderly device: A smartwatch or smartphone is used to send a trigger to start a conversation to the elderly at a set time and record the response voice.

[1560] User device: A smartphone or computer used by a family member or caregiver to receive notifications and view detailed data.

[1561] Server: A computer system that processes and stores data and runs programs for sentiment analysis and text conversion. Major software includes web frameworks such as Django and Flask, Google Cloud Speech-to-Text API, and TextBlob.

[1562] Processing flow

[1563] 1. Periodic transmission of conversation triggers: The server uses the scheduler function to send a conversation start trigger to the elderly device at a set time every day. For example, the server sends an HTTP request at 9:00 a.m. or 6:00 p.m.

[1564] Example: At 9:00 a.m., a trigger is sent from the server, causing an elderly person's smartphone to ask, "Good morning. What are your plans for today?"

[1565] 2. Conversation recording: The elderly device receives a trigger, and the AI ​​avatar app launches and speaks to the elderly. After the elderly responds, the audio is recorded and sent to the server.

[1566] Example: An elderly person replies, "I'm going to the hospital today." This is recorded and sent to the server.

[1567] 3. Audio data processing: The server converts the received audio data into text using a generative AI model, and then passes the same audio data to an emotion analysis engine to analyze the elderly person's emotional state.

[1568] Example: From the speech "I plan to go to the hospital today," the text "I plan to go to the hospital today" is generated and the emotional state "I am worried" is analyzed.

[1569] 4. Data Summarization: The generated text data is summarized and the emotional state is integrated. Software such as TextBlob is used to extract important information and keywords and compile them into a concise summary.

[1570] Example: The text "I'm going to the hospital today" is summarized as "I'm going to the hospital" and the emotion "I'm worried" is integrated.

[1571] 5. Notification to user device: The text data and emotion data stored in the database are sent to the user's device via email or app push notification.

[1572] Example: The server notifies the generated data, and a family member receives a notification on their smartphone saying, "An elderly person is scheduled to go to the hospital today, and they are worried."

[1573] 6. User review: Users can review summary data and emotion data through a dedicated app or web portal. They can also review detailed text and original audio data if necessary.

[1574] Example: A child opens their smartphone and checks summary data and emotional state such as "I'm going to the hospital" and "I'm worried" in a dedicated app.

[1575] Examples and prompts

[1576] Example: If an elderly person answers, "I plan to go to the hospital today," the text "I plan to go to the hospital" and the emotion data "I'm worried" are generated.

[1577] Example prompt sentence:

[1578] Example prompts for text conversion and summarization:

[1579] Summarize the following sentence: "I'm going to the hospital today."

[1580] Sentiment analysis prompt example:

[1581] Analyze the sentiment of the following sentence: "I'm going to the hospital today."

[1582] With the above configuration, the present invention realizes an effective solution for notifying family members of the elderly's living situation and emotional state in detail and quickly, and for providing appropriate support.

[1583] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1584] Step 1:

[1585] The server uses a scheduler function to periodically send a conversation start trigger to the elderly device at a set time. Here, the server's input is the scheduler setting, and its output is a conversation start trigger (HTTP request). This trigger starts a conversation on the elderly device.

[1586] Step 2:

[1587] Based on the received conversation start trigger, the elderly device launches the AI ​​avatar app and speaks to the elderly, saying, "Good morning. What are your plans for today?" Here, the input of the elderly device is the conversation start trigger, and its output is the start of voice recording and the subsequent voice data (the elderly person's response).

[1588] Step 3:

[1589] The elderly device sends the recorded voice data to the server, which receives the voice data and treats it as input. The output is a trigger to store the voice data and start the voice recognition process.

[1590] Step 4:

[1591] The server converts the voice data into text data using a generative AI model. The input in this process is voice data, and the output is the corresponding text data. In a specific example, the text "I plan to go to the hospital today" is generated from the voice "I plan to go to the hospital today."

[1592] Step 5:

[1593] The server then sends the text data to an emotion analysis engine to analyze the emotional state. The input is text data, and the output is the analyzed emotional state (e.g., "worried"). This allows the elderly person's emotional state to be estimated.

[1594] Step 6:

[1595] The text data is summarized and integrated with the emotional state. To do this, the server sends the text data to the summarization engine as input. The output is the integrated summary data. In a specific example, "I plan to go to the hospital today" is summarized as "I plan to go to the hospital" and integrated with the emotion "I'm worried."

[1596] Step 7:

[1597] The server stores this text data and emotion data in a database. The input is the integrated summary data, and the output is the results stored in the database. The stored data can be searched and analyzed later.

[1598] Step 8:

[1599] After the storage is completed, the server sends a notification of the summary data and emotion data to the user device, where the server's input is the stored data and its output is a notification message (push notification or email).

[1600] Step 9:

[1601] The user device displays the received notification, and the family member can check the summary data and emotional state through a dedicated app or web portal. The input here is the notification message, and the output is the displayed summary data and emotional state. In a specific example, the family member can open the smartphone app and check the data for "Plan to go to the hospital" and "Worried."

[1602] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1603] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1604] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1605] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1606] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1607] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1608] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1609] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1610] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1611] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1612] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1613] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1614] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1615] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1616] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1617] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1618] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1619] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1620] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1621] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1622] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1623] The following is further disclosed regarding the above embodiment.

[1624] (Claim 1)

[1625] a means for periodically initiating a conversation with the senior device;

[1626] means for transmitting audio data of the conversation from the senior citizen device;

[1627] a means for converting the audio data into text;

[1628] a means for summarizing text data;

[1629] a means for storing the summarized data in a database;

[1630] The system includes means for sending a notification of the summarized data to a user device.

[1631] (Claim 2)

[1632] 2. The system of claim 1, wherein the elderly device is a smartwatch or a smartphone.

[1633] (Claim 3)

[1634] 2. The system according to claim 1, wherein the speech recognition engine used to generate the text data is a generative AI.

[1635] "Example 1"

[1636] (Claim 1)

[1637] A means for sending a trigger to start a conversation at a predetermined time to the elderly person's device;

[1638] A means for automatically transmitting audio data of the conversation recorded from the elderly person's device to a server;

[1639] A means for converting voice data into text data using a generative AI model;

[1640] A means for summarizing text data using a generative AI model;

[1641] means for storing the summarized data in a relational database;

[1642] means for notifying the user device of the completion of generation of the summarized data;

[1643] A means for users to view summary data via a dedicated app or web portal

[1644] A system including:

[1645] (Claim 2)

[1646] 2. The system of claim 1, wherein the elderly device is a smartwatch or a smartphone.

[1647] (Claim 3)

[1648] 2. The system according to claim 1, wherein the speech recognition engine used to generate the text data is a generative AI.

[1649] "Application Example 1"

[1650] (Claim 1)

[1651] a means for periodically initiating a conversation with the senior device;

[1652] means for transmitting audio data of the conversation from the senior citizen device;

[1653] a means for converting the audio data into text;

[1654] a means for summarizing text data;

[1655] a means for storing the summarized data in a database;

[1656] means for sending a notification of the summarized data to the user device;

[1657] A means for extracting specific keywords from voice data and detecting anomalies;

[1658] The system includes a means for sending a notification to an emergency contact if an anomaly is detected.

[1659] (Claim 2)

[1660] 2. The system of claim 1, wherein the elderly device is a smartwatch or a smartphone.

[1661] (Claim 3)

[1662] 2. The system according to claim 1, wherein the speech recognition engine used to generate the text data is a generative AI.

[1663] "Example 2: Combining Emotion Engines"

[1664] (Claim 1)

[1665] a means for periodically initiating a conversation with the senior device;

[1666] means for transmitting audio data of the conversation from the senior citizen device;

[1667] a means for converting the audio data into text;

[1668] a means for performing sentiment analysis;

[1669] a means for summarizing text data;

[1670] means for storing the summarized data and the emotion data in a database;

[1671] The system includes means for transmitting a notification of the summarized data and emotion data to a user device.

[1672] (Claim 2)

[1673] 10. The system of claim 1, wherein the senior device is a personal digital assistant or a mobile electronic device.

[1674] (Claim 3)

[1675] 2. The system according to claim 1, wherein the speech recognition engine used to generate the text data is a generative model.

[1676] "Application example 2 when combining emotion engines"

[1677] (Claim 1)

[1678] a means for periodically initiating a conversation with the senior device;

[1679] means for transmitting audio data of the conversation from the senior citizen device;

[1680] a means for converting the audio data into text;

[1681] a means for summarizing text data;

[1682] a means for storing the summarized data in a database;

[1683] means for sending a notification of the summarized data to the user device;

[1684] A system comprising means for performing sentiment analysis of audio data.

[1685] (Claim 2)

[1686] 2. The system of claim 1, wherein the elderly device is a smartwatch or a smartphone, and the user device is a smartphone or a computer.

[1687] (Claim 3)

[1688] The system according to claim 1, wherein the speech recognition engine used for generating text data and analyzing emotions is a generative AI. [Explanation of symbols]

[1689] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for periodically initiating a conversation with the senior device; means for transmitting audio data of the conversation from the senior citizen device; a means for converting the audio data into text; a means for summarizing text data; a means for storing the summarized data in a database; The system includes means for sending a notification of the summarized data to a user device.

2. The system of claim 1 , wherein the elderly device is a smartwatch or a smartphone.

3. 2. The system according to claim 1, wherein the speech recognition engine used to generate the text data is a generative AI.

Images