System

Abstract

A system is provided.SOLUTION: A system for supporting seniors and caregivers, comprising: means for capturing speech of seniors and caregivers and converting the speech to text; means for analyzing the generated text data and generating appropriate responses; means for converting the generated responses to speech and presenting them to the user; means for monitoring speech and behavior of seniors and detecting signs of dementia; means for notifying caregivers when signs of dementia are detected; means for analyzing question content and generating advice to provide know-how about care; and means for reminding to take medicine or make an appointment at a hospital.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In an aging society, there are an increasing number of cases where elderly parents are cared for by a small number of people. As a result, the stress and mental burden on caregivers are becoming serious problems. Furthermore, it is important to detect signs of dementia in the elderly early and take appropriate measures, but this also places a heavy burden on caregivers. To address these issues, there is a need to provide a system that can provide effective support, improve relationships between family members, and reduce the stress of caregiving. [Means for solving the problem]

[0005] The present invention provides a system that utilizes AI to support elderly people and their caregivers. This system includes: a means for capturing conversations between the elderly and their caregivers and converting the speech to text; a means for analyzing the generated text data and generating appropriate responses; a means for converting the generated responses to speech and presenting them to the user; a means for monitoring the elderly's conversations and behaviors and detecting signs of dementia; a means for notifying the caregiver when signs of dementia are detected; a means for analyzing questions and generating advice to provide caregiving know-how; and a means for reminding the elderly to take medication and make hospital appointments. Furthermore, the system includes sensors that constantly monitor the conversations and behaviors of the elderly and their caregivers, transmits the monitoring results to a server, and analyzes signs of dementia. It also includes a database that provides information on local consultation centers and welfare systems, and generates appropriate advice based on local information. This reduces the burden on caregivers, improves communication between family members, extends the healthy lifespan of the elderly, and contributes to reducing medical costs.

[0006] "Elderly people" refers to people of an age group who require care in an aging society.

[0007] "Caregiver" refers to a family member, friend, or social welfare professional who provides support and care for an older adult.

[0008] "Conversation" refers to the verbal communication behavior of the elderly and their caregivers.

[0009] "Means for converting speech to text" refers to a function that uses speech recognition technology to convert a user's speech into text information.

[0010] "Text data" refers to text information converted from audio.

[0011] "Analysis" refers to the process of using AI technology to understand and evaluate text data and derive appropriate responses and actions.

[0012] "Response" refers to information or action provided to the user based on the analysis results.

[0013] "Monitoring" refers to the process of continuously observing the conversations and behavior of elderly people and collecting and analyzing data.

[0014] "Signs of dementia" refers to abnormal patterns of speech and behavior in an elderly person that are recognized as early symptoms of dementia.

[0015] "Means of notification" refers to the function of communicating information to caregivers when signs of dementia are detected.

[0016] "Caregiving know-how" refers to specific advice and measures based on knowledge and experience related to caregiving.

[0017] "Means for generating advice" refers to the function of receiving questions about caregiving and providing appropriate responses and advice.

[0018] "Reminding means" refers to a function that reminds users to take medicine, make doctor's appointments, etc.

[0019] A "sensor" refers to a device that observes a user's conversations and behavior in real time and collects data.

[0020] "Server" refers to a central computer that analyzes data sent from devices and generates a response.

[0021] "Database" refers to an information storage device within the system that stores information on local consultation centers and welfare systems and provides this information as needed. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[0044] System configuration

[0045] 1. Server

[0046] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[0047] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0048] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[0049] 2. Terminal

[0050] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[0051] The device has the ability to capture the user's conversation and convert the speech into text.

[0052] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[0053] 3. Users

[0054] The users mainly refer to elderly people and their caregivers.

[0055] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[0056] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[0057] Program processing details

[0058] The main functions of the system are as follows:

[0059] 1. Initial Setup

[0060] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[0061] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0062] The server stores the received initial setting information in a database and generates a profile.

[0063] 2. Conversational Interaction

[0064] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[0065] The device converts the voice data into text and sends it to the server.

[0066] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[0067] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[0068] 3. Detecting signs of dementia

[0069] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0070] When the device detects abnormal speech or behavior, it sends the data to the server.

[0071] The server analyzes the data and runs algorithms to determine signs of dementia.

[0072] If the server detects any signs, it will send an alert to the caregiver via the device.

[0073] 4. Providing nursing care know-how

[0074] The user (caregiver) asks questions about care to the terminal.

[0075] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[0076] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[0077] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0078] 5. Reminder function

[0079] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0080] The device sends the reminder information to the server, which stores it.

[0081] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[0082] The terminal converts the notification message into voice and reminds the user.

[0083] Specific examples

[0084] Concrete example of conversational interaction

[0085] 1. User (elderly person): "I went for a walk today."

[0086] 2. Device: Capture audio with the microphone and temporarily store it.

[0087] 3. Device: Converts speech to text and sends it to the server.

[0088] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[0089] 5. Server: Generates and sends the response to the device.

[0090] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[0091] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[0092] The processing flow will be explained below.

[0093] Program processing steps

[0094] Initial Setup

[0095] 1. Step 1:

[0096] The user turns on the device and enters basic configuration information, including Wi-Fi settings and the name, age, and health information of the senior and caregiver.

[0097] 2. Step 2:

[0098] The device will perform an initial calibration based on the basic configuration information you entered, a process that includes testing the sensors, microphone, and speaker.

[0099] 3. Step 3:

[0100] The terminal sends initial configuration information to the server, including the user's profile data.

[0101] 4. Step 4:

[0102] The server stores the received profile data in a database and generates and manages user profiles.

[0103] Conversational Interactions

[0104] 1. Step 1:

[0105] The user (elderly person or caregiver) speaks into the device. Example: "I went for a walk today."

[0106] 2. Step 2:

[0107] The device captures the user's voice with a microphone and temporarily stores it as audio data.

[0108] 3. Step 3:

[0109] The device converts the voice data into text, using voice recognition technology to accurately convert the user's words into text data.

[0110] 4. Step 4:

[0111] The terminal transmits the converted text data to the server.

[0112] 5. Step 5:

[0113] The server analyzes the received text data, using NLP (Natural Language Processing) algorithms to understand what the user is saying and search for data to generate an appropriate response.

[0114] 6. Step 6:

[0115] The server runs a response generation algorithm and generates a text response, e.g., "How was the weather today?"

[0116] 7. Step 7:

[0117] The server transmits the generated response data to the terminal.

[0118] 8. Step 8:

[0119] The device converts the response data into speech and responds to the user with speech. Example: "How was the weather today?"

[0120] Dementia symptom detection

[0121] 1. Step 1:

[0122] The device constantly monitors the elderly person's conversations and behavior, collecting data using sensors and microphones.

[0123] 2. Step 2:

[0124] The device transmits data on abnormal behavior and speech to a server in real time.

[0125] 3. Step 3:

[0126] The server analyzes the data it receives and runs algorithms to detect signs of dementia, using pattern-matching techniques to identify abnormal behavioral patterns.

[0127] 4. Step 4:

[0128] If the server detects signs of dementia, it generates an alert, including specific abnormal behaviors and their frequency.

[0129] 5. Step 5:

[0130] The server transmits the generated alert information to the terminal.

[0131] 6. Step 6:

[0132] The device notifies the caregiver of the alert information via audio or visual messages.

[0133] Providing nursing care know-how

[0134] 1. Step 1:

[0135] The user (caregiver) asks a question about caregiving to the device. Example: "How can I guide the elderly effectively?"

[0136] 2. Step 2:

[0137] The device captures the user's question with a microphone and saves it as audio data.

[0138] 3. Step 3:

[0139] The device converts the voice data into text and sends it to the server.

[0140] 4. Step 4:

[0141] The server analyzes the received text data and understands the intent of the question.

[0142] 5. Step 5:

[0143] The server searches the database for relevant care know-how and advice.

[0144] 6. Step 6:

[0145] The server generates appropriate advice in text format and sends it to the terminal.

[0146] 7. Step 7:

[0147] The device converts the received advice into speech and conveys it to the user. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[0148] Reminder function

[0149] 1. Step 1:

[0150] Users input information such as medication times and hospital appointment information into the device, either via voice or text.

[0151] 2. Step 2:

[0152] The device temporarily stores the entered reminder information and sends it to the server.

[0153] 3. Step 3:

[0154] The server stores the reminder information in a database and manages the schedule.

[0155] 4. Step 4:

[0156] When the reminder time approaches, the server generates a notification message.

[0157] 5. Step 5:

[0158] The server sends the generated notification message to the terminal.

[0159] 6. Step 6:

[0160] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[0161] In this way, the system functions effectively through each processing step, providing support to the elderly and their caregivers.

[0162] Example 1

[0163] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] In modern society, elderly people and their caregivers face many challenges in their daily lives. In particular, there is a need for early detection of dementia in the elderly and support in daily life. Furthermore, providing effective caregiving know-how and reducing the burden on caregivers are also important issues. Addressing these challenges requires a system that can monitor the conversations and behavior of elderly people and provide appropriate advice and reminders.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0166] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses to speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for generating caregiving know-how and response text using a generative AI model, means for processing the speech data using speech recognition and speech synthesis software, and means for inputting basic setting information of the elderly person and the caregiver and generating a profile. This enables early detection of signs of dementia in the elderly, support for daily life, and providing effective advice to caregivers.

[0167] The "means for capturing conversation" is a function for acquiring the voice uttered by the user using a voice input device such as a microphone.

[0168] "Means for converting voice to text" refers to a function that converts acquired voice data into text format using voice recognition technology.

[0169] The "means for analyzing the generated text data and generating an appropriate response" is a function for analyzing the text data using natural language processing technology and generating an appropriate response based on the analysis.

[0170] The "means for converting the generated response into speech and presenting it to the user" is a function for converting the generated text response into speech using speech synthesis technology and presenting it to the user.

[0171] "Means for monitoring conversations and behaviors and detecting signs of dementia" refers to a function that uses sensors to continuously monitor the user's conversations and behaviors and detects signs of dementia based on an algorithm.

[0172] "Means for notifying caregivers when signs of dementia are detected" is a function that sends alerts and notifications to caregivers via a server when signs of dementia are detected by the algorithm.

[0173] "Means for analyzing questions and generating advice in order to provide know-how on caregiving" refers to a function that analyzes questions on caregiving from users and generates appropriate advice using a database and a generative AI model.

[0174] "Means to remind you to take medicine or make hospital appointments" is a function that records the time the user sets to take medicine or hospital appointment information, and notifies you when the reminder time approaches.

[0175] "Means for generating care know-how and response text using a generative AI model" is a function that uses a generative AI model to generate know-how about care and responses to user questions in text format.

[0176] "Means for processing voice data using voice recognition and voice synthesis software" refers to the function of converting voice into text using voice recognition software and converting text into voice using voice synthesis software.

[0177] The "means for inputting basic setting information and generating a profile" is a function in which a user inputs initial setting information, and the server generates a profile for each user based on this information.

[0178] MODE FOR CARRYING OUT THE INVENTION

[0179] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[0180] System configuration

[0181] 1. Server

[0182] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[0183] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0184] The server uses generative AI models to analyze the information sent by the user and generate the necessary responses or alerts.

[0185] 2. Terminal

[0186] The device is designed as a stuffed toy device and is equipped with a microphone, speaker, and various sensors.

[0187] The device uses voice recognition software to capture the user's speech and convert it into text.

[0188] The terminal transmits data to the server as needed, and also conveys responses from the server to the user by voice.

[0189] 3. Users

[0190] The users mainly refer to elderly people and their caregivers.

[0191] Users can talk to the device to engage in conversational interactions and ask questions about caregiving.

[0192] Users receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[0193] Program processing details

[0194] 1. Initial Setup

[0195] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and senior's names, ages, and medication information.

[0196] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0197] The server stores the received initial setting information in a database and generates profiles for the elderly person and their caregiver.

[0198] 2. Conversational Interaction

[0199] When a user speaks to the device, the device captures the voice with its microphone and temporarily stores it. For example, if a user says, "I went for a walk today," the device's microphone will capture this.

[0200] The device converts the voice data into text using voice recognition software and sends it to the server.

[0201] The server analyzes the text data and uses a generative AI model (e.g., a GPT-based model) to generate an appropriate response, such as "How was the weather today?"

[0202] The server transmits the generated response data to the terminal, which converts it into voice using speech synthesis software and conveys the response to the user.

[0203] 3. Detecting signs of dementia

[0204] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0205] If the device detects abnormal speech or behavior, it will send that data to the server. For example, if an elderly person frequently asks the same question, the device will send that data to the server.

[0206] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[0207] If the server detects any signs, it will send an alert to the caregiver via the device.

[0208] 4. Providing nursing care know-how

[0209] The user (caregiver) asks a question about care to the terminal, such as "How should I assist with eating?"

[0210] The device captures the question with a microphone, converts it into text using speech recognition software, and sends it to a server.

[0211] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[0212] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0213] 5. Reminder function

[0214] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0215] The device sends the reminder information to the server, which stores it in a database.

[0216] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[0217] The terminal converts the notification message into voice and reminds the user.

[0218] Specific examples

[0219] Concrete example of conversational interaction

[0220] 1. User (elderly person): "I went for a walk today."

[0221] 2. Device: Capture audio with the microphone and temporarily store it.

[0222] 3. Device: Converts speech to text and sends it to the server.

[0223] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[0224] 5. Server: Generates and sends the response to the device.

[0225] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[0226] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[0227] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0228] Initial Setup

[0229] Step 1:

[0230] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and the senior's names, ages, and medication information.

[0231] Input: Wi-Fi settings, basic information of the caregiver and the elderly person (name, age, medication information)

[0232] Output: Initial settings information is saved on the device

[0233] Specific behavior:

[0234] The user manually enters the necessary information on the initial setup screen of the device.

[0235] Step 2:

[0236] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0237] Input: User-entered initial configuration information

[0238] Output: Configuration information sent to the server

[0239] Specific behavior:

[0240] The device establishes a Wi-Fi connection and sends the initial configuration information in packets to the server.

[0241] The device will perform calibration of its internal sensors and microphones.

[0242] Step 3:

[0243] The server stores the received initial setting information in a database and generates a profile.

[0244] Input: Setting information sent from the device

[0245] Output: Profiles stored in the database

[0246] Specific behavior:

[0247] The server writes the configuration information to a database and generates profile IDs for the elderly person and their caregiver.

[0248] Conversational Interactions

[0249] Step 1:

[0250] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[0251] Input: User speech

[0252] Output: Temporarily saved audio data

[0253] Specific behavior:

[0254] The device's microphone captures audio data in real time and temporarily stores it in local storage.

[0255] Step 2:

[0256] The device converts the voice data into text using voice recognition software and sends it to the server.

[0257] Input: Temporarily saved audio data

[0258] Output: Data converted to text

[0259] Specific behavior:

[0260] The device converts voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[0261] Step 3:

[0262] The server analyzes the text data and generates an appropriate response using a generative AI model.

[0263] Input: Text format data

[0264] Output: The generated response text

[0265] Specific behavior:

[0266] The server analyzes the text data using models such as OpenAI's GPT model and generates an appropriate response.

[0267] Step 4:

[0268] The server transmits the generated response data to the terminal, which converts it into speech using speech synthesis software and responds to the user.

[0269] Input: Generated response text

[0270] Output: The transcribed response

[0271] Specific behavior:

[0272] The server sends response data generated by the AI ​​model to the terminal.

[0273] The device uses voice synthesis technology such as Amazon Polly to convert text data into speech and transmits it to the user through the speaker.

[0274] Dementia symptom detection

[0275] Step 1:

[0276] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0277] Input: Elderly conversations and behavior

[0278] Output: Monitoring result data

[0279] Specific behavior:

[0280] The device's built-in microphones, accelerometers, and cameras capture the elderly's behavioral data in real time.

[0281] Step 2:

[0282] When the device detects abnormal speech or behavior, it sends the data to the server.

[0283] Input: Monitoring result data

[0284] Output: Behavioral data sent to the server

[0285] Specific behavior:

[0286] When a terminal detects an abnormality, it sends the data in packet form to the server.

[0287] Step 3:

[0288] The server analyzes the data and runs algorithms to determine signs of dementia.

[0289] Input: Received behavioral data

[0290] Output: Detected dementia symptoms

[0291] Specific behavior:

[0292] The server uses machine learning algorithms to analyze behavioral data and detect abnormal trends.

[0293] Step 4:

[0294] If the server detects any signs, it will send an alert to the caregiver via the device.

[0295] Input: Detected dementia symptoms

[0296] Output: Alert notification to caregiver

[0297] Specific behavior:

[0298] The server generates an alert message and sends it to the terminal.

[0299] The device will notify caregivers of alerts via speaker or display.

[0300] Providing nursing care know-how

[0301] Step 1:

[0302] The user (caregiver) asks questions about care to the terminal.

[0303] Input: Caregiver's voice question

[0304] Output: Temporarily saved audio data

[0305] Specific behavior:

[0306] The device captures the caregiver's questions and temporarily stores the audio.

[0307] Step 2:

[0308] The device uses voice recognition software to convert the question into text and send it to the server.

[0309] Input: Temporarily saved audio data

[0310] Output: Data converted to text

[0311] Specific behavior:

[0312] The device uses voice recognition software to convert the voice data into text and send it to the server.

[0313] Step 3:

[0314] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[0315] Input: Text question data

[0316] Output: Generated advice

[0317] Specific behavior:

[0318] The server uses a generative AI model to analyze the text data and generate appropriate care advice.

[0319] Step 4:

[0320] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0321] Input: Generated advice text

[0322] Output: Advice converted to speech

[0323] Specific behavior:

[0324] The server transmits the generated advice to the terminal.

[0325] The device uses speech synthesis software to convert advice into voice and convey it to the caregiver.

[0326] Reminder function

[0327] Step 1:

[0328] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0329] Input: Reminder information entered by the user

[0330] Output: Reminder information saved on the device

[0331] Specific behavior:

[0332] The user manually enters the reminder information into an input form on the device.

[0333] Step 2:

[0334] The device sends the reminder information to the server, which stores it in a database.

[0335] Input: Reminder information sent from the device

[0336] Output: Reminder information stored in the database

[0337] Specific behavior:

[0338] The device sends the reminder information to the server, which stores it in a database.

[0339] Step 3:

[0340] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[0341] Input: Reminder information stored in the database

[0342] Output: The generated notification message

[0343] Specific behavior:

[0344] The server generates a notification message based on the set reminder time and sends it to the terminal.

[0345] Step 4:

[0346] The terminal converts the notification message into voice and reminds the user.

[0347] Input: The generated notification message

[0348] Output: Reminder message converted to audio

[0349] Specific behavior:

[0350] The device uses speech synthesis software to convert notification messages into voice and provide reminders.

[0351] (Application example 1)

[0352] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] Supporting the elderly and their caregivers has become an important issue in modern society. In particular, there is a demand for conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. At the same time, the quality and functionality of the devices and robots used to provide this support are also extremely important. In the manufacturing process of elderly support robots, quality inspection, function testing, and progress management of manufacturing work are challenges.

[0354] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0355] In this invention, the server includes a means for capturing conversations between the elderly person and their caregiver and converting the speech into text, a means for analyzing the generated text data and generating appropriate responses, and a means for converting the generated responses into speech and presenting them to the user. This enables high-performance support for the elderly person and their caregiver, as well as the implementation of quality inspections and functional tests in the manufacturing process, progress management of manufacturing work using voice recognition, and scheduling and notifying quality inspections using a reminder function.

[0356] A "support system for older people and their caregivers" is an integrated system that provides the services and functions needed by both older people and their caregivers.

[0357] "Means for capturing conversations and converting speech to text" refers to technology that collects a user's speech and converts it into text information.

[0358] "Means for analyzing generated text data and generating an appropriate response" refers to a technology that analyzes text data and determines the optimal response or action based on its content.

[0359] "Means for converting the generated response into audio and presenting it to the user" refers to technology that converts the generated text-based response into audio format and provides it to the user.

[0360] "Means for monitoring conversations and behavior to detect signs of dementia" refers to technology that monitors a user's speech and behavior patterns and detects abnormalities.

[0361] "Means to notify caregivers if signs of dementia are detected" is a function that sends an alert to caregivers when early signs of dementia are confirmed.

[0362] "Means for analyzing questions and generating advice in order to provide know-how related to caregiving" refers to technology that analyzes questions from caregivers and generates and provides appropriate advice and information.

[0363] "Medication and medical appointment reminders" refers to features that remind users of specific times or events (e.g., medication time, doctor's appointment time).

[0364] "Means for conducting quality inspections and product functionality tests during the manufacturing process and recording the results" refers to a system that checks the quality and functionality of products at each stage of manufacturing and stores the results as a log.

[0365] "Means for managing the progress of manufacturing work using voice recognition" is a technology that uses voice input to track and manage the status and progress of manufacturing work.

[0366] "Means for scheduling and notifying quality inspections using a reminder function" is a function that allows you to schedule quality inspections and notify you when the time comes.

[0367] This invention is a system that supports elderly people and their caregivers, and in particular, it realizes conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[0368] System configuration

[0369] 1. Server

[0370] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia. The analysis uses NLP (natural language processing) technology.

[0371] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0372] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[0373] Manage quality inspections and product functionality tests during the manufacturing process and record the results.

[0374] 2. Terminal

[0375] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[0376] The device has the ability to capture the user's conversation and convert the speech into text.

[0377] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[0378] The terminal uses voice recognition to manage the progress of manufacturing work.

[0379] 3. Users

[0380] The users mainly refer to elderly people and their caregivers.

[0381] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[0382] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[0383] Program processing details

[0384] The main functions of the system are as follows:

[0385] 1. Initial Setup

[0386] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[0387] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0388] The server stores the received initial setting information in a database and generates a profile.

[0389] 2. Conversational Interaction

[0390] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[0391] The device converts the voice data into text and sends it to the server.

[0392] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[0393] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[0394] 3. Detecting signs of dementia

[0395] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0396] When the device detects abnormal speech or behavior, it sends the data to the server.

[0397] The server analyzes the data and runs algorithms to determine signs of dementia.

[0398] If the server detects any signs, it will send an alert to the caregiver via the device.

[0399] 4. Providing nursing care know-how

[0400] The user (caregiver) asks questions about care to the terminal.

[0401] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[0402] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[0403] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0404] 5. Reminder function

[0405] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0406] The device sends the reminder information to the server, which stores it.

[0407] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[0408] The terminal converts the notification message into voice and reminds the user.

[0409] 6. Quality inspection and function test

[0410] During the manufacturing process, the terminal performs quality inspections and product functionality tests and transmits the results to the server.

[0411] The server records the results and generates alerts for any defective or problematic products.

[0412] 7. Manufacturing work progress management

[0413] The terminal uses voice recognition to manage the progress of manufacturing work and transmits necessary information and instructions to the server.

[0414] The server tracks progress and provides timely feedback.

[0415] 8. Reminders feature expansion

[0416] The device can use the reminder function to schedule and notify quality inspections.

[0417] Based on this information, the server manages the schedule and sends notifications at the appropriate times.

[0418] Specific examples

[0419] Concrete example of conversational interaction

[0420] 1. User (elderly person): "I went for a walk today."

[0421] 2. Device: Capture audio with the microphone and temporarily save it.

[0422] 3. Device: Converts speech into text and sends it to the server.

[0423] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[0424] 5. Server: Sends the generated response to the terminal.

[0425] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[0426] Examples of quality inspections and functional tests

[0427] 1. Terminal: Inspects each part of the robot during production.

[0428] 2. Server: Records inspection results and generates alerts if there are any problems.

[0429] 3. Terminal: The test is completed and the results are sent to the server.

[0430] 4. Terminal: "The robot's serial number is R12345, and this test result is a pass."

[0431] This effectively provides support for both the elderly and their caregivers, while also ensuring quality control and progress management of the manufacturing process.

[0432] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0433] Step 1:

[0434] The user performs the initial setup of the system. The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and caregiver into the device. The device calibrates the sensors and microphone based on this information and sends the setup information to the server. The server generates a profile based on this initial setup information and saves it in a database. The input is the user's basic setup information, and the output is the server's creation and saving of the profile.

[0435] Step 2:

[0436] The user speaks into the device. The device captures the voice with a microphone and temporarily stores it. Since the voice data cannot be used as is, the device converts the voice into text. This converted text data is sent to the server. The input is the user's voice, and the output is text data sent to the server.

[0437] Step 3:

[0438] The server analyzes the transmitted text data and generates an appropriate response using NLP (natural language processing) technology. This requires understanding the context and intent from the text data and generating a corresponding response. The generated response data is then sent to the terminal. The input is the text data, and the output is the generated response.

[0439] Step 4:

[0440] The terminal converts the response received from the server into speech and presents it to the user. The user receives the speech response. The input is the response data from the server, and the output is the speech response to the user.

[0441] Step 5:

[0442] The device uses sensors to constantly monitor the user's conversations and behavior. If the sensors detect abnormal speech or behavior, the device temporarily stores it and sends it to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, the server sends an alert to the caregiver via the device. The input is sensor data, and the output is a dementia sign alert.

[0443] Step 6:

[0444] The user (caregiver) asks a question about caregiving to the device. The device captures the question with a microphone, converts the voice into text, and sends it to the server. The server analyzes the text data, searches a database for appropriate caregiving know-how and advice, and sends the generated advice to the device. The device then conveys this to the user by voice. The input is the voice of the question, and the output is caregiving advice.

[0445] Step 7:

[0446] The user sets the time to take medicine and hospital appointment information on the device. The device sends this reminder information to the server, which stores it. When the reminder time approaches, the server generates a notification message and sends it to the device. The device converts the notification message into audio and reminds the user. The input is the reminder setting information, and the output is the reminder notification.

[0447] Step 8:

[0448] During the manufacturing process, the terminal performs quality inspections and product functionality tests. The terminal transmits the inspection results for each item to the server, which records the results and generates alerts as needed. The input is the inspection data, and the output is the quality inspection result record and alerts.

[0449] Step 9:

[0450] The terminal uses voice recognition to manage the progress of manufacturing work. When the user speaks to report progress, the terminal converts the speech into text and sends it to the server. The server updates the progress based on the transmitted data and provides necessary feedback. The input is the voice progress report, and the output is progress updates and feedback.

[0451] Step 10:

[0452] The terminal uses the reminder function to schedule quality inspections and notify the user. The server manages the schedule based on this information and notifies the user at the appropriate time. The input is the inspection schedule setting information, and the output is the inspection schedule notification.

[0453] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0454] This invention is a system to support elderly people and their caregivers, and aims to provide more effective support by incorporating an emotion engine that recognizes the user's emotions, in addition to conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[0455] System configuration

[0456] 1. Server

[0457] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[0458] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0459] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[0460] The server also generates a response based on the emotional state sent from the emotion engine.

[0461] 2. Terminal

[0462] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, sensors, and camera.

[0463] The device has the ability to capture the user's conversation and convert the speech into text.

[0464] The terminal transmits information to the server as needed and also conveys responses from the server to the user by voice.

[0465] The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[0466] 3. Users

[0467] The users mainly refer to elderly people and their caregivers.

[0468] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[0469] Users receive reminders, alerts, and emotional responses from the device, which they can use as a reference for their daily lives and caregiving.

[0470] Program processing details

[0471] The main functions of the system are as follows:

[0472] 1. Initial Setup

[0473] The user resets the device and enters basic configuration information, including Wi-Fi settings and basic information about the senior and caregiver.

[0474] The device calibrates the sensors, microphone, and camera based on this information and sends the configuration information to the server.

[0475] The server stores the received profile data in a database and generates a user profile.

[0476] 2. Conversational Interaction

[0477] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[0478] The device converts the voice data into text and sends it to the server.

[0479] The server analyzes the text data and uses NLP (Natural Language Processing) algorithms to find data to generate an appropriate response.

[0480] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[0481] 3. Emotion Recognition by Emotion Engine

[0482] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them.

[0483] The device inputs this data into an emotion engine to recognize the user's emotional state in real time.

[0484] The emotion engine sends the recognized emotional state to the server.

[0485] 4. Generating Emotion-Based Responses

[0486] The server then executes an algorithm based on the received emotional state data to generate a response appropriate to the user's current state of mind.

[0487] For example, if the user is in a sad emotional state, a supportive or encouraging response may be generated.

[0488] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[0489] 5. Detecting signs of dementia

[0490] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0491] When the device detects abnormal speech or behavior, it sends the data to the server.

[0492] The server analyzes the data and runs algorithms to determine signs of dementia.

[0493] If the server detects a symptom, it generates alert information.

[0494] The device notifies the caregiver of the alert information.

[0495] 6. Providing nursing care know-how

[0496] The user (caregiver) asks questions about care to the terminal.

[0497] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[0498] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[0499] The server generates advice and sends it to the terminal.

[0500] The terminal converts the received advice into voice and conveys it to the user.

[0501] 7. Reminder function

[0502] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0503] The device sends the reminder information to the server, which stores it.

[0504] When the reminder time approaches, the server generates a notification message.

[0505] The server sends the generated notification message to the terminal.

[0506] The terminal converts the notification message into voice and reminds the user.

[0507] Specific examples

[0508] Specific examples of emotion recognition using emotion engines

[0509] 1. User (elderly): "I've been feeling unwell and depressed lately."

[0510] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[0511] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[0512] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[0513] 5. Server: Generates an encouraging response such as "Cheer up" based on the emotional state "sad."

[0514] 6. Server: Sends the generated response to the device.

[0515] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[0516] In this way, the system as a whole aims to reduce the mental and physical burden on elderly people and caregivers by incorporating conversational interaction, emotion recognition, and care support functions. Utilizing an emotion engine enables flexible responses according to the user's emotional state, providing more personalized support.

[0517] The processing flow will be explained below.

[0518] Program processing steps

[0519] Emotion recognition and response generation by emotion engine

[0520] 1. Step 1:

[0521] The user speaks to the device. Example: "I've been feeling unwell and depressed lately."

[0522] 2. Step 2:

[0523] The device uses a microphone to capture the user's voice and a camera to capture the user's facial expressions.

[0524] 3. Step 3:

[0525] The device converts the captured voice data into text using voice recognition technology and sends it to the emotion engine along with facial expression data.

[0526] 4. Step 4:

[0527] The emotion engine analyzes the tone of the voice data and facial expression data and recognizes the user's emotional state as "sad."

[0528] 5. Step 5:

[0529] The emotion engine transmits the recognized emotional state data to the server.

[0530] 6. Step 6:

[0531] The server analyzes the received emotional state data and generates an appropriate response, e.g., encouraging words such as "Cheer up!"

[0532] 7. Step 7:

[0533] The server transmits the generated response data to the terminal.

[0534] 8. Step 8:

[0535] The device converts the received response data into voice and responds to the user with voice. Example: "Cheer up."

[0536] Dementia symptom detection

[0537] 1. Step 1:

[0538] The device constantly monitors the elderly person's conversations and behavior using sensors and microphones.

[0539] 2. Step 2:

[0540] When the device detects abnormal behavior (e.g. repeating the same word multiple times), it sends the data to the server in real time.

[0541] 3. Step 3:

[0542] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[0543] 4. Step 4:

[0544] If the server detects signs of dementia, it generates alert information and sends data to the terminal to notify the caregiver.

[0545] 5. Step 5:

[0546] The device will notify the caregiver of the alert notification with audio and visual messages, for example, "The elderly person is repeating the same words over and over. Attention is needed."

[0547] Providing nursing care know-how

[0548] 1. Step 1:

[0549] The user (caregiver) asks a question about caregiving to the device. Example: "How can we effectively guide the elderly?"

[0550] 2. Step 2:

[0551] The device captures the question with a microphone and saves it as audio data.

[0552] 3. Step 3:

[0553] The device converts the voice data into text using voice recognition technology and sends it to the server.

[0554] 4. Step 4:

[0555] The server analyzes the received text data and runs NLP algorithms to understand the intent of the question.

[0556] 5. Step 5:

[0557] The server searches the database for relevant care know-how and advice.

[0558] 6. Step 6:

[0559] The server generates appropriate advice and sends it to the device. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[0560] 7. Step 7:

[0561] The terminal converts the received advice into voice and conveys it to the user.

[0562] Reminder function

[0563] 1. Step 1:

[0564] Users enter medication times and hospital appointment information into the terminal by voice or manually.

[0565] 2. Step 2:

[0566] The device temporarily stores the entered reminder information and sends it to the server.

[0567] 3. Step 3:

[0568] The server stores the reminder information in a database and manages the schedule.

[0569] 4. Step 4:

[0570] When the reminder time approaches, the server generates a reminder notification message.

[0571] 5. Step 5:

[0572] The server sends the generated remind notification message to the terminal.

[0573] 6. Step 6:

[0574] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[0575] This system combines multiple functions to provide multifaceted support to the elderly and their caregivers. By utilizing emotion recognition through an emotion engine, it is possible to provide more personalized responses, aiming to increase user satisfaction.

[0576] Example 2

[0577] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0578] Providing effective support to elderly people and their caregivers requires not only conversational interaction and information provision, but also responses that take into account the user's emotions and psychological state. However, conventional systems have had difficulty comprehensively recognizing emotions, detecting signs of dementia, and providing care know-how. Furthermore, there have been no systems that can reduce the user's mental burden and provide flexible responses that correspond to their emotions. This has made it difficult to improve the satisfaction of elderly people and caregivers.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0580] In this invention, the server

[0581] A means of capturing the conversations of the senior and caregiver and converting the speech to text;

[0582] means for analyzing the generated text data and generating an appropriate response;

[0583] means for converting the generated response into speech and presenting it to the user;

[0584] A means of monitoring the conversations and behavior of elderly people to detect signs of dementia,

[0585] A means to notify caregivers if signs of dementia are detected; and

[0586] A means for analyzing questions and generating advice to provide knowledge about care;

[0587] A way to remind you to take your medicine or make a doctor's appointment,

[0588] a means for recognizing a user's emotion using an emotion engine and generating a response according to the emotion;

[0589] means for transmitting the emotional state to a server for analysis and generating an appropriate response based on the user's psychological state;

[0590] This will enable comprehensive and personalized support for older people and their caregivers.

[0591] "Elderly people and caregivers" refers to people who play a role in supporting the daily lives and health management of older people.

[0592] "Capture" refers to the technology of collecting data such as audio and video and storing it in digital form.

[0593] "Speech-to-text" refers to the technology of analyzing voice data and converting its contents into text data.

[0594] "Generated text data" refers to data obtained by analyzing speech and converting it into character data.

[0595] "Response" refers to information or messages returned in response to user input.

[0596] "Monitoring" refers to the act of continuously observing the state or behavior of a specific object to detect abnormalities or changes.

[0597] "Detecting signs of dementia" refers to finding changes in specific behaviors or speech that may be indicative of dementia.

[0598] "Notify caregiver" refers to providing important detected information to the caregiver as a warning or alert.

[0599] "Providing knowledge" means providing appropriate advice and guidance based on the questions asked using accumulated information.

[0600] A "remind" is a notification in advance about a specific time or event so that you don't forget.

[0601] An "emotion engine" refers to technology that analyzes a user's tone of voice, facial expressions, etc. to identify their current emotional state.

[0602] "Responding based on psychological state" refers to generating a response appropriate to the user's mental state at the time based on their emotional data.

[0603] This invention is a system that supports elderly people and their caregivers, and provides comprehensive and personalized support by incorporating conversational interaction, dementia symptom detection, caregiving know-how provision, reminder functions, and an emotion engine.

[0604] System configuration

[0605] This system mainly consists of three elements: a server, a terminal, and a user.

[0606] 1. Server

[0607] The server analyzes voice data, generates text data, generates responses, and detects signs of dementia. The server is equipped with a database that stores information on care know-how, local consultation centers, and welfare systems. The server analyzes the information sent by the user and generates the necessary responses and alerts. It also generates responses based on the emotional state sent from the emotion engine.

[0608] 2. Terminal

[0609] The device is designed as a plush toy and is equipped with a microphone, speaker, sensors, and camera. The device has the ability to capture the user's conversation and convert the speech into text. If necessary, it sends information to the server and communicates the server's response to the user via voice. The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[0610] 3. Users

[0611] Users are primarily elderly people and their caregivers. By speaking into the device, users can engage in conversational interactions and ask questions about caregiving. Users receive reminders, alerts, and emotional responses from the device, which they can use as references for their daily lives and caregiving.

[0612] Detailed System Operation

[0613] The detailed operation of this system will be described below.

[0614] 1. Initial Setup

[0615] The user initializes the device and enters Wi-Fi settings and basic information (information about the senior and caregiver). The device calibrates the sensors, microphone, and camera based on this information and sends the setting information to the server. The server saves the received information and creates a user profile.

[0616] 2. Conversational Interaction

[0617] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is converted into text and sent to the server. The server analyzes the text data and generates an appropriate response using a natural language processing (NLP) algorithm. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[0618] 3. Emotion Recognition by Emotion Engine

[0619] The device captures and temporarily stores the user's voice tone and facial expressions. This data is input to the emotion engine, which recognizes the user's emotional state in real time. The emotion engine then transmits the recognized emotional state to the server.

[0620] 4. Generating Emotion-Based Responses

[0621] The server generates a response appropriate to the user's current psychological state based on the emotional state data. For example, if the emotional state is "sad," it generates a supportive or encouraging response. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[0622] 5. Detecting signs of dementia

[0623] The device uses sensors to monitor the elderly person's conversations and behavior. If it detects abnormal speech or behavior, it sends the data to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, it generates an alert and notifies the caregiver via the device.

[0624] 6. Providing nursing care know-how

[0625] When a user (caregiver) asks a question about caregiving to the device, the device captures the question, converts the voice to text, and sends it to the server. The server analyzes the data and searches a database for appropriate caregiving know-how and advice. It generates advice and sends it to the device. The device then converts the received advice into voice and conveys it to the user.

[0626] 7. Reminder function

[0627] The user inputs information about medication times and hospital appointment times into the device. The device then sends the reminder information to the server, which stores it. When the reminder time approaches, the server sends a notification message to the device, which then converts it into audio and notifies the user.

[0628] Specific examples

[0629] Specific examples of emotion recognition using emotion engines

[0630] 1. User (elderly): "I've been feeling unwell and depressed lately."

[0631] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[0632] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[0633] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[0634] 5. Server: Generates a supportive response such as "Cheer up" based on the emotional state "sad."

[0635] 6. Server: Generates and sends the response to the device.

[0636] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[0637] Examples of prompt statements

[0638] Below are some example prompts that utilize generative AI models (e.g., GPT-3) to generate natural-sounding responses.

[0639] "An elderly user says, 'I've been feeling unwell and depressed lately.' The emotion engine detects the emotional state as 'sad.' Please generate an appropriate message of encouragement for this user."

[0640] In this way, this system, with its conversational interaction, emotion recognition, and care support functions, reduces the mental and physical burden on elderly people and caregivers and provides personalized support.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1: Initial Setup

[0643] The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and their caregiver, including the user's name, age, and health status.

[0644] Your device uses the information you provide to calibrate sensors, microphones, and cameras, for example, adjusting microphone sensitivity and focusing the camera, optimizing device performance.

[0645] The terminal transmits the initial setting information to the server.

[0646] The server stores the received setting information in a database and generates a user profile, which prepares the data required for subsequent processing.

[0647] Step 2: Conversational Interaction

[0648] The user speaks to the terminal, for example, saying a greeting such as "Good morning."

[0649] The device captures the audio using a microphone and temporarily stores it as audio data.

[0650] The device converts the captured voice data into text to generate text data, which becomes input data.

[0651] The terminal transmits the generated text data to the server.

[0652] The server analyzes the received text data and generates an appropriate response using an NLP (natural language processing) algorithm. For example, for the text "Good morning," it generates the response "Good morning. What are your plans for today?" This becomes the output data.

[0653] The server transmits the generated response data to the terminal.

[0654] The terminal converts the received response data into voice and responds to the user.

[0655] Step 3: Emotion recognition by the emotion engine

[0656] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them, for example, as digital data, such as voice waveforms and facial movements.

[0657] The device inputs the captured data into an emotion engine to recognize the emotional state. The emotion engine uses a deep learning algorithm to analyze the data and label the emotional state as "sad" or "happy." This becomes the output data.

[0658] The emotion engine transmits the recognized emotional state data to a server.

[0659] Step 4: Generate an emotion-based response

[0660] The server generates a response appropriate to the user's current psychological state based on the received emotional state data. For example, if the user is feeling sad, the server generates a message of encouragement such as "Cheer up!"

[0661] The server transmits the generated response data to the terminal.

[0662] The terminal converts the received response data into voice and responds to the user.

[0663] Step 5: Detecting signs of dementia

[0664] The device uses sensors to monitor the elderly's conversations and behavior, such as motion sensors to monitor walking patterns and microphones to analyze conversation content.

[0665] When the device detects abnormal speech or behavior, it sends the data to the server.

[0666] The server analyzes the received data and runs algorithms to identify signs of dementia, such as by analyzing specific keywords or abnormal behavioral patterns.

[0667] If the server detects signs of dementia, it generates alert information and notifies the caregiver via the terminal.

[0668] Step 6: Providing nursing care know-how

[0669] The user (caregiver) asks a question about care to the device, for example, "What can I do to prevent falls?"

[0670] The device captures the question with a microphone and converts the voice into text to generate text data.

[0671] The terminal transmits the generated text data to the server.

[0672] The server analyzes the received text data and searches a database for appropriate care know-how and advice.

[0673] The server generates appropriate advice and sends it to the terminal, for example, generating a response such as "It would be a good idea to install handrails."

[0674] The terminal converts the received advice data into voice and conveys it to the user.

[0675] Step 7: Reminders

[0676] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0677] The terminal transmits the input remind information to the server.

[0678] The server stores the received reminder information in a database.

[0679] When the reminder time approaches, the server generates a notification message.

[0680] The server sends the generated notification message to the terminal, for example, "It's time to take your medicine at 2 PM."

[0681] The terminal converts the received notification message into voice and reminds the user.

[0682] These steps enable the system to provide comprehensive and personalized support to seniors and their caregivers.

[0683] (Application example 2)

[0684] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0685] Daily life and caregiving entail many burdens for elderly people and their caregivers. Meal preparation and delivery, health monitoring, and psychological support are particularly important. However, conventional systems often fail to adequately address these needs. For example, they lack meal suggestions that take the user's emotional state into account, and real-time mental support during delivery, making it difficult to improve user satisfaction. Furthermore, there are challenges with early detection of dementia symptoms and providing appropriate care know-how. Given these circumstances, there is a need for a comprehensive and personalized support system for elderly people and their caregivers.

[0686] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0687] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses into speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for recognizing the user's emotions in real time and suggesting appropriate meals based on the user's emotional state, and means for providing real-time mental support for situations during delivery. This enables personalized meal suggestions and mental support based on the user's emotional state, thereby improving the quality of life of the elderly person and their caregiver.

[0688] definition statement

[0689] The term "elderly" generally refers to people aged 65 and over.

[0690] A "caregiver" is a person who supports an elderly person in their daily lives and provides the care they need.

[0691] "Emotion engine" refers to a system that includes algorithms that recognize a user's emotional state from their tone of voice and facial expressions.

[0692] "Server" refers to a device that has data processing, analysis and storage functions and manages and controls the entire system.

[0693] "Terminal" refers to an input and output device used by a user that has the functions of voice capture, text conversion, and voice output.

[0694] "Text data" refers to digital information that has been converted from audio data into text format.

[0695] "Nursing care know-how" refers to knowledge, methods, and techniques related to nursing care.

[0696] "Monitoring" refers to the act of continuously observing and recording a subject's conversations and behavior.

[0697] "Signs of dementia" refer to characteristic symptoms and behaviors seen in the early stages or as dementia progresses.

[0698] "Notification" refers to the act of informing a user of important information in real time.

[0699] "Advice" refers to information that provides solutions or recommendations for a particular problem or situation.

[0700] "Remind" refers to the act of notifying and reminding a user in advance of a specific action or event.

[0701] "Meal suggestion" refers to the act of recommending appropriate meal content based on the user's preferences and health condition.

[0702] "Mental support" refers to the act of providing assistance and encouragement to promote the user's mental stability and sense of security.

[0703] invention specification

[0704] This invention is a system that supports elderly people and caregivers, and aims to provide more effective support by including functions such as conversational interaction, emotion recognition, dementia symptom detection, provision of care know-how, meal suggestions, reminders, and mental support.

[0705] System configuration

[0706] This system mainly consists of three elements: a server, a terminal, and a user.

[0707] 1. Server

[0708] The server has the following features:

[0709] Analyzing voice data, generating text data, generating responses

[0710] Dementia symptom detection

[0711] Managing the emotion engine and meal recommendation system

[0712] Real-time mental support during delivery

[0713] Management of various databases (care know-how, local consultation points, welfare system information, etc.)

[0714] 2. Terminal

[0715] The terminal mainly uses the following devices:

[0716] Smartphone, smart glasses, or head-mounted display

[0717] A stuffed toy device equipped with a microphone, speaker, sensor, and camera

[0718] The main features of the terminal are:

[0719] Capture user conversations in real time and convert speech to text

[0720] Sending data to the server and receiving the generated response

[0721] Collecting data to recognize user emotions (voice tone, facial expressions)

[0722] Providing meal suggestions and reminder functions

[0723] 3. Users

[0724] Users are the elderly and their caregivers, and they primarily use the system in the following ways:

[0725] By talking to the device, you can have conversational interactions and ask questions about caregiving.

[0726] Receive responses and reminders from your device

[0727] Receive meal suggestions and mental support from devices and applications

[0728] Program processing details

[0729] Initial Setup

[0730] The user initializes the device and enters basic configuration information, including Wi-Fi settings and a user profile (such as name, age, dietary preferences, and allergy information). The device uses this information to calibrate the sensors, microphone, and camera, and then sends the configuration information to the server. The server then stores the received profile data in a database and creates a user profile.

[0731] Conversational Interaction and Emotion Recognition

[0732] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is then converted into text and sent to the server. The server analyzes the text data and uses an NLP (natural language processing) algorithm to search for data to generate an appropriate response. The generated response data is sent to the device, which then converts it into voice and responds to the user. The device also inputs voice tone and facial expressions into an emotion engine to recognize the user's emotional state in real time.

[0733] Meal suggestions

[0734] Based on the user's emotional state and profile (food preferences and health status), the server runs an algorithm to suggest suitable meals. For example, if the user is in a "sad" emotional state, it will suggest nutritious meals.

[0735] Mental support

[0736] If a delay occurs during delivery, the server checks the user's emotional state again. If the user is anxious, it generates an encouraging message and sends it to the user via the terminal.

[0737] Specific examples

[0738] For example, if a user says, "I haven't had much of an appetite lately," the smartphone's microphone captures the voice. The captured voice data is input into the emotion engine, which recognizes the emotional state as "sad." As a result, the server generates a suggestion such as, "Cheer up! How about this nutritious menu?" and notifies the user via voice through the device.

[0739] Prompt Sentence Examples

[0740] Input: "I haven't had much of an appetite lately."

[0741] Prompt: "Use an emotion engine to recognize the user's emotional state and suggest suitable meals."

[0742] Sample output: "Cheer up! Would you like some of these nutritious options?"

[0743] In this way, the present invention improves the quality of life for seniors and their caregivers by providing personalized support based on the user's emotional state.

[0744] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0745] Program processing steps

[0746] Step 1:

[0747] The user speaks into the terminal.

[0748] Input: User's voice

[0749] What it does: The device captures audio using the built-in microphone.

[0750] Output: Audio data

[0751] Step 2:

[0752] The device converts the voice data into text.

[0753] Input: Audio data

[0754] How it works: The device's built-in software uses voice recognition technology to convert speech into text.

[0755] Output: Text data

[0756] Step 3:

[0757] Sends text data to the server.

[0758] Input: Text data

[0759] Specific operation: The terminal sends text data to the server via the network.

[0760] Output: Text data sent to the server

[0761] Step 4:

[0762] The server parses the received text data and generates an appropriate response.

[0763] Input: Text data

[0764] What it does: The server uses NLP (Natural Language Processing) algorithms to analyze the text data and generate a corresponding response, possibly referencing information in a database.

[0765] Output: Response data

[0766] Step 5:

[0767] The server transmits the generated response data to the terminal.

[0768] Input: Response data

[0769] Specific operation: The server sends response data to the terminal via the network.

[0770] Output: Response data sent to the terminal

[0771] Step 6:

[0772] The terminal converts the response data into voice and presents it to the user.

[0773] Input: Response data

[0774] Specific operation: The device's built-in software converts the text data into voice data and responds to the user audibly using the speaker.

[0775] Output: Voice response

[0776] Step 7:

[0777] The device recognizes the user's emotions in real time.

[0778] Input: User's voice tone and facial expression data

[0779] How it works: The device captures voice tone and facial expression data using a microphone and camera, and then analyzes the data using an emotion engine.

[0780] Output: Emotional state

[0781] Step 8:

[0782] The server suggests suitable meals to the user based on their emotional state.

[0783] Input: Emotional state and user profile data (e.g., food preferences)

[0784] Specific operation: The server uses an emotion engine and a meal suggestion algorithm to analyze the emotional state and profile data and select an appropriate meal.

[0785] Output: Meal suggestion data

[0786] Step 9:

[0787] The server transmits the generated meal suggestion data to the terminal.

[0788] Input: Meal suggestion data

[0789] Specific operation: The server sends meal suggestion data to the terminal via the network.

[0790] Output: Meal suggestion data sent to the device

[0791] Step 10:

[0792] The terminal converts the meal suggestion data into voice and notifies the user.

[0793] Input: Meal suggestion data

[0794] Specific operation: The device's built-in software converts text data into voice data and provides meal suggestions to the user using the speaker.

[0795] Output: Spoken meal suggestions

[0796] Step 11:

[0797] The server provides real-time mental support during delivery situations.

[0798] Input: Delivery status and user emotional state

[0799] Specific operation: The server monitors the delivery status and the user's emotional state in real time and generates mental support messages as needed.

[0800] Output: Mental support message

[0801] Step 12:

[0802] The server transmits the generated mental support message to the terminal.

[0803] Input: Mental support message

[0804] Specific operation: The server sends a mental support message to the terminal via the network.

[0805] Output: Mental support message sent to the device

[0806] Step 13:

[0807] The terminal converts the mental support message into voice and notifies the user.

[0808] Input: Mental support message

[0809] Specific operation: The device's built-in software converts text data into voice data and uses the speaker to provide the user with a mental support message.

[0810] Output: Audio mental support

[0811] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0813] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0814] [Second embodiment]

[0815] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0816] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0817] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0818] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0819] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0820] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0821] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0822] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0823] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0824] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0825] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0826] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0827] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[0828] System configuration

[0829] 1. Server

[0830] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[0831] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0832] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[0833] 2. Terminal

[0834] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[0835] The device has the ability to capture the user's conversation and convert the speech into text.

[0836] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[0837] 3. Users

[0838] The users mainly refer to elderly people and their caregivers.

[0839] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[0840] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[0841] Program processing details

[0842] The main functions of the system are as follows:

[0843] 1. Initial Setup

[0844] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[0845] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0846] The server stores the received initial setting information in a database and generates a profile.

[0847] 2. Conversational Interaction

[0848] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[0849] The device converts the voice data into text and sends it to the server.

[0850] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[0851] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[0852] 3. Detecting signs of dementia

[0853] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0854] When the device detects abnormal speech or behavior, it sends the data to the server.

[0855] The server analyzes the data and runs algorithms to determine signs of dementia.

[0856] If the server detects any signs, it will send an alert to the caregiver via the device.

[0857] 4. Providing nursing care know-how

[0858] The user (caregiver) asks questions about care to the terminal.

[0859] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[0860] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[0861] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0862] 5. Reminder function

[0863] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0864] The device sends the reminder information to the server, which stores it.

[0865] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[0866] The terminal converts the notification message into voice and reminds the user.

[0867] Specific examples

[0868] Concrete example of conversational interaction

[0869] 1. User (elderly person): "I went for a walk today."

[0870] 2. Device: Capture audio with the microphone and temporarily store it.

[0871] 3. Device: Converts speech to text and sends it to the server.

[0872] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[0873] 5. Server: Generates and sends the response to the device.

[0874] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[0875] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[0876] The processing flow will be explained below.

[0877] Program processing steps

[0878] Initial Setup

[0879] 1. Step 1:

[0880] The user turns on the device and enters basic configuration information, including Wi-Fi settings and the name, age, and health information of the senior and caregiver.

[0881] 2. Step 2:

[0882] The device will perform an initial calibration based on the basic configuration information you entered, a process that includes testing the sensors, microphone, and speaker.

[0883] 3. Step 3:

[0884] The terminal sends initial configuration information to the server, including the user's profile data.

[0885] 4. Step 4:

[0886] The server stores the received profile data in a database and generates and manages user profiles.

[0887] Conversational Interactions

[0888] 1. Step 1:

[0889] The user (elderly person or caregiver) speaks into the device. Example: "I went for a walk today."

[0890] 2. Step 2:

[0891] The device captures the user's voice with a microphone and temporarily stores it as audio data.

[0892] 3. Step 3:

[0893] The device converts the voice data into text, using voice recognition technology to accurately convert the user's words into text data.

[0894] 4. Step 4:

[0895] The terminal transmits the converted text data to the server.

[0896] 5. Step 5:

[0897] The server analyzes the received text data, using NLP (Natural Language Processing) algorithms to understand what the user is saying and search for data to generate an appropriate response.

[0898] 6. Step 6:

[0899] The server runs a response generation algorithm and generates a text response, e.g., "How was the weather today?"

[0900] 7. Step 7:

[0901] The server transmits the generated response data to the terminal.

[0902] 8. Step 8:

[0903] The device converts the response data into speech and responds to the user with speech. Example: "How was the weather today?"

[0904] Dementia symptom detection

[0905] 1. Step 1:

[0906] The device constantly monitors the elderly person's conversations and behavior, collecting data using sensors and microphones.

[0907] 2. Step 2:

[0908] The device transmits data on abnormal behavior and speech to a server in real time.

[0909] 3. Step 3:

[0910] The server analyzes the data it receives and runs algorithms to detect signs of dementia, using pattern-matching techniques to identify abnormal behavioral patterns.

[0911] 4. Step 4:

[0912] If the server detects signs of dementia, it generates an alert, including specific abnormal behaviors and their frequency.

[0913] 5. Step 5:

[0914] The server transmits the generated alert information to the terminal.

[0915] 6. Step 6:

[0916] The device notifies the caregiver of the alert information via audio or visual messages.

[0917] Providing nursing care know-how

[0918] 1. Step 1:

[0919] The user (caregiver) asks a question about caregiving to the device. Example: "How can I guide the elderly effectively?"

[0920] 2. Step 2:

[0921] The device captures the user's question with a microphone and saves it as audio data.

[0922] 3. Step 3:

[0923] The device converts the voice data into text and sends it to the server.

[0924] 4. Step 4:

[0925] The server analyzes the received text data and understands the intent of the question.

[0926] 5. Step 5:

[0927] The server searches the database for relevant care know-how and advice.

[0928] 6. Step 6:

[0929] The server generates appropriate advice in text format and sends it to the terminal.

[0930] 7. Step 7:

[0931] The device converts the received advice into speech and conveys it to the user. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[0932] Reminder function

[0933] 1. Step 1:

[0934] Users input information such as medication times and hospital appointment information into the device, either via voice or text.

[0935] 2. Step 2:

[0936] The device temporarily stores the entered reminder information and sends it to the server.

[0937] 3. Step 3:

[0938] The server stores the reminder information in a database and manages the schedule.

[0939] 4. Step 4:

[0940] When the reminder time approaches, the server generates a notification message.

[0941] 5. Step 5:

[0942] The server sends the generated notification message to the terminal.

[0943] 6. Step 6:

[0944] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[0945] In this way, the system functions effectively through each processing step, providing support to the elderly and their caregivers.

[0946] Example 1

[0947] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0948] In modern society, elderly people and their caregivers face many challenges in their daily lives. In particular, there is a need for early detection of dementia in the elderly and support in daily life. Furthermore, providing effective caregiving know-how and reducing the burden on caregivers are also important issues. Addressing these challenges requires a system that can monitor the conversations and behavior of elderly people and provide appropriate advice and reminders.

[0949] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0950] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses to speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for generating caregiving know-how and response text using a generative AI model, means for processing the speech data using speech recognition and speech synthesis software, and means for inputting basic setting information of the elderly person and the caregiver and generating a profile. This enables early detection of signs of dementia in the elderly, support for daily life, and providing effective advice to caregivers.

[0951] The "means for capturing conversation" is a function for acquiring the voice uttered by the user using a voice input device such as a microphone.

[0952] "Means for converting voice to text" refers to a function that converts acquired voice data into text format using voice recognition technology.

[0953] The "means for analyzing the generated text data and generating an appropriate response" is a function for analyzing the text data using natural language processing technology and generating an appropriate response based on the analysis.

[0954] The "means for converting the generated response into speech and presenting it to the user" is a function for converting the generated text response into speech using speech synthesis technology and presenting it to the user.

[0955] "Means for monitoring conversations and behaviors and detecting signs of dementia" refers to a function that uses sensors to continuously monitor the user's conversations and behaviors and detects signs of dementia based on an algorithm.

[0956] "Means for notifying caregivers when signs of dementia are detected" is a function that sends alerts and notifications to caregivers via a server when signs of dementia are detected by the algorithm.

[0957] "Means for analyzing questions and generating advice in order to provide know-how on caregiving" refers to a function that analyzes questions on caregiving from users and generates appropriate advice using a database and a generative AI model.

[0958] "Means to remind you to take medicine or make hospital appointments" is a function that records the time the user sets to take medicine or hospital appointment information, and notifies you when the reminder time approaches.

[0959] "Means for generating care know-how and response text using a generative AI model" is a function that uses a generative AI model to generate know-how about care and responses to user questions in text format.

[0960] "Means for processing voice data using voice recognition and voice synthesis software" refers to the function of converting voice into text using voice recognition software and converting text into voice using voice synthesis software.

[0961] The "means for inputting basic setting information and generating a profile" is a function in which a user inputs initial setting information, and the server generates a profile for each user based on this information.

[0962] MODE FOR CARRYING OUT THE INVENTION

[0963] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[0964] System configuration

[0965] 1. Server

[0966] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[0967] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[0968] The server uses generative AI models to analyze the information sent by the user and generate the necessary responses or alerts.

[0969] 2. Terminal

[0970] The device is designed as a stuffed toy device and is equipped with a microphone, speaker, and various sensors.

[0971] The device uses voice recognition software to capture the user's speech and convert it into text.

[0972] The terminal transmits data to the server as needed, and also conveys responses from the server to the user by voice.

[0973] 3. Users

[0974] The users mainly refer to elderly people and their caregivers.

[0975] Users can talk to the device to engage in conversational interactions and ask questions about caregiving.

[0976] Users receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[0977] Program processing details

[0978] 1. Initial Setup

[0979] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and senior's names, ages, and medication information.

[0980] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[0981] The server stores the received initial setting information in a database and generates profiles for the elderly person and their caregiver.

[0982] 2. Conversational Interaction

[0983] When a user speaks to the device, the device captures the voice with its microphone and temporarily stores it. For example, if a user says, "I went for a walk today," the device's microphone will capture this.

[0984] The device converts the voice data into text using voice recognition software and sends it to the server.

[0985] The server analyzes the text data and uses a generative AI model (e.g., a GPT-based model) to generate an appropriate response, such as "How was the weather today?"

[0986] The server transmits the generated response data to the terminal, which converts it into voice using speech synthesis software and conveys the response to the user.

[0987] 3. Detecting signs of dementia

[0988] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[0989] If the device detects abnormal speech or behavior, it will send that data to the server. For example, if an elderly person frequently asks the same question, the device will send that data to the server.

[0990] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[0991] If the server detects any signs, it will send an alert to the caregiver via the device.

[0992] 4. Providing nursing care know-how

[0993] The user (caregiver) asks a question about care to the terminal, such as "How should I assist with eating?"

[0994] The device captures the question with a microphone, converts it into text using speech recognition software, and sends it to a server.

[0995] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[0996] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[0997] 5. Reminder function

[0998] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[0999] The device sends the reminder information to the server, which stores it in a database.

[1000] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1001] The terminal converts the notification message into voice and reminds the user.

[1002] Specific examples

[1003] Concrete example of conversational interaction

[1004] 1. User (elderly person): "I went for a walk today."

[1005] 2. Device: Capture audio with the microphone and temporarily store it.

[1006] 3. Device: Converts speech to text and sends it to the server.

[1007] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[1008] 5. Server: Generates and sends the response to the device.

[1009] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[1010] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[1011] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1012] Initial Setup

[1013] Step 1:

[1014] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and the senior's names, ages, and medication information.

[1015] Input: Wi-Fi settings, basic information of the caregiver and the elderly person (name, age, medication information)

[1016] Output: Initial settings information is saved on the device

[1017] Specific behavior:

[1018] The user manually enters the necessary information on the initial setup screen of the device.

[1019] Step 2:

[1020] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1021] Input: User-entered initial configuration information

[1022] Output: Configuration information sent to the server

[1023] Specific behavior:

[1024] The device establishes a Wi-Fi connection and sends the initial configuration information in packets to the server.

[1025] The device will perform calibration of its internal sensors and microphones.

[1026] Step 3:

[1027] The server stores the received initial setting information in a database and generates a profile.

[1028] Input: Setting information sent from the device

[1029] Output: Profiles stored in the database

[1030] Specific behavior:

[1031] The server writes the configuration information to a database and generates profile IDs for the elderly person and their caregiver.

[1032] Conversational Interactions

[1033] Step 1:

[1034] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1035] Input: User speech

[1036] Output: Temporarily saved audio data

[1037] Specific behavior:

[1038] The device's microphone captures audio data in real time and temporarily stores it in local storage.

[1039] Step 2:

[1040] The device converts the voice data into text using voice recognition software and sends it to the server.

[1041] Input: Temporarily saved audio data

[1042] Output: Data converted to text

[1043] Specific behavior:

[1044] The device converts voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[1045] Step 3:

[1046] The server analyzes the text data and generates an appropriate response using a generative AI model.

[1047] Input: Text format data

[1048] Output: The generated response text

[1049] Specific behavior:

[1050] The server analyzes the text data using models such as OpenAI's GPT model and generates an appropriate response.

[1051] Step 4:

[1052] The server transmits the generated response data to the terminal, which converts it into speech using speech synthesis software and responds to the user.

[1053] Input: Generated response text

[1054] Output: The transcribed response

[1055] Specific behavior:

[1056] The server sends response data generated by the AI ​​model to the terminal.

[1057] The device uses voice synthesis technology such as Amazon Polly to convert text data into speech and transmits it to the user through the speaker.

[1058] Dementia symptom detection

[1059] Step 1:

[1060] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1061] Input: Elderly conversations and behavior

[1062] Output: Monitoring result data

[1063] Specific behavior:

[1064] The device's built-in microphones, accelerometers, and cameras capture the elderly's behavioral data in real time.

[1065] Step 2:

[1066] When the device detects abnormal speech or behavior, it sends the data to the server.

[1067] Input: Monitoring result data

[1068] Output: Behavioral data sent to the server

[1069] Specific behavior:

[1070] When a terminal detects an abnormality, it sends the data in packet form to the server.

[1071] Step 3:

[1072] The server analyzes the data and runs algorithms to determine signs of dementia.

[1073] Input: Received behavioral data

[1074] Output: Detected dementia symptoms

[1075] Specific behavior:

[1076] The server uses machine learning algorithms to analyze behavioral data and detect abnormal trends.

[1077] Step 4:

[1078] If the server detects any signs, it will send an alert to the caregiver via the device.

[1079] Input: Detected dementia symptoms

[1080] Output: Alert notification to caregiver

[1081] Specific behavior:

[1082] The server generates an alert message and sends it to the terminal.

[1083] The device will notify caregivers of alerts via speaker or display.

[1084] Providing nursing care know-how

[1085] Step 1:

[1086] The user (caregiver) asks questions about care to the terminal.

[1087] Input: Caregiver's voice question

[1088] Output: Temporarily saved audio data

[1089] Specific behavior:

[1090] The device captures the caregiver's questions and temporarily stores the audio.

[1091] Step 2:

[1092] The device uses voice recognition software to convert the question into text and send it to the server.

[1093] Input: Temporarily saved audio data

[1094] Output: Data converted to text

[1095] Specific behavior:

[1096] The device uses voice recognition software to convert the voice data into text and send it to the server.

[1097] Step 3:

[1098] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[1099] Input: Text question data

[1100] Output: Generated advice

[1101] Specific behavior:

[1102] The server uses a generative AI model to analyze the text data and generate appropriate care advice.

[1103] Step 4:

[1104] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1105] Input: Generated advice text

[1106] Output: Advice converted to speech

[1107] Specific behavior:

[1108] The server transmits the generated advice to the terminal.

[1109] The device uses speech synthesis software to convert advice into voice and convey it to the caregiver.

[1110] Reminder function

[1111] Step 1:

[1112] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1113] Input: Reminder information entered by the user

[1114] Output: Reminder information saved on the device

[1115] Specific behavior:

[1116] The user manually enters the reminder information into an input form on the device.

[1117] Step 2:

[1118] The device sends the reminder information to the server, which stores it in a database.

[1119] Input: Reminder information sent from the device

[1120] Output: Reminder information stored in the database

[1121] Specific behavior:

[1122] The device sends the reminder information to the server, which stores it in a database.

[1123] Step 3:

[1124] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1125] Input: Reminder information stored in the database

[1126] Output: The generated notification message

[1127] Specific behavior:

[1128] The server generates a notification message based on the set reminder time and sends it to the terminal.

[1129] Step 4:

[1130] The terminal converts the notification message into voice and reminds the user.

[1131] Input: The generated notification message

[1132] Output: Reminder message converted to audio

[1133] Specific behavior:

[1134] The device uses speech synthesis software to convert notification messages into voice and provide reminders.

[1135] (Application example 1)

[1136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] Supporting the elderly and their caregivers has become an important issue in modern society. In particular, there is a demand for conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. At the same time, the quality and functionality of the devices and robots used to provide this support are also extremely important. In the manufacturing process of elderly support robots, quality inspection, function testing, and progress management of manufacturing work are challenges.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1139] In this invention, the server includes a means for capturing conversations between the elderly person and their caregiver and converting the speech into text, a means for analyzing the generated text data and generating appropriate responses, and a means for converting the generated responses into speech and presenting them to the user. This enables high-performance support for the elderly person and their caregiver, as well as the implementation of quality inspections and functional tests in the manufacturing process, progress management of manufacturing work using voice recognition, and scheduling and notifying quality inspections using a reminder function.

[1140] A "support system for older people and their caregivers" is an integrated system that provides the services and functions needed by both older people and their caregivers.

[1141] "Means for capturing conversations and converting speech to text" refers to technology that collects a user's speech and converts it into text information.

[1142] "Means for analyzing generated text data and generating an appropriate response" refers to a technology that analyzes text data and determines the optimal response or action based on its content.

[1143] "Means for converting the generated response into audio and presenting it to the user" refers to technology that converts the generated text-based response into audio format and provides it to the user.

[1144] "Means for monitoring conversations and behavior to detect signs of dementia" refers to technology that monitors a user's speech and behavior patterns and detects abnormalities.

[1145] "Means to notify caregivers if signs of dementia are detected" is a function that sends an alert to caregivers when early signs of dementia are confirmed.

[1146] "Means for analyzing questions and generating advice in order to provide know-how related to caregiving" refers to technology that analyzes questions from caregivers and generates and provides appropriate advice and information.

[1147] "Medication and medical appointment reminders" refers to features that remind users of specific times or events (e.g., medication time, doctor's appointment time).

[1148] "Means for conducting quality inspections and product functionality tests during the manufacturing process and recording the results" refers to a system that checks the quality and functionality of products at each stage of manufacturing and stores the results as a log.

[1149] "Means for managing the progress of manufacturing work using voice recognition" is a technology that uses voice input to track and manage the status and progress of manufacturing work.

[1150] "Means for scheduling and notifying quality inspections using a reminder function" is a function that allows you to schedule quality inspections and notify you when the time comes.

[1151] This invention is a system that supports elderly people and their caregivers, and in particular, it realizes conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[1152] System configuration

[1153] 1. Server

[1154] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia. The analysis uses NLP (natural language processing) technology.

[1155] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[1156] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[1157] Manage quality inspections and product functionality tests during the manufacturing process and record the results.

[1158] 2. Terminal

[1159] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[1160] The device has the ability to capture the user's conversation and convert the speech into text.

[1161] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[1162] The terminal uses voice recognition to manage the progress of manufacturing work.

[1163] 3. Users

[1164] The users mainly refer to elderly people and their caregivers.

[1165] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[1166] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[1167] Program processing details

[1168] The main functions of the system are as follows:

[1169] 1. Initial Setup

[1170] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[1171] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1172] The server stores the received initial setting information in a database and generates a profile.

[1173] 2. Conversational Interaction

[1174] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1175] The device converts the voice data into text and sends it to the server.

[1176] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[1177] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[1178] 3. Detecting signs of dementia

[1179] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1180] When the device detects abnormal speech or behavior, it sends the data to the server.

[1181] The server analyzes the data and runs algorithms to determine signs of dementia.

[1182] If the server detects any signs, it will send an alert to the caregiver via the device.

[1183] 4. Providing nursing care know-how

[1184] The user (caregiver) asks questions about care to the terminal.

[1185] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[1186] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[1187] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1188] 5. Reminder function

[1189] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1190] The device sends the reminder information to the server, which stores it.

[1191] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1192] The terminal converts the notification message into voice and reminds the user.

[1193] 6. Quality inspection and function test

[1194] During the manufacturing process, the terminal performs quality inspections and product functionality tests and transmits the results to the server.

[1195] The server records the results and generates alerts for any defective or problematic products.

[1196] 7. Manufacturing work progress management

[1197] The terminal uses voice recognition to manage the progress of manufacturing work and transmits necessary information and instructions to the server.

[1198] The server tracks progress and provides timely feedback.

[1199] 8. Reminders feature expansion

[1200] The device can use the reminder function to schedule and notify quality inspections.

[1201] Based on this information, the server manages the schedule and sends notifications at the appropriate times.

[1202] Specific examples

[1203] Concrete example of conversational interaction

[1204] 1. User (elderly person): "I went for a walk today."

[1205] 2. Device: Capture audio with the microphone and temporarily save it.

[1206] 3. Device: Converts speech into text and sends it to the server.

[1207] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[1208] 5. Server: Sends the generated response to the terminal.

[1209] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[1210] Examples of quality inspections and functional tests

[1211] 1. Terminal: Inspects each part of the robot during production.

[1212] 2. Server: Records inspection results and generates alerts if there are any problems.

[1213] 3. Terminal: The test is completed and the results are sent to the server.

[1214] 4. Terminal: "The robot's serial number is R12345, and this test result is a pass."

[1215] This effectively provides support for both the elderly and their caregivers, while also ensuring quality control and progress management of the manufacturing process.

[1216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1217] Step 1:

[1218] The user performs the initial setup of the system. The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and caregiver into the device. The device calibrates the sensors and microphone based on this information and sends the setup information to the server. The server generates a profile based on this initial setup information and saves it in a database. The input is the user's basic setup information, and the output is the server's creation and saving of the profile.

[1219] Step 2:

[1220] The user speaks into the device. The device captures the voice with a microphone and temporarily stores it. Since the voice data cannot be used as is, the device converts the voice into text. This converted text data is sent to the server. The input is the user's voice, and the output is text data sent to the server.

[1221] Step 3:

[1222] The server analyzes the transmitted text data and generates an appropriate response using NLP (natural language processing) technology. This requires understanding the context and intent from the text data and generating a corresponding response. The generated response data is then sent to the terminal. The input is the text data, and the output is the generated response.

[1223] Step 4:

[1224] The terminal converts the response received from the server into speech and presents it to the user. The user receives the speech response. The input is the response data from the server, and the output is the speech response to the user.

[1225] Step 5:

[1226] The device uses sensors to constantly monitor the user's conversations and behavior. If the sensors detect abnormal speech or behavior, the device temporarily stores it and sends it to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, the server sends an alert to the caregiver via the device. The input is sensor data, and the output is a dementia sign alert.

[1227] Step 6:

[1228] The user (caregiver) asks a question about caregiving to the device. The device captures the question with a microphone, converts the voice into text, and sends it to the server. The server analyzes the text data, searches a database for appropriate caregiving know-how and advice, and sends the generated advice to the device. The device then conveys this to the user by voice. The input is the voice of the question, and the output is caregiving advice.

[1229] Step 7:

[1230] The user sets the time to take medicine and hospital appointment information on the device. The device sends this reminder information to the server, which stores it. When the reminder time approaches, the server generates a notification message and sends it to the device. The device converts the notification message into audio and reminds the user. The input is the reminder setting information, and the output is the reminder notification.

[1231] Step 8:

[1232] During the manufacturing process, the terminal performs quality inspections and product functionality tests. The terminal transmits the inspection results for each item to the server, which records the results and generates alerts as needed. The input is the inspection data, and the output is the quality inspection result record and alerts.

[1233] Step 9:

[1234] The terminal uses voice recognition to manage the progress of manufacturing work. When the user speaks to report progress, the terminal converts the speech into text and sends it to the server. The server updates the progress based on the transmitted data and provides necessary feedback. The input is the voice progress report, and the output is progress updates and feedback.

[1235] Step 10:

[1236] The terminal uses the reminder function to schedule quality inspections and notify the user. The server manages the schedule based on this information and notifies the user at the appropriate time. The input is the inspection schedule setting information, and the output is the inspection schedule notification.

[1237] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1238] This invention is a system to support elderly people and their caregivers, and aims to provide more effective support by incorporating an emotion engine that recognizes the user's emotions, in addition to conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[1239] System configuration

[1240] 1. Server

[1241] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[1242] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[1243] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[1244] The server also generates a response based on the emotional state sent from the emotion engine.

[1245] 2. Terminal

[1246] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, sensors, and camera.

[1247] The device has the ability to capture the user's conversation and convert the speech into text.

[1248] The terminal transmits information to the server as needed and also conveys responses from the server to the user by voice.

[1249] The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[1250] 3. Users

[1251] The users mainly refer to elderly people and their caregivers.

[1252] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[1253] Users receive reminders, alerts, and emotional responses from the device, which they can use as a reference for their daily lives and caregiving.

[1254] Program processing details

[1255] The main functions of the system are as follows:

[1256] 1. Initial Setup

[1257] The user resets the device and enters basic configuration information, including Wi-Fi settings and basic information about the senior and caregiver.

[1258] The device calibrates the sensors, microphone, and camera based on this information and sends the configuration information to the server.

[1259] The server stores the received profile data in a database and generates a user profile.

[1260] 2. Conversational Interaction

[1261] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1262] The device converts the voice data into text and sends it to the server.

[1263] The server analyzes the text data and uses NLP (Natural Language Processing) algorithms to find data to generate an appropriate response.

[1264] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[1265] 3. Emotion Recognition by Emotion Engine

[1266] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them.

[1267] The device inputs this data into an emotion engine to recognize the user's emotional state in real time.

[1268] The emotion engine sends the recognized emotional state to the server.

[1269] 4. Generating Emotion-Based Responses

[1270] The server then executes an algorithm based on the received emotional state data to generate a response appropriate to the user's current state of mind.

[1271] For example, if the user is in a sad emotional state, a supportive or encouraging response may be generated.

[1272] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[1273] 5. Detecting signs of dementia

[1274] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1275] When the device detects abnormal speech or behavior, it sends the data to the server.

[1276] The server analyzes the data and runs algorithms to determine signs of dementia.

[1277] If the server detects a symptom, it generates alert information.

[1278] The device notifies the caregiver of the alert information.

[1279] 6. Providing nursing care know-how

[1280] The user (caregiver) asks questions about care to the terminal.

[1281] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[1282] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[1283] The server generates advice and sends it to the terminal.

[1284] The terminal converts the received advice into voice and conveys it to the user.

[1285] 7. Reminder function

[1286] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1287] The device sends the reminder information to the server, which stores it.

[1288] When the reminder time approaches, the server generates a notification message.

[1289] The server sends the generated notification message to the terminal.

[1290] The terminal converts the notification message into voice and reminds the user.

[1291] Specific examples

[1292] Specific examples of emotion recognition using emotion engines

[1293] 1. User (elderly): "I've been feeling unwell and depressed lately."

[1294] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[1295] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[1296] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[1297] 5. Server: Generates an encouraging response such as "Cheer up" based on the emotional state "sad."

[1298] 6. Server: Sends the generated response to the device.

[1299] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[1300] In this way, the system as a whole aims to reduce the mental and physical burden on elderly people and caregivers by incorporating conversational interaction, emotion recognition, and care support functions. Utilizing an emotion engine enables flexible responses according to the user's emotional state, providing more personalized support.

[1301] The processing flow will be explained below.

[1302] Program processing steps

[1303] Emotion recognition and response generation by emotion engine

[1304] 1. Step 1:

[1305] The user speaks to the device. Example: "I've been feeling unwell and depressed lately."

[1306] 2. Step 2:

[1307] The device uses a microphone to capture the user's voice and a camera to capture the user's facial expressions.

[1308] 3. Step 3:

[1309] The device converts the captured voice data into text using voice recognition technology and sends it to the emotion engine along with facial expression data.

[1310] 4. Step 4:

[1311] The emotion engine analyzes the tone of the voice data and facial expression data and recognizes the user's emotional state as "sad."

[1312] 5. Step 5:

[1313] The emotion engine transmits the recognized emotional state data to the server.

[1314] 6. Step 6:

[1315] The server analyzes the received emotional state data and generates an appropriate response, e.g., encouraging words such as "Cheer up!"

[1316] 7. Step 7:

[1317] The server transmits the generated response data to the terminal.

[1318] 8. Step 8:

[1319] The device converts the received response data into voice and responds to the user with voice. Example: "Cheer up."

[1320] Dementia symptom detection

[1321] 1. Step 1:

[1322] The device constantly monitors the elderly person's conversations and behavior using sensors and microphones.

[1323] 2. Step 2:

[1324] When the device detects abnormal behavior (e.g. repeating the same word multiple times), it sends the data to the server in real time.

[1325] 3. Step 3:

[1326] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[1327] 4. Step 4:

[1328] If the server detects signs of dementia, it generates alert information and sends data to the terminal to notify the caregiver.

[1329] 5. Step 5:

[1330] The device will notify the caregiver of the alert notification with audio and visual messages, for example, "The elderly person is repeating the same words over and over. Attention is needed."

[1331] Providing nursing care know-how

[1332] 1. Step 1:

[1333] The user (caregiver) asks a question about caregiving to the device. Example: "How can we effectively guide the elderly?"

[1334] 2. Step 2:

[1335] The device captures the question with a microphone and saves it as audio data.

[1336] 3. Step 3:

[1337] The device converts the voice data into text using voice recognition technology and sends it to the server.

[1338] 4. Step 4:

[1339] The server analyzes the received text data and runs NLP algorithms to understand the intent of the question.

[1340] 5. Step 5:

[1341] The server searches the database for relevant care know-how and advice.

[1342] 6. Step 6:

[1343] The server generates appropriate advice and sends it to the device. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[1344] 7. Step 7:

[1345] The terminal converts the received advice into voice and conveys it to the user.

[1346] Reminder function

[1347] 1. Step 1:

[1348] Users enter medication times and hospital appointment information into the terminal by voice or manually.

[1349] 2. Step 2:

[1350] The device temporarily stores the entered reminder information and sends it to the server.

[1351] 3. Step 3:

[1352] The server stores the reminder information in a database and manages the schedule.

[1353] 4. Step 4:

[1354] When the reminder time approaches, the server generates a reminder notification message.

[1355] 5. Step 5:

[1356] The server sends the generated remind notification message to the terminal.

[1357] 6. Step 6:

[1358] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[1359] This system combines multiple functions to provide multifaceted support to the elderly and their caregivers. By utilizing emotion recognition through an emotion engine, it is possible to provide more personalized responses, aiming to increase user satisfaction.

[1360] Example 2

[1361] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1362] Providing effective support to elderly people and their caregivers requires not only conversational interaction and information provision, but also responses that take into account the user's emotions and psychological state. However, conventional systems have had difficulty comprehensively recognizing emotions, detecting signs of dementia, and providing care know-how. Furthermore, there have been no systems that can reduce the user's mental burden and provide flexible responses that correspond to their emotions. This has made it difficult to improve the satisfaction of elderly people and caregivers.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1364] In this invention, the server

[1365] A means of capturing the conversations of the senior and caregiver and converting the speech to text;

[1366] means for analyzing the generated text data and generating an appropriate response;

[1367] means for converting the generated response into speech and presenting it to the user;

[1368] A means of monitoring the conversations and behavior of elderly people to detect signs of dementia,

[1369] A means to notify caregivers if signs of dementia are detected; and

[1370] A means for analyzing questions and generating advice to provide knowledge about care;

[1371] A way to remind you to take your medicine or make a doctor's appointment,

[1372] a means for recognizing a user's emotion using an emotion engine and generating a response according to the emotion;

[1373] means for transmitting the emotional state to a server for analysis and generating an appropriate response based on the user's psychological state;

[1374] This will enable comprehensive and personalized support for older people and their caregivers.

[1375] "Elderly people and caregivers" refers to people who play a role in supporting the daily lives and health management of older people.

[1376] "Capture" refers to the technology of collecting data such as audio and video and storing it in digital form.

[1377] "Speech-to-text" refers to the technology of analyzing voice data and converting its contents into text data.

[1378] "Generated text data" refers to data obtained by analyzing speech and converting it into character data.

[1379] "Response" refers to information or messages returned in response to user input.

[1380] "Monitoring" refers to the act of continuously observing the state or behavior of a specific object to detect abnormalities or changes.

[1381] "Detecting signs of dementia" refers to finding changes in specific behaviors or speech that may be indicative of dementia.

[1382] "Notify caregiver" refers to providing important detected information to the caregiver as a warning or alert.

[1383] "Providing knowledge" means providing appropriate advice and guidance based on the questions asked using accumulated information.

[1384] A "remind" is a notification in advance about a specific time or event so that you don't forget.

[1385] An "emotion engine" refers to technology that analyzes a user's tone of voice, facial expressions, etc. to identify their current emotional state.

[1386] "Responding based on psychological state" refers to generating a response appropriate to the user's mental state at the time based on their emotional data.

[1387] This invention is a system that supports elderly people and their caregivers, and provides comprehensive and personalized support by incorporating conversational interaction, dementia symptom detection, caregiving know-how provision, reminder functions, and an emotion engine.

[1388] System configuration

[1389] This system mainly consists of three elements: a server, a terminal, and a user.

[1390] 1. Server

[1391] The server analyzes voice data, generates text data, generates responses, and detects signs of dementia. The server is equipped with a database that stores information on care know-how, local consultation centers, and welfare systems. The server analyzes the information sent by the user and generates the necessary responses and alerts. It also generates responses based on the emotional state sent from the emotion engine.

[1392] 2. Terminal

[1393] The device is designed as a plush toy and is equipped with a microphone, speaker, sensors, and camera. The device has the ability to capture the user's conversation and convert the speech into text. If necessary, it sends information to the server and communicates the server's response to the user via voice. The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[1394] 3. Users

[1395] Users are primarily elderly people and their caregivers. By speaking into the device, users can engage in conversational interactions and ask questions about caregiving. Users receive reminders, alerts, and emotional responses from the device, which they can use as references for their daily lives and caregiving.

[1396] Detailed System Operation

[1397] The detailed operation of this system will be described below.

[1398] 1. Initial Setup

[1399] The user initializes the device and enters Wi-Fi settings and basic information (information about the senior and caregiver). The device calibrates the sensors, microphone, and camera based on this information and sends the setting information to the server. The server saves the received information and creates a user profile.

[1400] 2. Conversational Interaction

[1401] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is converted into text and sent to the server. The server analyzes the text data and generates an appropriate response using a natural language processing (NLP) algorithm. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[1402] 3. Emotion Recognition by Emotion Engine

[1403] The device captures and temporarily stores the user's voice tone and facial expressions. This data is input to the emotion engine, which recognizes the user's emotional state in real time. The emotion engine then transmits the recognized emotional state to the server.

[1404] 4. Generating Emotion-Based Responses

[1405] The server generates a response appropriate to the user's current psychological state based on the emotional state data. For example, if the emotional state is "sad," it generates a supportive or encouraging response. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[1406] 5. Detecting signs of dementia

[1407] The device uses sensors to monitor the elderly person's conversations and behavior. If it detects abnormal speech or behavior, it sends the data to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, it generates an alert and notifies the caregiver via the device.

[1408] 6. Providing nursing care know-how

[1409] When a user (caregiver) asks a question about caregiving to the device, the device captures the question, converts the voice to text, and sends it to the server. The server analyzes the data and searches a database for appropriate caregiving know-how and advice. It generates advice and sends it to the device. The device then converts the received advice into voice and conveys it to the user.

[1410] 7. Reminder function

[1411] The user inputs information about medication times and hospital appointment times into the device. The device then sends the reminder information to the server, which stores it. When the reminder time approaches, the server sends a notification message to the device, which then converts it into audio and notifies the user.

[1412] Specific examples

[1413] Specific examples of emotion recognition using emotion engines

[1414] 1. User (elderly): "I've been feeling unwell and depressed lately."

[1415] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[1416] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[1417] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[1418] 5. Server: Generates a supportive response such as "Cheer up" based on the emotional state "sad."

[1419] 6. Server: Generates and sends the response to the device.

[1420] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[1421] Examples of prompt statements

[1422] Below are some example prompts that utilize generative AI models (e.g., GPT-3) to generate natural-sounding responses.

[1423] "An elderly user says, 'I've been feeling unwell and depressed lately.' The emotion engine detects the emotional state as 'sad.' Please generate an appropriate message of encouragement for this user."

[1424] In this way, this system, with its conversational interaction, emotion recognition, and care support functions, reduces the mental and physical burden on elderly people and caregivers and provides personalized support.

[1425] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1426] Step 1: Initial Setup

[1427] The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and their caregiver, including the user's name, age, and health status.

[1428] Your device uses the information you provide to calibrate sensors, microphones, and cameras, for example, adjusting microphone sensitivity and focusing the camera, optimizing device performance.

[1429] The terminal transmits the initial setting information to the server.

[1430] The server stores the received setting information in a database and generates a user profile, which prepares the data required for subsequent processing.

[1431] Step 2: Conversational Interaction

[1432] The user speaks to the terminal, for example, saying a greeting such as "Good morning."

[1433] The device captures the audio using a microphone and temporarily stores it as audio data.

[1434] The device converts the captured voice data into text to generate text data, which becomes input data.

[1435] The terminal transmits the generated text data to the server.

[1436] The server analyzes the received text data and generates an appropriate response using an NLP (natural language processing) algorithm. For example, for the text "Good morning," it generates the response "Good morning. What are your plans for today?" This becomes the output data.

[1437] The server transmits the generated response data to the terminal.

[1438] The terminal converts the received response data into voice and responds to the user.

[1439] Step 3: Emotion recognition by the emotion engine

[1440] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them, for example, as digital data, such as voice waveforms and facial movements.

[1441] The device inputs the captured data into an emotion engine to recognize the emotional state. The emotion engine uses a deep learning algorithm to analyze the data and label the emotional state as "sad" or "happy." This becomes the output data.

[1442] The emotion engine transmits the recognized emotional state data to a server.

[1443] Step 4: Generate an emotion-based response

[1444] The server generates a response appropriate to the user's current psychological state based on the received emotional state data. For example, if the user is feeling sad, the server generates a message of encouragement such as "Cheer up!"

[1445] The server transmits the generated response data to the terminal.

[1446] The terminal converts the received response data into voice and responds to the user.

[1447] Step 5: Detecting signs of dementia

[1448] The device uses sensors to monitor the elderly's conversations and behavior, such as motion sensors to monitor walking patterns and microphones to analyze conversation content.

[1449] When the device detects abnormal speech or behavior, it sends the data to the server.

[1450] The server analyzes the received data and runs algorithms to identify signs of dementia, such as by analyzing specific keywords or abnormal behavioral patterns.

[1451] If the server detects signs of dementia, it generates alert information and notifies the caregiver via the terminal.

[1452] Step 6: Providing nursing care know-how

[1453] The user (caregiver) asks a question about care to the device, for example, "What can I do to prevent falls?"

[1454] The device captures the question with a microphone and converts the voice into text to generate text data.

[1455] The terminal transmits the generated text data to the server.

[1456] The server analyzes the received text data and searches a database for appropriate care know-how and advice.

[1457] The server generates appropriate advice and sends it to the terminal, for example, generating a response such as "It would be a good idea to install handrails."

[1458] The terminal converts the received advice data into voice and conveys it to the user.

[1459] Step 7: Reminders

[1460] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1461] The terminal transmits the input remind information to the server.

[1462] The server stores the received reminder information in a database.

[1463] When the reminder time approaches, the server generates a notification message.

[1464] The server sends the generated notification message to the terminal, for example, "It's time to take your medicine at 2 PM."

[1465] The terminal converts the received notification message into voice and reminds the user.

[1466] These steps enable the system to provide comprehensive and personalized support to seniors and their caregivers.

[1467] (Application example 2)

[1468] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1469] Daily life and caregiving entail many burdens for elderly people and their caregivers. Meal preparation and delivery, health monitoring, and psychological support are particularly important. However, conventional systems often fail to adequately address these needs. For example, they lack meal suggestions that take the user's emotional state into account, and real-time mental support during delivery, making it difficult to improve user satisfaction. Furthermore, there are challenges with early detection of dementia symptoms and providing appropriate care know-how. Given these circumstances, there is a need for a comprehensive and personalized support system for elderly people and their caregivers.

[1470] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1471] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses into speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for recognizing the user's emotions in real time and suggesting appropriate meals based on the user's emotional state, and means for providing real-time mental support for situations during delivery. This enables personalized meal suggestions and mental support based on the user's emotional state, thereby improving the quality of life of the elderly person and their caregiver.

[1472] definition statement

[1473] The term "elderly" generally refers to people aged 65 and over.

[1474] A "caregiver" is a person who supports an elderly person in their daily lives and provides the care they need.

[1475] "Emotion engine" refers to a system that includes algorithms that recognize a user's emotional state from their tone of voice and facial expressions.

[1476] "Server" refers to a device that has data processing, analysis and storage functions and manages and controls the entire system.

[1477] "Terminal" refers to an input and output device used by a user that has the functions of voice capture, text conversion, and voice output.

[1478] "Text data" refers to digital information that has been converted from audio data into text format.

[1479] "Nursing care know-how" refers to knowledge, methods, and techniques related to nursing care.

[1480] "Monitoring" refers to the act of continuously observing and recording a subject's conversations and behavior.

[1481] "Signs of dementia" refer to characteristic symptoms and behaviors seen in the early stages or as dementia progresses.

[1482] "Notification" refers to the act of informing a user of important information in real time.

[1483] "Advice" refers to information that provides solutions or recommendations for a particular problem or situation.

[1484] "Remind" refers to the act of notifying and reminding a user in advance of a specific action or event.

[1485] "Meal suggestion" refers to the act of recommending appropriate meal content based on the user's preferences and health condition.

[1486] "Mental support" refers to the act of providing assistance and encouragement to promote the user's mental stability and sense of security.

[1487] invention specification

[1488] This invention is a system that supports elderly people and caregivers, and aims to provide more effective support by including functions such as conversational interaction, emotion recognition, dementia symptom detection, provision of care know-how, meal suggestions, reminders, and mental support.

[1489] System configuration

[1490] This system mainly consists of three elements: a server, a terminal, and a user.

[1491] 1. Server

[1492] The server has the following features:

[1493] Analyzing voice data, generating text data, generating responses

[1494] Dementia symptom detection

[1495] Managing the emotion engine and meal recommendation system

[1496] Real-time mental support during delivery

[1497] Management of various databases (care know-how, local consultation points, welfare system information, etc.)

[1498] 2. Terminal

[1499] The terminal mainly uses the following devices:

[1500] Smartphone, smart glasses, or head-mounted display

[1501] A stuffed toy device equipped with a microphone, speaker, sensor, and camera

[1502] The main features of the terminal are:

[1503] Capture user conversations in real time and convert speech to text

[1504] Sending data to the server and receiving the generated response

[1505] Collecting data to recognize user emotions (voice tone, facial expressions)

[1506] Providing meal suggestions and reminder functions

[1507] 3. Users

[1508] Users are the elderly and their caregivers, and they primarily use the system in the following ways:

[1509] By talking to the device, you can have conversational interactions and ask questions about caregiving.

[1510] Receive responses and reminders from your device

[1511] Receive meal suggestions and mental support from devices and applications

[1512] Program processing details

[1513] Initial Setup

[1514] The user initializes the device and enters basic configuration information, including Wi-Fi settings and a user profile (such as name, age, dietary preferences, and allergy information). The device uses this information to calibrate the sensors, microphone, and camera, and then sends the configuration information to the server. The server then stores the received profile data in a database and creates a user profile.

[1515] Conversational Interaction and Emotion Recognition

[1516] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is then converted into text and sent to the server. The server analyzes the text data and uses an NLP (natural language processing) algorithm to search for data to generate an appropriate response. The generated response data is sent to the device, which then converts it into voice and responds to the user. The device also inputs voice tone and facial expressions into an emotion engine to recognize the user's emotional state in real time.

[1517] Meal suggestions

[1518] Based on the user's emotional state and profile (food preferences and health status), the server runs an algorithm to suggest suitable meals. For example, if the user is in a "sad" emotional state, it will suggest nutritious meals.

[1519] Mental support

[1520] If a delay occurs during delivery, the server checks the user's emotional state again. If the user is anxious, it generates an encouraging message and sends it to the user via the terminal.

[1521] Specific examples

[1522] For example, if a user says, "I haven't had much of an appetite lately," the smartphone's microphone captures the voice. The captured voice data is input into the emotion engine, which recognizes the emotional state as "sad." As a result, the server generates a suggestion such as, "Cheer up! How about this nutritious menu?" and notifies the user via voice through the device.

[1523] Prompt Sentence Examples

[1524] Input: "I haven't had much of an appetite lately."

[1525] Prompt: "Use an emotion engine to recognize the user's emotional state and suggest suitable meals."

[1526] Sample output: "Cheer up! Would you like some of these nutritious options?"

[1527] In this way, the present invention improves the quality of life for seniors and their caregivers by providing personalized support based on the user's emotional state.

[1528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1529] Program processing steps

[1530] Step 1:

[1531] The user speaks into the terminal.

[1532] Input: User's voice

[1533] What it does: The device captures audio using the built-in microphone.

[1534] Output: Audio data

[1535] Step 2:

[1536] The device converts the voice data into text.

[1537] Input: Audio data

[1538] How it works: The device's built-in software uses voice recognition technology to convert speech into text.

[1539] Output: Text data

[1540] Step 3:

[1541] Sends text data to the server.

[1542] Input: Text data

[1543] Specific operation: The terminal sends text data to the server via the network.

[1544] Output: Text data sent to the server

[1545] Step 4:

[1546] The server parses the received text data and generates an appropriate response.

[1547] Input: Text data

[1548] What it does: The server uses NLP (Natural Language Processing) algorithms to analyze the text data and generate a corresponding response, possibly referencing information in a database.

[1549] Output: Response data

[1550] Step 5:

[1551] The server transmits the generated response data to the terminal.

[1552] Input: Response data

[1553] Specific operation: The server sends response data to the terminal via the network.

[1554] Output: Response data sent to the terminal

[1555] Step 6:

[1556] The terminal converts the response data into voice and presents it to the user.

[1557] Input: Response data

[1558] Specific operation: The device's built-in software converts the text data into voice data and responds to the user audibly using the speaker.

[1559] Output: Voice response

[1560] Step 7:

[1561] The device recognizes the user's emotions in real time.

[1562] Input: User's voice tone and facial expression data

[1563] How it works: The device captures voice tone and facial expression data using a microphone and camera, and then analyzes the data using an emotion engine.

[1564] Output: Emotional state

[1565] Step 8:

[1566] The server suggests suitable meals to the user based on their emotional state.

[1567] Input: Emotional state and user profile data (e.g., food preferences)

[1568] Specific operation: The server uses an emotion engine and a meal suggestion algorithm to analyze the emotional state and profile data and select an appropriate meal.

[1569] Output: Meal suggestion data

[1570] Step 9:

[1571] The server transmits the generated meal suggestion data to the terminal.

[1572] Input: Meal suggestion data

[1573] Specific operation: The server sends meal suggestion data to the terminal via the network.

[1574] Output: Meal suggestion data sent to the device

[1575] Step 10:

[1576] The terminal converts the meal suggestion data into voice and notifies the user.

[1577] Input: Meal suggestion data

[1578] Specific operation: The device's built-in software converts text data into voice data and provides meal suggestions to the user using the speaker.

[1579] Output: Spoken meal suggestions

[1580] Step 11:

[1581] The server provides real-time mental support during delivery situations.

[1582] Input: Delivery status and user emotional state

[1583] Specific operation: The server monitors the delivery status and the user's emotional state in real time and generates mental support messages as needed.

[1584] Output: Mental support message

[1585] Step 12:

[1586] The server transmits the generated mental support message to the terminal.

[1587] Input: Mental support message

[1588] Specific operation: The server sends a mental support message to the terminal via the network.

[1589] Output: Mental support message sent to the device

[1590] Step 13:

[1591] The terminal converts the mental support message into voice and notifies the user.

[1592] Input: Mental support message

[1593] Specific operation: The device's built-in software converts text data into voice data and uses the speaker to provide the user with a mental support message.

[1594] Output: Audio mental support

[1595] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1597] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1598] [Third embodiment]

[1599] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1600] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1601] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1602] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1603] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1605] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1606] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1607] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1608] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1609] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1610] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1611] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[1612] System configuration

[1613] 1. Server

[1614] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[1615] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[1616] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[1617] 2. Terminal

[1618] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[1619] The device has the ability to capture the user's conversation and convert the speech into text.

[1620] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[1621] 3. Users

[1622] The users mainly refer to elderly people and their caregivers.

[1623] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[1624] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[1625] Program processing details

[1626] The main functions of the system are as follows:

[1627] 1. Initial Setup

[1628] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[1629] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1630] The server stores the received initial setting information in a database and generates a profile.

[1631] 2. Conversational Interaction

[1632] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1633] The device converts the voice data into text and sends it to the server.

[1634] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[1635] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[1636] 3. Detecting signs of dementia

[1637] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1638] When the device detects abnormal speech or behavior, it sends the data to the server.

[1639] The server analyzes the data and runs algorithms to determine signs of dementia.

[1640] If the server detects any signs, it will send an alert to the caregiver via the device.

[1641] 4. Providing nursing care know-how

[1642] The user (caregiver) asks questions about care to the terminal.

[1643] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[1644] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[1645] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1646] 5. Reminder function

[1647] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1648] The device sends the reminder information to the server, which stores it.

[1649] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1650] The terminal converts the notification message into voice and reminds the user.

[1651] Specific examples

[1652] Concrete example of conversational interaction

[1653] 1. User (elderly person): "I went for a walk today."

[1654] 2. Device: Capture audio with the microphone and temporarily store it.

[1655] 3. Device: Converts speech to text and sends it to the server.

[1656] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[1657] 5. Server: Generates and sends the response to the device.

[1658] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[1659] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[1660] The processing flow will be explained below.

[1661] Program processing steps

[1662] Initial Setup

[1663] 1. Step 1:

[1664] The user turns on the device and enters basic configuration information, including Wi-Fi settings and the name, age, and health information of the senior and caregiver.

[1665] 2. Step 2:

[1666] The device will perform an initial calibration based on the basic configuration information you entered, a process that includes testing the sensors, microphone, and speaker.

[1667] 3. Step 3:

[1668] The terminal sends initial configuration information to the server, including the user's profile data.

[1669] 4. Step 4:

[1670] The server stores the received profile data in a database and generates and manages user profiles.

[1671] Conversational Interactions

[1672] 1. Step 1:

[1673] The user (elderly person or caregiver) speaks into the device. Example: "I went for a walk today."

[1674] 2. Step 2:

[1675] The device captures the user's voice with a microphone and temporarily stores it as audio data.

[1676] 3. Step 3:

[1677] The device converts the voice data into text, using voice recognition technology to accurately convert the user's words into text data.

[1678] 4. Step 4:

[1679] The terminal transmits the converted text data to the server.

[1680] 5. Step 5:

[1681] The server analyzes the received text data, using NLP (Natural Language Processing) algorithms to understand what the user is saying and search for data to generate an appropriate response.

[1682] 6. Step 6:

[1683] The server runs a response generation algorithm and generates a text response, e.g., "How was the weather today?"

[1684] 7. Step 7:

[1685] The server transmits the generated response data to the terminal.

[1686] 8. Step 8:

[1687] The device converts the response data into speech and responds to the user with speech. Example: "How was the weather today?"

[1688] Dementia symptom detection

[1689] 1. Step 1:

[1690] The device constantly monitors the elderly person's conversations and behavior, collecting data using sensors and microphones.

[1691] 2. Step 2:

[1692] The device transmits data on abnormal behavior and speech to a server in real time.

[1693] 3. Step 3:

[1694] The server analyzes the data it receives and runs algorithms to detect signs of dementia, using pattern-matching techniques to identify abnormal behavioral patterns.

[1695] 4. Step 4:

[1696] If the server detects signs of dementia, it generates an alert, including specific abnormal behaviors and their frequency.

[1697] 5. Step 5:

[1698] The server transmits the generated alert information to the terminal.

[1699] 6. Step 6:

[1700] The device notifies the caregiver of the alert information via audio or visual messages.

[1701] Providing nursing care know-how

[1702] 1. Step 1:

[1703] The user (caregiver) asks a question about caregiving to the device. Example: "How can I guide the elderly effectively?"

[1704] 2. Step 2:

[1705] The device captures the user's question with a microphone and saves it as audio data.

[1706] 3. Step 3:

[1707] The device converts the voice data into text and sends it to the server.

[1708] 4. Step 4:

[1709] The server analyzes the received text data and understands the intent of the question.

[1710] 5. Step 5:

[1711] The server searches the database for relevant care know-how and advice.

[1712] 6. Step 6:

[1713] The server generates appropriate advice in text format and sends it to the terminal.

[1714] 7. Step 7:

[1715] The device converts the received advice into speech and conveys it to the user. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[1716] Reminder function

[1717] 1. Step 1:

[1718] Users input information such as medication times and hospital appointment information into the device, either via voice or text.

[1719] 2. Step 2:

[1720] The device temporarily stores the entered reminder information and sends it to the server.

[1721] 3. Step 3:

[1722] The server stores the reminder information in a database and manages the schedule.

[1723] 4. Step 4:

[1724] When the reminder time approaches, the server generates a notification message.

[1725] 5. Step 5:

[1726] The server sends the generated notification message to the terminal.

[1727] 6. Step 6:

[1728] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[1729] In this way, the system functions effectively through each processing step, providing support to the elderly and their caregivers.

[1730] Example 1

[1731] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1732] In modern society, elderly people and their caregivers face many challenges in their daily lives. In particular, there is a need for early detection of dementia in the elderly and support in daily life. Furthermore, providing effective caregiving know-how and reducing the burden on caregivers are also important issues. Addressing these challenges requires a system that can monitor the conversations and behavior of elderly people and provide appropriate advice and reminders.

[1733] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1734] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses to speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for generating caregiving know-how and response text using a generative AI model, means for processing the speech data using speech recognition and speech synthesis software, and means for inputting basic setting information of the elderly person and the caregiver and generating a profile. This enables early detection of signs of dementia in the elderly, support for daily life, and providing effective advice to caregivers.

[1735] The "means for capturing conversation" is a function for acquiring the voice uttered by the user using a voice input device such as a microphone.

[1736] "Means for converting voice to text" refers to a function that converts acquired voice data into text format using voice recognition technology.

[1737] The "means for analyzing the generated text data and generating an appropriate response" is a function for analyzing the text data using natural language processing technology and generating an appropriate response based on the analysis.

[1738] The "means for converting the generated response into speech and presenting it to the user" is a function for converting the generated text response into speech using speech synthesis technology and presenting it to the user.

[1739] "Means for monitoring conversations and behaviors and detecting signs of dementia" refers to a function that uses sensors to continuously monitor the user's conversations and behaviors and detects signs of dementia based on an algorithm.

[1740] "Means for notifying caregivers when signs of dementia are detected" is a function that sends alerts and notifications to caregivers via a server when signs of dementia are detected by the algorithm.

[1741] "Means for analyzing questions and generating advice in order to provide know-how on caregiving" refers to a function that analyzes questions on caregiving from users and generates appropriate advice using a database and a generative AI model.

[1742] "Means to remind you to take medicine or make hospital appointments" is a function that records the time the user sets to take medicine or hospital appointment information, and notifies you when the reminder time approaches.

[1743] "Means for generating care know-how and response text using a generative AI model" is a function that uses a generative AI model to generate know-how about care and responses to user questions in text format.

[1744] "Means for processing voice data using voice recognition and voice synthesis software" refers to the function of converting voice into text using voice recognition software and converting text into voice using voice synthesis software.

[1745] The "means for inputting basic setting information and generating a profile" is a function in which a user inputs initial setting information, and the server generates a profile for each user based on this information.

[1746] MODE FOR CARRYING OUT THE INVENTION

[1747] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[1748] System configuration

[1749] 1. Server

[1750] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[1751] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[1752] The server uses generative AI models to analyze the information sent by the user and generate the necessary responses or alerts.

[1753] 2. Terminal

[1754] The device is designed as a stuffed toy device and is equipped with a microphone, speaker, and various sensors.

[1755] The device uses voice recognition software to capture the user's speech and convert it into text.

[1756] The terminal transmits data to the server as needed, and also conveys responses from the server to the user by voice.

[1757] 3. Users

[1758] The users mainly refer to elderly people and their caregivers.

[1759] Users can talk to the device to engage in conversational interactions and ask questions about caregiving.

[1760] Users receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[1761] Program processing details

[1762] 1. Initial Setup

[1763] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and senior's names, ages, and medication information.

[1764] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1765] The server stores the received initial setting information in a database and generates profiles for the elderly person and their caregiver.

[1766] 2. Conversational Interaction

[1767] When a user speaks to the device, the device captures the voice with its microphone and temporarily stores it. For example, if a user says, "I went for a walk today," the device's microphone will capture this.

[1768] The device converts the voice data into text using voice recognition software and sends it to the server.

[1769] The server analyzes the text data and uses a generative AI model (e.g., a GPT-based model) to generate an appropriate response, such as "How was the weather today?"

[1770] The server transmits the generated response data to the terminal, which converts it into voice using speech synthesis software and conveys the response to the user.

[1771] 3. Detecting signs of dementia

[1772] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1773] If the device detects abnormal speech or behavior, it will send that data to the server. For example, if an elderly person frequently asks the same question, the device will send that data to the server.

[1774] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[1775] If the server detects any signs, it will send an alert to the caregiver via the device.

[1776] 4. Providing nursing care know-how

[1777] The user (caregiver) asks a question about care to the terminal, such as "How should I assist with eating?"

[1778] The device captures the question with a microphone, converts it into text using speech recognition software, and sends it to a server.

[1779] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[1780] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1781] 5. Reminder function

[1782] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1783] The device sends the reminder information to the server, which stores it in a database.

[1784] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1785] The terminal converts the notification message into voice and reminds the user.

[1786] Specific examples

[1787] Concrete example of conversational interaction

[1788] 1. User (elderly person): "I went for a walk today."

[1789] 2. Device: Capture audio with the microphone and temporarily store it.

[1790] 3. Device: Converts speech to text and sends it to the server.

[1791] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[1792] 5. Server: Generates and sends the response to the device.

[1793] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[1794] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[1795] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1796] Initial Setup

[1797] Step 1:

[1798] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and the senior's names, ages, and medication information.

[1799] Input: Wi-Fi settings, basic information of the caregiver and the elderly person (name, age, medication information)

[1800] Output: Initial settings information is saved on the device

[1801] Specific behavior:

[1802] The user manually enters the necessary information on the initial setup screen of the device.

[1803] Step 2:

[1804] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1805] Input: User-entered initial configuration information

[1806] Output: Configuration information sent to the server

[1807] Specific behavior:

[1808] The device establishes a Wi-Fi connection and sends the initial configuration information in packets to the server.

[1809] The device will perform calibration of its internal sensors and microphones.

[1810] Step 3:

[1811] The server stores the received initial setting information in a database and generates a profile.

[1812] Input: Setting information sent from the device

[1813] Output: Profiles stored in the database

[1814] Specific behavior:

[1815] The server writes the configuration information to a database and generates profile IDs for the elderly person and their caregiver.

[1816] Conversational Interactions

[1817] Step 1:

[1818] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1819] Input: User speech

[1820] Output: Temporarily saved audio data

[1821] Specific behavior:

[1822] The device's microphone captures audio data in real time and temporarily stores it in local storage.

[1823] Step 2:

[1824] The device converts the voice data into text using voice recognition software and sends it to the server.

[1825] Input: Temporarily saved audio data

[1826] Output: Data converted to text

[1827] Specific behavior:

[1828] The device converts voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text.

[1829] Step 3:

[1830] The server analyzes the text data and generates an appropriate response using a generative AI model.

[1831] Input: Text format data

[1832] Output: The generated response text

[1833] Specific behavior:

[1834] The server analyzes the text data using models such as OpenAI's GPT model and generates an appropriate response.

[1835] Step 4:

[1836] The server transmits the generated response data to the terminal, which converts it into speech using speech synthesis software and responds to the user.

[1837] Input: Generated response text

[1838] Output: The transcribed response

[1839] Specific behavior:

[1840] The server sends response data generated by the AI ​​model to the terminal.

[1841] The device uses voice synthesis technology such as Amazon Polly to convert text data into speech and transmits it to the user through the speaker.

[1842] Dementia symptom detection

[1843] Step 1:

[1844] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1845] Input: Elderly conversations and behavior

[1846] Output: Monitoring result data

[1847] Specific behavior:

[1848] The device's built-in microphones, accelerometers, and cameras capture the elderly's behavioral data in real time.

[1849] Step 2:

[1850] When the device detects abnormal speech or behavior, it sends the data to the server.

[1851] Input: Monitoring result data

[1852] Output: Behavioral data sent to the server

[1853] Specific behavior:

[1854] When a terminal detects an abnormality, it sends the data in packet form to the server.

[1855] Step 3:

[1856] The server analyzes the data and runs algorithms to determine signs of dementia.

[1857] Input: Received behavioral data

[1858] Output: Detected dementia symptoms

[1859] Specific behavior:

[1860] The server uses machine learning algorithms to analyze behavioral data and detect abnormal trends.

[1861] Step 4:

[1862] If the server detects any signs, it will send an alert to the caregiver via the device.

[1863] Input: Detected dementia symptoms

[1864] Output: Alert notification to caregiver

[1865] Specific behavior:

[1866] The server generates an alert message and sends it to the terminal.

[1867] The device will notify caregivers of alerts via speaker or display.

[1868] Providing nursing care know-how

[1869] Step 1:

[1870] The user (caregiver) asks questions about care to the terminal.

[1871] Input: Caregiver's voice question

[1872] Output: Temporarily saved audio data

[1873] Specific behavior:

[1874] The device captures the caregiver's questions and temporarily stores the audio.

[1875] Step 2:

[1876] The device uses voice recognition software to convert the question into text and send it to the server.

[1877] Input: Temporarily saved audio data

[1878] Output: Data converted to text

[1879] Specific behavior:

[1880] The device uses voice recognition software to convert the voice data into text and send it to the server.

[1881] Step 3:

[1882] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[1883] Input: Text question data

[1884] Output: Generated advice

[1885] Specific behavior:

[1886] The server uses a generative AI model to analyze the text data and generate appropriate care advice.

[1887] Step 4:

[1888] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1889] Input: Generated advice text

[1890] Output: Advice converted to speech

[1891] Specific behavior:

[1892] The server transmits the generated advice to the terminal.

[1893] The device uses speech synthesis software to convert advice into voice and convey it to the caregiver.

[1894] Reminder function

[1895] Step 1:

[1896] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1897] Input: Reminder information entered by the user

[1898] Output: Reminder information saved on the device

[1899] Specific behavior:

[1900] The user manually enters the reminder information into an input form on the device.

[1901] Step 2:

[1902] The device sends the reminder information to the server, which stores it in a database.

[1903] Input: Reminder information sent from the device

[1904] Output: Reminder information stored in the database

[1905] Specific behavior:

[1906] The device sends the reminder information to the server, which stores it in a database.

[1907] Step 3:

[1908] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1909] Input: Reminder information stored in the database

[1910] Output: The generated notification message

[1911] Specific behavior:

[1912] The server generates a notification message based on the set reminder time and sends it to the terminal.

[1913] Step 4:

[1914] The terminal converts the notification message into voice and reminds the user.

[1915] Input: The generated notification message

[1916] Output: Reminder message converted to audio

[1917] Specific behavior:

[1918] The device uses speech synthesis software to convert notification messages into voice and provide reminders.

[1919] (Application example 1)

[1920] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1921] Supporting the elderly and their caregivers has become an important issue in modern society. In particular, there is a demand for conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. At the same time, the quality and functionality of the devices and robots used to provide this support are also extremely important. In the manufacturing process of elderly support robots, quality inspection, function testing, and progress management of manufacturing work are challenges.

[1922] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1923] In this invention, the server includes a means for capturing conversations between the elderly person and their caregiver and converting the speech into text, a means for analyzing the generated text data and generating appropriate responses, and a means for converting the generated responses into speech and presenting them to the user. This enables high-performance support for the elderly person and their caregiver, as well as the implementation of quality inspections and functional tests in the manufacturing process, progress management of manufacturing work using voice recognition, and scheduling and notifying quality inspections using a reminder function.

[1924] A "support system for older people and their caregivers" is an integrated system that provides the services and functions needed by both older people and their caregivers.

[1925] "Means for capturing conversations and converting speech to text" refers to technology that collects a user's speech and converts it into text information.

[1926] "Means for analyzing generated text data and generating an appropriate response" refers to a technology that analyzes text data and determines the optimal response or action based on its content.

[1927] "Means for converting the generated response into audio and presenting it to the user" refers to technology that converts the generated text-based response into audio format and provides it to the user.

[1928] "Means for monitoring conversations and behavior to detect signs of dementia" refers to technology that monitors a user's speech and behavior patterns and detects abnormalities.

[1929] "Means to notify caregivers if signs of dementia are detected" is a function that sends an alert to caregivers when early signs of dementia are confirmed.

[1930] "Means for analyzing questions and generating advice in order to provide know-how related to caregiving" refers to technology that analyzes questions from caregivers and generates and provides appropriate advice and information.

[1931] "Medication and medical appointment reminders" refers to features that remind users of specific times or events (e.g., medication time, doctor's appointment time).

[1932] "Means for conducting quality inspections and product functionality tests during the manufacturing process and recording the results" refers to a system that checks the quality and functionality of products at each stage of manufacturing and stores the results as a log.

[1933] "Means for managing the progress of manufacturing work using voice recognition" is a technology that uses voice input to track and manage the status and progress of manufacturing work.

[1934] "Means for scheduling and notifying quality inspections using a reminder function" is a function that allows you to schedule quality inspections and notify you when the time comes.

[1935] This invention is a system that supports elderly people and their caregivers, and in particular, it realizes conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[1936] System configuration

[1937] 1. Server

[1938] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia. The analysis uses NLP (natural language processing) technology.

[1939] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[1940] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[1941] Manage quality inspections and product functionality tests during the manufacturing process and record the results.

[1942] 2. Terminal

[1943] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[1944] The device has the ability to capture the user's conversation and convert the speech into text.

[1945] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[1946] The terminal uses voice recognition to manage the progress of manufacturing work.

[1947] 3. Users

[1948] The users mainly refer to elderly people and their caregivers.

[1949] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[1950] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[1951] Program processing details

[1952] The main functions of the system are as follows:

[1953] 1. Initial Setup

[1954] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[1955] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[1956] The server stores the received initial setting information in a database and generates a profile.

[1957] 2. Conversational Interaction

[1958] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[1959] The device converts the voice data into text and sends it to the server.

[1960] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[1961] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[1962] 3. Detecting signs of dementia

[1963] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[1964] When the device detects abnormal speech or behavior, it sends the data to the server.

[1965] The server analyzes the data and runs algorithms to determine signs of dementia.

[1966] If the server detects any signs, it will send an alert to the caregiver via the device.

[1967] 4. Providing nursing care know-how

[1968] The user (caregiver) asks questions about care to the terminal.

[1969] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[1970] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[1971] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[1972] 5. Reminder function

[1973] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[1974] The device sends the reminder information to the server, which stores it.

[1975] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[1976] The terminal converts the notification message into voice and reminds the user.

[1977] 6. Quality inspection and function test

[1978] During the manufacturing process, the terminal performs quality inspections and product functionality tests and transmits the results to the server.

[1979] The server records the results and generates alerts for any defective or problematic products.

[1980] 7. Manufacturing work progress management

[1981] The terminal uses voice recognition to manage the progress of manufacturing work and transmits necessary information and instructions to the server.

[1982] The server tracks progress and provides timely feedback.

[1983] 8. Reminders feature expansion

[1984] The device can use the reminder function to schedule and notify quality inspections.

[1985] Based on this information, the server manages the schedule and sends notifications at the appropriate times.

[1986] Specific examples

[1987] Concrete example of conversational interaction

[1988] 1. User (elderly person): "I went for a walk today."

[1989] 2. Device: Capture audio with the microphone and temporarily save it.

[1990] 3. Device: Converts speech into text and sends it to the server.

[1991] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[1992] 5. Server: Sends the generated response to the terminal.

[1993] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[1994] Examples of quality inspections and functional tests

[1995] 1. Terminal: Inspects each part of the robot during production.

[1996] 2. Server: Records inspection results and generates alerts if there are any problems.

[1997] 3. Terminal: The test is completed and the results are sent to the server.

[1998] 4. Terminal: "The robot's serial number is R12345, and this test result is a pass."

[1999] This effectively provides support for both the elderly and their caregivers, while also ensuring quality control and progress management of the manufacturing process.

[2000] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2001] Step 1:

[2002] The user performs the initial setup of the system. The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and caregiver into the device. The device calibrates the sensors and microphone based on this information and sends the setup information to the server. The server generates a profile based on this initial setup information and saves it in a database. The input is the user's basic setup information, and the output is the server's creation and saving of the profile.

[2003] Step 2:

[2004] The user speaks into the device. The device captures the voice with a microphone and temporarily stores it. Since the voice data cannot be used as is, the device converts the voice into text. This converted text data is sent to the server. The input is the user's voice, and the output is text data sent to the server.

[2005] Step 3:

[2006] The server analyzes the transmitted text data and generates an appropriate response using NLP (natural language processing) technology. This requires understanding the context and intent from the text data and generating a corresponding response. The generated response data is then sent to the terminal. The input is the text data, and the output is the generated response.

[2007] Step 4:

[2008] The terminal converts the response received from the server into speech and presents it to the user. The user receives the speech response. The input is the response data from the server, and the output is the speech response to the user.

[2009] Step 5:

[2010] The device uses sensors to constantly monitor the user's conversations and behavior. If the sensors detect abnormal speech or behavior, the device temporarily stores it and sends it to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, the server sends an alert to the caregiver via the device. The input is sensor data, and the output is a dementia sign alert.

[2011] Step 6:

[2012] The user (caregiver) asks a question about caregiving to the device. The device captures the question with a microphone, converts the voice into text, and sends it to the server. The server analyzes the text data, searches a database for appropriate caregiving know-how and advice, and sends the generated advice to the device. The device then conveys this to the user by voice. The input is the voice of the question, and the output is caregiving advice.

[2013] Step 7:

[2014] The user sets the time to take medicine and hospital appointment information on the device. The device sends this reminder information to the server, which stores it. When the reminder time approaches, the server generates a notification message and sends it to the device. The device converts the notification message into audio and reminds the user. The input is the reminder setting information, and the output is the reminder notification.

[2015] Step 8:

[2016] During the manufacturing process, the terminal performs quality inspections and product functionality tests. The terminal transmits the inspection results for each item to the server, which records the results and generates alerts as needed. The input is the inspection data, and the output is the quality inspection result record and alerts.

[2017] Step 9:

[2018] The terminal uses voice recognition to manage the progress of manufacturing work. When the user speaks to report progress, the terminal converts the speech into text and sends it to the server. The server updates the progress based on the transmitted data and provides necessary feedback. The input is the voice progress report, and the output is progress updates and feedback.

[2019] Step 10:

[2020] The terminal uses the reminder function to schedule quality inspections and notify the user. The server manages the schedule based on this information and notifies the user at the appropriate time. The input is the inspection schedule setting information, and the output is the inspection schedule notification.

[2021] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2022] This invention is a system to support elderly people and their caregivers, and aims to provide more effective support by incorporating an emotion engine that recognizes the user's emotions, in addition to conversational interaction, dementia symptom detection, provision of care know-how, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[2023] System configuration

[2024] 1. Server

[2025] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[2026] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[2027] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[2028] The server also generates a response based on the emotional state sent from the emotion engine.

[2029] 2. Terminal

[2030] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, sensors, and camera.

[2031] The device has the ability to capture the user's conversation and convert the speech into text.

[2032] The terminal transmits information to the server as needed and also conveys responses from the server to the user by voice.

[2033] The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[2034] 3. Users

[2035] The users mainly refer to elderly people and their caregivers.

[2036] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[2037] Users receive reminders, alerts, and emotional responses from the device, which they can use as a reference for their daily lives and caregiving.

[2038] Program processing details

[2039] The main functions of the system are as follows:

[2040] 1. Initial Setup

[2041] The user resets the device and enters basic configuration information, including Wi-Fi settings and basic information about the senior and caregiver.

[2042] The device calibrates the sensors, microphone, and camera based on this information and sends the configuration information to the server.

[2043] The server stores the received profile data in a database and generates a user profile.

[2044] 2. Conversational Interaction

[2045] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[2046] The device converts the voice data into text and sends it to the server.

[2047] The server analyzes the text data and uses NLP (Natural Language Processing) algorithms to find data to generate an appropriate response.

[2048] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[2049] 3. Emotion Recognition by Emotion Engine

[2050] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them.

[2051] The device inputs this data into an emotion engine to recognize the user's emotional state in real time.

[2052] The emotion engine sends the recognized emotional state to the server.

[2053] 4. Generating Emotion-Based Responses

[2054] The server then executes an algorithm based on the received emotional state data to generate a response appropriate to the user's current state of mind.

[2055] For example, if the user is in a sad emotional state, a supportive or encouraging response may be generated.

[2056] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[2057] 5. Detecting signs of dementia

[2058] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[2059] When the device detects abnormal speech or behavior, it sends the data to the server.

[2060] The server analyzes the data and runs algorithms to determine signs of dementia.

[2061] If the server detects a symptom, it generates alert information.

[2062] The device notifies the caregiver of the alert information.

[2063] 6. Providing nursing care know-how

[2064] The user (caregiver) asks questions about care to the terminal.

[2065] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[2066] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[2067] The server generates advice and sends it to the terminal.

[2068] The terminal converts the received advice into voice and conveys it to the user.

[2069] 7. Reminder function

[2070] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[2071] The device sends the reminder information to the server, which stores it.

[2072] When the reminder time approaches, the server generates a notification message.

[2073] The server sends the generated notification message to the terminal.

[2074] The terminal converts the notification message into voice and reminds the user.

[2075] Specific examples

[2076] Specific examples of emotion recognition using emotion engines

[2077] 1. User (elderly): "I've been feeling unwell and depressed lately."

[2078] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[2079] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[2080] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[2081] 5. Server: Generates an encouraging response such as "Cheer up" based on the emotional state "sad."

[2082] 6. Server: Sends the generated response to the device.

[2083] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[2084] In this way, the system as a whole aims to reduce the mental and physical burden on elderly people and caregivers by incorporating conversational interaction, emotion recognition, and care support functions. Utilizing an emotion engine enables flexible responses according to the user's emotional state, providing more personalized support.

[2085] The processing flow will be explained below.

[2086] Program processing steps

[2087] Emotion recognition and response generation by emotion engine

[2088] 1. Step 1:

[2089] The user speaks to the device. Example: "I've been feeling unwell and depressed lately."

[2090] 2. Step 2:

[2091] The device uses a microphone to capture the user's voice and a camera to capture the user's facial expressions.

[2092] 3. Step 3:

[2093] The device converts the captured voice data into text using voice recognition technology and sends it to the emotion engine along with facial expression data.

[2094] 4. Step 4:

[2095] The emotion engine analyzes the tone of the voice data and facial expression data and recognizes the user's emotional state as "sad."

[2096] 5. Step 5:

[2097] The emotion engine transmits the recognized emotional state data to the server.

[2098] 6. Step 6:

[2099] The server analyzes the received emotional state data and generates an appropriate response, e.g., encouraging words such as "Cheer up!"

[2100] 7. Step 7:

[2101] The server transmits the generated response data to the terminal.

[2102] 8. Step 8:

[2103] The device converts the received response data into voice and responds to the user with voice. Example: "Cheer up."

[2104] Dementia symptom detection

[2105] 1. Step 1:

[2106] The device constantly monitors the elderly person's conversations and behavior using sensors and microphones.

[2107] 2. Step 2:

[2108] When the device detects abnormal behavior (e.g. repeating the same word multiple times), it sends the data to the server in real time.

[2109] 3. Step 3:

[2110] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[2111] 4. Step 4:

[2112] If the server detects signs of dementia, it generates alert information and sends data to the terminal to notify the caregiver.

[2113] 5. Step 5:

[2114] The device will notify the caregiver of the alert notification with audio and visual messages, for example, "The elderly person is repeating the same words over and over. Attention is needed."

[2115] Providing nursing care know-how

[2116] 1. Step 1:

[2117] The user (caregiver) asks a question about caregiving to the device. Example: "How can we effectively guide the elderly?"

[2118] 2. Step 2:

[2119] The device captures the question with a microphone and saves it as audio data.

[2120] 3. Step 3:

[2121] The device converts the voice data into text using voice recognition technology and sends it to the server.

[2122] 4. Step 4:

[2123] The server analyzes the received text data and runs NLP algorithms to understand the intent of the question.

[2124] 5. Step 5:

[2125] The server searches the database for relevant care know-how and advice.

[2126] 6. Step 6:

[2127] The server generates appropriate advice and sends it to the device. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[2128] 7. Step 7:

[2129] The terminal converts the received advice into voice and conveys it to the user.

[2130] Reminder function

[2131] 1. Step 1:

[2132] Users enter medication times and hospital appointment information into the terminal by voice or manually.

[2133] 2. Step 2:

[2134] The device temporarily stores the entered reminder information and sends it to the server.

[2135] 3. Step 3:

[2136] The server stores the reminder information in a database and manages the schedule.

[2137] 4. Step 4:

[2138] When the reminder time approaches, the server generates a reminder notification message.

[2139] 5. Step 5:

[2140] The server sends the generated remind notification message to the terminal.

[2141] 6. Step 6:

[2142] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[2143] This system combines multiple functions to provide multifaceted support to the elderly and their caregivers. By utilizing emotion recognition through an emotion engine, it is possible to provide more personalized responses, aiming to increase user satisfaction.

[2144] Example 2

[2145] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2146] Providing effective support to elderly people and their caregivers requires not only conversational interaction and information provision, but also responses that take into account the user's emotions and psychological state. However, conventional systems have had difficulty comprehensively recognizing emotions, detecting signs of dementia, and providing care know-how. Furthermore, there have been no systems that can reduce the user's mental burden and provide flexible responses that correspond to their emotions. This has made it difficult to improve the satisfaction of elderly people and caregivers.

[2147] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2148] In this invention, the server

[2149] A means of capturing the conversations of the senior and caregiver and converting the speech to text;

[2150] means for analyzing the generated text data and generating an appropriate response;

[2151] means for converting the generated response into speech and presenting it to the user;

[2152] A means of monitoring the conversations and behavior of elderly people to detect signs of dementia,

[2153] A means to notify caregivers if signs of dementia are detected; and

[2154] A means for analyzing questions and generating advice to provide knowledge about care;

[2155] A way to remind you to take your medicine or make a doctor's appointment,

[2156] a means for recognizing a user's emotion using an emotion engine and generating a response according to the emotion;

[2157] means for transmitting the emotional state to a server for analysis and generating an appropriate response based on the user's psychological state;

[2158] This will enable comprehensive and personalized support for older people and their caregivers.

[2159] "Elderly people and caregivers" refers to people who play a role in supporting the daily lives and health management of older people.

[2160] "Capture" refers to the technology of collecting data such as audio and video and storing it in digital form.

[2161] "Speech-to-text" refers to the technology of analyzing voice data and converting its contents into text data.

[2162] "Generated text data" refers to data obtained by analyzing speech and converting it into character data.

[2163] "Response" refers to information or messages returned in response to user input.

[2164] "Monitoring" refers to the act of continuously observing the state or behavior of a specific object to detect abnormalities or changes.

[2165] "Detecting signs of dementia" refers to finding changes in specific behaviors or speech that may be indicative of dementia.

[2166] "Notify caregiver" refers to providing important detected information to the caregiver as a warning or alert.

[2167] "Providing knowledge" means providing appropriate advice and guidance based on the questions asked using accumulated information.

[2168] A "remind" is a notification in advance about a specific time or event so that you don't forget.

[2169] An "emotion engine" refers to technology that analyzes a user's tone of voice, facial expressions, etc. to identify their current emotional state.

[2170] "Responding based on psychological state" refers to generating a response appropriate to the user's mental state at the time based on their emotional data.

[2171] This invention is a system that supports elderly people and their caregivers, and provides comprehensive and personalized support by incorporating conversational interaction, dementia symptom detection, caregiving know-how provision, reminder functions, and an emotion engine.

[2172] System configuration

[2173] This system mainly consists of three elements: a server, a terminal, and a user.

[2174] 1. Server

[2175] The server analyzes voice data, generates text data, generates responses, and detects signs of dementia. The server is equipped with a database that stores information on care know-how, local consultation centers, and welfare systems. The server analyzes the information sent by the user and generates the necessary responses and alerts. It also generates responses based on the emotional state sent from the emotion engine.

[2176] 2. Terminal

[2177] The device is designed as a plush toy and is equipped with a microphone, speaker, sensors, and camera. The device has the ability to capture the user's conversation and convert the speech into text. If necessary, it sends information to the server and communicates the server's response to the user via voice. The device captures the user's facial expressions and voice tone and recognizes their emotional state through an emotion engine.

[2178] 3. Users

[2179] Users are primarily elderly people and their caregivers. By speaking into the device, users can engage in conversational interactions and ask questions about caregiving. Users receive reminders, alerts, and emotional responses from the device, which they can use as references for their daily lives and caregiving.

[2180] Detailed System Operation

[2181] The detailed operation of this system will be described below.

[2182] 1. Initial Setup

[2183] The user initializes the device and enters Wi-Fi settings and basic information (information about the senior and caregiver). The device calibrates the sensors, microphone, and camera based on this information and sends the setting information to the server. The server saves the received information and creates a user profile.

[2184] 2. Conversational Interaction

[2185] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is converted into text and sent to the server. The server analyzes the text data and generates an appropriate response using a natural language processing (NLP) algorithm. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[2186] 3. Emotion Recognition by Emotion Engine

[2187] The device captures and temporarily stores the user's voice tone and facial expressions. This data is input to the emotion engine, which recognizes the user's emotional state in real time. The emotion engine then transmits the recognized emotional state to the server.

[2188] 4. Generating Emotion-Based Responses

[2189] The server generates a response appropriate to the user's current psychological state based on the emotional state data. For example, if the emotional state is "sad," it generates a supportive or encouraging response. The generated response data is sent to the device, which then converts it into voice and responds to the user.

[2190] 5. Detecting signs of dementia

[2191] The device uses sensors to monitor the elderly person's conversations and behavior. If it detects abnormal speech or behavior, it sends the data to a server. The server analyzes the data and runs an algorithm to determine signs of dementia. If signs are detected, it generates an alert and notifies the caregiver via the device.

[2192] 6. Providing nursing care know-how

[2193] When a user (caregiver) asks a question about caregiving to the device, the device captures the question, converts the voice to text, and sends it to the server. The server analyzes the data and searches a database for appropriate caregiving know-how and advice. It generates advice and sends it to the device. The device then converts the received advice into voice and conveys it to the user.

[2194] 7. Reminder function

[2195] The user inputs information about medication times and hospital appointment times into the device. The device then sends the reminder information to the server, which stores it. When the reminder time approaches, the server sends a notification message to the device, which then converts it into audio and notifies the user.

[2196] Specific examples

[2197] Specific examples of emotion recognition using emotion engines

[2198] 1. User (elderly): "I've been feeling unwell and depressed lately."

[2199] 2. Device: Capture and temporarily store audio using a microphone, and capture and temporarily store the user's facial expressions using a camera.

[2200] 3. Terminal: Inputs voice and facial expression data into the emotion engine to recognize the emotional state.

[2201] 4. Emotion engine: The recognized emotional state “sad” is sent to the server.

[2202] 5. Server: Generates a supportive response such as "Cheer up" based on the emotional state "sad."

[2203] 6. Server: Generates and sends the response to the device.

[2204] 7. Terminal: Converts the response into voice and tells the elderly person to "cheer up."

[2205] Examples of prompt statements

[2206] Below are some example prompts that utilize generative AI models (e.g., GPT-3) to generate natural-sounding responses.

[2207] "An elderly user says, 'I've been feeling unwell and depressed lately.' The emotion engine detects the emotional state as 'sad.' Please generate an appropriate message of encouragement for this user."

[2208] In this way, this system, with its conversational interaction, emotion recognition, and care support functions, reduces the mental and physical burden on elderly people and caregivers and provides personalized support.

[2209] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2210] Step 1: Initial Setup

[2211] The user initializes the device and enters Wi-Fi settings and basic information about the elderly person and their caregiver, including the user's name, age, and health status.

[2212] Your device uses the information you provide to calibrate sensors, microphones, and cameras, for example, adjusting microphone sensitivity and focusing the camera, optimizing device performance.

[2213] The terminal transmits the initial setting information to the server.

[2214] The server stores the received setting information in a database and generates a user profile, which prepares the data required for subsequent processing.

[2215] Step 2: Conversational Interaction

[2216] The user speaks to the terminal, for example, saying a greeting such as "Good morning."

[2217] The device captures the audio using a microphone and temporarily stores it as audio data.

[2218] The device converts the captured voice data into text to generate text data, which becomes input data.

[2219] The terminal transmits the generated text data to the server.

[2220] The server analyzes the received text data and generates an appropriate response using an NLP (natural language processing) algorithm. For example, for the text "Good morning," it generates the response "Good morning. What are your plans for today?" This becomes the output data.

[2221] The server transmits the generated response data to the terminal.

[2222] The terminal converts the received response data into voice and responds to the user.

[2223] Step 3: Emotion recognition by the emotion engine

[2224] The device captures the user's voice tone and facial expressions using a microphone and camera and temporarily stores them, for example, as digital data, such as voice waveforms and facial movements.

[2225] The device inputs the captured data into an emotion engine to recognize the emotional state. The emotion engine uses a deep learning algorithm to analyze the data and label the emotional state as "sad" or "happy." This becomes the output data.

[2226] The emotion engine transmits the recognized emotional state data to a server.

[2227] Step 4: Generate an emotion-based response

[2228] The server generates a response appropriate to the user's current psychological state based on the received emotional state data. For example, if the user is feeling sad, the server generates a message of encouragement such as "Cheer up!"

[2229] The server transmits the generated response data to the terminal.

[2230] The terminal converts the received response data into voice and responds to the user.

[2231] Step 5: Detecting signs of dementia

[2232] The device uses sensors to monitor the elderly's conversations and behavior, such as motion sensors to monitor walking patterns and microphones to analyze conversation content.

[2233] When the device detects abnormal speech or behavior, it sends the data to the server.

[2234] The server analyzes the received data and runs algorithms to identify signs of dementia, such as by analyzing specific keywords or abnormal behavioral patterns.

[2235] If the server detects signs of dementia, it generates alert information and notifies the caregiver via the terminal.

[2236] Step 6: Providing nursing care know-how

[2237] The user (caregiver) asks a question about care to the device, for example, "What can I do to prevent falls?"

[2238] The device captures the question with a microphone and converts the voice into text to generate text data.

[2239] The terminal transmits the generated text data to the server.

[2240] The server analyzes the received text data and searches a database for appropriate care know-how and advice.

[2241] The server generates appropriate advice and sends it to the terminal, for example, generating a response such as "It would be a good idea to install handrails."

[2242] The terminal converts the received advice data into voice and conveys it to the user.

[2243] Step 7: Reminders

[2244] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[2245] The terminal transmits the input remind information to the server.

[2246] The server stores the received reminder information in a database.

[2247] When the reminder time approaches, the server generates a notification message.

[2248] The server sends the generated notification message to the terminal, for example, "It's time to take your medicine at 2 PM."

[2249] The terminal converts the received notification message into voice and reminds the user.

[2250] These steps enable the system to provide comprehensive and personalized support to seniors and their caregivers.

[2251] (Application example 2)

[2252] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2253] Daily life and caregiving entail many burdens for elderly people and their caregivers. Meal preparation and delivery, health monitoring, and psychological support are particularly important. However, conventional systems often fail to adequately address these needs. For example, they lack meal suggestions that take the user's emotional state into account, and real-time mental support during delivery, making it difficult to improve user satisfaction. Furthermore, there are challenges with early detection of dementia symptoms and providing appropriate care know-how. Given these circumstances, there is a need for a comprehensive and personalized support system for elderly people and their caregivers.

[2254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2255] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses into speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for recognizing the user's emotions in real time and suggesting appropriate meals based on the user's emotional state, and means for providing real-time mental support for situations during delivery. This enables personalized meal suggestions and mental support based on the user's emotional state, thereby improving the quality of life of the elderly person and their caregiver.

[2256] definition statement

[2257] The term "elderly" generally refers to people aged 65 and over.

[2258] A "caregiver" is a person who supports an elderly person in their daily lives and provides the care they need.

[2259] "Emotion engine" refers to a system that includes algorithms that recognize a user's emotional state from their tone of voice and facial expressions.

[2260] "Server" refers to a device that has data processing, analysis and storage functions and manages and controls the entire system.

[2261] "Terminal" refers to an input and output device used by a user that has the functions of voice capture, text conversion, and voice output.

[2262] "Text data" refers to digital information that has been converted from audio data into text format.

[2263] "Nursing care know-how" refers to knowledge, methods, and techniques related to nursing care.

[2264] "Monitoring" refers to the act of continuously observing and recording a subject's conversations and behavior.

[2265] "Signs of dementia" refer to characteristic symptoms and behaviors seen in the early stages or as dementia progresses.

[2266] "Notification" refers to the act of informing a user of important information in real time.

[2267] "Advice" refers to information that provides solutions or recommendations for a particular problem or situation.

[2268] "Remind" refers to the act of notifying and reminding a user in advance of a specific action or event.

[2269] "Meal suggestion" refers to the act of recommending appropriate meal content based on the user's preferences and health condition.

[2270] "Mental support" refers to the act of providing assistance and encouragement to promote the user's mental stability and sense of security.

[2271] invention specification

[2272] This invention is a system that supports elderly people and caregivers, and aims to provide more effective support by including functions such as conversational interaction, emotion recognition, dementia symptom detection, provision of care know-how, meal suggestions, reminders, and mental support.

[2273] System configuration

[2274] This system mainly consists of three elements: a server, a terminal, and a user.

[2275] 1. Server

[2276] The server has the following features:

[2277] Analyzing voice data, generating text data, generating responses

[2278] Dementia symptom detection

[2279] Managing the emotion engine and meal recommendation system

[2280] Real-time mental support during delivery

[2281] Management of various databases (care know-how, local consultation points, welfare system information, etc.)

[2282] 2. Terminal

[2283] The terminal mainly uses the following devices:

[2284] Smartphone, smart glasses, or head-mounted display

[2285] A stuffed toy device equipped with a microphone, speaker, sensor, and camera

[2286] The main features of the terminal are:

[2287] Capture user conversations in real time and convert speech to text

[2288] Sending data to the server and receiving the generated response

[2289] Collecting data to recognize user emotions (voice tone, facial expressions)

[2290] Providing meal suggestions and reminder functions

[2291] 3. Users

[2292] Users are the elderly and their caregivers, and they primarily use the system in the following ways:

[2293] By talking to the device, you can have conversational interactions and ask questions about caregiving.

[2294] Receive responses and reminders from your device

[2295] Receive meal suggestions and mental support from devices and applications

[2296] Program processing details

[2297] Initial Setup

[2298] The user initializes the device and enters basic configuration information, including Wi-Fi settings and a user profile (such as name, age, dietary preferences, and allergy information). The device uses this information to calibrate the sensors, microphone, and camera, and then sends the configuration information to the server. The server then stores the received profile data in a database and creates a user profile.

[2299] Conversational Interaction and Emotion Recognition

[2300] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it. The voice data is then converted into text and sent to the server. The server analyzes the text data and uses an NLP (natural language processing) algorithm to search for data to generate an appropriate response. The generated response data is sent to the device, which then converts it into voice and responds to the user. The device also inputs voice tone and facial expressions into an emotion engine to recognize the user's emotional state in real time.

[2301] Meal suggestions

[2302] Based on the user's emotional state and profile (food preferences and health status), the server runs an algorithm to suggest suitable meals. For example, if the user is in a "sad" emotional state, it will suggest nutritious meals.

[2303] Mental support

[2304] If a delay occurs during delivery, the server checks the user's emotional state again. If the user is anxious, it generates an encouraging message and sends it to the user via the terminal.

[2305] Specific examples

[2306] For example, if a user says, "I haven't had much of an appetite lately," the smartphone's microphone captures the voice. The captured voice data is input into the emotion engine, which recognizes the emotional state as "sad." As a result, the server generates a suggestion such as, "Cheer up! How about this nutritious menu?" and notifies the user via voice through the device.

[2307] Prompt Sentence Examples

[2308] Input: "I haven't had much of an appetite lately."

[2309] Prompt: "Use an emotion engine to recognize the user's emotional state and suggest suitable meals."

[2310] Sample output: "Cheer up! Would you like some of these nutritious options?"

[2311] In this way, the present invention improves the quality of life for seniors and their caregivers by providing personalized support based on the user's emotional state.

[2312] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2313] Program processing steps

[2314] Step 1:

[2315] The user speaks into the terminal.

[2316] Input: User's voice

[2317] What it does: The device captures audio using the built-in microphone.

[2318] Output: Audio data

[2319] Step 2:

[2320] The device converts the voice data into text.

[2321] Input: Audio data

[2322] How it works: The device's built-in software uses voice recognition technology to convert speech into text.

[2323] Output: Text data

[2324] Step 3:

[2325] Sends text data to the server.

[2326] Input: Text data

[2327] Specific operation: The terminal sends text data to the server via the network.

[2328] Output: Text data sent to the server

[2329] Step 4:

[2330] The server parses the received text data and generates an appropriate response.

[2331] Input: Text data

[2332] What it does: The server uses NLP (Natural Language Processing) algorithms to analyze the text data and generate a corresponding response, possibly referencing information in a database.

[2333] Output: Response data

[2334] Step 5:

[2335] The server transmits the generated response data to the terminal.

[2336] Input: Response data

[2337] Specific operation: The server sends response data to the terminal via the network.

[2338] Output: Response data sent to the terminal

[2339] Step 6:

[2340] The terminal converts the response data into voice and presents it to the user.

[2341] Input: Response data

[2342] Specific operation: The device's built-in software converts the text data into voice data and responds to the user audibly using the speaker.

[2343] Output: Voice response

[2344] Step 7:

[2345] The device recognizes the user's emotions in real time.

[2346] Input: User's voice tone and facial expression data

[2347] How it works: The device captures voice tone and facial expression data using a microphone and camera, and then analyzes the data using an emotion engine.

[2348] Output: Emotional state

[2349] Step 8:

[2350] The server suggests suitable meals to the user based on their emotional state.

[2351] Input: Emotional state and user profile data (e.g., food preferences)

[2352] Specific operation: The server uses an emotion engine and a meal suggestion algorithm to analyze the emotional state and profile data and select an appropriate meal.

[2353] Output: Meal suggestion data

[2354] Step 9:

[2355] The server transmits the generated meal suggestion data to the terminal.

[2356] Input: Meal suggestion data

[2357] Specific operation: The server sends meal suggestion data to the terminal via the network.

[2358] Output: Meal suggestion data sent to the device

[2359] Step 10:

[2360] The terminal converts the meal suggestion data into voice and notifies the user.

[2361] Input: Meal suggestion data

[2362] Specific operation: The device's built-in software converts text data into voice data and provides meal suggestions to the user using the speaker.

[2363] Output: Spoken meal suggestions

[2364] Step 11:

[2365] The server provides real-time mental support during delivery situations.

[2366] Input: Delivery status and user emotional state

[2367] Specific operation: The server monitors the delivery status and the user's emotional state in real time and generates mental support messages as needed.

[2368] Output: Mental support message

[2369] Step 12:

[2370] The server transmits the generated mental support message to the terminal.

[2371] Input: Mental support message

[2372] Specific operation: The server sends a mental support message to the terminal via the network.

[2373] Output: Mental support message sent to the device

[2374] Step 13:

[2375] The terminal converts the mental support message into voice and notifies the user.

[2376] Input: Mental support message

[2377] Specific operation: The device's built-in software converts text data into voice data and uses the speaker to provide the user with a mental support message.

[2378] Output: Audio mental support

[2379] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2380] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2381] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2382] [Fourth embodiment]

[2383] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2384] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2385] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2386] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2387] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2388] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2389] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2390] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2391] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2392] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2393] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2394] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2395] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2396] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system mainly consists of three elements: a server, a terminal, and a user.

[2397] System configuration

[2398] 1. Server

[2399] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[2400] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[2401] The server analyzes the information sent by the user and generates the necessary responses or alerts.

[2402] 2. Terminal

[2403] The device is designed as a plush toy-like device and is equipped with a microphone, speaker, and sensors.

[2404] The device has the ability to capture the user's conversation and convert the speech into text.

[2405] The terminal transmits data to the server as required and also conveys responses from the server to the user by voice.

[2406] 3. Users

[2407] The users mainly refer to elderly people and their caregivers.

[2408] Users can engage in conversational interactions and ask questions about caregiving by speaking into the terminal.

[2409] Users can receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[2410] Program processing details

[2411] The main functions of the system are as follows:

[2412] 1. Initial Setup

[2413] The user initializes the device and enters basic configuration information, including Wi-Fi settings and basic information about the caregiver and the senior.

[2414] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[2415] The server stores the received initial setting information in a database and generates a profile.

[2416] 2. Conversational Interaction

[2417] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[2418] The device converts the voice data into text and sends it to the server.

[2419] The server analyzes the text data and uses NLP (natural language processing) to generate an appropriate response.

[2420] The server transmits the generated response data to the terminal, and the terminal converts it into voice and responds to the user.

[2421] 3. Detecting signs of dementia

[2422] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[2423] When the device detects abnormal speech or behavior, it sends the data to the server.

[2424] The server analyzes the data and runs algorithms to determine signs of dementia.

[2425] If the server detects any signs, it will send an alert to the caregiver via the device.

[2426] 4. Providing nursing care know-how

[2427] The user (caregiver) asks questions about care to the terminal.

[2428] The device captures the question with a microphone, converts the speech into text, and sends it to a server.

[2429] The server analyzes the text data and searches a database for appropriate care know-how and advice.

[2430] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[2431] 5. Reminder function

[2432] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[2433] The device sends the reminder information to the server, which stores it.

[2434] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[2435] The terminal converts the notification message into voice and reminds the user.

[2436] Specific examples

[2437] Concrete example of conversational interaction

[2438] 1. User (elderly person): "I went for a walk today."

[2439] 2. Device: Capture audio with the microphone and temporarily store it.

[2440] 3. Device: Converts speech to text and sends it to the server.

[2441] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[2442] 5. Server: Generates and sends the response to the device.

[2443] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[2444] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[2445] The processing flow will be explained below.

[2446] Program processing steps

[2447] Initial Setup

[2448] 1. Step 1:

[2449] The user turns on the device and enters basic configuration information, including Wi-Fi settings and the name, age, and health information of the senior and caregiver.

[2450] 2. Step 2:

[2451] The device will perform an initial calibration based on the basic configuration information you entered, a process that includes testing the sensors, microphone, and speaker.

[2452] 3. Step 3:

[2453] The terminal sends initial configuration information to the server, including the user's profile data.

[2454] 4. Step 4:

[2455] The server stores the received profile data in a database and generates and manages user profiles.

[2456] Conversational Interactions

[2457] 1. Step 1:

[2458] The user (elderly person or caregiver) speaks into the device. Example: "I went for a walk today."

[2459] 2. Step 2:

[2460] The device captures the user's voice with a microphone and temporarily stores it as audio data.

[2461] 3. Step 3:

[2462] The device converts the voice data into text, using voice recognition technology to accurately convert the user's words into text data.

[2463] 4. Step 4:

[2464] The terminal transmits the converted text data to the server.

[2465] 5. Step 5:

[2466] The server analyzes the received text data, using NLP (Natural Language Processing) algorithms to understand what the user is saying and search for data to generate an appropriate response.

[2467] 6. Step 6:

[2468] The server runs a response generation algorithm and generates a text response, e.g., "How was the weather today?"

[2469] 7. Step 7:

[2470] The server transmits the generated response data to the terminal.

[2471] 8. Step 8:

[2472] The device converts the response data into speech and responds to the user with speech. Example: "How was the weather today?"

[2473] Dementia symptom detection

[2474] 1. Step 1:

[2475] The device constantly monitors the elderly person's conversations and behavior, collecting data using sensors and microphones.

[2476] 2. Step 2:

[2477] The device transmits data on abnormal behavior and speech to a server in real time.

[2478] 3. Step 3:

[2479] The server analyzes the data it receives and runs algorithms to detect signs of dementia, using pattern-matching techniques to identify abnormal behavioral patterns.

[2480] 4. Step 4:

[2481] If the server detects signs of dementia, it generates an alert, including specific abnormal behaviors and their frequency.

[2482] 5. Step 5:

[2483] The server transmits the generated alert information to the terminal.

[2484] 6. Step 6:

[2485] The device notifies the caregiver of the alert information via audio or visual messages.

[2486] Providing nursing care know-how

[2487] 1. Step 1:

[2488] The user (caregiver) asks a question about caregiving to the device. Example: "How can I guide the elderly effectively?"

[2489] 2. Step 2:

[2490] The device captures the user's question with a microphone and saves it as audio data.

[2491] 3. Step 3:

[2492] The device converts the voice data into text and sends it to the server.

[2493] 4. Step 4:

[2494] The server analyzes the received text data and understands the intent of the question.

[2495] 5. Step 5:

[2496] The server searches the database for relevant care know-how and advice.

[2497] 6. Step 6:

[2498] The server generates appropriate advice in text format and sends it to the terminal.

[2499] 7. Step 7:

[2500] The device converts the received advice into speech and conveys it to the user. For example, "When guiding elderly people, it is important to speak slowly and clearly."

[2501] Reminder function

[2502] 1. Step 1:

[2503] Users input information such as medication times and hospital appointment information into the device, either via voice or text.

[2504] 2. Step 2:

[2505] The device temporarily stores the entered reminder information and sends it to the server.

[2506] 3. Step 3:

[2507] The server stores the reminder information in a database and manages the schedule.

[2508] 4. Step 4:

[2509] When the reminder time approaches, the server generates a notification message.

[2510] 5. Step 5:

[2511] The server sends the generated notification message to the terminal.

[2512] 6. Step 6:

[2513] The device converts the notification message into voice and reminds the user, for example, "It's time for your medicine."

[2514] In this way, the system functions effectively through each processing step, providing support to the elderly and their caregivers.

[2515] Example 1

[2516] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2517] In modern society, elderly people and their caregivers face many challenges in their daily lives. In particular, there is a need for early detection of dementia in the elderly and support in daily life. Furthermore, providing effective caregiving know-how and reducing the burden on caregivers are also important issues. Addressing these challenges requires a system that can monitor the conversations and behavior of elderly people and provide appropriate advice and reminders.

[2518] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2519] In this invention, the server includes means for capturing conversations between the elderly person and the caregiver and converting the speech to text, means for analyzing the generated text data and generating appropriate responses, means for converting the generated responses to speech and presenting them to the user, means for monitoring the elderly person's conversations and behavior and detecting signs of dementia, means for notifying the caregiver when signs of dementia are detected, means for analyzing questions and generating advice to provide caregiving know-how, means for reminding the elderly person to take medication or make hospital appointments, means for generating caregiving know-how and response text using a generative AI model, means for processing the speech data using speech recognition and speech synthesis software, and means for inputting basic setting information of the elderly person and the caregiver and generating a profile. This enables early detection of signs of dementia in the elderly, support for daily life, and providing effective advice to caregivers.

[2520] The "means for capturing conversation" is a function for acquiring the voice uttered by the user using a voice input device such as a microphone.

[2521] "Means for converting voice to text" refers to a function that converts acquired voice data into text format using voice recognition technology.

[2522] The "means for analyzing the generated text data and generating an appropriate response" is a function for analyzing the text data using natural language processing technology and generating an appropriate response based on the analysis.

[2523] The "means for converting the generated response into speech and presenting it to the user" is a function for converting the generated text response into speech using speech synthesis technology and presenting it to the user.

[2524] "Means for monitoring conversations and behaviors and detecting signs of dementia" refers to a function that uses sensors to continuously monitor the user's conversations and behaviors and detects signs of dementia based on an algorithm.

[2525] "Means for notifying caregivers when signs of dementia are detected" is a function that sends alerts and notifications to caregivers via a server when signs of dementia are detected by the algorithm.

[2526] "Means for analyzing questions and generating advice in order to provide know-how on caregiving" refers to a function that analyzes questions on caregiving from users and generates appropriate advice using a database and a generative AI model.

[2527] "Means to remind you to take medicine or make hospital appointments" is a function that records the time the user sets to take medicine or hospital appointment information, and notifies you when the reminder time approaches.

[2528] "Means for generating care know-how and response text using a generative AI model" is a function that uses a generative AI model to generate know-how about care and responses to user questions in text format.

[2529] "Means for processing voice data using voice recognition and voice synthesis software" refers to the function of converting voice into text using voice recognition software and converting text into voice using voice synthesis software.

[2530] The "means for inputting basic setting information and generating a profile" is a function in which a user inputs initial setting information, and the server generates a profile for each user based on this information.

[2531] MODE FOR CARRYING OUT THE INVENTION

[2532] This invention is a system to support elderly people and their caregivers, and aims to realize conversational interaction, dementia symptom detection, care know-how provision, and reminder functions. This system is mainly composed of three elements: a server, a terminal, and a user.

[2533] System configuration

[2534] 1. Server

[2535] The server analyzes the voice data, generates text data, generates responses, and detects signs of dementia.

[2536] The server is equipped with a database that stores information on nursing care know-how, local consultation centers, and welfare systems.

[2537] The server uses generative AI models to analyze the information sent by the user and generate the necessary responses or alerts.

[2538] 2. Terminal

[2539] The device is designed as a stuffed toy device and is equipped with a microphone, speaker, and various sensors.

[2540] The device uses voice recognition software to capture the user's speech and convert it into text.

[2541] The terminal transmits data to the server as needed, and also conveys responses from the server to the user by voice.

[2542] 3. Users

[2543] The users mainly refer to elderly people and their caregivers.

[2544] Users can talk to the device to engage in conversational interactions and ask questions about caregiving.

[2545] Users receive reminders and alerts from the device and use them as a reference for daily life and caregiving.

[2546] Program processing details

[2547] 1. Initial Setup

[2548] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and senior's names, ages, and medication information.

[2549] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[2550] The server stores the received initial setting information in a database and generates profiles for the elderly person and their caregiver.

[2551] 2. Conversational Interaction

[2552] When a user speaks to the device, the device captures the voice with its microphone and temporarily stores it. For example, if a user says, "I went for a walk today," the device's microphone will capture this.

[2553] The device converts the voice data into text using voice recognition software and sends it to the server.

[2554] The server analyzes the text data and uses a generative AI model (e.g., a GPT-based model) to generate an appropriate response, such as "How was the weather today?"

[2555] The server transmits the generated response data to the terminal, which converts it into voice using speech synthesis software and conveys the response to the user.

[2556] 3. Detecting signs of dementia

[2557] The device uses sensors to constantly monitor the elderly person's conversations and behavior.

[2558] If the device detects abnormal speech or behavior, it will send that data to the server. For example, if an elderly person frequently asks the same question, the device will send that data to the server.

[2559] The server analyzes the received data and runs an algorithm to determine signs of dementia.

[2560] If the server detects any signs, it will send an alert to the caregiver via the device.

[2561] 4. Providing nursing care know-how

[2562] The user (caregiver) asks a question about care to the terminal, such as "How should I assist with eating?"

[2563] The device captures the question with a microphone, converts it into text using speech recognition software, and sends it to a server.

[2564] The server analyzes the text data and uses a generative AI model to search a database for appropriate care know-how and advice.

[2565] The server generates advice and sends it to the terminal, which then conveys it to the user by voice.

[2566] 5. Reminder function

[2567] The user inputs the time to take the medicine and hospital appointment information into the terminal.

[2568] The device sends the reminder information to the server, which stores it in a database.

[2569] When the reminder time approaches, the server generates a notification message and sends it to the terminal.

[2570] The terminal converts the notification message into voice and reminds the user.

[2571] Specific examples

[2572] Concrete example of conversational interaction

[2573] 1. User (elderly person): "I went for a walk today."

[2574] 2. Device: Capture audio with the microphone and temporarily store it.

[2575] 3. Device: Converts speech to text and sends it to the server.

[2576] 4. Server: Analyzes the text data and generates a response: "How was the weather today?"

[2577] 5. Server: Generates and sends the response to the device.

[2578] 6. Terminal: Converts the response into speech and tells the elderly, "How was the weather today?"

[2579] Based on the embodiments of the present invention, support is effectively provided to both the elderly and their caregivers, which is expected to reduce the mental and physical burden on the caregivers and extend the healthy lifespan of the elderly.

[2580] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2581] Initial Setup

[2582] Step 1:

[2583] The user initializes the device and enters basic configuration information, such as Wi-Fi settings, the caregiver's and the senior's names, ages, and medication information.

[2584] Input: Wi-Fi settings, basic information of the caregiver and the elderly person (name, age, medication information)

[2585] Output: Initial settings information is saved on the device

[2586] Specific behavior:

[2587] The user manually enters the necessary information on the initial setup screen of the device.

[2588] Step 2:

[2589] The device calibrates the sensors and microphone based on this information and sends the configuration information to the server.

[2590] Input: User-entered initial configuration information

[2591] Output: Configuration information sent to the server

[2592] Specific behavior:

[2593] The device establishes a Wi-Fi connection and sends the initial configuration information in packets to the server.

[2594] The device will perform calibration of its internal sensors and microphones.

[2595] Step 3:

[2596] The server stores the received initial setting information in a database and generates a profile.

[2597] Input: Setting information sent from the device

[2598] Output: Profiles stored in the database

[2599] Specific behavior:

[2600] The server writes the configuration information to a database and generates profile IDs for the elderly person and their caregiver.

[2601] Conversational Interactions

[2602] Step 1:

[2603] When a user speaks into the device, the device captures the voice with a microphone and temporarily stores it.

[2604] Input: User speech

[2605] Output: Temporarily saved audio data

[2606] Specific behavior:

[2607] The device's microphone captures audio data in real time and temporarily stores it in local storage.

[2608] Step 2:

[2609] The device converts the voice data into text using voice recognition software and sends it to the serv...

Claims

1. A system for supporting elderly people and their caregivers, comprising: A means of capturing the conversations of the senior and caregiver and converting the speech to text; means for analyzing the generated text data and generating an appropriate response; means for converting the generated response into speech and presenting it to the user; A means of monitoring the conversations and behavior of elderly people to detect signs of dementia, A means to notify caregivers if signs of dementia are detected; and A means for analyzing questions and generating advice to provide know-how on caregiving; A system that includes reminders for taking medication and making doctor appointments.

2. Equipped with sensors that constantly monitor the conversations and behavioral patterns of the elderly and their caregivers, The system according to claim 1, wherein the monitoring results are transmitted to a server and analyzed for signs of dementia.

3. It has a database that provides information on local consultation services and welfare systems, The system according to claim 1, wherein appropriate advice is generated based on local information.

Images