System

Abstract

A system is provided.SOLUTION: A system for personalized care support for elderly people and people in need of care, comprising: means for collecting data of voices, facial expressions, and behaviors of the elderly people; means for analyzing the collected data and evaluating stress levels and loneliness of the elderly people; means for detecting abnormalities based on analysis results and notifying caregivers; means for managing timing of medicines to be taken by the elderly people and providing reminders; and means for predicting health conditions and risks of the elderly people and suggesting appropriate care plans.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As our society ages, there is a growing need for personalized care and support for the elderly and those in need of nursing care. However, caregivers are busy on a daily basis, making it difficult for them to provide appropriate care plans tailored to each individual's needs. This creates a need for an efficient system that can reduce the burden on caregivers and protect the safety and health of the elderly. Specifically, there is a need for a system that can monitor the condition of the elderly in real time, detect abnormalities early, and take appropriate action. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes a means for collecting data on the elderly's voice, facial expressions, and behavior; a means for analyzing the collected data and evaluating the elderly's stress level and sense of loneliness; a means for detecting abnormalities based on the analysis results and notifying caregivers; a means for managing the timing of medication intake by the elderly and providing reminders; and a means for predicting the elderly's health status and risks and proposing an appropriate care plan. Specifically, a terminal collects the elderly's voice, facial expressions, and behavior data, and a server analyzes the data to detect and notify abnormalities, provide reminders for medication administration, and predict the elderly's health status and risks, and then proposes an optimal care plan based on the results. This system enables caregivers to provide efficient and accurate care for the elderly.

[0006] "Elderly" refers to people who are of an age that requires special care and nursing care due to aging.

[0007] A "caregiver" is someone who provides daily support and care to older people and those in need of care.

[0008] "Voice data" refers to audio information collected through a microphone and is primarily used to analyze the tone and patterns of a speaker's voice.

[0009] "Facial expression data" refers to facial expression information collected through the camera and is used for emotion analysis and health assessment.

[0010] "Behavioral data" refers to information about your body movements collected through accelerometers and other motion sensors.

[0011] "Stress level" refers to an indicator that assesses the level of mental tension and anxiety that an elderly person feels.

[0012] "Loneliness" refers to the degree of isolation that older people feel due to a lack of social connections and support.

[0013] "Abnormality detection" refers to the early detection of unusual changes or problems in the behavior or condition of elderly people.

[0014] "Notification" refers to the act of sending a warning to a caregiver or family member when an abnormality is detected.

[0015] "Reminders" refer to notifications that prompt seniors to take medication or take other actions based on pre-set times or conditions.

[0016] A "care plan" refers to a plan for optimal care and support for an elderly person, created based on their individual health condition and risks.

[0017] "Server" refers to a centralized device that stores collected data and performs analytical processing.

[0018] "Terminal" refers to a device worn or used directly by the elderly person, which collects data and displays reminders.

[0019] "Health status" refers to the sum of various physiological data used to diagnose the physical and mental state of elderly people.

[0020] "Risk prediction" refers to predicting the probability of future health problems or accidents based on collected data. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate stress levels and feelings of loneliness, and detects and notifies users of abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[0043] System program and processing flow

[0044] This system consists of three main components: a server, a terminal, and an elderly person (user). Below, we explain in natural language the functions and processing flow of each component.

[0045] Data collection

[0046] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[0047] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[0048] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[0049] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[0050] The collected data is sent from the terminal to a server.

[0051] Data analysis

[0052] Server: The server receives the transmitted data and performs the following analysis:

[0053] Facial expression analysis: Analyze facial expression data to estimate the emotional state and stress level of elderly people.

[0054] Voice analysis: Analyze voice data and assess the psychological state of the elderly person based on their tone and voice patterns.

[0055] Movement analysis: Analyzing behavioral data to assess elderly people's activity levels and fall risk.

[0056] Anomaly detection and notification

[0057] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[0058] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0059] Notification: If an abnormality is detected, a notification will be sent to the caregiver, containing details of the abnormality and instructions on how to respond.

[0060] Medication administration management

[0061] Server: Manages the timing of medication based on the elderly person's health data. Specific operations are as follows:

[0062] Generate reminders: Generates reminders when it's time to take your dose.

[0063] Send Reminder: Send the generated reminder to your device.

[0064] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0065] Health Status and Risk Prediction

[0066] Server: Predicts the health status and risks of elderly people based on collected data.

[0067] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0068] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver.

[0069] Specific examples

[0070] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0071] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[0072] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[0073] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[0074] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and promptly notify in the event of an abnormality.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] Data collection

[0078] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[0079] Step 2:

[0080] Data transmission

[0081] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[0082] Step 3:

[0083] Data reception

[0084] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[0085] Step 4:

[0086] facial expression analysis

[0087] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to an emotion recognition model.

[0088] Step 5:

[0089] Audio analysis

[0090] Server: Analyzes voice data and evaluates the psychological state of elderly people based on their voice tone and patterns. For voice analysis, acoustic features are extracted and applied to models that detect emotions and loneliness.

[0091] Step 6:

[0092] Motion analysis

[0093] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[0094] Step 7:

[0095] Anomaly detection

[0096] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[0097] Step 8:

[0098] Notification generation

[0099] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[0100] Step 9:

[0101] Send notifications

[0102] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[0103] Step 10:

[0104] Reminder generation

[0105] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[0106] Step 11:

[0107] Send reminders

[0108] Server: Sends the generated reminders to the device.

[0109] Step 12:

[0110] Reminder display

[0111] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[0112] Step 13:

[0113] Enter intake status

[0114] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[0115] Step 14:

[0116] Recording intake status

[0117] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[0118] Step 15:

[0119] Risk prediction

[0120] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[0121] Step 16:

[0122] Care plan proposal

[0123] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0124] Step 17:

[0125] Task Management

[0126] Server: Prioritizes caregiver tasks and optimizes schedules.

[0127] Step 18:

[0128] Sending reminders and notifications

[0129] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[0130] Step 19:

[0131] Schedule adjustments

[0132] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[0133] Example 1

[0134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] In order to improve the efficiency and accuracy of nursing care support for the elderly and those who require nursing care, it is necessary to monitor the health and psychological state of the elderly in real time, quickly detect and notify abnormalities, and provide appropriate nursing care plans. The purpose of this invention is to solve these problems and provide a new nursing care system that comprehensively supports the safety and health of the elderly.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0137] In this invention, the server includes a means for collecting biometric data, voice, facial expression, and behavioral data, a means for analyzing the collected data to estimate the elderly person's emotional state and stress level, and a means for detecting abnormalities based on the analysis results and notifying the caregiver. This makes it possible to accurately predict the elderly person's health condition and risks, and quickly provide an appropriate care plan.

[0138] "Biometric data" refers to data that indicates the physical condition of an elderly person, such as their heart rate, blood pressure, and body temperature.

[0139] "Voice" refers to the words, tone, and pitch of the voice spoken by the elderly, and is data used to evaluate their psychological state and emotions.

[0140] "Facial expression" refers to data showing the facial expressions of elderly people and is used to evaluate their emotional and psychological states.

[0141] "Behavior" refers to data that records the daily movements and activity patterns of elderly people, and is used to evaluate the amount of exercise and abnormal behavior.

[0142] "Stress level" is an indicator of the degree of mental and physical stress experienced by elderly people.

[0143] "Abnormal" refers to a state that is different from the normal state of an elderly person, and refers to phenomena or behaviors that may pose a risk to their health or safety.

[0144] A "care plan" is a nursing care plan formulated according to the individual needs of an elderly person, and is a plan for efficiently managing their health and supporting their daily lives.

[0145] "Reminders" are notifications that remind elderly people to take their medicine or take important actions without forgetting.

[0146] MODE FOR CARRYING OUT THE INVENTION

[0147] This invention is an individual care support system for elderly people and those who need care. This system is composed of a server, terminals, and users, and realizes specific functions based on their respective roles.

[0148] System Components

[0149] 1. Terminal

[0150] The terminal is a wristwatch-type device worn by elderly people and has built-in multi-function sensors as follows:

[0151] Camera: The elderly person's face is photographed periodically to collect facial expression data.

[0152] Microphone: Records the elderly person's voice and performs speaker identification if necessary.

[0153] Accelerometer: Detects the movements of the elderly and records their walking patterns and body movements.

[0154] This data is sent from the device to the server using communication protocols such as Wi-Fi or Bluetooth.

[0155] 2. Server

[0156] The server is a central management system that receives and analyzes data sent from the terminals. Specifically, it uses the following software:

[0157] Facial expression analysis: Facial expression data is analyzed using facial recognition software such as OpenCV to estimate the emotional state and stress level of the elderly.

[0158] Voice analysis: Analyze voice data using voice processing libraries such as LibROSA to assess psychological state from voice tone and patterns.

[0159] Movement analysis: Analyze behavioral data using machine learning algorithms such as TENSORFLOW® to assess movement patterns of older adults and predict fall risk.

[0160] If an abnormality is detected from the analysis results, the server will send a notification to the caregiver, including details of the abnormality and how to respond.

[0161] 3. Users (elderly people)

[0162] Elderly people wear the device to receive appropriate care support. When it is time to take their medicine, the device will send reminder notifications. For example, the device will vibrate and the display will say, "Please take your medicine."

[0163] Specific examples

[0164] For example, when an elderly person takes medicine at a fixed time every day, the system operates as follows.

[0165] Device: Periodically takes photos of the elderly person's face and records their voice, and uses an accelerometer to detect their walking patterns.

[0166] Server: This data is received and analyzed using OpenCV, LibROSA, and TensorFlow to analyze facial expressions, voice, and movements. If an abnormality is detected, a caregiver is immediately notified.

[0167] User (elderly): The device will remind the user when it is time to take their medicine, and the user will take the medicine as instructed. This information will be sent to the server, and the intake status will be recorded.

[0168] An example of a prompt sentence is as follows:

[0169] If a senior needs to take their medication at a specific time each day, explain the steps below. Be sure to include specific actions.

[0170] 1. How the device collects data and sends it to the server (e.g., capturing facial expressions with a camera, recording audio with a microphone, detecting walking with an accelerometer, and sending this via Wi-Fi).

[0171] 2. How the server receives and analyzes the data (e.g., facial expression analysis using OpenCV, voice analysis using LibROSA, motion analysis using TensorFlow).

[0172] 3. How the server detects and notifies anomalies (e.g., threshold checks, SMS notifications to caregivers).

[0173] 4. How the server generates and sends medication reminders, and how the device displays them (e.g., checking medication schedules, formatting reminders, and displaying vibration reminders on the device).

[0174] In this way, this invention is a system that comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and quickly notify in the event of an abnormality.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Specific steps of the program's processing

[0177] Step 1: Collect data from the device

[0178] Input: Daily activities and environmental conditions of elderly people

[0179] Output: facial expression data, voice data, behavior data

[0180] 1. Specific actions:

[0181] Facial expression capture: The device's built-in camera captures the elderly person's face every minute.

[0182] Voice Recording: The built-in microphone records the elderly person's voice in real time. Recording starts automatically the moment they start speaking.

[0183] Motion detection: Accelerometers detect the movements of the elderly and record their walking patterns and body movements.

[0184] Step 2: Send data from the device to the server

[0185] Input: facial expression data, voice data, and behavioral data collected on the device

[0186] Output: Data packets sent to the server

[0187] 1. Specific actions:

[0188] Data packet generation: The collected data is converted into a certain format and a data packet is generated.

[0189] Use of communication protocols: Wi-Fi or Bluetooth is used to send collected data packets to the server. For example, if Wi-Fi is used, data is uploaded every two minutes.

[0190] Step 3: Server receives data

[0191] Input: Data packets sent from the device

[0192] Output: Buffering of received data and time-stamped data

[0193] 1. Specific actions:

[0194] Use of Data Buffer: The server uses a data buffer to temporarily store the received data.

[0195] Add timestamp: Add a timestamp to the received data and save it.

[0196] Step 4: Facial expression analysis by the server

[0197] Input: Received facial expression data

[0198] Output: Estimated emotional state and stress level

[0199] 1. Specific actions:

[0200] Facial Recognition: The server uses facial recognition software such as OpenCV to extract faces from images.

[0201] Emotion estimation: Based on the extracted facial feature points, an algorithm is run to estimate the emotional state (happiness, sadness, anger, etc.) and stress level.

[0202] Step 5: Audio analysis by the server

[0203] Input: Received audio data

[0204] Output: Estimated psychological state

[0205] 1. Specific actions:

[0206] Speech feature extraction: Extract basic characteristics of speech (e.g., pitch, tone, rhythm) using tools such as LibROSA.

[0207] Mood state estimation: Based on the extracted speech features, we use a machine learning model to estimate the mood.

[0208] Step 6: Server analysis

[0209] Input: Received behavioral data

[0210] Output: Activity level and fall risk assessment results

[0211] 1. Specific actions:

[0212] Walking pattern analysis: The server analyzes walking data using TensorFlow and other tools to detect deviations from normal walking patterns.

[0213] Risk assessment: Calculate a risk score based on the signs of falling.

[0214] Step 7: Anomaly detection and notification

[0215] Input: Results of facial expression analysis, voice analysis, and movement analysis

[0216] Output: Notification of abnormality and how to respond

[0217] 1. Specific actions:

[0218] Threshold check: Detects abnormalities when emotional state or behavior data exceeds a set threshold.

[0219] Generate notification: Generate a notification message containing the anomaly information.

[0220] Send Notifications: Send notifications to caregivers via SMS and email protocols.

[0221] Step 8: Manage your medication reminders

[0222] Input: Elderly person's health data and medication schedule

[0223] Output: Reminder notification

[0224] 1. Specific actions:

[0225] Reminder generation: The server checks the time to take the medicine and generates a reminder.

[0226] Send Reminder: Send the generated reminder to your device.

[0227] Reminder display: The device notifies the elderly of reminders by voice or vibration and displays instructions on the screen.

[0228] Step 9: Regular monitoring and database updates

[0229] Input: Continuously collected data

[0230] Output: Updated user profile and analytics history

[0231] 1. Specific actions:

[0232] Data Storage: The server adds the newly received data to the existing user profile and updates the database.

[0233] Accumulation of analysis results: Accumulate and manage past analysis results to monitor long-term health trends.

[0234] Thus, through the above steps, the present invention is a system that comprehensively supports the safety and health of the elderly.

[0235] (Application example 1)

[0236] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0237] While individual care support systems exist for the elderly and those who require care, there are not enough systems in place to monitor the safety and health of workers in factories and work environments in real time. Another issue is the lack of a system that can comprehensively analyze workers' voices, facial expressions, and behavioral data to not only detect abnormalities but also optimally allocate tasks and suggest break times. This makes it difficult to improve work efficiency and manage worker health.

[0238] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0239] In this invention, the server includes means for collecting data on the voice, facial expression, and behavior of users (workers), means for analyzing the collected data and evaluating the user's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver (manager), means for managing the timing of medication intake by the user and providing reminders, means for predicting the user's health condition and risks and proposing an appropriate care plan, means for collecting data on the voice, facial expression, and behavior of workers and monitoring their safety and health condition, means for predicting work risks according to the work environment and optimizing the work schedule, and means for proposing work allocation and break timing based on the prediction results. This makes it possible to comprehensively understand the health and safety of workers and perform appropriate labor management.

[0240] "User" refers to the person who will use the system, such as the elderly, people who need care, or workers in factories or work environments.

[0241] "Voice data" refers to information used to record a user's voice and, if necessary, analyze the tone and patterns of that voice.

[0242] "Facial expression data" is information used to capture a user's facial expressions and estimate their emotional state and stress level.

[0243] "Behavioral data" is information used to detect a user's physical movements and work patterns, and to assess their activity level and risk of falling.

[0244] The "analysis means" refers to technology that analyzes collected voice, facial expression, and behavioral data to assess the user's stress level, loneliness, activity level, etc.

[0245] "Anomaly detection means" is a technology for detecting abnormal conditions or behavior from the analysis results and issuing necessary notifications.

[0246] "Notification means" refers to the method for detecting abnormalities and creating necessary reminders to notify caregivers and administrators.

[0247] "Reminder provision means" refers to technology that notifies users of the timing of when they should take their medication, the need for rest, and so on.

[0248] A "health condition prediction means" is a technology that predicts the user's health condition and risks based on collected data and proposes future care plans.

[0249] "Work risk prediction means" is a technology for predicting work-related risks in the work environment based on the health status and work patterns of workers.

[0250] The "work schedule optimization method" is a method for analyzing workers' voices, facial expressions, behavioral data, etc., and proposing optimal work allocation and break timing.

[0251] MODE FOR CARRYING OUT THE INVENTION

[0252] This invention is a system for comprehensively supporting the safety and health of the elderly, people who need care, and workers. The invention has components that mainly consist of a server, a terminal, and a user.

[0253] System Structure and Operation

[0254] Data collection

[0255] Device: The device is worn by the user and has the function of collecting the following data:

[0256] Facial Expression Data: The built-in camera periodically photographs your face and captures your facial expressions.

[0257] Voice data: A microphone records your voice and performs speaker identification if necessary.

[0258] Behavioral data: Accelerometers detect user movements and record walking patterns and body movements.

[0259] The collected data is sent from the terminal to a server.

[0260] Data analysis

[0261] Server: The server receives the transmitted data and performs the following analysis:

[0262] Facial expression analysis: Analyzes facial expression data to estimate the user's emotional state and stress level.

[0263] Voice analysis: Analyzes voice data and evaluates the user's psychological state based on their tone and voice patterns.

[0264] Movement analysis: Analyzes behavioral data to assess the user's activity level and fall risk.

[0265] Anomaly detection and notification

[0266] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver or work manager.

[0267] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0268] Notification: If an abnormality is detected, a notification will be sent to the caregiver or work manager, containing details of the abnormality and how to respond.

[0269] Medication administration management and work schedule management

[0270] Server: Manages medication timing and work schedules based on health data of elderly people and workers. Specific operations are as follows:

[0271] Generate reminders: Generate reminders when it's time to take a dose or take a break.

[0272] Send Reminder: Send the generated reminder to your device.

[0273] Display reminders: The device will notify the user of reminders by sound or vibration.

[0274] Health Status and Risk Prediction

[0275] Server: Predicts the user's health status and risks based on collected data.

[0276] Risk prediction: Using machine learning algorithms to predict risk of falls, risk of developing dementia, and occupational risks.

[0277] Proposing care plans and work schedules: Based on the prediction results, optimal care plans and work schedules are created and proposed to caregivers and work managers.

[0278] Adding specific examples

[0279] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0280] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[0281] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[0282] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[0283] Example prompts to input to the generative AI model

[0284] "Design a system that collects data on the voices, facial expressions, and behavior of workers in a factory, and detects and notifies them of abnormalities. Please use the configuration of an elderly care support system as a reference."

[0285] In this way, the system provides comprehensive support for safety and health through a series of processes that collect and analyze the voice, facial expressions, and behavioral data of users (elderly people and workers) in real time and quickly notify users in the event of an abnormality.

[0286] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0287] Step 1:

[0288] Data collection

[0289] The device collects data on the user's voice, facial expressions, and behavior.

[0290] Specifically, the built-in camera periodically takes pictures of the user's face, the microphone records their voice, and the accelerometer detects their movements.

[0291] Input: User's face image, voice data, movement data

[0292] Output: Collected data

[0293] Step 2:

[0294] Data transmission

[0295] The terminal transmits the collected data to the server.

[0296] Specifically, the collected data is securely encrypted and uploaded to a cloud server.

[0297] Input: Collected data (face images, voice data, movement data)

[0298] Output: Data sent to the server

[0299] Step 3:

[0300] Data analysis

[0301] The server parses the received data.

[0302] Specifically, facial expression analysis, voice analysis, and movement analysis are performed to evaluate the user's emotional state, stress level, and activity level.

[0303] Input: Data received by the server (face image, voice data, movement data)

[0304] Output: Analysis results (emotional state, stress level, activity level)

[0305] Step 4:

[0306] Anomaly detection

[0307] The server detects abnormalities from the analysis results.

[0308] Specifically, it detects abnormal behavior, changes in facial color, or unusual tones of voice, and raises a flag if action is required.

[0309] Input: Analysis results

[0310] Output: Whether an anomaly was detected and its details

[0311] Step 5:

[0312] notification

[0313] If the server detects an abnormality, it sends a notification to the caregiver or work manager.

[0314] Specifically, a message containing details of the abnormality and appropriate countermeasures is created and notified in real time.

[0315] Input: Anomaly detection details

[0316] Output: Notification sent to caregiver or work manager

[0317] Step 6:

[0318] Reminder generation

[0319] The server uses health data from elderly people and workers to remind them when to take their medicine and when they need to take a break.

[0320] Specifically, reminders are generated based on the schedule and sent to the device.

[0321] Input: Health data, schedule information

[0322] Output: Generated reminders

[0323] Step 7:

[0324] Send and display reminders

[0325] The terminal notifies the user of the reminder received from the server.

[0326] Specifically, reminders are notified by voice or vibration.

[0327] Input: Generated reminder

[0328] Output: Reminder notification to user

[0329] Step 8:

[0330] Health Status and Risk Prediction

[0331] The server predicts the user's health status and risks based on the collected data.

[0332] Specifically, machine learning algorithms are used to assess the risk of falls, the risk of developing dementia, and work-related risks.

[0333] Input: Collected data (face images, voice data, movement data)

[0334] Output: Health status prediction, risk assessment results

[0335] Step 9:

[0336] Proposing care plans and work schedules

[0337] Based on the prediction results, the server creates optimal care plans and work schedules and proposes them to caregivers and work managers.

[0338] Specifically, the system automatically generates care plans and work schedules and sends out proposals.

[0339] Input: Health status prediction, risk assessment results

[0340] Output: Optimal care plan and work schedule

[0341] Through these steps, the system is able to provide comprehensive support for the safety and health of users.

[0342] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0343] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate the user's emotional state, stress level, and sense of loneliness, and detects abnormalities and notifies the caregiver. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. Furthermore, by combining an emotion engine, the system can evaluate the elderly's emotional state in detail and provide suggestions for stress relief and real-time notifications.

[0344] System program and processing flow

[0345] This system consists of the following main components: a server, a terminal, an emotion engine, and a user (elderly person). Below, we explain in natural language the functions and processing flow of each component.

[0346] Data collection

[0347] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[0348] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[0349] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[0350] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[0351] The collected data is sent from the terminal to a server.

[0352] Data analysis

[0353] Server: The server receives the transmitted data and performs the following analysis:

[0354] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[0355] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[0356] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[0357] Emotional state assessment

[0358] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[0359] Anomaly detection and real-time notifications

[0360] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[0361] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0362] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[0363] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[0364] Stress relief suggestions

[0365] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[0366] Medication administration management

[0367] Server: Manages the timing of medication use based on the elderly person's health data.

[0368] Generate reminders: Generates reminders when it's time to take your dose.

[0369] Send Reminder: Send the generated reminder to your device.

[0370] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0371] Health Status and Risk Prediction

[0372] Server: Predicts the health status and risks of elderly people based on collected data.

[0373] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0374] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0375] Specific examples

[0376] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0377] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[0378] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[0379] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[0380] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[0381] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[0382] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes: collecting and analyzing voice, facial expression, and behavioral data of the elderly, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying in the event of an abnormality.

[0383] The processing flow will be explained below.

[0384] Step 1:

[0385] Data collection

[0386] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[0387] Step 2:

[0388] Data transmission

[0389] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[0390] Step 3:

[0391] Data reception

[0392] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[0393] Step 4:

[0394] facial expression analysis

[0395] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[0396] Step 5:

[0397] Audio analysis

[0398] Server: Analyzes voice data and evaluates the psychological state of the elderly person based on their tone and voice patterns. For voice analysis, acoustic features are extracted and applied to an emotion engine that detects emotions and feelings of loneliness.

[0399] Step 6:

[0400] Motion analysis

[0401] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[0402] Step 7:

[0403] Emotional state assessment

[0404] Emotion engine: The emotion engine integrated into the server uses the collected facial and voice data to assess the detailed emotional state of the elderly. The assessment results identify the elderly's stress, loneliness, joy, sadness, anger, etc.

[0405] Step 8:

[0406] Anomaly detection

[0407] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[0408] Step 9:

[0409] Notification generation

[0410] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[0411] Step 10:

[0412] Send notifications

[0413] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[0414] Step 11:

[0415] Real-time notifications

[0416] Server: When the evaluation results of the emotion engine are updated in real time, a notification is sent to the caregiver immediately in case of an emergency.

[0417] Step 12:

[0418] Stress relief suggestions

[0419] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[0420] Step 13:

[0421] Reminder generation

[0422] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[0423] Step 14:

[0424] Send reminders

[0425] Server: Sends the generated reminders to the device.

[0426] Step 15:

[0427] Reminder display

[0428] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[0429] Step 16:

[0430] Enter intake status

[0431] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[0432] Step 17:

[0433] Recording intake status

[0434] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[0435] Step 18:

[0436] Risk prediction

[0437] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[0438] Step 19:

[0439] Care plan proposal

[0440] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0441] Step 20:

[0442] Task Management

[0443] Server: Prioritizes caregiver tasks and optimizes schedules.

[0444] Step 21:

[0445] Sending reminders and notifications

[0446] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[0447] Step 22:

[0448] Schedule adjustments

[0449] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[0450] Example 2

[0451] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] For the elderly and those who require care, accidents and changes in health status in daily life are serious problems. It is also extremely difficult for caregivers to grasp the situation in real time and respond appropriately. Given this background, there is a need for a system that can efficiently collect and analyze data on the voice, facial expressions, and behavior of the elderly, detect abnormalities, and quickly notify them. In addition, suggestions for medication management and stress relief are also important for improving the quality of life for the elderly.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0454] In this invention, the server includes means for collecting data on the voice, facial expressions, and behavior of the elderly person, means for transmitting the collected data to the server, means for the server to receive and analyze the transmitted data, means for evaluating the elderly person's emotional state using an emotion engine based on the analyzed data, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of medication intake by the elderly person and providing reminders, and means for predicting the elderly person's health condition and risks and proposing an appropriate care plan. This enables comprehensive management of the safety and health of the elderly person, and enables prompt response and appropriate care.

[0455] "Data collection tools" refer to devices and systems used to collect data on the voices, facial expressions, and behavior of older adults.

[0456] The "server" is the central system that receives and analyzes data collected from the elderly.

[0457] "Data transmission means" refers to the functions and protocols for sending collected data to the server.

[0458] "Data analysis means" refers to algorithms or models that the server uses to analyze the data it receives and extract useful information.

[0459] The "emotion engine" is an analysis system that evaluates the emotional state of elderly people based on their facial expressions and voice data.

[0460] "Anomaly detection means" refers to a system or method for detecting abnormal conditions or behaviors from analysis results.

[0461] "Notification means" refers to a device or system that sends information to caregivers or family members when an abnormality is detected.

[0462] "Reminder provision means" refers to a function or device that manages the timing of when elderly people take their medicine and notifies them at the appropriate time.

[0463] A "health status prediction tool" is an algorithm or model that predicts the health status and risks of older adults based on collected data.

[0464] A "care plan proposal means" refers to a system or method for creating an optimal care plan based on the predicted health status and proposing it to a caregiver.

[0465] This invention relates to an individualized care support system for elderly people and those in need of care. It assesses the user's emotional state, stress level, and sense of loneliness through the collection and analysis of data on the elderly's voice, facial expressions, and behavior, and notifies caregivers of any abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. The following describes the specific hardware and software configurations and their operation.

[0466] System configuration

[0467] The system of this invention is composed of the following main components: a server, a terminal (wristwatch type), an emotion engine, and a user (elderly person).

[0468] Data collection

[0469] Device: A wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[0470] Facial expression data: The built-in camera periodically photographs the elderly person's face and captures their facial expressions, for example, automatically taking a photo once every hour.

[0471] Voice data: The built-in microphone records the elderly person's voice and performs speaker identification if necessary, for example, by recording the voice of everyday conversations.

[0472] Behavioral data: Accelerometers detect the movements of seniors and record their walking patterns and physical movements, such as the number of steps taken each day and the distance traveled.

[0473] The collected data is transmitted from the terminal to a server.

[0474] Data analysis

[0475] Server: The server receives the transmitted data and performs the following analysis:

[0476] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[0477] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[0478] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[0479] Emotional state assessment

[0480] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[0481] Anomaly detection and real-time notifications

[0482] Server: If an abnormality is detected from the analysis results, the following actions will be taken.

[0483] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0484] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[0485] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[0486] Stress relief suggestions

[0487] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[0488] Medication administration management

[0489] Server: Manages the timing of medication use based on the elderly person's health data.

[0490] Generate reminders: Generates reminders when it's time to take your dose.

[0491] Send Reminder: Send the generated reminder to your device.

[0492] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0493] Health Status and Risk Prediction

[0494] Server: Predicts the health status and risks of elderly people based on collected data.

[0495] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0496] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0497] Specific examples

[0498] For example, if an elderly person needs to take medicine at a specific time each day, the following steps would be taken:

[0499] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[0500] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[0501] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[0502] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[0503] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[0504] Example prompts for generative AI models:

[0505] "Design a system that analyzes the behavioral data of elderly people and notifies them when the risk of falling increases."

[0506] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0507] Step 1: Data collection

[0508] Device: A wristwatch-type device worn by the elderly collects data using built-in sensors and cameras.

[0509] Facial expression data collection: A camera periodically photographs the elderly person's face and captures their facial expressions. The input is the image data of the elderly person's face, and the output is the storage and transmission of the image data to a server.

[0510] Voice data collection: The built-in microphone records the elderly person's voice and collects the voice data. The input is the recorded voice, and the output is the storage and transmission of the voice data to the server.

[0511] Behavioral data collection: The accelerometer detects the elderly's movements and records their walking patterns and body movements. The input is acceleration data, and the output is motion data that is stored and sent to a server.

[0512] Step 2: Send data

[0513] Terminal: Collected facial expression data, voice data, and behavioral data are encrypted and sent to the server. The input is the various data collected by the terminal, and the output is the data sent to the server.

[0514] Step 3: Receive and save data

[0515] Server: Receives and securely stores data sent from the device. The input is facial expression, voice, and behavioral data sent from the device, and the output is stored in a database.

[0516] Step 4: Data analysis

[0517] Server: Analyzes the stored data and performs the following operations:

[0518] Facial Expression Analysis: Facial expression data is analyzed using facial recognition technology, and the emotion engine estimates the emotional state and stress level. The input is the stored facial expression data, and the output is the evaluation result of the emotional state and stress level.

[0519] Voice analysis: Extracts acoustic features from voice data and evaluates psychological state from the voice data using an emotion engine. The input is stored voice data, and the output is an evaluation result of loneliness and stress level.

[0520] Behavior analysis: A machine learning model is used to analyze behavior data and detect anomalous behavior. The input is the stored behavior data, and the output is the detection result of anomalous behavior.

[0521] Step 5: Assess your emotional state

[0522] Server: Using the emotion engine, the detailed emotional state is evaluated from the facial expression data and voice data. The input is the analyzed facial expression data and voice data, and the output is the detailed emotional state evaluation result.

[0523] Step 6: Anomaly detection and notification

[0524] Server: If an abnormality is detected from the analysis results, the following process is performed:

[0525] Anomaly detection: Detects falls, abnormal facial color, changes in voice tone, etc. The input is various analysis results, and the output is abnormality detection information.

[0526] Notification generation: If an abnormality is detected, a notification message is generated for the caregiver. The input is the abnormality detection information, and the output is the generation of the notification message.

[0527] Send Notification: Send the generated notification message to caregivers and family members. The input is the notification message, and the output is the sent notification. The notification is sent via smartphone app, email, SMS, etc.

[0528] Step 7: Stress reduction suggestions

[0529] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. The input is the emotional state assessment result, and the output is suggestions for stress reduction. Suggestions include music therapy, relaxation techniques, and recommendations for social activities.

[0530] Step 8: Medication Administration

[0531] Server: Manages the timing of medication use based on the elderly person's health data.

[0532] Generate reminders: Generate reminders when it's time to take a medication. The input is a regular medication schedule, and the output is a reminder.

[0533] Send reminder: Sends the generated reminder to the device. The input is the generated reminder and the output is the send to the device.

[0534] Displaying reminders: The device notifies the elderly of reminders by voice or vibration. The input is the sent reminder, and the output is the display and notification of the reminder. The elderly inputs into the device that they have taken their medicine, and the information is sent to the server.

[0535] Step 9: Health Status and Risk Prediction

[0536] Server: Predicts the health status and risks of elderly people based on collected data.

[0537] Risk prediction: Using machine learning algorithms to predict the risk of falls or dementia. The input is the collected data, and the output is the risk prediction result.

[0538] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The input is the risk prediction results, and the output is a proposed care plan. The care plan includes suggestions for exercise, nutrition, and social activities.

[0539] (Application example 2)

[0540] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0541] With the increasing number of elderly people and those requiring care, there is a growing need for individualized care support systems to ensure their safety and security. However, current care systems lack comprehensive functionality for monitoring the emotional state and health risks of elderly people in real time and providing appropriate responses. They also lack the functionality to provide real-time notifications and suggest stress relief measures to enable caregivers to quickly grasp the elderly's situation and respond immediately. Therefore, there is a need for the development of a new care support system that comprehensively supports the safety and health of elderly people and reduces the burden on caregivers.

[0542] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0543] In this invention, the server includes means for collecting data on the elderly person's voice, facial expressions, and behavior, means for analyzing the collected data and evaluating the elderly person's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of the elderly person's medication intake and providing reminders, means for predicting the elderly person's health status and risks and proposing an appropriate care plan, means for collecting user data using sensors in a smart device (smartphone, smartwatch, etc.) and processing it in real time, and means for proposing stress relief measures for the elderly person based on emotional information obtained from the collected data and providing relaxation methods such as music. This provides comprehensive support for the safety and health of the elderly, enabling caregivers to quickly understand the elderly person's situation and respond immediately.

[0544] "Elderly" refers to people who are older and require physical or mental support.

[0545] "Nursing care" refers to providing assistance with daily living and medical services to elderly people and people with physical or mental disabilities.

[0546] A "sensor" refers to a device or equipment used to measure the physical condition and environmental information of elderly people and collect the data.

[0547] "Real-time" refers to data collection, analysis, notification, and other processing being carried out on the spot without delay.

[0548] "Stress level" refers to an indicator of the degree of mental or physical pressure felt by older adults.

[0549] "Loneliness" refers to the psychological state in which older people feel a lack of social connection with others.

[0550] "Reminders" refer to measures that inform seniors when to take their medications or other important events.

[0551] A "care plan" is a plan that specifically outlines the most appropriate care methods and support measures based on the health and living conditions of an elderly person.

[0552] "Smart device" refers to an electronic device (e.g., smartphone, smartwatch, etc.) that has internet connectivity and is capable of collecting and analyzing data.

[0553] "Music therapy" refers to a therapeutic method that uses music to relieve mental stress and promote relaxation.

[0554] "Emotional information" refers to data that indicates the emotional state of an elderly person analyzed from their facial expressions and voice.

[0555] This invention relates to an individualized care support system for the elderly and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, and analyzes this data to evaluate the elderly's emotional state, stress level, and loneliness. It detects abnormalities and notifies caregivers. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[0556] Data collection

[0557] Device: A smart device (e.g., a smartwatch) worn by an elderly person uses built-in sensors and cameras to collect the following data:

[0558] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[0559] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[0560] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[0561] The collected data is sent from the terminal to a server.

[0562] Data analysis

[0563] Server: The server receives the transmitted data and performs the following analysis:

[0564] Facial Expression Analysis: Analyze facial expression data and use an emotion engine to estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine. OpenCV and TensorFlow are used.

[0565] Speech analysis: Analyzes speech data and uses an emotion engine to assess the psychological state of elderly people based on their tone and patterns of voice. For speech analysis, acoustic features are extracted and applied to models that detect emotions and loneliness. Scikit-learn is used.

[0566] Motion analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Motion analysis uses machine learning models to analyze movement patterns and detect abnormal behavior. TensorFlow is used.

[0567] Emotional state assessment

[0568] Emotion Engine: The emotion engine integrated into the server uses collected facial and voice data to assess the detailed emotional state of the elderly, identifying not only levels of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[0569] Anomaly detection and real-time notifications

[0570] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[0571] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0572] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[0573] Notification sending: The generated notification message is sent to caregivers and family members via smartphone app, email, or SMS.

[0574] Stress relief suggestions

[0575] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. These suggestions include music therapy, relaxation techniques, and social activity recommendations. For example, if an elderly person is feeling stressed, the server plays relaxing music on their smartphone.

[0576] Medication administration management

[0577] Server: Manages the timing of medication use based on the elderly person's health data.

[0578] Generate reminders: Generates reminders when it's time to take your dose.

[0579] Send Reminder: Send the generated reminder to your device.

[0580] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0581] Health Status and Risk Prediction

[0582] Server: Predicts the health status and risks of elderly people based on collected data.

[0583] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0584] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0585] Specific examples

[0586] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0587] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[0588] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[0589] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[0590] 4. Abnormality detection and notification: If an abnormality is detected, a notification will be sent to the caregiver immediately.

[0591] 5. Medication administration management: The server manages medication intake times, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records the information. In this way, the present invention comprehensively supports the safety and health of the elderly by collecting and analyzing the elderly's voice, facial expressions, and behavioral data, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying the elderly in the event of an abnormality.

[0592] Prompt Sentence Examples

[0593] "Please analyze this image and estimate the user's emotional state."

[0594] "Analyze this audio data and assess your stress level."

[0595] "Detect abnormal behavior based on this walking data."

[0596] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0597] Step 1: Data collection

[0598] Device: A smart device (smartwatch or smartphone) collects data on the elderly person's voice, facial expressions, and behavior. The camera periodically takes pictures of the elderly person's face, the microphone records their voice, and the accelerometer records their movement data.

[0599] Input: Raw data of the elderly person's voice, facial expressions, and movements.

[0600] Output: Raw data showing the emotional state and behavior patterns of the elderly.

[0601] Step 2: Send data

[0602] Terminal: Sends collected raw data to the server. Data is transferred using a secure protocol.

[0603] Input: Raw data collected.

[0604] Output: The raw data sent to the server.

[0605] Step 3: Facial Expression Analysis

[0606] Server: Analyzes the received facial expression data and evaluates the emotional state using an emotion engine. Specifically, it extracts facial features and analyzes them using a machine learning model (e.g., TensorFlow).

[0607] Input: Facial expression data of elderly people.

[0608] Output: The emotional state of the elderly person (e.g., happy, sad, angry).

[0609] Step 4: Audio analysis

[0610] Server: Analyzes the received voice data and evaluates the elderly person's psychological state based on the tone and pattern of the voice. Acoustic features are extracted and analyzed using a machine learning model (e.g., Scikit-learn).

[0611] Input: Elderly speech data.

[0612] Output: Psychological state of the elderly (stress level, loneliness, etc.).

[0613] Step 5: Behavior analysis

[0614] Server: Analyzes the received movement data to evaluate the elderly person's activity level and abnormal behavior. Analyzes movement patterns using a machine learning model (e.g., TensorFlow) to detect abnormal behavior.

[0615] Input: Elderly person's movement data.

[0616] Output: Activity level of elderly people, abnormal behavior detection results.

[0617] Step 6: Anomaly detection and notification generation

[0618] Server: If an anomaly is detected from the analysis results, a notification message is immediately generated, containing detailed information such as the type of anomaly, date and time, and location.

[0619] Input: Results of facial expression analysis, voice analysis, and movement analysis.

[0620] Output: An error notification message.

[0621] Step 7: Send notification

[0622] Server: Generates and sends notification messages to caregivers and family members via smartphone app, email, SMS, etc.

[0623] Input: Anomaly notification message.

[0624] Output: Notification to caregivers and family members.

[0625] Step 8: Emotional state assessment and stress reduction suggestions

[0626] Server: Based on the evaluation results of the emotion engine, the server makes suggestions for stress reduction, including recommendations for music therapy and relaxation techniques.

[0627] Input: Assessment results of the emotional state of older adults.

[0628] Output: Stress relief suggestions.

[0629] Step 9: Medication Administration

[0630] Server: Manages medication timing based on the elderly person's health data. Generates reminders and sends them to the device.

[0631] Input: Health data of elderly people.

[0632] Output: Reminder notification.

[0633] Step 10: Health Status and Risk Prediction

[0634] Server: Based on collected data, the server predicts the health status and risks of elderly people. It uses machine learning algorithms to evaluate risks such as falls and dementia, and proposes appropriate care plans.

[0635] Input: Integrated data of elderly people (facial expression, voice, and movement data).

[0636] Output: Health status prediction results and recommendations for optimal care plans.

[0637] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0638] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0639] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0640] [Second embodiment]

[0641] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0642] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0643] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0644] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0645] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0646] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0647] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0648] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0649] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0650] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0651] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0652] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0653] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate stress levels and feelings of loneliness, and detects and notifies users of abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[0654] System program and processing flow

[0655] This system consists of three main components: a server, a terminal, and an elderly person (user). Below, we explain in natural language the functions and processing flow of each component.

[0656] Data collection

[0657] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[0658] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[0659] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[0660] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[0661] The collected data is sent from the terminal to a server.

[0662] Data analysis

[0663] Server: The server receives the transmitted data and performs the following analysis:

[0664] Facial expression analysis: Analyze facial expression data to estimate the emotional state and stress level of elderly people.

[0665] Voice analysis: Analyze voice data and assess the psychological state of the elderly person based on their tone and voice patterns.

[0666] Movement analysis: Analyzing behavioral data to assess elderly people's activity levels and fall risk.

[0667] Anomaly detection and notification

[0668] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[0669] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0670] Notification: If an abnormality is detected, a notification will be sent to the caregiver, containing details of the abnormality and instructions on how to respond.

[0671] Medication administration management

[0672] Server: Manages the timing of medication based on the elderly person's health data. Specific operations are as follows:

[0673] Generate reminders: Generates reminders when it's time to take your dose.

[0674] Send Reminder: Send the generated reminder to your device.

[0675] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0676] Health Status and Risk Prediction

[0677] Server: Predicts the health status and risks of elderly people based on collected data.

[0678] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0679] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver.

[0680] Specific examples

[0681] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0682] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[0683] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[0684] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[0685] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and promptly notify in the event of an abnormality.

[0686] The processing flow will be explained below.

[0687] Step 1:

[0688] Data collection

[0689] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[0690] Step 2:

[0691] Data transmission

[0692] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[0693] Step 3:

[0694] Data reception

[0695] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[0696] Step 4:

[0697] facial expression analysis

[0698] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to an emotion recognition model.

[0699] Step 5:

[0700] Audio analysis

[0701] Server: Analyzes voice data and evaluates the psychological state of elderly people based on their voice tone and patterns. For voice analysis, acoustic features are extracted and applied to models that detect emotions and loneliness.

[0702] Step 6:

[0703] Motion analysis

[0704] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[0705] Step 7:

[0706] Anomaly detection

[0707] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[0708] Step 8:

[0709] Notification generation

[0710] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[0711] Step 9:

[0712] Send notifications

[0713] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[0714] Step 10:

[0715] Reminder generation

[0716] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[0717] Step 11:

[0718] Send reminders

[0719] Server: Sends the generated reminders to the device.

[0720] Step 12:

[0721] Reminder display

[0722] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[0723] Step 13:

[0724] Enter intake status

[0725] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[0726] Step 14:

[0727] Recording intake status

[0728] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[0729] Step 15:

[0730] Risk prediction

[0731] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[0732] Step 16:

[0733] Care plan proposal

[0734] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0735] Step 17:

[0736] Task Management

[0737] Server: Prioritizes caregiver tasks and optimizes schedules.

[0738] Step 18:

[0739] Sending reminders and notifications

[0740] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[0741] Step 19:

[0742] Schedule adjustments

[0743] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[0744] Example 1

[0745] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] In order to improve the efficiency and accuracy of nursing care support for the elderly and those who require nursing care, it is necessary to monitor the health and psychological state of the elderly in real time, quickly detect and notify abnormalities, and provide appropriate nursing care plans. The purpose of this invention is to solve these problems and provide a new nursing care system that comprehensively supports the safety and health of the elderly.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0748] In this invention, the server includes a means for collecting biometric data, voice, facial expression, and behavioral data, a means for analyzing the collected data to estimate the elderly person's emotional state and stress level, and a means for detecting abnormalities based on the analysis results and notifying the caregiver. This makes it possible to accurately predict the elderly person's health condition and risks, and quickly provide an appropriate care plan.

[0749] "Biometric data" refers to data that indicates the physical condition of an elderly person, such as their heart rate, blood pressure, and body temperature.

[0750] "Voice" refers to the words, tone, and pitch of the voice spoken by the elderly, and is data used to evaluate their psychological state and emotions.

[0751] "Facial expression" refers to data showing the facial expressions of elderly people and is used to evaluate their emotional and psychological states.

[0752] "Behavior" refers to data that records the daily movements and activity patterns of elderly people, and is used to evaluate the amount of exercise and abnormal behavior.

[0753] "Stress level" is an indicator of the degree of mental and physical stress experienced by elderly people.

[0754] "Abnormal" refers to a state that is different from the normal state of an elderly person, and refers to phenomena or behaviors that may pose a risk to their health or safety.

[0755] A "care plan" is a nursing care plan formulated according to the individual needs of an elderly person, and is a plan for efficiently managing their health and supporting their daily lives.

[0756] "Reminders" are notifications that remind elderly people to take their medicine or take important actions without forgetting.

[0757] MODE FOR CARRYING OUT THE INVENTION

[0758] This invention is an individual care support system for elderly people and those who need care. This system is composed of a server, terminals, and users, and realizes specific functions based on their respective roles.

[0759] System Components

[0760] 1. Terminal

[0761] The terminal is a wristwatch-type device worn by elderly people and has built-in multi-function sensors as follows:

[0762] Camera: The elderly person's face is photographed periodically to collect facial expression data.

[0763] Microphone: Records the elderly person's voice and performs speaker identification if necessary.

[0764] Accelerometer: Detects the movements of the elderly and records their walking patterns and body movements.

[0765] This data is sent from the device to the server using communication protocols such as Wi-Fi or Bluetooth.

[0766] 2. Server

[0767] The server is a central management system that receives and analyzes data sent from the terminals. Specifically, it uses the following software:

[0768] Facial expression analysis: Facial expression data is analyzed using facial recognition software such as OpenCV to estimate the emotional state and stress level of the elderly.

[0769] Voice analysis: Analyze voice data using voice processing libraries such as LibROSA to assess psychological state from voice tone and patterns.

[0770] Movement analysis: Using machine learning algorithms such as TensorFlow, behavioral data is analyzed to evaluate the movement patterns of elderly people and predict their risk of falls.

[0771] If an abnormality is detected from the analysis results, the server will send a notification to the caregiver, including details of the abnormality and how to respond.

[0772] 3. Users (elderly people)

[0773] Elderly people wear the device to receive appropriate care support. When it is time to take their medicine, the device will send reminder notifications. For example, the device will vibrate and the display will say, "Please take your medicine."

[0774] Specific examples

[0775] For example, when an elderly person takes medicine at a fixed time every day, the system operates as follows.

[0776] Device: Periodically takes photos of the elderly person's face and records their voice, and uses an accelerometer to detect their walking patterns.

[0777] Server: This data is received and analyzed using OpenCV, LibROSA, and TensorFlow to analyze facial expressions, voice, and movements. If an abnormality is detected, a caregiver is immediately notified.

[0778] User (elderly): The device will remind the user when it is time to take their medicine, and the user will take the medicine as instructed. This information will be sent to the server, and the intake status will be recorded.

[0779] An example of a prompt sentence is as follows:

[0780] If a senior needs to take their medication at a specific time each day, explain the steps below. Be sure to include specific actions.

[0781] 1. How the device collects data and sends it to the server (e.g., capturing facial expressions with a camera, recording audio with a microphone, detecting walking with an accelerometer, and sending this via Wi-Fi).

[0782] 2. How the server receives and analyzes the data (e.g., facial expression analysis using OpenCV, voice analysis using LibROSA, motion analysis using TensorFlow).

[0783] 3. How the server detects and notifies anomalies (e.g., threshold checks, SMS notifications to caregivers).

[0784] 4. How the server generates and sends medication reminders, and how the device displays them (e.g., checking medication schedules, formatting reminders, and displaying vibration reminders on the device).

[0785] In this way, this invention is a system that comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and quickly notify in the event of an abnormality.

[0786] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0787] Specific steps of the program's processing

[0788] Step 1: Collect data from the device

[0789] Input: Daily activities and environmental conditions of elderly people

[0790] Output: facial expression data, voice data, behavior data

[0791] 1. Specific actions:

[0792] Facial expression capture: The device's built-in camera captures the elderly person's face every minute.

[0793] Voice Recording: The built-in microphone records the elderly person's voice in real time. Recording starts automatically the moment they start speaking.

[0794] Motion detection: Accelerometers detect the movements of the elderly and record their walking patterns and body movements.

[0795] Step 2: Send data from the device to the server

[0796] Input: facial expression data, voice data, and behavioral data collected on the device

[0797] Output: Data packets sent to the server

[0798] 1. Specific actions:

[0799] Data packet generation: The collected data is converted into a certain format and a data packet is generated.

[0800] Use of communication protocols: Wi-Fi or Bluetooth is used to send collected data packets to the server. For example, if Wi-Fi is used, data is uploaded every two minutes.

[0801] Step 3: Server receives data

[0802] Input: Data packets sent from the device

[0803] Output: Buffering of received data and time-stamped data

[0804] 1. Specific actions:

[0805] Use of Data Buffer: The server uses a data buffer to temporarily store the received data.

[0806] Add timestamp: Add a timestamp to the received data and save it.

[0807] Step 4: Facial expression analysis by the server

[0808] Input: Received facial expression data

[0809] Output: Estimated emotional state and stress level

[0810] 1. Specific actions:

[0811] Facial Recognition: The server uses facial recognition software such as OpenCV to extract faces from images.

[0812] Emotion estimation: Based on the extracted facial feature points, an algorithm is run to estimate the emotional state (happiness, sadness, anger, etc.) and stress level.

[0813] Step 5: Audio analysis by the server

[0814] Input: Received audio data

[0815] Output: Estimated psychological state

[0816] 1. Specific actions:

[0817] Speech feature extraction: Extract basic characteristics of speech (e.g., pitch, tone, rhythm) using tools such as LibROSA.

[0818] Mood state estimation: Based on the extracted speech features, we use a machine learning model to estimate the mood.

[0819] Step 6: Server analysis

[0820] Input: Received behavioral data

[0821] Output: Activity level and fall risk assessment results

[0822] 1. Specific actions:

[0823] Walking pattern analysis: The server analyzes walking data using TensorFlow and other tools to detect deviations from normal walking patterns.

[0824] Risk assessment: Calculate a risk score based on the signs of falling.

[0825] Step 7: Anomaly detection and notification

[0826] Input: Results of facial expression analysis, voice analysis, and movement analysis

[0827] Output: Notification of abnormality and how to respond

[0828] 1. Specific actions:

[0829] Threshold check: Detects abnormalities when emotional state or behavior data exceeds a set threshold.

[0830] Generate notification: Generate a notification message containing the anomaly information.

[0831] Send Notifications: Send notifications to caregivers via SMS and email protocols.

[0832] Step 8: Manage your medication reminders

[0833] Input: Elderly person's health data and medication schedule

[0834] Output: Reminder notification

[0835] 1. Specific actions:

[0836] Reminder generation: The server checks the time to take the medicine and generates a reminder.

[0837] Send Reminder: Send the generated reminder to your device.

[0838] Reminder display: The device notifies the elderly of reminders by voice or vibration and displays instructions on the screen.

[0839] Step 9: Regular monitoring and database updates

[0840] Input: Continuously collected data

[0841] Output: Updated user profile and analytics history

[0842] 1. Specific actions:

[0843] Data Storage: The server adds the newly received data to the existing user profile and updates the database.

[0844] Accumulation of analysis results: Accumulate and manage past analysis results to monitor long-term health trends.

[0845] Thus, through the above steps, the present invention is a system that comprehensively supports the safety and health of the elderly.

[0846] (Application example 1)

[0847] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0848] While individual care support systems exist for the elderly and those who require care, there are not enough systems in place to monitor the safety and health of workers in factories and work environments in real time. Another issue is the lack of a system that can comprehensively analyze workers' voices, facial expressions, and behavioral data to not only detect abnormalities but also optimally allocate tasks and suggest break times. This makes it difficult to improve work efficiency and manage worker health.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0850] In this invention, the server includes means for collecting data on the voice, facial expression, and behavior of users (workers), means for analyzing the collected data and evaluating the user's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver (manager), means for managing the timing of medication intake by the user and providing reminders, means for predicting the user's health condition and risks and proposing an appropriate care plan, means for collecting data on the voice, facial expression, and behavior of workers and monitoring their safety and health condition, means for predicting work risks according to the work environment and optimizing the work schedule, and means for proposing work allocation and break timing based on the prediction results. This makes it possible to comprehensively understand the health and safety of workers and perform appropriate labor management.

[0851] "User" refers to the person who will use the system, such as the elderly, people who need care, or workers in factories or work environments.

[0852] "Voice data" refers to information used to record a user's voice and, if necessary, analyze the tone and patterns of that voice.

[0853] "Facial expression data" is information used to capture a user's facial expressions and estimate their emotional state and stress level.

[0854] "Behavioral data" is information used to detect a user's physical movements and work patterns, and to assess their activity level and risk of falling.

[0855] The "analysis means" refers to technology that analyzes collected voice, facial expression, and behavioral data to assess the user's stress level, loneliness, activity level, etc.

[0856] "Anomaly detection means" is a technology for detecting abnormal conditions or behavior from the analysis results and issuing necessary notifications.

[0857] "Notification means" refers to the method for detecting abnormalities and creating necessary reminders to notify caregivers and administrators.

[0858] "Reminder provision means" refers to technology that notifies users of the timing of when they should take their medication, the need for rest, and so on.

[0859] A "health condition prediction means" is a technology that predicts the user's health condition and risks based on collected data and proposes future care plans.

[0860] "Work risk prediction means" is a technology for predicting work-related risks in the work environment based on the health status and work patterns of workers.

[0861] The "work schedule optimization method" is a method for analyzing workers' voices, facial expressions, behavioral data, etc., and proposing optimal work allocation and break timing.

[0862] MODE FOR CARRYING OUT THE INVENTION

[0863] This invention is a system for comprehensively supporting the safety and health of the elderly, people who need care, and workers. The invention has components that mainly consist of a server, a terminal, and a user.

[0864] System Structure and Operation

[0865] Data collection

[0866] Device: The device is worn by the user and has the function of collecting the following data:

[0867] Facial Expression Data: The built-in camera periodically photographs your face and captures your facial expressions.

[0868] Voice data: A microphone records your voice and performs speaker identification if necessary.

[0869] Behavioral data: Accelerometers detect user movements and record walking patterns and body movements.

[0870] The collected data is sent from the terminal to a server.

[0871] Data analysis

[0872] Server: The server receives the transmitted data and performs the following analysis:

[0873] Facial expression analysis: Analyzes facial expression data to estimate the user's emotional state and stress level.

[0874] Voice analysis: Analyzes voice data and evaluates the user's psychological state based on their tone and voice patterns.

[0875] Movement analysis: Analyzes behavioral data to assess the user's activity level and fall risk.

[0876] Anomaly detection and notification

[0877] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver or work manager.

[0878] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0879] Notification: If an abnormality is detected, a notification will be sent to the caregiver or work manager, containing details of the abnormality and how to respond.

[0880] Medication administration management and work schedule management

[0881] Server: Manages medication timing and work schedules based on health data of elderly people and workers. Specific operations are as follows:

[0882] Generate reminders: Generate reminders when it's time to take a dose or take a break.

[0883] Send Reminder: Send the generated reminder to your device.

[0884] Display reminders: The device will notify the user of reminders by sound or vibration.

[0885] Health Status and Risk Prediction

[0886] Server: Predicts the user's health status and risks based on collected data.

[0887] Risk prediction: Using machine learning algorithms to predict risk of falls, risk of developing dementia, and occupational risks.

[0888] Proposing care plans and work schedules: Based on the prediction results, optimal care plans and work schedules are created and proposed to caregivers and work managers.

[0889] Adding specific examples

[0890] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0891] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[0892] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[0893] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[0894] Example prompts to input to the generative AI model

[0895] "Design a system that collects data on the voices, facial expressions, and behavior of workers in a factory, and detects and notifies them of abnormalities. Please use this as a reference for the configuration of an elderly care support system."

[0896] In this way, the system provides comprehensive support for safety and health through a series of processes that collect and analyze the voice, facial expressions, and behavioral data of users (elderly people and workers) in real time and quickly notify users in the event of an abnormality.

[0897] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0898] Step 1:

[0899] Data collection

[0900] The device collects data on the user's voice, facial expressions, and behavior.

[0901] Specifically, the built-in camera periodically takes pictures of the user's face, the microphone records their voice, and the accelerometer detects their movements.

[0902] Input: User's face image, voice data, movement data

[0903] Output: Collected data

[0904] Step 2:

[0905] Data transmission

[0906] The terminal transmits the collected data to the server.

[0907] Specifically, the collected data is securely encrypted and uploaded to a cloud server.

[0908] Input: Collected data (face images, voice data, movement data)

[0909] Output: Data sent to the server

[0910] Step 3:

[0911] Data analysis

[0912] The server parses the received data.

[0913] Specifically, facial expression analysis, voice analysis, and movement analysis are performed to evaluate the user's emotional state, stress level, and activity level.

[0914] Input: Data received by the server (face image, voice data, movement data)

[0915] Output: Analysis results (emotional state, stress level, activity level)

[0916] Step 4:

[0917] Anomaly detection

[0918] The server detects abnormalities from the analysis results.

[0919] Specifically, it detects abnormal behavior, changes in facial color, or unusual tones of voice, and raises a flag if action is required.

[0920] Input: Analysis results

[0921] Output: Whether an anomaly was detected and its details

[0922] Step 5:

[0923] notification

[0924] If the server detects an abnormality, it sends a notification to the caregiver or work manager.

[0925] Specifically, a message containing details of the abnormality and appropriate countermeasures is created and notified in real time.

[0926] Input: Anomaly detection details

[0927] Output: Notification sent to caregiver or work manager

[0928] Step 6:

[0929] Reminder generation

[0930] The server uses health data from elderly people and workers to remind them when to take their medicine and when they need to take a break.

[0931] Specifically, reminders are generated based on the schedule and sent to the device.

[0932] Input: Health data, schedule information

[0933] Output: Generated reminders

[0934] Step 7:

[0935] Send and display reminders

[0936] The terminal notifies the user of the reminder received from the server.

[0937] Specifically, reminders are notified by voice or vibration.

[0938] Input: Generated reminder

[0939] Output: Reminder notification to user

[0940] Step 8:

[0941] Health Status and Risk Prediction

[0942] The server predicts the user's health status and risks based on the collected data.

[0943] Specifically, machine learning algorithms are used to assess the risk of falls, the risk of developing dementia, and work-related risks.

[0944] Input: Collected data (face images, voice data, movement data)

[0945] Output: Health status prediction, risk assessment results

[0946] Step 9:

[0947] Proposing care plans and work schedules

[0948] Based on the prediction results, the server creates optimal care plans and work schedules and proposes them to caregivers and work managers.

[0949] Specifically, the system automatically generates care plans and work schedules and sends out proposals.

[0950] Input: Health status prediction, risk assessment results

[0951] Output: Optimal care plan and work schedule

[0952] Through these steps, the system is able to provide comprehensive support for the safety and health of users.

[0953] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0954] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate the user's emotional state, stress level, and sense of loneliness, and detects abnormalities and notifies the caregiver. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. Furthermore, by combining an emotion engine, the system can evaluate the elderly's emotional state in detail and provide suggestions for stress relief and real-time notifications.

[0955] System program and processing flow

[0956] This system consists of the following main components: a server, a terminal, an emotion engine, and a user (elderly person). Below, we explain in natural language the functions and processing flow of each component.

[0957] Data collection

[0958] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[0959] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[0960] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[0961] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[0962] The collected data is sent from the terminal to a server.

[0963] Data analysis

[0964] Server: The server receives the transmitted data and performs the following analysis:

[0965] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[0966] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[0967] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[0968] Emotional state assessment

[0969] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[0970] Anomaly detection and real-time notifications

[0971] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[0972] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[0973] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[0974] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[0975] Stress relief suggestions

[0976] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[0977] Medication administration management

[0978] Server: Manages the timing of medication use based on the elderly person's health data.

[0979] Generate reminders: Generates reminders when it's time to take your dose.

[0980] Send Reminder: Send the generated reminder to your device.

[0981] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[0982] Health Status and Risk Prediction

[0983] Server: Predicts the health status and risks of elderly people based on collected data.

[0984] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[0985] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[0986] Specific examples

[0987] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[0988] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[0989] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[0990] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[0991] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[0992] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[0993] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes: collecting and analyzing voice, facial expression, and behavioral data of the elderly, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying in the event of an abnormality.

[0994] The processing flow will be explained below.

[0995] Step 1:

[0996] Data collection

[0997] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[0998] Step 2:

[0999] Data transmission

[1000] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[1001] Step 3:

[1002] Data reception

[1003] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[1004] Step 4:

[1005] facial expression analysis

[1006] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[1007] Step 5:

[1008] Audio analysis

[1009] Server: Analyzes voice data and evaluates the psychological state of the elderly person based on their tone and voice patterns. For voice analysis, acoustic features are extracted and applied to an emotion engine that detects emotions and feelings of loneliness.

[1010] Step 6:

[1011] Motion analysis

[1012] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[1013] Step 7:

[1014] Emotional state assessment

[1015] Emotion engine: The emotion engine integrated into the server uses the collected facial and voice data to assess the detailed emotional state of the elderly. The assessment results identify the elderly's stress, loneliness, joy, sadness, anger, etc.

[1016] Step 8:

[1017] Anomaly detection

[1018] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[1019] Step 9:

[1020] Notification generation

[1021] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[1022] Step 10:

[1023] Send notifications

[1024] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[1025] Step 11:

[1026] Real-time notifications

[1027] Server: When the evaluation results of the emotion engine are updated in real time, a notification is sent to the caregiver immediately in case of an emergency.

[1028] Step 12:

[1029] Stress relief suggestions

[1030] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[1031] Step 13:

[1032] Reminder generation

[1033] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[1034] Step 14:

[1035] Send reminders

[1036] Server: Sends the generated reminders to the device.

[1037] Step 15:

[1038] Reminder display

[1039] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[1040] Step 16:

[1041] Enter intake status

[1042] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[1043] Step 17:

[1044] Recording intake status

[1045] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[1046] Step 18:

[1047] Risk prediction

[1048] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[1049] Step 19:

[1050] Care plan proposal

[1051] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1052] Step 20:

[1053] Task Management

[1054] Server: Prioritizes caregiver tasks and optimizes schedules.

[1055] Step 21:

[1056] Sending reminders and notifications

[1057] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[1058] Step 22:

[1059] Schedule adjustments

[1060] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[1061] Example 2

[1062] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1063] For the elderly and those who require care, accidents and changes in health status in daily life are serious problems. It is also extremely difficult for caregivers to grasp the situation in real time and respond appropriately. Given this background, there is a need for a system that can efficiently collect and analyze data on the voice, facial expressions, and behavior of the elderly, detect abnormalities, and quickly notify them. In addition, suggestions for medication management and stress relief are also important for improving the quality of life for the elderly.

[1064] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1065] In this invention, the server includes means for collecting data on the voice, facial expressions, and behavior of the elderly person, means for transmitting the collected data to the server, means for the server to receive and analyze the transmitted data, means for evaluating the elderly person's emotional state using an emotion engine based on the analyzed data, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of medication intake by the elderly person and providing reminders, and means for predicting the elderly person's health condition and risks and proposing an appropriate care plan. This enables comprehensive management of the safety and health of the elderly person, and enables prompt response and appropriate care.

[1066] "Data collection tools" refer to devices and systems used to collect data on the voices, facial expressions, and behavior of older adults.

[1067] The "server" is the central system that receives and analyzes data collected from the elderly.

[1068] "Data transmission means" refers to the functions and protocols for sending collected data to the server.

[1069] "Data analysis means" refers to algorithms or models that the server uses to analyze the data it receives and extract useful information.

[1070] The "emotion engine" is an analysis system that evaluates the emotional state of elderly people based on their facial expressions and voice data.

[1071] "Anomaly detection means" refers to a system or method for detecting abnormal conditions or behaviors from analysis results.

[1072] "Notification means" refers to a device or system that sends information to caregivers or family members when an abnormality is detected.

[1073] "Reminder provision means" refers to a function or device that manages the timing of when elderly people take their medicine and notifies them at the appropriate time.

[1074] A "health status prediction tool" is an algorithm or model that predicts the health status and risks of older adults based on collected data.

[1075] A "care plan proposal means" refers to a system or method for creating an optimal care plan based on the predicted health status and proposing it to a caregiver.

[1076] This invention relates to an individualized care support system for elderly people and those in need of care. It assesses the user's emotional state, stress level, and sense of loneliness through the collection and analysis of data on the elderly's voice, facial expressions, and behavior, and notifies caregivers of any abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. The following describes the specific hardware and software configurations and their operation.

[1077] System configuration

[1078] The system of this invention is composed of the following main components: a server, a terminal (wristwatch type), an emotion engine, and a user (elderly person).

[1079] Data collection

[1080] Device: A wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[1081] Facial expression data: The built-in camera periodically photographs the elderly person's face and captures their facial expressions, for example, automatically taking a photo once every hour.

[1082] Voice data: The built-in microphone records the elderly person's voice and performs speaker identification if necessary, for example, by recording the voice of everyday conversations.

[1083] Behavioral data: Accelerometers detect the movements of seniors and record their walking patterns and physical movements, such as the number of steps taken each day and the distance traveled.

[1084] The collected data is transmitted from the terminal to a server.

[1085] Data analysis

[1086] Server: The server receives the transmitted data and performs the following analysis:

[1087] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[1088] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[1089] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[1090] Emotional state assessment

[1091] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[1092] Anomaly detection and real-time notifications

[1093] Server: If an abnormality is detected from the analysis results, the following actions will be taken.

[1094] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1095] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[1096] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[1097] Stress relief suggestions

[1098] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[1099] Medication administration management

[1100] Server: Manages the timing of medication use based on the elderly person's health data.

[1101] Generate reminders: Generates reminders when it's time to take your dose.

[1102] Send Reminder: Send the generated reminder to your device.

[1103] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1104] Health Status and Risk Prediction

[1105] Server: Predicts the health status and risks of elderly people based on collected data.

[1106] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1107] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1108] Specific examples

[1109] For example, if an elderly person needs to take medicine at a specific time each day, the following steps would be taken:

[1110] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[1111] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[1112] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[1113] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[1114] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[1115] Example prompts for generative AI models:

[1116] "Design a system that analyzes the behavioral data of elderly people and notifies them when the risk of falling increases."

[1117] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1118] Step 1: Data collection

[1119] Device: A wristwatch-type device worn by the elderly collects data using built-in sensors and cameras.

[1120] Facial expression data collection: A camera periodically photographs the elderly person's face and captures their facial expressions. The input is the image data of the elderly person's face, and the output is the storage and transmission of the image data to a server.

[1121] Voice data collection: The built-in microphone records the elderly person's voice and collects the voice data. The input is the recorded voice, and the output is the storage and transmission of the voice data to the server.

[1122] Behavioral data collection: The accelerometer detects the elderly's movements and records their walking patterns and body movements. The input is acceleration data, and the output is motion data that is stored and sent to a server.

[1123] Step 2: Send data

[1124] Terminal: Collected facial expression data, voice data, and behavioral data are encrypted and sent to the server. The input is the various data collected by the terminal, and the output is the data sent to the server.

[1125] Step 3: Receive and save data

[1126] Server: Receives and securely stores data sent from the device. The input is facial expression, voice, and behavioral data sent from the device, and the output is stored in a database.

[1127] Step 4: Data analysis

[1128] Server: Analyzes the stored data and performs the following operations:

[1129] Facial Expression Analysis: Facial expression data is analyzed using facial recognition technology, and the emotion engine estimates the emotional state and stress level. The input is the stored facial expression data, and the output is the evaluation result of the emotional state and stress level.

[1130] Voice analysis: Extracts acoustic features from voice data and evaluates psychological state from the voice data using an emotion engine. The input is stored voice data, and the output is an evaluation result of loneliness and stress level.

[1131] Behavior analysis: A machine learning model is used to analyze behavior data and detect anomalous behavior. The input is the stored behavior data, and the output is the detection result of anomalous behavior.

[1132] Step 5: Assess your emotional state

[1133] Server: Using the emotion engine, the detailed emotional state is evaluated from the facial expression data and voice data. The input is the analyzed facial expression data and voice data, and the output is the detailed emotional state evaluation result.

[1134] Step 6: Anomaly detection and notification

[1135] Server: If an abnormality is detected from the analysis results, the following process is performed:

[1136] Anomaly detection: Detects falls, abnormal facial color, changes in voice tone, etc. The input is various analysis results, and the output is abnormality detection information.

[1137] Notification generation: If an abnormality is detected, a notification message is generated for the caregiver. The input is the abnormality detection information, and the output is the generation of the notification message.

[1138] Send Notification: Send the generated notification message to caregivers and family members. The input is the notification message, and the output is the sent notification. The notification is sent via smartphone app, email, SMS, etc.

[1139] Step 7: Stress reduction suggestions

[1140] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. The input is the emotional state assessment result, and the output is suggestions for stress reduction. Suggestions include music therapy, relaxation techniques, and recommendations for social activities.

[1141] Step 8: Medication Administration

[1142] Server: Manages the timing of medication use based on the elderly person's health data.

[1143] Generate reminders: Generate reminders when it's time to take a medication. The input is a regular medication schedule, and the output is a reminder.

[1144] Send reminder: Sends the generated reminder to the device. The input is the generated reminder and the output is the send to the device.

[1145] Displaying reminders: The device notifies the elderly of reminders by voice or vibration. The input is the sent reminder, and the output is the display and notification of the reminder. The elderly inputs into the device that they have taken their medicine, and the information is sent to the server.

[1146] Step 9: Health Status and Risk Prediction

[1147] Server: Predicts the health status and risks of elderly people based on collected data.

[1148] Risk prediction: Using machine learning algorithms to predict the risk of falls or dementia. The input is the collected data, and the output is the risk prediction result.

[1149] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The input is the risk prediction results, and the output is a proposed care plan. The care plan includes suggestions for exercise, nutrition, and social activities.

[1150] (Application example 2)

[1151] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1152] With the increasing number of elderly people and those requiring care, there is a growing need for individualized care support systems to ensure their safety and security. However, current care systems lack comprehensive functionality for monitoring the emotional state and health risks of elderly people in real time and providing appropriate responses. They also lack the functionality to provide real-time notifications and suggest stress relief measures to enable caregivers to quickly grasp the elderly's situation and respond immediately. Therefore, there is a need for the development of a new care support system that comprehensively supports the safety and health of elderly people and reduces the burden on caregivers.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1154] In this invention, the server includes means for collecting data on the elderly person's voice, facial expressions, and behavior, means for analyzing the collected data and evaluating the elderly person's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of the elderly person's medication intake and providing reminders, means for predicting the elderly person's health status and risks and proposing an appropriate care plan, means for collecting user data using sensors in a smart device (smartphone, smartwatch, etc.) and processing it in real time, and means for proposing stress relief measures for the elderly person based on emotional information obtained from the collected data and providing relaxation methods such as music. This provides comprehensive support for the safety and health of the elderly, enabling caregivers to quickly understand the elderly person's situation and respond immediately.

[1155] "Elderly" refers to people who are older and require physical or mental support.

[1156] "Nursing care" refers to providing assistance with daily living and medical services to elderly people and people with physical or mental disabilities.

[1157] A "sensor" refers to a device or equipment used to measure the physical condition and environmental information of elderly people and collect the data.

[1158] "Real-time" refers to data collection, analysis, notification, and other processing being carried out on the spot without delay.

[1159] "Stress level" refers to an indicator of the degree of mental or physical pressure felt by older adults.

[1160] "Loneliness" refers to the psychological state in which older people feel a lack of social connection with others.

[1161] "Reminders" refer to measures that inform seniors when to take their medications or other important events.

[1162] A "care plan" is a plan that specifically outlines the most appropriate care methods and support measures based on the health and living conditions of an elderly person.

[1163] "Smart device" refers to an electronic device (e.g., smartphone, smartwatch, etc.) that has internet connectivity and is capable of collecting and analyzing data.

[1164] "Music therapy" refers to a therapeutic method that uses music to relieve mental stress and promote relaxation.

[1165] "Emotional information" refers to data that indicates the emotional state of an elderly person analyzed from their facial expressions and voice.

[1166] This invention relates to an individualized care support system for the elderly and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, and analyzes this data to evaluate the elderly's emotional state, stress level, and loneliness. It detects abnormalities and notifies caregivers. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[1167] Data collection

[1168] Device: A smart device (e.g., a smartwatch) worn by an elderly person uses built-in sensors and cameras to collect the following data:

[1169] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[1170] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[1171] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[1172] The collected data is sent from the terminal to a server.

[1173] Data analysis

[1174] Server: The server receives the transmitted data and performs the following analysis:

[1175] Facial Expression Analysis: Analyze facial expression data and use an emotion engine to estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine. OpenCV and TensorFlow are used.

[1176] Speech analysis: Analyzes speech data and uses an emotion engine to assess the psychological state of elderly people based on their tone and patterns of voice. For speech analysis, acoustic features are extracted and applied to models that detect emotions and loneliness. Scikit-learn is used.

[1177] Motion analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Motion analysis uses machine learning models to analyze movement patterns and detect abnormal behavior. TensorFlow is used.

[1178] Emotional state assessment

[1179] Emotion Engine: The emotion engine integrated into the server uses collected facial and voice data to assess the detailed emotional state of the elderly, identifying not only levels of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[1180] Anomaly detection and real-time notifications

[1181] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[1182] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1183] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[1184] Notification sending: The generated notification message is sent to caregivers and family members via smartphone app, email, or SMS.

[1185] Stress relief suggestions

[1186] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. These suggestions include music therapy, relaxation techniques, and social activity recommendations. For example, if an elderly person is feeling stressed, the server plays relaxing music on their smartphone.

[1187] Medication administration management

[1188] Server: Manages the timing of medication use based on the elderly person's health data.

[1189] Generate reminders: Generates reminders when it's time to take your dose.

[1190] Send Reminder: Send the generated reminder to your device.

[1191] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1192] Health Status and Risk Prediction

[1193] Server: Predicts the health status and risks of elderly people based on collected data.

[1194] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1195] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1196] Specific examples

[1197] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1198] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[1199] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[1200] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[1201] 4. Abnormality detection and notification: If an abnormality is detected, a notification will be sent to the caregiver immediately.

[1202] 5. Medication administration management: The server manages medication intake times, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records the information. In this way, the present invention comprehensively supports the safety and health of the elderly by collecting and analyzing the elderly's voice, facial expressions, and behavioral data, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying the elderly in the event of an abnormality.

[1203] Prompt Sentence Examples

[1204] "Please analyze this image and estimate the user's emotional state."

[1205] "Analyze this audio data and assess your stress level."

[1206] "Detect abnormal behavior based on this walking data."

[1207] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1208] Step 1: Data collection

[1209] Device: A smart device (smartwatch or smartphone) collects data on the elderly person's voice, facial expressions, and behavior. The camera periodically takes pictures of the elderly person's face, the microphone records their voice, and the accelerometer records their movement data.

[1210] Input: Raw data of the elderly person's voice, facial expressions, and movements.

[1211] Output: Raw data showing the emotional state and behavior patterns of the elderly.

[1212] Step 2: Send data

[1213] Terminal: Sends collected raw data to the server. Data is transferred using a secure protocol.

[1214] Input: Raw data collected.

[1215] Output: The raw data sent to the server.

[1216] Step 3: Facial Expression Analysis

[1217] Server: Analyzes the received facial expression data and evaluates the emotional state using an emotion engine. Specifically, it extracts facial features and analyzes them using a machine learning model (e.g., TensorFlow).

[1218] Input: Facial expression data of elderly people.

[1219] Output: The emotional state of the elderly person (e.g., happy, sad, angry).

[1220] Step 4: Audio analysis

[1221] Server: Analyzes the received voice data and evaluates the elderly person's psychological state based on the tone and pattern of the voice. Acoustic features are extracted and analyzed using a machine learning model (e.g., Scikit-learn).

[1222] Input: Elderly speech data.

[1223] Output: Psychological state of the elderly (stress level, loneliness, etc.).

[1224] Step 5: Behavior analysis

[1225] Server: Analyzes the received movement data to evaluate the elderly person's activity level and abnormal behavior. Analyzes movement patterns using a machine learning model (e.g., TensorFlow) to detect abnormal behavior.

[1226] Input: Elderly person's movement data.

[1227] Output: Activity level of elderly people, abnormal behavior detection results.

[1228] Step 6: Anomaly detection and notification generation

[1229] Server: If an anomaly is detected from the analysis results, a notification message is immediately generated, containing detailed information such as the type of anomaly, date and time, and location.

[1230] Input: Results of facial expression analysis, voice analysis, and movement analysis.

[1231] Output: An error notification message.

[1232] Step 7: Send notification

[1233] Server: Generates and sends notification messages to caregivers and family members via smartphone app, email, SMS, etc.

[1234] Input: Anomaly notification message.

[1235] Output: Notification to caregivers and family members.

[1236] Step 8: Emotional state assessment and stress reduction suggestions

[1237] Server: Based on the evaluation results of the emotion engine, the server makes suggestions for stress reduction, including recommendations for music therapy and relaxation techniques.

[1238] Input: Assessment results of the emotional state of older adults.

[1239] Output: Stress relief suggestions.

[1240] Step 9: Medication Administration

[1241] Server: Manages medication timing based on the elderly person's health data. Generates reminders and sends them to the device.

[1242] Input: Health data of elderly people.

[1243] Output: Reminder notification.

[1244] Step 10: Health Status and Risk Prediction

[1245] Server: Based on collected data, the server predicts the health status and risks of elderly people. It uses machine learning algorithms to evaluate risks such as falls and dementia, and proposes appropriate care plans.

[1246] Input: Integrated data of elderly people (facial expression, voice, and movement data).

[1247] Output: Health status prediction results and recommendations for optimal care plans.

[1248] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1249] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1250] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1251] [Third embodiment]

[1252] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1253] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1254] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1255] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1256] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1257] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1258] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1259] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1260] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1261] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1262] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1263] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1264] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate stress levels and feelings of loneliness, and detects and notifies users of abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[1265] System program and processing flow

[1266] This system consists of three main components: a server, a terminal, and an elderly person (user). Below, we explain in natural language the functions and processing flow of each component.

[1267] Data collection

[1268] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[1269] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[1270] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[1271] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[1272] The collected data is sent from the terminal to a server.

[1273] Data analysis

[1274] Server: The server receives the transmitted data and performs the following analysis:

[1275] Facial expression analysis: Analyze facial expression data to estimate the emotional state and stress level of elderly people.

[1276] Voice analysis: Analyze voice data and assess the psychological state of the elderly person based on their tone and voice patterns.

[1277] Movement analysis: Analyzing behavioral data to assess elderly people's activity levels and fall risk.

[1278] Anomaly detection and notification

[1279] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[1280] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1281] Notification: If an abnormality is detected, a notification will be sent to the caregiver, containing details of the abnormality and instructions on how to respond.

[1282] Medication administration management

[1283] Server: Manages the timing of medication based on the elderly person's health data. Specific operations are as follows:

[1284] Generate reminders: Generates reminders when it's time to take your dose.

[1285] Send Reminder: Send the generated reminder to your device.

[1286] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1287] Health Status and Risk Prediction

[1288] Server: Predicts the health status and risks of elderly people based on collected data.

[1289] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1290] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver.

[1291] Specific examples

[1292] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1293] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[1294] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[1295] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[1296] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and promptly notify in the event of an abnormality.

[1297] The processing flow will be explained below.

[1298] Step 1:

[1299] Data collection

[1300] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[1301] Step 2:

[1302] Data transmission

[1303] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[1304] Step 3:

[1305] Data reception

[1306] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[1307] Step 4:

[1308] facial expression analysis

[1309] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to an emotion recognition model.

[1310] Step 5:

[1311] Audio analysis

[1312] Server: Analyzes voice data and evaluates the psychological state of elderly people based on their voice tone and patterns. For voice analysis, acoustic features are extracted and applied to models that detect emotions and loneliness.

[1313] Step 6:

[1314] Motion analysis

[1315] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[1316] Step 7:

[1317] Anomaly detection

[1318] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[1319] Step 8:

[1320] Notification generation

[1321] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[1322] Step 9:

[1323] Send notifications

[1324] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[1325] Step 10:

[1326] Reminder generation

[1327] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[1328] Step 11:

[1329] Send reminders

[1330] Server: Sends the generated reminders to the device.

[1331] Step 12:

[1332] Reminder display

[1333] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[1334] Step 13:

[1335] Enter intake status

[1336] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[1337] Step 14:

[1338] Recording intake status

[1339] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[1340] Step 15:

[1341] Risk prediction

[1342] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[1343] Step 16:

[1344] Care plan proposal

[1345] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1346] Step 17:

[1347] Task Management

[1348] Server: Prioritizes caregiver tasks and optimizes schedules.

[1349] Step 18:

[1350] Sending reminders and notifications

[1351] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[1352] Step 19:

[1353] Schedule adjustments

[1354] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[1355] Example 1

[1356] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1357] In order to improve the efficiency and accuracy of nursing care support for the elderly and those who require nursing care, it is necessary to monitor the health and psychological state of the elderly in real time, quickly detect and notify abnormalities, and provide appropriate nursing care plans. The purpose of this invention is to solve these problems and provide a new nursing care system that comprehensively supports the safety and health of the elderly.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1359] In this invention, the server includes a means for collecting biometric data, voice, facial expression, and behavioral data, a means for analyzing the collected data to estimate the elderly person's emotional state and stress level, and a means for detecting abnormalities based on the analysis results and notifying the caregiver. This makes it possible to accurately predict the elderly person's health condition and risks, and quickly provide an appropriate care plan.

[1360] "Biometric data" refers to data that indicates the physical condition of an elderly person, such as their heart rate, blood pressure, and body temperature.

[1361] "Voice" refers to the words, tone, and pitch of the voice spoken by the elderly, and is data used to evaluate their psychological state and emotions.

[1362] "Facial expression" refers to data showing the facial expressions of elderly people and is used to evaluate their emotional and psychological states.

[1363] "Behavior" refers to data that records the daily movements and activity patterns of elderly people, and is used to evaluate the amount of exercise and abnormal behavior.

[1364] "Stress level" is an indicator of the degree of mental and physical stress experienced by elderly people.

[1365] "Abnormal" refers to a state that is different from the normal state of an elderly person, and refers to phenomena or behaviors that may pose a risk to their health or safety.

[1366] A "care plan" is a nursing care plan formulated according to the individual needs of an elderly person, and is a plan for efficiently managing their health and supporting their daily lives.

[1367] "Reminders" are notifications that remind elderly people to take their medicine or take important actions without forgetting.

[1368] MODE FOR CARRYING OUT THE INVENTION

[1369] This invention is an individual care support system for elderly people and those who need care. This system is composed of a server, terminals, and users, and realizes specific functions based on their respective roles.

[1370] System Components

[1371] 1. Terminal

[1372] The terminal is a wristwatch-type device worn by elderly people and has built-in multi-function sensors as follows:

[1373] Camera: The elderly person's face is photographed periodically to collect facial expression data.

[1374] Microphone: Records the elderly person's voice and performs speaker identification if necessary.

[1375] Accelerometer: Detects the movements of the elderly and records their walking patterns and body movements.

[1376] This data is sent from the device to the server using communication protocols such as Wi-Fi or Bluetooth.

[1377] 2. Server

[1378] The server is a central management system that receives and analyzes data sent from the terminals. Specifically, it uses the following software:

[1379] Facial expression analysis: Facial expression data is analyzed using facial recognition software such as OpenCV to estimate the emotional state and stress level of the elderly.

[1380] Voice analysis: Analyze voice data using voice processing libraries such as LibROSA to assess psychological state from voice tone and patterns.

[1381] Movement analysis: Using machine learning algorithms such as TensorFlow, behavioral data is analyzed to evaluate the movement patterns of elderly people and predict their risk of falls.

[1382] If an abnormality is detected from the analysis results, the server will send a notification to the caregiver, including details of the abnormality and how to respond.

[1383] 3. Users (elderly people)

[1384] Elderly people wear the device to receive appropriate care support. When it is time to take their medicine, the device will send reminder notifications. For example, the device will vibrate and the display will say, "Please take your medicine."

[1385] Specific examples

[1386] For example, when an elderly person takes medicine at a fixed time every day, the system operates as follows.

[1387] Device: Periodically takes photos of the elderly person's face and records their voice, and uses an accelerometer to detect their walking patterns.

[1388] Server: This data is received and analyzed using OpenCV, LibROSA, and TensorFlow to analyze facial expressions, voice, and movements. If an abnormality is detected, a caregiver is immediately notified.

[1389] User (elderly): The device will remind the user when it is time to take their medicine, and the user will take the medicine as instructed. This information will be sent to the server, and the intake status will be recorded.

[1390] An example of a prompt sentence is as follows:

[1391] If a senior needs to take their medication at a specific time each day, explain the steps below. Be sure to include specific actions.

[1392] 1. How the device collects data and sends it to the server (e.g., capturing facial expressions with a camera, recording audio with a microphone, detecting walking with an accelerometer, and sending this via Wi-Fi).

[1393] 2. How the server receives and analyzes the data (e.g., facial expression analysis using OpenCV, voice analysis using LibROSA, motion analysis using TensorFlow).

[1394] 3. How the server detects and notifies anomalies (e.g., threshold checks, SMS notifications to caregivers).

[1395] 4. How the server generates and sends medication reminders, and how the device displays them (e.g., checking medication schedules, formatting reminders, and displaying vibration reminders on the device).

[1396] In this way, this invention is a system that comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and quickly notify in the event of an abnormality.

[1397] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1398] Specific steps of the program's processing

[1399] Step 1: Collect data from the device

[1400] Input: Daily activities and environmental conditions of elderly people

[1401] Output: facial expression data, voice data, behavior data

[1402] 1. Specific actions:

[1403] Facial expression capture: The device's built-in camera captures the elderly person's face every minute.

[1404] Voice Recording: The built-in microphone records the elderly person's voice in real time. Recording starts automatically the moment they start speaking.

[1405] Motion detection: Accelerometers detect the movements of the elderly and record their walking patterns and body movements.

[1406] Step 2: Send data from the device to the server

[1407] Input: facial expression data, voice data, and behavioral data collected on the device

[1408] Output: Data packets sent to the server

[1409] 1. Specific actions:

[1410] Data packet generation: The collected data is converted into a certain format and a data packet is generated.

[1411] Use of communication protocols: Wi-Fi or Bluetooth is used to send collected data packets to the server. For example, if Wi-Fi is used, data is uploaded every two minutes.

[1412] Step 3: Server receives data

[1413] Input: Data packets sent from the device

[1414] Output: Buffering of received data and time-stamped data

[1415] 1. Specific actions:

[1416] Use of Data Buffer: The server uses a data buffer to temporarily store the received data.

[1417] Add timestamp: Add a timestamp to the received data and save it.

[1418] Step 4: Facial expression analysis by the server

[1419] Input: Received facial expression data

[1420] Output: Estimated emotional state and stress level

[1421] 1. Specific actions:

[1422] Facial Recognition: The server uses facial recognition software such as OpenCV to extract faces from images.

[1423] Emotion estimation: Based on the extracted facial feature points, an algorithm is run to estimate the emotional state (happiness, sadness, anger, etc.) and stress level.

[1424] Step 5: Audio analysis by the server

[1425] Input: Received audio data

[1426] Output: Estimated psychological state

[1427] 1. Specific actions:

[1428] Speech feature extraction: Extract basic characteristics of speech (e.g., pitch, tone, rhythm) using tools such as LibROSA.

[1429] Mood state estimation: Based on the extracted speech features, we use a machine learning model to estimate the mood.

[1430] Step 6: Server analysis

[1431] Input: Received behavioral data

[1432] Output: Activity level and fall risk assessment results

[1433] 1. Specific actions:

[1434] Walking pattern analysis: The server analyzes walking data using TensorFlow and other tools to detect deviations from normal walking patterns.

[1435] Risk assessment: Calculate a risk score based on the signs of falling.

[1436] Step 7: Anomaly detection and notification

[1437] Input: Results of facial expression analysis, voice analysis, and movement analysis

[1438] Output: Notification of abnormality and how to respond

[1439] 1. Specific actions:

[1440] Threshold check: Detects abnormalities when emotional state or behavior data exceeds a set threshold.

[1441] Generate notification: Generate a notification message containing the anomaly information.

[1442] Send Notifications: Send notifications to caregivers via SMS and email protocols.

[1443] Step 8: Manage your medication reminders

[1444] Input: Elderly person's health data and medication schedule

[1445] Output: Reminder notification

[1446] 1. Specific actions:

[1447] Reminder generation: The server checks the time to take the medicine and generates a reminder.

[1448] Send Reminder: Send the generated reminder to your device.

[1449] Reminder display: The device notifies the elderly of reminders by voice or vibration and displays instructions on the screen.

[1450] Step 9: Regular monitoring and database updates

[1451] Input: Continuously collected data

[1452] Output: Updated user profile and analytics history

[1453] 1. Specific actions:

[1454] Data Storage: The server adds the newly received data to the existing user profile and updates the database.

[1455] Accumulation of analysis results: Accumulate and manage past analysis results to monitor long-term health trends.

[1456] Thus, through the above steps, the present invention is a system that comprehensively supports the safety and health of the elderly.

[1457] (Application example 1)

[1458] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1459] While individual care support systems exist for the elderly and those who require care, there are not enough systems in place to monitor the safety and health of workers in factories and work environments in real time. Another issue is the lack of a system that can comprehensively analyze workers' voices, facial expressions, and behavioral data to not only detect abnormalities but also optimally allocate tasks and suggest break times. This makes it difficult to improve work efficiency and manage worker health.

[1460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1461] In this invention, the server includes means for collecting data on the voice, facial expression, and behavior of users (workers), means for analyzing the collected data and evaluating the user's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver (manager), means for managing the timing of medication intake by the user and providing reminders, means for predicting the user's health condition and risks and proposing an appropriate care plan, means for collecting data on the voice, facial expression, and behavior of workers and monitoring their safety and health condition, means for predicting work risks according to the work environment and optimizing the work schedule, and means for proposing work allocation and break timing based on the prediction results. This makes it possible to comprehensively understand the health and safety of workers and perform appropriate labor management.

[1462] "User" refers to the person who will use the system, such as the elderly, people who need care, or workers in factories or work environments.

[1463] "Voice data" refers to information used to record a user's voice and, if necessary, analyze the tone and patterns of that voice.

[1464] "Facial expression data" is information used to capture a user's facial expressions and estimate their emotional state and stress level.

[1465] "Behavioral data" is information used to detect a user's physical movements and work patterns, and to assess their activity level and risk of falling.

[1466] The "analysis means" refers to technology that analyzes collected voice, facial expression, and behavioral data to assess the user's stress level, loneliness, activity level, etc.

[1467] "Anomaly detection means" is a technology for detecting abnormal conditions or behavior from the analysis results and issuing necessary notifications.

[1468] "Notification means" refers to the method for detecting abnormalities and creating necessary reminders to notify caregivers and administrators.

[1469] "Reminder provision means" refers to technology that notifies users of the timing of when they should take their medication, the need for rest, and so on.

[1470] A "health condition prediction means" is a technology that predicts the user's health condition and risks based on collected data and proposes future care plans.

[1471] "Work risk prediction means" is a technology for predicting work-related risks in the work environment based on the health status and work patterns of workers.

[1472] The "work schedule optimization method" is a method for analyzing workers' voices, facial expressions, behavioral data, etc., and proposing optimal work allocation and break timing.

[1473] MODE FOR CARRYING OUT THE INVENTION

[1474] This invention is a system for comprehensively supporting the safety and health of the elderly, people who need care, and workers. The invention has components that mainly consist of a server, a terminal, and a user.

[1475] System Structure and Operation

[1476] Data collection

[1477] Device: The device is worn by the user and has the function of collecting the following data:

[1478] Facial Expression Data: The built-in camera periodically photographs your face and captures your facial expressions.

[1479] Voice data: A microphone records your voice and performs speaker identification if necessary.

[1480] Behavioral data: Accelerometers detect user movements and record walking patterns and body movements.

[1481] The collected data is sent from the terminal to a server.

[1482] Data analysis

[1483] Server: The server receives the transmitted data and performs the following analysis:

[1484] Facial expression analysis: Analyzes facial expression data to estimate the user's emotional state and stress level.

[1485] Voice analysis: Analyzes voice data and evaluates the user's psychological state based on their tone and voice patterns.

[1486] Movement analysis: Analyzes behavioral data to assess the user's activity level and fall risk.

[1487] Anomaly detection and notification

[1488] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver or work manager.

[1489] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1490] Notification: If an abnormality is detected, a notification will be sent to the caregiver or work manager, containing details of the abnormality and how to respond.

[1491] Medication administration management and work schedule management

[1492] Server: Manages medication timing and work schedules based on health data of elderly people and workers. Specific operations are as follows:

[1493] Generate reminders: Generate reminders when it's time to take a dose or take a break.

[1494] Send Reminder: Send the generated reminder to your device.

[1495] Display reminders: The device will notify the user of reminders by sound or vibration.

[1496] Health Status and Risk Prediction

[1497] Server: Predicts the user's health status and risks based on collected data.

[1498] Risk prediction: Using machine learning algorithms to predict risk of falls, risk of developing dementia, and occupational risks.

[1499] Proposing care plans and work schedules: Based on the prediction results, optimal care plans and work schedules are created and proposed to caregivers and work managers.

[1500] Adding specific examples

[1501] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1502] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[1503] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[1504] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[1505] Example prompts to input to the generative AI model

[1506] "Design a system that collects data on the voices, facial expressions, and behavior of workers in a factory, and detects and notifies them of abnormalities. Please use this as a reference for the configuration of an elderly care support system."

[1507] In this way, the system provides comprehensive support for safety and health through a series of processes that collect and analyze the voice, facial expressions, and behavioral data of users (elderly people and workers) in real time and quickly notify users in the event of an abnormality.

[1508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1509] Step 1:

[1510] Data collection

[1511] The device collects data on the user's voice, facial expressions, and behavior.

[1512] Specifically, the built-in camera periodically takes pictures of the user's face, the microphone records their voice, and the accelerometer detects their movements.

[1513] Input: User's face image, voice data, movement data

[1514] Output: Collected data

[1515] Step 2:

[1516] Data transmission

[1517] The terminal transmits the collected data to the server.

[1518] Specifically, the collected data is securely encrypted and uploaded to a cloud server.

[1519] Input: Collected data (face images, voice data, movement data)

[1520] Output: Data sent to the server

[1521] Step 3:

[1522] Data analysis

[1523] The server parses the received data.

[1524] Specifically, facial expression analysis, voice analysis, and movement analysis are performed to evaluate the user's emotional state, stress level, and activity level.

[1525] Input: Data received by the server (face image, voice data, movement data)

[1526] Output: Analysis results (emotional state, stress level, activity level)

[1527] Step 4:

[1528] Anomaly detection

[1529] The server detects abnormalities from the analysis results.

[1530] Specifically, it detects abnormal behavior, changes in facial color, or unusual tones of voice, and raises a flag if action is required.

[1531] Input: Analysis results

[1532] Output: Whether an anomaly was detected and its details

[1533] Step 5:

[1534] notification

[1535] If the server detects an abnormality, it sends a notification to the caregiver or work manager.

[1536] Specifically, a message containing details of the abnormality and appropriate countermeasures is created and notified in real time.

[1537] Input: Anomaly detection details

[1538] Output: Notification sent to caregiver or work manager

[1539] Step 6:

[1540] Reminder generation

[1541] The server uses health data from elderly people and workers to remind them when to take their medicine and when they need to take a break.

[1542] Specifically, reminders are generated based on the schedule and sent to the device.

[1543] Input: Health data, schedule information

[1544] Output: Generated reminders

[1545] Step 7:

[1546] Send and display reminders

[1547] The terminal notifies the user of the reminder received from the server.

[1548] Specifically, reminders are notified by voice or vibration.

[1549] Input: Generated reminder

[1550] Output: Reminder notification to user

[1551] Step 8:

[1552] Health Status and Risk Prediction

[1553] The server predicts the user's health status and risks based on the collected data.

[1554] Specifically, machine learning algorithms are used to assess the risk of falls, the risk of developing dementia, and work-related risks.

[1555] Input: Collected data (face images, voice data, movement data)

[1556] Output: Health status prediction, risk assessment results

[1557] Step 9:

[1558] Proposing care plans and work schedules

[1559] Based on the prediction results, the server creates optimal care plans and work schedules and proposes them to caregivers and work managers.

[1560] Specifically, the system automatically generates care plans and work schedules and sends out proposals.

[1561] Input: Health status prediction, risk assessment results

[1562] Output: Optimal care plan and work schedule

[1563] Through these steps, the system is able to provide comprehensive support for the safety and health of users.

[1564] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1565] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate the user's emotional state, stress level, and sense of loneliness, and detects abnormalities and notifies the caregiver. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. Furthermore, by combining an emotion engine, the system can evaluate the elderly's emotional state in detail and provide suggestions for stress relief and real-time notifications.

[1566] System program and processing flow

[1567] This system consists of the following main components: a server, a terminal, an emotion engine, and a user (elderly person). Below, we explain in natural language the functions and processing flow of each component.

[1568] Data collection

[1569] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[1570] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[1571] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[1572] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[1573] The collected data is sent from the terminal to a server.

[1574] Data analysis

[1575] Server: The server receives the transmitted data and performs the following analysis:

[1576] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[1577] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[1578] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[1579] Emotional state assessment

[1580] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[1581] Anomaly detection and real-time notifications

[1582] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[1583] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1584] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[1585] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[1586] Stress relief suggestions

[1587] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[1588] Medication administration management

[1589] Server: Manages the timing of medication use based on the elderly person's health data.

[1590] Generate reminders: Generates reminders when it's time to take your dose.

[1591] Send Reminder: Send the generated reminder to your device.

[1592] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1593] Health Status and Risk Prediction

[1594] Server: Predicts the health status and risks of elderly people based on collected data.

[1595] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1596] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1597] Specific examples

[1598] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1599] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[1600] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[1601] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[1602] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[1603] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[1604] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes: collecting and analyzing voice, facial expression, and behavioral data of the elderly, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying in the event of an abnormality.

[1605] The processing flow will be explained below.

[1606] Step 1:

[1607] Data collection

[1608] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[1609] Step 2:

[1610] Data transmission

[1611] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[1612] Step 3:

[1613] Data reception

[1614] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[1615] Step 4:

[1616] facial expression analysis

[1617] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[1618] Step 5:

[1619] Audio analysis

[1620] Server: Analyzes voice data and evaluates the psychological state of the elderly person based on their tone and voice patterns. For voice analysis, acoustic features are extracted and applied to an emotion engine that detects emotions and feelings of loneliness.

[1621] Step 6:

[1622] Motion analysis

[1623] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[1624] Step 7:

[1625] Emotional state assessment

[1626] Emotion engine: The emotion engine integrated into the server uses the collected facial and voice data to assess the detailed emotional state of the elderly. The assessment results identify the elderly's stress, loneliness, joy, sadness, anger, etc.

[1627] Step 8:

[1628] Anomaly detection

[1629] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[1630] Step 9:

[1631] Notification generation

[1632] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[1633] Step 10:

[1634] Send notifications

[1635] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[1636] Step 11:

[1637] Real-time notifications

[1638] Server: When the evaluation results of the emotion engine are updated in real time, a notification is sent to the caregiver immediately in case of an emergency.

[1639] Step 12:

[1640] Stress relief suggestions

[1641] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[1642] Step 13:

[1643] Reminder generation

[1644] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[1645] Step 14:

[1646] Send reminders

[1647] Server: Sends the generated reminders to the device.

[1648] Step 15:

[1649] Reminder display

[1650] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[1651] Step 16:

[1652] Enter intake status

[1653] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[1654] Step 17:

[1655] Recording intake status

[1656] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[1657] Step 18:

[1658] Risk prediction

[1659] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[1660] Step 19:

[1661] Care plan proposal

[1662] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1663] Step 20:

[1664] Task Management

[1665] Server: Prioritizes caregiver tasks and optimizes schedules.

[1666] Step 21:

[1667] Sending reminders and notifications

[1668] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[1669] Step 22:

[1670] Schedule adjustments

[1671] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[1672] Example 2

[1673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1674] For the elderly and those who require care, accidents and changes in health status in daily life are serious problems. It is also extremely difficult for caregivers to grasp the situation in real time and respond appropriately. Given this background, there is a need for a system that can efficiently collect and analyze data on the voice, facial expressions, and behavior of the elderly, detect abnormalities, and quickly notify them. In addition, suggestions for medication management and stress relief are also important for improving the quality of life for the elderly.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1676] In this invention, the server includes means for collecting data on the voice, facial expressions, and behavior of the elderly person, means for transmitting the collected data to the server, means for the server to receive and analyze the transmitted data, means for evaluating the elderly person's emotional state using an emotion engine based on the analyzed data, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of medication intake by the elderly person and providing reminders, and means for predicting the elderly person's health condition and risks and proposing an appropriate care plan. This enables comprehensive management of the safety and health of the elderly person, and enables prompt response and appropriate care.

[1677] "Data collection tools" refer to devices and systems used to collect data on the voices, facial expressions, and behavior of older adults.

[1678] The "server" is the central system that receives and analyzes data collected from the elderly.

[1679] "Data transmission means" refers to the functions and protocols for sending collected data to the server.

[1680] "Data analysis means" refers to algorithms or models that the server uses to analyze the data it receives and extract useful information.

[1681] The "emotion engine" is an analysis system that evaluates the emotional state of elderly people based on their facial expressions and voice data.

[1682] "Anomaly detection means" refers to a system or method for detecting abnormal conditions or behaviors from analysis results.

[1683] "Notification means" refers to a device or system that sends information to caregivers or family members when an abnormality is detected.

[1684] "Reminder provision means" refers to a function or device that manages the timing of when elderly people take their medicine and notifies them at the appropriate time.

[1685] A "health status prediction tool" is an algorithm or model that predicts the health status and risks of older adults based on collected data.

[1686] A "care plan proposal means" refers to a system or method for creating an optimal care plan based on the predicted health status and proposing it to a caregiver.

[1687] This invention relates to an individualized care support system for elderly people and those in need of care. It assesses the user's emotional state, stress level, and sense of loneliness through the collection and analysis of data on the elderly's voice, facial expressions, and behavior, and notifies caregivers of any abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. The following describes the specific hardware and software configurations and their operation.

[1688] System configuration

[1689] The system of this invention is composed of the following main components: a server, a terminal (wristwatch type), an emotion engine, and a user (elderly person).

[1690] Data collection

[1691] Device: A wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[1692] Facial expression data: The built-in camera periodically photographs the elderly person's face and captures their facial expressions, for example, automatically taking a photo once every hour.

[1693] Voice data: The built-in microphone records the elderly person's voice and performs speaker identification if necessary, for example, by recording the voice of everyday conversations.

[1694] Behavioral data: Accelerometers detect the movements of seniors and record their walking patterns and physical movements, such as the number of steps taken each day and the distance traveled.

[1695] The collected data is transmitted from the terminal to a server.

[1696] Data analysis

[1697] Server: The server receives the transmitted data and performs the following analysis:

[1698] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[1699] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[1700] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[1701] Emotional state assessment

[1702] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[1703] Anomaly detection and real-time notifications

[1704] Server: If an abnormality is detected from the analysis results, the following actions will be taken.

[1705] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1706] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[1707] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[1708] Stress relief suggestions

[1709] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[1710] Medication administration management

[1711] Server: Manages the timing of medication use based on the elderly person's health data.

[1712] Generate reminders: Generates reminders when it's time to take your dose.

[1713] Send Reminder: Send the generated reminder to your device.

[1714] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1715] Health Status and Risk Prediction

[1716] Server: Predicts the health status and risks of elderly people based on collected data.

[1717] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1718] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1719] Specific examples

[1720] For example, if an elderly person needs to take medicine at a specific time each day, the following steps would be taken:

[1721] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[1722] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[1723] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[1724] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[1725] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[1726] Example prompts for generative AI models:

[1727] "Design a system that analyzes the behavioral data of elderly people and notifies them when the risk of falling increases."

[1728] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1729] Step 1: Data collection

[1730] Device: A wristwatch-type device worn by the elderly collects data using built-in sensors and cameras.

[1731] Facial expression data collection: A camera periodically photographs the elderly person's face and captures their facial expressions. The input is the image data of the elderly person's face, and the output is the storage and transmission of the image data to a server.

[1732] Voice data collection: The built-in microphone records the elderly person's voice and collects the voice data. The input is the recorded voice, and the output is the storage and transmission of the voice data to the server.

[1733] Behavioral data collection: The accelerometer detects the elderly's movements and records their walking patterns and body movements. The input is acceleration data, and the output is motion data that is stored and sent to a server.

[1734] Step 2: Send data

[1735] Terminal: Collected facial expression data, voice data, and behavioral data are encrypted and sent to the server. The input is the various data collected by the terminal, and the output is the data sent to the server.

[1736] Step 3: Receive and save data

[1737] Server: Receives and securely stores data sent from the device. The input is facial expression, voice, and behavioral data sent from the device, and the output is stored in a database.

[1738] Step 4: Data analysis

[1739] Server: Analyzes the stored data and performs the following operations:

[1740] Facial Expression Analysis: Facial expression data is analyzed using facial recognition technology, and the emotion engine estimates the emotional state and stress level. The input is the stored facial expression data, and the output is the evaluation result of the emotional state and stress level.

[1741] Voice analysis: Extracts acoustic features from voice data and evaluates psychological state from the voice data using an emotion engine. The input is stored voice data, and the output is an evaluation result of loneliness and stress level.

[1742] Behavior analysis: A machine learning model is used to analyze behavior data and detect anomalous behavior. The input is the stored behavior data, and the output is the detection result of anomalous behavior.

[1743] Step 5: Assess your emotional state

[1744] Server: Using the emotion engine, the detailed emotional state is evaluated from the facial expression data and voice data. The input is the analyzed facial expression data and voice data, and the output is the detailed emotional state evaluation result.

[1745] Step 6: Anomaly detection and notification

[1746] Server: If an abnormality is detected from the analysis results, the following process is performed:

[1747] Anomaly detection: Detects falls, abnormal facial color, changes in voice tone, etc. The input is various analysis results, and the output is abnormality detection information.

[1748] Notification generation: If an abnormality is detected, a notification message is generated for the caregiver. The input is the abnormality detection information, and the output is the generation of the notification message.

[1749] Send Notification: Send the generated notification message to caregivers and family members. The input is the notification message, and the output is the sent notification. The notification is sent via smartphone app, email, SMS, etc.

[1750] Step 7: Stress reduction suggestions

[1751] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. The input is the emotional state assessment result, and the output is suggestions for stress reduction. Suggestions include music therapy, relaxation techniques, and recommendations for social activities.

[1752] Step 8: Medication Administration

[1753] Server: Manages the timing of medication use based on the elderly person's health data.

[1754] Generate reminders: Generate reminders when it's time to take a medication. The input is a regular medication schedule, and the output is a reminder.

[1755] Send reminder: Sends the generated reminder to the device. The input is the generated reminder and the output is the send to the device.

[1756] Displaying reminders: The device notifies the elderly of reminders by voice or vibration. The input is the sent reminder, and the output is the display and notification of the reminder. The elderly inputs into the device that they have taken their medicine, and the information is sent to the server.

[1757] Step 9: Health Status and Risk Prediction

[1758] Server: Predicts the health status and risks of elderly people based on collected data.

[1759] Risk prediction: Using machine learning algorithms to predict the risk of falls or dementia. The input is the collected data, and the output is the risk prediction result.

[1760] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The input is the risk prediction results, and the output is a proposed care plan. The care plan includes suggestions for exercise, nutrition, and social activities.

[1761] (Application example 2)

[1762] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1763] With the increasing number of elderly people and those requiring care, there is a growing need for individualized care support systems to ensure their safety and security. However, current care systems lack comprehensive functionality for monitoring the emotional state and health risks of elderly people in real time and providing appropriate responses. They also lack the functionality to provide real-time notifications and suggest stress relief measures to enable caregivers to quickly grasp the elderly's situation and respond immediately. Therefore, there is a need for the development of a new care support system that comprehensively supports the safety and health of elderly people and reduces the burden on caregivers.

[1764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1765] In this invention, the server includes means for collecting data on the elderly person's voice, facial expressions, and behavior, means for analyzing the collected data and evaluating the elderly person's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of the elderly person's medication intake and providing reminders, means for predicting the elderly person's health status and risks and proposing an appropriate care plan, means for collecting user data using sensors in a smart device (smartphone, smartwatch, etc.) and processing it in real time, and means for proposing stress relief measures for the elderly person based on emotional information obtained from the collected data and providing relaxation methods such as music. This provides comprehensive support for the safety and health of the elderly, enabling caregivers to quickly understand the elderly person's situation and respond immediately.

[1766] "Elderly" refers to people who are older and require physical or mental support.

[1767] "Nursing care" refers to providing assistance with daily living and medical services to elderly people and people with physical or mental disabilities.

[1768] A "sensor" refers to a device or equipment used to measure the physical condition and environmental information of elderly people and collect the data.

[1769] "Real-time" refers to data collection, analysis, notification, and other processing being carried out on the spot without delay.

[1770] "Stress level" refers to an indicator of the degree of mental or physical pressure felt by older adults.

[1771] "Loneliness" refers to the psychological state in which older people feel a lack of social connection with others.

[1772] "Reminders" refer to measures that inform seniors when to take their medications or other important events.

[1773] A "care plan" is a plan that specifically outlines the most appropriate care methods and support measures based on the health and living conditions of an elderly person.

[1774] "Smart device" refers to an electronic device (e.g., smartphone, smartwatch, etc.) that has internet connectivity and is capable of collecting and analyzing data.

[1775] "Music therapy" refers to a therapeutic method that uses music to relieve mental stress and promote relaxation.

[1776] "Emotional information" refers to data that indicates the emotional state of an elderly person analyzed from their facial expressions and voice.

[1777] This invention relates to an individualized care support system for the elderly and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, and analyzes this data to evaluate the elderly's emotional state, stress level, and loneliness. It detects abnormalities and notifies caregivers. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[1778] Data collection

[1779] Device: A smart device (e.g., a smartwatch) worn by an elderly person uses built-in sensors and cameras to collect the following data:

[1780] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[1781] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[1782] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[1783] The collected data is sent from the terminal to a server.

[1784] Data analysis

[1785] Server: The server receives the transmitted data and performs the following analysis:

[1786] Facial Expression Analysis: Analyze facial expression data and use an emotion engine to estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine. OpenCV and TensorFlow are used.

[1787] Speech analysis: Analyzes speech data and uses an emotion engine to assess the psychological state of elderly people based on their tone and patterns of voice. For speech analysis, acoustic features are extracted and applied to models that detect emotions and loneliness. Scikit-learn is used.

[1788] Motion analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Motion analysis uses machine learning models to analyze movement patterns and detect abnormal behavior. TensorFlow is used.

[1789] Emotional state assessment

[1790] Emotion Engine: The emotion engine integrated into the server uses collected facial and voice data to assess the detailed emotional state of the elderly, identifying not only levels of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[1791] Anomaly detection and real-time notifications

[1792] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[1793] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1794] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[1795] Notification sending: The generated notification message is sent to caregivers and family members via smartphone app, email, or SMS.

[1796] Stress relief suggestions

[1797] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction. These suggestions include music therapy, relaxation techniques, and social activity recommendations. For example, if an elderly person is feeling stressed, the server plays relaxing music on their smartphone.

[1798] Medication administration management

[1799] Server: Manages the timing of medication use based on the elderly person's health data.

[1800] Generate reminders: Generates reminders when it's time to take your dose.

[1801] Send Reminder: Send the generated reminder to your device.

[1802] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1803] Health Status and Risk Prediction

[1804] Server: Predicts the health status and risks of elderly people based on collected data.

[1805] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1806] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1807] Specific examples

[1808] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1809] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[1810] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[1811] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[1812] 4. Abnormality detection and notification: If an abnormality is detected, a notification will be sent to the caregiver immediately.

[1813] 5. Medication administration management: The server manages medication intake times, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records the information. In this way, the present invention comprehensively supports the safety and health of the elderly by collecting and analyzing the elderly's voice, facial expressions, and behavioral data, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying the elderly in the event of an abnormality.

[1814] Prompt Sentence Examples

[1815] "Please analyze this image and estimate the user's emotional state."

[1816] "Analyze this audio data and assess your stress level."

[1817] "Detect abnormal behavior based on this walking data."

[1818] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1819] Step 1: Data collection

[1820] Device: A smart device (smartwatch or smartphone) collects data on the elderly person's voice, facial expressions, and behavior. The camera periodically takes pictures of the elderly person's face, the microphone records their voice, and the accelerometer records their movement data.

[1821] Input: Raw data of the elderly person's voice, facial expressions, and movements.

[1822] Output: Raw data showing the emotional state and behavior patterns of the elderly.

[1823] Step 2: Send data

[1824] Terminal: Sends collected raw data to the server. Data is transferred using a secure protocol.

[1825] Input: Raw data collected.

[1826] Output: The raw data sent to the server.

[1827] Step 3: Facial Expression Analysis

[1828] Server: Analyzes the received facial expression data and evaluates the emotional state using an emotion engine. Specifically, it extracts facial features and analyzes them using a machine learning model (e.g., TensorFlow).

[1829] Input: Facial expression data of elderly people.

[1830] Output: The emotional state of the elderly person (e.g., happy, sad, angry).

[1831] Step 4: Audio analysis

[1832] Server: Analyzes the received voice data and evaluates the elderly person's psychological state based on the tone and pattern of the voice. Acoustic features are extracted and analyzed using a machine learning model (e.g., Scikit-learn).

[1833] Input: Elderly speech data.

[1834] Output: Psychological state of the elderly (stress level, loneliness, etc.).

[1835] Step 5: Behavior analysis

[1836] Server: Analyzes the received movement data to evaluate the elderly person's activity level and abnormal behavior. Analyzes movement patterns using a machine learning model (e.g., TensorFlow) to detect abnormal behavior.

[1837] Input: Elderly person's movement data.

[1838] Output: Activity level of elderly people, abnormal behavior detection results.

[1839] Step 6: Anomaly detection and notification generation

[1840] Server: If an anomaly is detected from the analysis results, a notification message is immediately generated, containing detailed information such as the type of anomaly, date and time, and location.

[1841] Input: Results of facial expression analysis, voice analysis, and movement analysis.

[1842] Output: An error notification message.

[1843] Step 7: Send notification

[1844] Server: Generates and sends notification messages to caregivers and family members via smartphone app, email, SMS, etc.

[1845] Input: Anomaly notification message.

[1846] Output: Notification to caregivers and family members.

[1847] Step 8: Emotional state assessment and stress reduction suggestions

[1848] Server: Based on the evaluation results of the emotion engine, the server makes suggestions for stress reduction, including recommendations for music therapy and relaxation techniques.

[1849] Input: Assessment results of the emotional state of older adults.

[1850] Output: Stress relief suggestions.

[1851] Step 9: Medication Administration

[1852] Server: Manages medication timing based on the elderly person's health data. Generates reminders and sends them to the device.

[1853] Input: Health data of elderly people.

[1854] Output: Reminder notification.

[1855] Step 10: Health Status and Risk Prediction

[1856] Server: Based on collected data, the server predicts the health status and risks of elderly people. It uses machine learning algorithms to evaluate risks such as falls and dementia, and proposes appropriate care plans.

[1857] Input: Integrated data of elderly people (facial expression, voice, and movement data).

[1858] Output: Health status prediction results and recommendations for optimal care plans.

[1859] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1860] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1861] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1862] [Fourth embodiment]

[1863] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1864] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1865] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1866] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1867] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1868] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1869] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1870] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1871] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1872] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1873] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1874] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1875] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1876] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate stress levels and feelings of loneliness, and detects and notifies users of abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration.

[1877] System program and processing flow

[1878] This system consists of three main components: a server, a terminal, and an elderly person (user). Below, we explain in natural language the functions and processing flow of each component.

[1879] Data collection

[1880] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[1881] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[1882] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[1883] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[1884] The collected data is sent from the terminal to a server.

[1885] Data analysis

[1886] Server: The server receives the transmitted data and performs the following analysis:

[1887] Facial expression analysis: Analyze facial expression data to estimate the emotional state and stress level of elderly people.

[1888] Voice analysis: Analyze voice data and assess the psychological state of the elderly person based on their tone and voice patterns.

[1889] Movement analysis: Analyzing behavioral data to assess elderly people's activity levels and fall risk.

[1890] Anomaly detection and notification

[1891] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[1892] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[1893] Notification: If an abnormality is detected, a notification will be sent to the caregiver, containing details of the abnormality and instructions on how to respond.

[1894] Medication administration management

[1895] Server: Manages the timing of medication based on the elderly person's health data. Specific operations are as follows:

[1896] Generate reminders: Generates reminders when it's time to take your dose.

[1897] Send Reminder: Send the generated reminder to your device.

[1898] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[1899] Health Status and Risk Prediction

[1900] Server: Predicts the health status and risks of elderly people based on collected data.

[1901] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[1902] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver.

[1903] Specific examples

[1904] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[1905] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[1906] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[1907] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[1908] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and promptly notify in the event of an abnormality.

[1909] The processing flow will be explained below.

[1910] Step 1:

[1911] Data collection

[1912] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[1913] Step 2:

[1914] Data transmission

[1915] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[1916] Step 3:

[1917] Data reception

[1918] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[1919] Step 4:

[1920] facial expression analysis

[1921] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to an emotion recognition model.

[1922] Step 5:

[1923] Audio analysis

[1924] Server: Analyzes voice data and evaluates the psychological state of elderly people based on their voice tone and patterns. For voice analysis, acoustic features are extracted and applied to models that detect emotions and loneliness.

[1925] Step 6:

[1926] Motion analysis

[1927] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[1928] Step 7:

[1929] Anomaly detection

[1930] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[1931] Step 8:

[1932] Notification generation

[1933] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[1934] Step 9:

[1935] Send notifications

[1936] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[1937] Step 10:

[1938] Reminder generation

[1939] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[1940] Step 11:

[1941] Send reminders

[1942] Server: Sends the generated reminders to the device.

[1943] Step 12:

[1944] Reminder display

[1945] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[1946] Step 13:

[1947] Enter intake status

[1948] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[1949] Step 14:

[1950] Recording intake status

[1951] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[1952] Step 15:

[1953] Risk prediction

[1954] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[1955] Step 16:

[1956] Care plan proposal

[1957] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[1958] Step 17:

[1959] Task Management

[1960] Server: Prioritizes caregiver tasks and optimizes schedules.

[1961] Step 18:

[1962] Sending reminders and notifications

[1963] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[1964] Step 19:

[1965] Schedule adjustments

[1966] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[1967] Example 1

[1968] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1969] In order to improve the efficiency and accuracy of nursing care support for the elderly and those who require nursing care, it is necessary to monitor the health and psychological state of the elderly in real time, quickly detect and notify abnormalities, and provide appropriate nursing care plans. The purpose of this invention is to solve these problems and provide a new nursing care system that comprehensively supports the safety and health of the elderly.

[1970] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1971] In this invention, the server includes a means for collecting biometric data, voice, facial expression, and behavioral data, a means for analyzing the collected data to estimate the elderly person's emotional state and stress level, and a means for detecting abnormalities based on the analysis results and notifying the caregiver. This makes it possible to accurately predict the elderly person's health condition and risks, and quickly provide an appropriate care plan.

[1972] "Biometric data" refers to data that indicates the physical condition of an elderly person, such as their heart rate, blood pressure, and body temperature.

[1973] "Voice" refers to the words, tone, and pitch of the voice spoken by the elderly, and is data used to evaluate their psychological state and emotions.

[1974] "Facial expression" refers to data showing the facial expressions of elderly people and is used to evaluate their emotional and psychological states.

[1975] "Behavior" refers to data that records the daily movements and activity patterns of elderly people, and is used to evaluate the amount of exercise and abnormal behavior.

[1976] "Stress level" is an indicator of the degree of mental and physical stress experienced by elderly people.

[1977] "Abnormal" refers to a state that is different from the normal state of an elderly person, and refers to phenomena or behaviors that may pose a risk to their health or safety.

[1978] A "care plan" is a nursing care plan formulated according to the individual needs of an elderly person, and is a plan for efficiently managing their health and supporting their daily lives.

[1979] "Reminders" are notifications that remind elderly people to take their medicine or take important actions without forgetting.

[1980] MODE FOR CARRYING OUT THE INVENTION

[1981] This invention is an individual care support system for elderly people and those who need care. This system is composed of a server, terminals, and users, and realizes specific functions based on their respective roles.

[1982] System Components

[1983] 1. Terminal

[1984] The terminal is a wristwatch-type device worn by elderly people and has built-in multi-function sensors as follows:

[1985] Camera: The elderly person's face is photographed periodically to collect facial expression data.

[1986] Microphone: Records the elderly person's voice and performs speaker identification if necessary.

[1987] Accelerometer: Detects the movements of the elderly and records their walking patterns and body movements.

[1988] This data is sent from the device to the server using communication protocols such as Wi-Fi or Bluetooth.

[1989] 2. Server

[1990] The server is a central management system that receives and analyzes data sent from the terminals. Specifically, it uses the following software:

[1991] Facial expression analysis: Facial expression data is analyzed using facial recognition software such as OpenCV to estimate the emotional state and stress level of the elderly.

[1992] Voice analysis: Analyze voice data using voice processing libraries such as LibROSA to assess psychological state from voice tone and patterns.

[1993] Movement analysis: Using machine learning algorithms such as TensorFlow, behavioral data is analyzed to evaluate the movement patterns of elderly people and predict their risk of falls.

[1994] If an abnormality is detected from the analysis results, the server will send a notification to the caregiver, including details of the abnormality and how to respond.

[1995] 3. Users (elderly people)

[1996] Elderly people wear the device to receive appropriate care support. When it is time to take their medicine, the device will send reminder notifications. For example, the device will vibrate and the display will say, "Please take your medicine."

[1997] Specific examples

[1998] For example, when an elderly person takes medicine at a fixed time every day, the system operates as follows.

[1999] Device: Periodically takes photos of the elderly person's face and records their voice, and uses an accelerometer to detect their walking patterns.

[2000] Server: This data is received and analyzed using OpenCV, LibROSA, and TensorFlow to analyze facial expressions, voice, and movements. If an abnormality is detected, a caregiver is immediately notified.

[2001] User (elderly): The device will remind the user when it is time to take their medicine, and the user will take the medicine as instructed. This information will be sent to the server, and the intake status will be recorded.

[2002] An example of a prompt sentence is as follows:

[2003] If a senior needs to take their medication at a specific time each day, explain the steps below. Be sure to include specific actions.

[2004] 1. How the device collects data and sends it to the server (e.g., capturing facial expressions with a camera, recording audio with a microphone, detecting walking with an accelerometer, and sending this via Wi-Fi).

[2005] 2. How the server receives and analyzes the data (e.g., facial expression analysis using OpenCV, voice analysis using LibROSA, motion analysis using TensorFlow).

[2006] 3. How the server detects and notifies anomalies (e.g., threshold checks, SMS notifications to caregivers).

[2007] 4. How the server generates and sends medication reminders, and how the device displays them (e.g., checking medication schedules, formatting reminders, and displaying vibration reminders on the device).

[2008] In this way, this invention is a system that comprehensively supports the safety and health of the elderly through a series of processes that collect and analyze voice, facial expression, and behavioral data of the elderly in real time and quickly notify in the event of an abnormality.

[2009] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2010] Specific steps of the program's processing

[2011] Step 1: Collect data from the device

[2012] Input: Daily activities and environmental conditions of elderly people

[2013] Output: facial expression data, voice data, behavior data

[2014] 1. Specific actions:

[2015] Facial expression capture: The device's built-in camera captures the elderly person's face every minute.

[2016] Voice Recording: The built-in microphone records the elderly person's voice in real time. Recording starts automatically the moment they start speaking.

[2017] Motion detection: Accelerometers detect the movements of the elderly and record their walking patterns and body movements.

[2018] Step 2: Send data from the device to the server

[2019] Input: facial expression data, voice data, and behavioral data collected on the device

[2020] Output: Data packets sent to the server

[2021] 1. Specific actions:

[2022] Data packet generation: The collected data is converted into a certain format and a data packet is generated.

[2023] Use of communication protocols: Wi-Fi or Bluetooth is used to send collected data packets to the server. For example, if Wi-Fi is used, data is uploaded every two minutes.

[2024] Step 3: Server receives data

[2025] Input: Data packets sent from the device

[2026] Output: Buffering of received data and time-stamped data

[2027] 1. Specific actions:

[2028] Use of Data Buffer: The server uses a data buffer to temporarily store the received data.

[2029] Add timestamp: Add a timestamp to the received data and save it.

[2030] Step 4: Facial expression analysis by the server

[2031] Input: Received facial expression data

[2032] Output: Estimated emotional state and stress level

[2033] 1. Specific actions:

[2034] Facial Recognition: The server uses facial recognition software such as OpenCV to extract faces from images.

[2035] Emotion estimation: Based on the extracted facial feature points, an algorithm is run to estimate the emotional state (happiness, sadness, anger, etc.) and stress level.

[2036] Step 5: Audio analysis by the server

[2037] Input: Received audio data

[2038] Output: Estimated psychological state

[2039] 1. Specific actions:

[2040] Speech feature extraction: Extract basic characteristics of speech (e.g., pitch, tone, rhythm) using tools such as LibROSA.

[2041] Mood state estimation: Based on the extracted speech features, we use a machine learning model to estimate the mood.

[2042] Step 6: Server analysis

[2043] Input: Received behavioral data

[2044] Output: Activity level and fall risk assessment results

[2045] 1. Specific actions:

[2046] Walking pattern analysis: The server analyzes walking data using TensorFlow and other tools to detect deviations from normal walking patterns.

[2047] Risk assessment: Calculate a risk score based on the signs of falling.

[2048] Step 7: Anomaly detection and notification

[2049] Input: Results of facial expression analysis, voice analysis, and movement analysis

[2050] Output: Notification of abnormality and how to respond

[2051] 1. Specific actions:

[2052] Threshold check: Detects abnormalities when emotional state or behavior data exceeds a set threshold.

[2053] Generate notification: Generate a notification message containing the anomaly information.

[2054] Send Notifications: Send notifications to caregivers via SMS and email protocols.

[2055] Step 8: Manage your medication reminders

[2056] Input: Elderly person's health data and medication schedule

[2057] Output: Reminder notification

[2058] 1. Specific actions:

[2059] Reminder generation: The server checks the time to take the medicine and generates a reminder.

[2060] Send Reminder: Send the generated reminder to your device.

[2061] Reminder display: The device notifies the elderly of reminders by voice or vibration and displays instructions on the screen.

[2062] Step 9: Regular monitoring and database updates

[2063] Input: Continuously collected data

[2064] Output: Updated user profile and analytics history

[2065] 1. Specific actions:

[2066] Data Storage: The server adds the newly received data to the existing user profile and updates the database.

[2067] Accumulation of analysis results: Accumulate and manage past analysis results to monitor long-term health trends.

[2068] Thus, through the above steps, the present invention is a system that comprehensively supports the safety and health of the elderly.

[2069] (Application example 1)

[2070] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2071] While individual care support systems exist for the elderly and those who require care, there are not enough systems in place to monitor the safety and health of workers in factories and work environments in real time. Another issue is the lack of a system that can comprehensively analyze workers' voices, facial expressions, and behavioral data to not only detect abnormalities but also optimally allocate tasks and suggest break times. This makes it difficult to improve work efficiency and manage worker health.

[2072] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2073] In this invention, the server includes means for collecting data on the voice, facial expression, and behavior of users (workers), means for analyzing the collected data and evaluating the user's stress level and sense of loneliness, means for detecting abnormalities based on the analysis results and notifying the caregiver (manager), means for managing the timing of medication intake by the user and providing reminders, means for predicting the user's health condition and risks and proposing an appropriate care plan, means for collecting data on the voice, facial expression, and behavior of workers and monitoring their safety and health condition, means for predicting work risks according to the work environment and optimizing the work schedule, and means for proposing work allocation and break timing based on the prediction results. This makes it possible to comprehensively understand the health and safety of workers and perform appropriate labor management.

[2074] "User" refers to the person who will use the system, such as the elderly, people who need care, or workers in factories or work environments.

[2075] "Voice data" refers to information used to record a user's voice and, if necessary, analyze the tone and patterns of that voice.

[2076] "Facial expression data" is information used to capture a user's facial expressions and estimate their emotional state and stress level.

[2077] "Behavioral data" is information used to detect a user's physical movements and work patterns, and to assess their activity level and risk of falling.

[2078] The "analysis means" refers to technology that analyzes collected voice, facial expression, and behavioral data to assess the user's stress level, loneliness, activity level, etc.

[2079] "Anomaly detection means" is a technology for detecting abnormal conditions or behavior from the analysis results and issuing necessary notifications.

[2080] "Notification means" refers to the method for detecting abnormalities and creating necessary reminders to notify caregivers and administrators.

[2081] "Reminder provision means" refers to technology that notifies users of the timing of when they should take their medication, the need for rest, and so on.

[2082] A "health condition prediction means" is a technology that predicts the user's health condition and risks based on collected data and proposes future care plans.

[2083] "Work risk prediction means" is a technology for predicting work-related risks in the work environment based on the health status and work patterns of workers.

[2084] The "work schedule optimization method" is a method for analyzing workers' voices, facial expressions, behavioral data, etc., and proposing optimal work allocation and break timing.

[2085] MODE FOR CARRYING OUT THE INVENTION

[2086] This invention is a system for comprehensively supporting the safety and health of the elderly, people who need care, and workers. The invention has components that mainly consist of a server, a terminal, and a user.

[2087] System Structure and Operation

[2088] Data collection

[2089] Device: The device is worn by the user and has the function of collecting the following data:

[2090] Facial Expression Data: The built-in camera periodically photographs your face and captures your facial expressions.

[2091] Voice data: A microphone records your voice and performs speaker identification if necessary.

[2092] Behavioral data: Accelerometers detect user movements and record walking patterns and body movements.

[2093] The collected data is sent from the terminal to a server.

[2094] Data analysis

[2095] Server: The server receives the transmitted data and performs the following analysis:

[2096] Facial expression analysis: Analyzes facial expression data to estimate the user's emotional state and stress level.

[2097] Voice analysis: Analyzes voice data and evaluates the user's psychological state based on their tone and voice patterns.

[2098] Movement analysis: Analyzes behavioral data to assess the user's activity level and fall risk.

[2099] Anomaly detection and notification

[2100] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver or work manager.

[2101] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[2102] Notification: If an abnormality is detected, a notification will be sent to the caregiver or work manager, containing details of the abnormality and how to respond.

[2103] Medication administration management and work schedule management

[2104] Server: Manages medication timing and work schedules based on health data of elderly people and workers. Specific operations are as follows:

[2105] Generate reminders: Generate reminders when it's time to take a dose or take a break.

[2106] Send Reminder: Send the generated reminder to your device.

[2107] Display reminders: The device will notify the user of reminders by sound or vibration.

[2108] Health Status and Risk Prediction

[2109] Server: Predicts the user's health status and risks based on collected data.

[2110] Risk prediction: Using machine learning algorithms to predict risk of falls, risk of developing dementia, and occupational risks.

[2111] Proposing care plans and work schedules: Based on the prediction results, optimal care plans and work schedules are created and proposed to caregivers and work managers.

[2112] Adding specific examples

[2113] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[2114] Device: Periodically takes photos of the elderly person's face and records their voice. An accelerometer detects their walking patterns.

[2115] Server: Receives this data and analyzes facial expressions and movements. If abnormal movements are detected, it immediately notifies the caregiver.

[2116] User (elderly): The device reminds the elderly to take their medicine, and they take it as instructed. This information is then sent back to the server, where the intake status is recorded.

[2117] Example prompts to input to the generative AI model

[2118] "Design a system that collects data on the voices, facial expressions, and behavior of workers in a factory, and detects and notifies them of abnormalities. Please use this as a reference for the configuration of an elderly care support system."

[2119] In this way, the system provides comprehensive support for safety and health through a series of processes that collect and analyze the voice, facial expressions, and behavioral data of users (elderly people and workers) in real time and quickly notify users in the event of an abnormality.

[2120] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2121] Step 1:

[2122] Data collection

[2123] The device collects data on the user's voice, facial expressions, and behavior.

[2124] Specifically, the built-in camera periodically takes pictures of the user's face, the microphone records their voice, and the accelerometer detects their movements.

[2125] Input: User's face image, voice data, movement data

[2126] Output: Collected data

[2127] Step 2:

[2128] Data transmission

[2129] The terminal transmits the collected data to the server.

[2130] Specifically, the collected data is securely encrypted and uploaded to a cloud server.

[2131] Input: Collected data (face images, voice data, movement data)

[2132] Output: Data sent to the server

[2133] Step 3:

[2134] Data analysis

[2135] The server parses the received data.

[2136] Specifically, facial expression analysis, voice analysis, and movement analysis are performed to evaluate the user's emotional state, stress level, and activity level.

[2137] Input: Data received by the server (face image, voice data, movement data)

[2138] Output: Analysis results (emotional state, stress level, activity level)

[2139] Step 4:

[2140] Anomaly detection

[2141] The server detects abnormalities from the analysis results.

[2142] Specifically, it detects abnormal behavior, changes in facial color, or unusual tones of voice, and raises a flag if action is required.

[2143] Input: Analysis results

[2144] Output: Whether an anomaly was detected and its details

[2145] Step 5:

[2146] notification

[2147] If the server detects an abnormality, it sends a notification to the caregiver or work manager.

[2148] Specifically, a message containing details of the abnormality and appropriate countermeasures is created and notified in real time.

[2149] Input: Anomaly detection details

[2150] Output: Notification sent to caregiver or work manager

[2151] Step 6:

[2152] Reminder generation

[2153] The server uses health data from elderly people and workers to remind them when to take their medicine and when they need to take a break.

[2154] Specifically, reminders are generated based on the schedule and sent to the device.

[2155] Input: Health data, schedule information

[2156] Output: Generated reminders

[2157] Step 7:

[2158] Send and display reminders

[2159] The terminal notifies the user of the reminder received from the server.

[2160] Specifically, reminders are notified by voice or vibration.

[2161] Input: Generated reminder

[2162] Output: Reminder notification to user

[2163] Step 8:

[2164] Health Status and Risk Prediction

[2165] The server predicts the user's health status and risks based on the collected data.

[2166] Specifically, machine learning algorithms are used to assess the risk of falls, the risk of developing dementia, and work-related risks.

[2167] Input: Collected data (face images, voice data, movement data)

[2168] Output: Health status prediction, risk assessment results

[2169] Step 9:

[2170] Proposing care plans and work schedules

[2171] Based on the prediction results, the server creates optimal care plans and work schedules and proposes them to caregivers and work managers.

[2172] Specifically, the system automatically generates care plans and work schedules and sends out proposals.

[2173] Input: Health status prediction, risk assessment results

[2174] Output: Optimal care plan and work schedule

[2175] Through these steps, the system is able to provide comprehensive support for the safety and health of users.

[2176] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2177] This invention relates to an individualized care support system for elderly people and those in need of care. Specifically, the system collects data on the elderly's voice, facial expressions, and behavior, analyzes this data to evaluate the user's emotional state, stress level, and sense of loneliness, and detects abnormalities and notifies the caregiver. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. Furthermore, by combining an emotion engine, the system can evaluate the elderly's emotional state in detail and provide suggestions for stress relief and real-time notifications.

[2178] System program and processing flow

[2179] This system consists of the following main components: a server, a terminal, an emotion engine, and a user (elderly person). Below, we explain in natural language the functions and processing flow of each component.

[2180] Data collection

[2181] Device: The wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[2182] Facial expression data: The built-in camera periodically photographs the senior's face and captures their facial expressions.

[2183] Voice data: A microphone records the elderly person's voice and performs speaker identification if necessary.

[2184] Behavioral data: Accelerometers detect the elderly's movements and record their walking patterns and physical movements.

[2185] The collected data is sent from the terminal to a server.

[2186] Data analysis

[2187] Server: The server receives the transmitted data and performs the following analysis:

[2188] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[2189] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For speech analysis, we extract acoustic features and apply them to models that detect emotions and loneliness.

[2190] Movement analysis: Analyzes behavioral data to assess elderly people's activity levels and fall risk. Movement analysis involves analyzing movement patterns using machine learning models to detect abnormal behavior.

[2191] Emotional state assessment

[2192] Emotion Engine: The emotion engine integrated into the server utilizes the collected facial and voice data to assess the detailed emotional state of the elderly. This assessment identifies not only the level of stress and loneliness, but also subtle emotions such as joy, sadness, and anger.

[2193] Anomaly detection and real-time notifications

[2194] Server: If an abnormality is detected from the analysis results, the server will take the following actions for the caregiver.

[2195] Anomaly detection: Detects falls, changes in facial color, abnormal voice tones, etc.

[2196] Notification Generation: When an abnormality is detected, a notification message is generated for the caregiver, including details such as the type of abnormality, date, time, and location.

[2197] Notification sending: Send the generated notification message to caregivers and family members via smartphone app, email, SMS, etc.

[2198] Stress relief suggestions

[2199] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[2200] Medication administration management

[2201] Server: Manages the timing of medication use based on the elderly person's health data.

[2202] Generate reminders: Generates reminders when it's time to take your dose.

[2203] Send Reminder: Send the generated reminder to your device.

[2204] Displaying reminders: The device will notify the elderly of reminders by sound or vibration.

[2205] Health Status and Risk Prediction

[2206] Server: Predicts the health status and risks of elderly people based on collected data.

[2207] Risk prediction: Using machine learning algorithms to predict the risk of falls or developing dementia.

[2208] Care plan proposal: Based on the prediction results, an optimal care plan is created and proposed to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[2209] Specific examples

[2210] For example, if an elderly person needs to take medicine at a specific time each day, the following steps are taken:

[2211] 1. Data collection: The device takes pictures of the elderly person's face, records their voice, and detects their walking patterns.

[2212] 2. Data analysis: The server receives this data and analyzes facial expressions and voice using an emotion engine.

[2213] 3. Evaluation and Suggestion: The emotion engine evaluates the emotional state of the elderly and suggests relaxing music if the elderly is feeling stressed.

[2214] 4. Abnormality detection and notification: If abnormal behavior is detected, a notification will be sent to the caregiver immediately.

[2215] 5. Medication administration management: The server manages the time to take medication, and the device displays reminders. The elderly person enters the fact that they have taken their medication into the device, and the server records that information.

[2216] In this way, the present invention comprehensively supports the safety and health of the elderly through a series of processes: collecting and analyzing voice, facial expression, and behavioral data of the elderly, using an emotion engine to evaluate the elderly's emotional state in detail, and promptly notifying in the event of an abnormality.

[2217] The processing flow will be explained below.

[2218] Step 1:

[2219] Data collection

[2220] Device: A wristwatch-type device worn by the elderly uses a built-in camera, microphone, and accelerometer to collect facial, voice, and behavioral data. The camera periodically captures the elderly's face, the microphone records the elderly's voice, and the accelerometer detects the elderly's movements.

[2221] Step 2:

[2222] Data transmission

[2223] Terminal: The terminal transmits collected data to the server via Wi-Fi or Bluetooth. The transmitted data includes facial expression data, voice data, and behavioral data.

[2224] Step 3:

[2225] Data reception

[2226] Server: Receives data sent from the device and stores it in an internal database. The data is then prepared for analysis.

[2227] Step 4:

[2228] facial expression analysis

[2229] Server: Analyzes the stored facial expression data and evaluates the emotional state and stress level of the elderly. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[2230] Step 5:

[2231] Audio analysis

[2232] Server: Analyzes voice data and evaluates the psychological state of the elderly person based on their tone and voice patterns. For voice analysis, acoustic features are extracted and applied to an emotion engine that detects emotions and feelings of loneliness.

[2233] Step 6:

[2234] Motion analysis

[2235] Server: Analyzes behavioral data to assess elderly people's activity levels and fall risk. For behavior analysis, machine learning models are used to analyze movement patterns and detect abnormal behavior.

[2236] Step 7:

[2237] Emotional state assessment

[2238] Emotion engine: The emotion engine integrated into the server uses the collected facial and voice data to assess the detailed emotional state of the elderly. The assessment results identify the elderly's stress, loneliness, joy, sadness, anger, etc.

[2239] Step 8:

[2240] Anomaly detection

[2241] Server: Detects anomalies from the data analysis results and flags any anomalies detected, such as falls, extreme changes in facial expression, or unusual voice tones.

[2242] Step 9:

[2243] Notification generation

[2244] Server: Generates a notification message to the caregiver when an abnormality is detected. The notification includes details such as the type of abnormality, date, time, and location.

[2245] Step 10:

[2246] Send notifications

[2247] Server: Generates and sends notification messages to caregivers and family members via smartphone apps, email, SMS, etc.

[2248] Step 11:

[2249] Real-time notifications

[2250] Server: When the evaluation results of the emotion engine are updated in real time, a notification is sent to the caregiver immediately in case of an emergency.

[2251] Step 12:

[2252] Stress relief suggestions

[2253] Server: Based on the emotional state assessed by the emotion engine, the server makes suggestions for stress reduction, such as music therapy, relaxation techniques, and recommended social activities.

[2254] Step 13:

[2255] Reminder generation

[2256] Server: Based on the elderly person's health data, manages the timing of taking medicine and generates reminders when it is nearing the time to take the medicine.

[2257] Step 14:

[2258] Send reminders

[2259] Server: Sends the generated reminders to the device.

[2260] Step 15:

[2261] Reminder display

[2262] Device: Reminders are displayed to the elderly, and notifications are sent via voice and vibration. The name of the medication and instructions for taking it are displayed on the screen.

[2263] Step 16:

[2264] Enter intake status

[2265] User (elderly): Enters the fact that he / she has taken medicine into the terminal and sends the information to the server.

[2266] Step 17:

[2267] Recording intake status

[2268] Server: Records the intake status sent by the elderly in a database and manages the risk of deficiencies or overdoses.

[2269] Step 18:

[2270] Risk prediction

[2271] Server: Predicts future health risks based on collected data. Machine learning algorithms are used to predict the risk of falls and dementia.

[2272] Step 19:

[2273] Care plan proposal

[2274] Server: Based on the prediction results, the server creates an individualized care plan for the elderly person and proposes it to the caregiver. The care plan includes suggestions for exercise, nutrition, and social activities.

[2275] Step 20:

[2276] Task Management

[2277] Server: Prioritizes caregiver tasks and optimizes schedules.

[2278] Step 21:

[2279] Sending reminders and notifications

[2280] Server: Generates appropriate reminders and notifications based on the elderly person's activities and sends them to the device.

[2281] Step 22:

[2282] Schedule adjustments

[2283] User (caregiver): Review the proposed schedule and care plan and adjust as needed.

[2284] Example 2

[2285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2286] For the elderly and those who require care, accidents and changes in health status in daily life are serious problems. It is also extremely difficult for caregivers to grasp the situation in real time and respond appropriately. Given this background, there is a need for a system that can efficiently collect and analyze data on the voice, facial expressions, and behavior of the elderly, detect abnormalities, and quickly notify them. In addition, suggestions for medication management and stress relief are also important for improving the quality of life for the elderly.

[2287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2288] In this invention, the server includes means for collecting data on the voice, facial expressions, and behavior of the elderly person, means for transmitting the collected data to the server, means for the server to receive and analyze the transmitted data, means for evaluating the elderly person's emotional state using an emotion engine based on the analyzed data, means for detecting abnormalities based on the analysis results and notifying the caregiver, means for managing the timing of medication intake by the elderly person and providing reminders, and means for predicting the elderly person's health condition and risks and proposing an appropriate care plan. This enables comprehensive management of the safety and health of the elderly person, and enables prompt response and appropriate care.

[2289] "Data collection tools" refer to devices and systems used to collect data on the voices, facial expressions, and behavior of older adults.

[2290] The "server" is the central system that receives and analyzes data collected from the elderly.

[2291] "Data transmission means" refers to the functions and protocols for sending collected data to the server.

[2292] "Data analysis means" refers to algorithms or models that the server uses to analyze the data it receives and extract useful information.

[2293] The "emotion engine" is an analysis system that evaluates the emotional state of elderly people based on their facial expressions and voice data.

[2294] "Anomaly detection means" refers to a system or method for detecting abnormal conditions or behaviors from analysis results.

[2295] "Notification means" refers to a device or system that sends information to caregivers or family members when an abnormality is detected.

[2296] "Reminder provision means" refers to a function or device that manages the timing of when elderly people take their medicine and notifies them at the appropriate time.

[2297] A "health status prediction tool" is an algorithm or model that predicts the health status and risks of older adults based on collected data.

[2298] A "care plan proposal means" refers to a system or method for creating an optimal care plan based on the predicted health status and proposing it to a caregiver.

[2299] This invention relates to an individualized care support system for elderly people and those in need of care. It assesses the user's emotional state, stress level, and sense of loneliness through the collection and analysis of data on the elderly's voice, facial expressions, and behavior, and notifies caregivers of any abnormalities. It also proposes appropriate care plans, predicts the elderly's health status and risks, and manages the timing of medication administration. The following describes the specific hardware and software configurations and their operation.

[2300] System configuration

[2301] The system of this invention is composed of the following main components: a server, a terminal (wristwatch type), an emotion engine, and a user (elderly person).

[2302] Data collection

[2303] Device: A wristwatch-type device worn by the elderly uses built-in sensors and cameras to collect the following data:

[2304] Facial expression data: The built-in camera periodically photographs the elderly person's face and captures their facial expressions, for example, automatically taking a photo once every hour.

[2305] Voice data: The built-in microphone records the elderly person's voice and performs speaker identification if necessary, for example, by recording the voice of everyday conversations.

[2306] Behavioral data: Accelerometers detect the movements of seniors and record their walking patterns and physical movements, such as the number of steps taken each day and the distance traveled.

[2307] The collected data is transmitted from the terminal to a server.

[2308] Data analysis

[2309] Server: The server receives the transmitted data and performs the following analysis:

[2310] Facial Expression Analysis: An emotion engine is used to analyze facial expression data and estimate the emotional state and stress level of elderly people. For facial expression analysis, facial recognition technology is used to extract facial features and apply them to the emotion engine.

[2311] Speech analysis: We analyze speech data and use an emotion engine to assess the psychological state of the elderly based on their tone and patterns of voice. For sp...

Claims

1. A system aimed at providing individual care support for the elderly and people who need care, A means of collecting data on the voices, facial expressions, and behaviors of elderly people; A method for analyzing the collected data and assessing the stress levels and loneliness of the elderly. A means to detect abnormalities based on the analysis results and notify caregivers; A means to manage the timing and provide reminders for medications that seniors should take; A means to predict the health status and risks of elderly people and propose appropriate care plans, A system including:

2. 10. The system of claim 1, further comprising analyzing the movement patterns of the elderly person to detect falls and abnormal movements.

3. The system according to claim 1, wherein the system uses voice analysis technology to evaluate the psychological state of an elderly person based on the tone and pattern of their voice.

Images