system

Abstract

Provide a system. 【Solution means】 Means for collecting biological data, Means for acquiring image data, Means for processing voice data, Means for analyzing the biological data and image data to detect abnormalities, Means for generating an emergency notification based on the detection of an abnormality, Means for performing natural language processing based on the voice data, Means for providing voice feedback to the user, A system including these.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In response to the increasing demand for care in an aging society, the shortage of care personnel has become a serious problem. Therefore, individualized life support and health management for the elderly are required in each household and facility, but it has been difficult to provide efficient and comprehensive monitoring with conventional technologies. In addition, there is a lack of means for real-time information sharing and safety assurance that can respond quickly in an emergency, and maintaining the safety and independence of the elderly is an issue due to this.

Means for Solving the Problems

[0005] This invention provides a system that collects biometric data from multiple sensors and uses it to predict health status, and analyzes image data from a camera and audio data from a microphone to perform natural language processing. An anomaly detection module detects abnormalities such as falls by the user in real time, and generates an emergency notification based on this detection, which is then sent to family members or caregivers. Furthermore, by providing real-time feedback to the user using speech synthesis, the system realizes an environment in which elderly people can live independently while feeling secure in their daily lives.

[0006] "Biometric data" refers to data that indicates an individual's physiological and health status, such as heart rate, body temperature, and blood pressure.

[0007] "Image data" refers to visual information acquired by a camera and is used to analyze the user's actions and environment.

[0008] "Voice data" refers to voice information obtained through user speech, which is data intended for analysis using natural language processing.

[0009] "Anomaly detection" refers to the process of analyzing collected data to recognize unusual states or actions, such as falls.

[0010] An "emergency notification" is a real-time alert generated based on anomaly detection, providing immediate notification to family members or caregivers.

[0011] "Natural language processing" is a technology that analyzes speech data, understands the user's intent, and generates responses.

[0012] "Voice feedback" is a method of conveying information and instructions to the user using speech synthesis.

[0013] "Information sharing" refers to sharing data collected by the system and information on detected anomalies with family members and caregivers in real time. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The elderly care system of the present invention combines biometric data, image data, and voice data to monitor the user's daily life in real time and manage their health. Specific embodiments are described below.

[0036] First, the sensor devices equipped in the terminal continuously monitor the user's heart rate, body temperature, activity level, etc., and collect biometric data. Next, cameras installed in the living room or living space capture images of the user's movements and environment, acquiring image data. This data is then transmitted to a server via wireless communication.

[0037] The server analyzes biometric and image data in real time. Specifically, it uses image processing algorithms to recognize user movements and postures and detect abnormal behavior. If an abnormality is detected, the server immediately generates an emergency notification and sends it to family members or caregivers. Furthermore, it predicts the user's health status based on the collected biometric data and provides optimal health advice.

[0038] Furthermore, the voice spoken by the user is collected through the device's microphone and analyzed on the server using natural language processing. This makes it possible to understand the user's intent and generate appropriate voice feedback.

[0039] As a concrete example, consider the case of a user falling. The device's camera captures the user's fall, and image data is sent to the server. The server detects the fall through image analysis, generates an emergency notification, and sends it to the family. At the same time, voice feedback can be used to confirm with the user, "Are you injured?", and further instructions can be given as needed.

[0040] This system not only monitors the user's status in real time but also allows for rapid response in case of abnormalities. This helps support the safe living of the elderly and reduces the burden on family members and caregivers.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device collects biometric data through wearable devices and smart home sensors. This data includes heart rate, body temperature, and activity level. The collected data is transmitted to a server in real time.

[0044] Step 2:

[0045] The device uses its built-in camera to capture image data of the user's movements and environment. For example, it records the user's posture and movement as video and sends the data to a server.

[0046] Step 3:

[0047] The server monitors the user's current health status based on the biometric data it receives. Predictive algorithms are executed to detect abnormal values ​​or changes.

[0048] Step 4:

[0049] The server processes image data and analyzes user behavior and environmental changes. Deep learning technology is used to detect falls and abnormal movements.

[0050] Step 5:

[0051] The device uses its microphone to collect voice data and sends it to the server. Voice input is used to recognize the user's natural speech.

[0052] Step 6:

[0053] The server performs speech recognition and natural language processing to analyze the user's utterances. This allows it to accurately understand the user's requests and instructions.

[0054] Step 7:

[0055] Based on the analysis results, the server will notify family members or caregivers of any anomalies detected in emergencies. Real-time alerts will be sent via email or app notifications.

[0056] Step 8:

[0057] The device uses speech synthesis technology to provide feedback to the user. For example, it can deliver reminders or health advice to the user via voice.

[0058] Step 9:

[0059] Users act according to the advice and instructions provided, which then provides feedback for the next data collection cycle, ensuring continued personalized care.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] There is a need to provide a system that ensures the safe living of the elderly while allowing family members and caregivers in remote locations to monitor their health and potential dangers in real time. However, existing systems lack sufficient real-time capabilities and immediate response to emergencies, making it difficult to monitor them with peace of mind.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes a device for collecting biometric information, a device for acquiring visual information, and a device for processing audio information. This enables real-time monitoring of the user's health status, immediate response to abnormalities, and provision of appropriate feedback.

[0065] "Biometric information" refers to data collected from the human body, such as heart rate, body temperature, and activity level.

[0066] "Visual information" refers to video and image data acquired through cameras and sensors.

[0067] "Voice information" refers to user speech and voice data collected using a microphone.

[0068] A "device that recognizes abnormalities" is a device that uses collected data to detect conditions that are different from the normal state.

[0069] A "device for generating emergency notifications" is a device that promptly issues warnings or notifications to relevant parties when an anomaly is detected.

[0070] A "language processing device" is a device that analyzes speech information as natural language and understands its meaning and intent.

[0071] A "device that provides voice response" is a device that provides appropriate voice feedback to the user based on analyzed information.

[0072] A "device that transmits wirelessly" is a device that uses wireless technology to transmit collected information to other devices or servers without using cables.

[0073] "Supporter" refers to the user's family, caregiver, or someone responsible for their health management.

[0074] "Real-time information sharing" refers to sharing information with people in remote locations in near real-time.

[0075] This invention relates to a system that monitors a user's health status in real time and enables immediate response in the event of an abnormality. The system mainly consists of terminals and servers.

[0076] The device incorporates a sensor device for collecting biometric information, continuously acquiring data such as heart rate, body temperature, and activity level. This sensor device is integrated into typical wearable devices. In addition, the device is equipped with a camera to acquire visual information, monitoring the user's movements and environment. Audio information is collected through the device's microphone. This information is transmitted from the device to a server via wireless communication.

[0077] The server analyzes received biometric and visual information in real time. Image processing algorithms running on the server accurately recognize the user's movements and posture, and immediately generate an emergency notification if an anomaly is detected. The emergency notification is sent to family members or caregivers via email or SMS. At the same time, the system predicts the user's health status based on biometric information and provides appropriate advice. Voice information is analyzed using natural language processing technology and used to understand the user's intentions and requests. Based on the results, a voice response is generated and fed back to the user through the terminal.

[0078] For example, if a user falls, the device's camera captures the situation and sends image data to the server. The server generates an emergency notification and sends a warning to family members such as, "The user has fallen." At the same time, it can check with the user via voice response, asking, "Are you injured?" and provide further instructions as needed.

[0079] An example of a prompt to input into the generating AI model is, "Please describe in detail the response system for when an elderly person falls." This system will efficiently manage the safety and health of users.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensor devices. The input is biometric data from the sensors, which is temporarily stored and prepared for periodic transmission to the server. The output is encrypted biometric data for transfer to the server. Specifically, the wearable device tracks the movements of the user wearing it and updates the data at regular intervals.

[0083] Step 2:

[0084] The device acquires the user's visual information using a camera. The input is real-time video captured by the camera, which is converted into a format suitable for image processing algorithms. The output is processed image data, which is prepared to be sent to the server. Specifically, the camera scans the user's environment at regular intervals and captures changes in movement as needed.

[0085] Step 3:

[0086] The terminal collects user speech using a microphone to acquire audio information. The input is audio data, which is converted into an audio format and configured for speech recognition. The output is an audio file to be later sent to the server. Specifically, it performs noise reduction and necessary amplification processing on the audio to prepare clear audio data.

[0087] Step 4:

[0088] The terminal transfers collected biometric, visual, and audio information to the server using wireless communication. Input consists of various data temporarily stored within the terminal. Output is an encrypted, comprehensive data packet sent to the server via a wireless communication protocol. Specifically, the terminal calculates the optimal timing and sends the data in a batch processing format.

[0089] Step 5:

[0090] The server analyzes the user's health status and behavior using various received information. Inputs are biometric and visual information transmitted from the terminal. Data processing involves applying an anomaly detection algorithm, and if an anomaly is detected, its nature is determined. Outputs include the anomaly detection result and its details. Specifically, it uses image data to perform motion recognition and identify abnormal postures.

[0091] Step 6:

[0092] The server immediately generates an emergency notification and sends it to family members or caregivers upon detecting an anomaly. The input is the anomaly detection result. The output is an emergency notification message that provides a detailed explanation of the user's situation. Specifically, it prioritizes notifications and delivers the necessary information in a single notification message.

[0093] Step 7:

[0094] The server analyzes voice information using natural language processing techniques to understand the user's intent. The input is voice data transmitted from the terminal. The output is the analyzed intent and request content. Specifically, it extracts keywords from the voice and analyzes their context.

[0095] Step 8:

[0096] The server generates an appropriate voice response based on the analysis results and provides feedback to the user through the terminal. The input is the result of the voice analysis, and the output is the generated voice response data. Specifically, it selects the appropriate voice to play based on the user's state and issues a playback command on the terminal.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] There is a need to monitor the health and safety of the elderly in real time, detect abnormalities early, and respond quickly. However, existing systems lack sufficient technology to integrate and analyze individual biometric, image, and audio information, making it difficult for caregivers to quickly grasp the situation. Furthermore, there is a lack of easy ways for caregivers to check the condition of the elderly.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to comprehensively analyze multiple pieces of information and detect user abnormalities in real time. Furthermore, caregivers can quickly check the user's condition via a portable device.

[0102] "Biometric information" refers to data that shows the user's physical characteristics, such as heart rate, body temperature, and activity level.

[0103] "Image information" refers to visual data acquired by a camera to understand the user's movements and posture.

[0104] "Audio information" refers to data that includes the voice spoken by the user, and is used to extract meaning and intent.

[0105] "Means for detecting anomalies" refers to the process of analyzing biological and image information to identify unexpected conditions or dangerous situations.

[0106] "Means of generating notifications" refer to actions taken to quickly communicate important information based on the detection of anomalies.

[0107] "Means of language processing" refers to technologies that analyze natural language from audio information to understand the user's intent.

[0108] "Means of providing audio output" refers to the process of conveying information and instructions by voice in a format that is easy for users to understand.

[0109] A "portable device" is a device that caregivers can carry around and use to check information about the user in real time.

[0110] "Means of providing time-series information" refers to the process of managing and presenting a series of information, including historical data.

[0111] This invention is a system for monitoring the health status and daily life of elderly people in real time. The system mainly consists of a terminal, a server, and a group of sensors installed in the user's living space.

[0112] The device is equipped with sensors that continuously collect biometric information such as heart rate and body temperature. This biometric information is transmitted to the server via wireless communication. Cameras are installed in the living space to capture image information of the user's movements and posture. Similarly, the device's microphone collects voice information emitted by the user. The server processes this data in real time to detect anomalies. In particular, image processing libraries such as OpenCV and scikit-learn are used to analyze the user's movements.

[0113] If an anomaly is detected, the server generates an emergency notification and sends it to the caregiver via a portable device. This portable device is a smartphone or smart glasses, allowing the caregiver to check it immediately. Furthermore, based on collected biometric information, the system predicts the user's health status and provides voice feedback. This voice feedback is generated using natural language processing, specifically NLP functions from Google® Dialogflow and Microsoft® Azure®.

[0114] For example, if a user falls, the server analyzes the camera data, immediately sends a notification to the caregiver, and plays a voice message to the user asking, "Are you injured?" This enables a quick response.

[0115] An example of a prompt is: "Generate a notification and follow-up scenario regarding fall detection in elderly individuals. Create notification messages and voice prompts, and describe in detail the steps to take after a fall is detected."

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensors. This biometric information is then transmitted to the server using wireless communication. At this stage, the data is still in raw format.

[0119] Step 2:

[0120] Cameras installed in the living space capture the user's movements and posture, acquiring image information. The captured images become input and are immediately sent to the server. At this point, the system is ready for image processing algorithms to be applied for motion recognition.

[0121] Step 3:

[0122] The device's microphone collects the user's voice and sends it to the server as audio data. The audio data becomes the input, and the server is ready to apply natural language processing.

[0123] Step 4:

[0124] The server initiates real-time analysis using biometric and image data. An algorithm for detecting anomalies in biometric data is applied using scikit-learn. Simultaneously, image data is analyzed using OpenCV to detect abnormal behavior. If an anomaly is detected, data reporting the anomaly is generated as output.

[0125] Step 5:

[0126] If an anomaly is detected, the server immediately generates an emergency notification. The notification data is used as input and output to a portable device. This notification is quickly sent to the caregiver's smartphone or glasses using the NF communication protocol, alerting the caregiver.

[0127] Step 6:

[0128] The server performs natural language processing based on the collected audio information. Using Google Dialogflow and Microsoft Azure's NLP capabilities, it analyzes the audio data to understand the user's intent. The resulting audio feedback is then sent to the device as output.

[0129] Step 7:

[0130] When a user needs feedback in an abnormal situation, the server generates voice feedback and provides it to the user via the terminal. For example, when a fall is detected, a voice message such as "Are you injured?" is output. This output allows the user to receive instructions appropriate to the situation.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] The elderly care system of the present invention not only comprehensively monitors the user's daily life by combining biometric data, image data, and voice data, but also recognizes the user's emotional state in real time using an additional emotion engine and provides appropriate feedback based on that.

[0133] First, the device uses sensor devices to collect biometric data such as the user's heart rate, body temperature, and activity level. Next, cameras installed in the room record the user's movements and facial expressions, acquiring image data. This image data plays a crucial role in analyzing the user's facial expressions. In addition, the device's microphone captures audio data, accurately recording the user's speech.

[0134] All of this data is transmitted to the server in real time. The server combines and analyzes the biometric and image data to detect abnormalities in the user's health and behavior. The emotion engine uses image and audio data to analyze the user's facial expressions and tone of voice to recognize the user's emotions. For example, if a user is stressed, this is detected as a change in facial expression and tone of voice.

[0135] Based on the identified emotions, the server adjusts the voice feedback to the user to provide more appropriate communication. For example, if the user is feeling anxious, the server provides feedback in a calm and reassuring tone. Furthermore, by sharing the user's emotional state with family members and caregivers, the server supports the provision of more appropriate care.

[0136] As a concrete example, consider a case where a user is feeling unwell and anxious. Biometric data indicates an abnormality, and the emotion engine detects anxiety from the user's facial expressions and voice. Based on this information, the server generates an emergency notification and sends an alert to family members. At the same time, the device plays a reassuring message to the user through voice feedback, such as, "Are you okay? If you're worried, we'll call support right away."

[0137] As a result, the present invention not only ensures the safety and security of users, but also enables meticulous care that takes their emotions into consideration.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The terminal collects biometric data from the user's wearable device. This data includes heart rate, body temperature, and activity level, and is transferred to a server using a secure communication protocol.

[0141] Step 2:

[0142] A camera installed on the terminal records video of the room, capturing the user's movements and facial expressions. The video data is transmitted to the server in real time. This allows for continuous monitoring of the user's posture and changes in the environment.

[0143] Step 3:

[0144] The device acquires user voice data via its microphone. It captures the user's everyday conversations and speech content and accurately transmits it to the server.

[0145] Step 4:

[0146] The server analyzes the received biometric data and determines whether it falls within the normal range. If an abnormal value is detected, an alert is immediately generated and preparations are made.

[0147] Step 5:

[0148] The server analyzes image and audio data and uses an emotion engine to determine the user's emotional state. It analyzes changes in facial expressions from the image data and the tone and volume of the voice from the audio data to identify emotions.

[0149] Step 6:

[0150] Based on the results of anomaly detection and emotion assessment, the server decides whether to send an emergency notification to family members or caregivers. Information is shared via email or push notifications as needed.

[0151] Step 7:

[0152] The server adjusts the voice feedback it provides to the user, generating feedback in an appropriate tone and content based on the user's emotions. For example, if anxiety is detected, it selects a calming message to reassure the user.

[0153] Step 8:

[0154] The device uses speech synthesis technology to provide the user with voice feedback generated from the server. It then provides necessary support through interaction with the user.

[0155] Step 9:

[0156] Users react to voice feedback and adjust their daily behaviors accordingly. This allows the entire system to receive useful feedback in the next observation cycle, improving the quality of care provided to the user.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] In elderly care, there is a need to monitor individual health and emotional states in real time and to respond quickly and appropriately when abnormalities are detected. However, conventional systems have difficulty providing integrated and immediate feedback of biometric and emotional data, making it challenging to adequately ensure user confidence.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for analyzing biometric information to detect abnormalities, means for performing emotion analysis and adjusting feedback according to the emotional state, and means for generating voice feedback based on the user's emotional state using a generative AI model. This enables a comprehensive understanding of the health and emotional state of elderly individuals, and allows for rapid and appropriate feedback tailored to individual circumstances.

[0162] "Biometric information" refers to numerical values ​​or data that indicate a person's physical state, such as heart rate, body temperature, and activity level.

[0163] "Image information" refers to visual data used to capture the user's facial expressions and movements.

[0164] "Audio information" refers to acoustic data that includes the user's voice, including the content of their speech and the tone of their voice.

[0165] "Anomaly detection" refers to the process of analyzing biological and image information to identify abnormal physical or behavioral conditions.

[0166] An "emergency notification" is a notification that is immediately sent to family members or caregivers in an emergency to ensure the user's safety.

[0167] "Speech analysis" is a process that uses natural language processing based on speech information to understand the content of speech and the tone of voice.

[0168] "Voice response" refers to voice feedback provided to the user, intended to convey information or soothe emotions.

[0169] "Emotional analysis" is a technology that identifies a user's emotional state by analyzing their facial expressions and tone of voice.

[0170] A "generative AI model" is a model that uses artificial intelligence techniques to generate output in response to a specific input.

[0171] This invention is a system for elderly care that monitors the user's daily health and emotional state in real time and provides appropriate responses. The system mainly consists of a "terminal" and a "server".

[0172] The device is hardware that is worn directly by the user or installed in their room, and is equipped with sensors for collecting biometric information (e.g., heart rate monitor and temperature sensor), a camera for recording movements and facial expressions, and a microphone for capturing audio. This device transmits the collected data to a server in real time.

[0173] The server integrates and processes received biometric, image, and audio information. Biometric information is used for detecting abnormalities in health status, image information is particularly used for analyzing the user's facial expressions, and audio information is used to analyze speech content and tone. The server also incorporates an emotion analysis engine and a generative AI model, which are used to analyze the user's emotional state and generate appropriate audio feedback based on the results. The generative AI model can receive prompts such as: "The user appears anxious; please generate reassuring words."

[0174] As a concrete example, if a user feels unwell and anxious, the device records an abnormally high heart rate and sends it to the server. Simultaneously, the camera captures the user's troubled expression, and the microphone records a tense voice. The server comprehensively analyzes this data, and if it detects both physical abnormality and anxiety, it uses a generative AI model to generate voice feedback such as, "Are you okay? We'll call support right away, so please don't worry," and conveys it to the user via the device.

[0175] This system allows users to centrally manage their health and emotional state and receive prompt and appropriate care as needed. Furthermore, family members and caregivers can monitor the user's condition in real time, enabling them to provide better support.

[0176] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0177] Step 1:

[0178] The device collects biometric information from the user. Specifically, it collects data such as heart rate, body temperature, and activity level using heart rate monitors and temperature sensors. The input is a digital signal from the biosensor, which is converted into a visualizeable format within the device. The output is formatted biometric data.

[0179] Step 2:

[0180] The device acquires image information using a camera installed in the room. This allows the user's movements and facial expressions to be captured in real time. The input is visual data received by the camera sensor, which is then processed and converted into a format that is easy to analyze. The output is the processed image data.

[0181] Step 3:

[0182] The device captures audio information using a microphone. The user's speech and tone are recorded, forming the basis for speech recognition. The input is an audio signal, which undergoes noise reduction and sampling. The output is clean audio data.

[0183] Step 4:

[0184] The device transmits all collected biometric, image, and audio information to the server in real time. Input is integrated user data transmitted using a secure protocol. Output is the data batch received by the server.

[0185] Step 5:

[0186] The server analyzes received biometric data to detect anomalies. Specifically, it analyzes sudden increases in heart rate and fluctuations in body temperature. The input is received biometric data, and data mining techniques are used to identify anomaly patterns. The output is an anomaly detection flag.

[0187] Step 6:

[0188] The server performs sentiment analysis using image and audio information. It uses facial expression recognition technology and voice sentiment analysis to determine the user's emotional state. The input consists of features extracted from image and audio data, and a machine learning model is used to identify emotion categories. The output is a label representing the emotional state.

[0189] Step 7:

[0190] The server uses an AI model to generate voice feedback that responds to the user's emotions. A prompt is input to the model, and a feedback message is output. The input is a prompt such as, "The user is feeling anxious; please generate a reassuring message." The output is the generated voice response message.

[0191] Step 8:

[0192] The device receives feedback from the server and provides voice feedback to the user. The input is a generated voice message, which is transmitted to the user through the speaker. The output is the voice feedback that the user hears.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] In elderly care, it is necessary not only to monitor health status but also to understand emotional states in real time. However, conventional technologies have made it difficult to comprehensively analyze biometric information, images, and audio data to provide appropriate feedback tailored to the emotional state of individual users. Furthermore, there has been a challenge in sharing timely and effective information with family members and caregivers, which prevents prompt and appropriate responses.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to analyze the user's health and emotional state in real time and quickly share necessary information with family members and caregivers.

[0198] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and activity level.

[0199] "Image information" refers to visual data of a user's face and movements acquired using a camera.

[0200] "Voice information" refers to data about the user's speech content and tone collected by a microphone.

[0201] "Emotion recognition means" refers to an algorithm that analyzes image and audio information to identify the user's emotional state.

[0202] An "emergency notification" is an alert message generated when an abnormality is detected, and is sent to family members or caregivers.

[0203] "Voice feedback" refers to reactions or instructions provided to the user via voice.

[0204] "Natural language processing" is a technology that analyzes speech information to understand the intent behind a user's utterances.

[0205] "Means of sharing emotional states with family and caregivers" refers to methods for transmitting a user's emotional information to relevant parties in real time.

[0206] The system for implementing this invention comprehensively collects and analyzes the user's biometric information, image information, and voice information to monitor the user's health and emotional state in real time and provide necessary information to family members and caregivers.

[0207] The server runs a program that collects biometric information such as heart rate, body temperature, and activity level from sensor devices. This data is necessary for analyzing the user's health status. Furthermore, it uses a camera to acquire image information of the user and analyzes the user's movements and facial expressions. Image recognition algorithms are used for this analysis. Audio information is acquired via a microphone, and natural language processing technology is used to analyze the user's speech content and tone.

[0208] The server integrates this data and uses a generative AI model to recognize the user's emotions. Once the emotional state is determined, it generates voice feedback based on the result and provides appropriate feedback to the user. For example, if the user is feeling anxious, it communicates with them in a calm voice to provide reassurance. Furthermore, if emotional information is important, it is shared in real time with family members or caregivers.

[0209] For example, suppose the system detects surprise from the user's facial expression along with an increase in heart rate while the user is in the living room. The server analyzes this information and provides voice feedback such as, "Calm down, take a deep breath," while simultaneously sending a notification to the caregiver.

[0210] An example of a prompt message would be, "If an elderly person experiences anxiety, how would you provide feedback using heart rate, facial expression, and voice data?"

[0211] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0212] Step 1:

[0213] The device collects biometric information from sensor devices. This input data includes heart rate, body temperature, and activity level. The device records the digital biometric data and prepares to send it to the server.

[0214] Step 2:

[0215] Using the camera, the device acquires image information of the user. This input consists of image data of the user's face and body movements. The device applies an image recognition algorithm to extract features for analyzing facial expressions and movements.

[0216] Step 3:

[0217] The device acquires voice information using a microphone. The input data includes the user's speech content and voice tone. The device uses natural language processing technology to convert the voice data into text and perform voice tone analysis.

[0218] Step 4:

[0219] The server integrates the data collected in steps 1 through 3. The input includes biometric information, image information, and audio information. The server uses a generative AI model to infer the user's health and emotional state from this data.

[0220] Step 5:

[0221] The server generates voice feedback for the user based on the emotion recognition results. The input is data indicating the emotional state, and the output is instructions for the voice feedback to be delivered to the user. The server performs speech synthesis to prepare the feedback with the appropriate tone and content.

[0222] Step 6:

[0223] The user receives generated audio feedback. The device plays the feedback through its speaker and delivers a message to the user to provide reassurance.

[0224] Step 7:

[0225] The server shares information about the user's health and emotional state with family and caregivers in real time. Inputs include analysis results, and outputs generate alerts and notifications that are sent to family and caregiver devices.

[0226] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0227] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0228] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0229] [Second Embodiment]

[0230] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0231] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0232] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0233] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0234] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0236] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0237] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0238] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0239] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0240] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0241] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] The elderly care system of the present invention combines biometric data, image data, and voice data to monitor the user's daily life in real time and manage their health. Specific embodiments are described below.

[0243] First, the sensor devices equipped in the terminal continuously monitor the user's heart rate, body temperature, activity level, etc., and collect biometric data. Next, cameras installed in the living room or living space capture images of the user's movements and environment, acquiring image data. This data is then transmitted to a server via wireless communication.

[0244] The server analyzes biometric and image data in real time. Specifically, it uses image processing algorithms to recognize user movements and postures and detect abnormal behavior. If an abnormality is detected, the server immediately generates an emergency notification and sends it to family members or caregivers. Furthermore, it predicts the user's health status based on the collected biometric data and provides optimal health advice.

[0245] Furthermore, the voice spoken by the user is collected through the device's microphone and analyzed on the server using natural language processing. This makes it possible to understand the user's intent and generate appropriate voice feedback.

[0246] As a concrete example, consider the case of a user falling. The device's camera captures the user's fall, and image data is sent to the server. The server detects the fall through image analysis, generates an emergency notification, and sends it to the family. At the same time, voice feedback can be used to confirm with the user, "Are you injured?", and further instructions can be given as needed.

[0247] This system not only monitors the user's status in real time but also allows for rapid response in case of abnormalities. This helps support the safe living of the elderly and reduces the burden on family members and caregivers.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The device collects biometric data through wearable devices and smart home sensors. This data includes heart rate, body temperature, and activity level. The collected data is transmitted to a server in real time.

[0251] Step 2:

[0252] The device uses its built-in camera to capture image data of the user's movements and environment. For example, it records the user's posture and movement as video and sends the data to a server.

[0253] Step 3:

[0254] The server monitors the user's current health status based on the biometric data it receives. Predictive algorithms are executed to detect abnormal values ​​or changes.

[0255] Step 4:

[0256] The server processes image data and analyzes user behavior and environmental changes. Deep learning technology is used to detect falls and abnormal movements.

[0257] Step 5:

[0258] The device uses its microphone to collect voice data and sends it to the server. Voice input is used to recognize the user's natural speech.

[0259] Step 6:

[0260] The server performs speech recognition and natural language processing to analyze the user's utterances. This allows it to accurately understand the user's requests and instructions.

[0261] Step 7:

[0262] Based on the analysis results, the server will notify family members or caregivers of any anomalies detected in emergencies. Real-time alerts will be sent via email or app notifications.

[0263] Step 8:

[0264] The device uses speech synthesis technology to provide feedback to the user. For example, it can deliver reminders or health advice to the user via voice.

[0265] Step 9:

[0266] Users act according to the advice and instructions provided, which then provides feedback for the next data collection cycle, ensuring continued personalized care.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] There is a need to provide a system that ensures the safe living of the elderly while allowing family members and caregivers in remote locations to monitor their health and potential dangers in real time. However, existing systems lack sufficient real-time capabilities and immediate response to emergencies, making it difficult to monitor them with peace of mind.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes a device for collecting biometric information, a device for acquiring visual information, and a device for processing audio information. This enables real-time monitoring of the user's health status, immediate response to abnormalities, and provision of appropriate feedback.

[0272] "Biometric information" refers to data collected from the human body, such as heart rate, body temperature, and activity level.

[0273] "Visual information" refers to video and image data acquired through cameras and sensors.

[0274] "Voice information" refers to user speech and voice data collected using a microphone.

[0275] A "device that recognizes abnormalities" is a device that uses collected data to detect conditions that are different from the normal state.

[0276] A "device for generating emergency notifications" is a device that promptly issues warnings or notifications to relevant parties when an anomaly is detected.

[0277] A "language processing device" is a device that analyzes speech information as natural language and understands its meaning and intent.

[0278] A "device that provides voice response" is a device that provides appropriate voice feedback to the user based on analyzed information.

[0279] A "device that transmits wirelessly" is a device that uses wireless technology to transmit collected information to other devices or servers without using cables.

[0280] "Supporter" refers to the user's family, caregiver, or someone responsible for their health management.

[0281] "Real-time information sharing" refers to sharing information with people in remote locations in near real-time.

[0282] This invention relates to a system that monitors a user's health status in real time and enables immediate response in the event of an abnormality. The system mainly consists of terminals and servers.

[0283] The device incorporates a sensor device for collecting biometric information, continuously acquiring data such as heart rate, body temperature, and activity level. This sensor device is integrated into typical wearable devices. In addition, the device is equipped with a camera to acquire visual information, monitoring the user's movements and environment. Audio information is collected through the device's microphone. This information is transmitted from the device to a server via wireless communication.

[0284] The server analyzes the received biometric information and visual information in real time. Through the image processing algorithm operating on the server, it accurately recognizes the user's actions and postures, and generates an emergency notification immediately when an abnormality is detected. The emergency notification is sent to family members and supporters via email or SMS. At the same time, it predicts the user's health status based on the biometric information and provides appropriate advice. The voice information is analyzed using natural language processing technology and utilized to understand the user's intentions and requests. Based on the results, a voice response is generated and fed back to the user through the terminal.

[0285] As a specific example, when the user falls, the camera of the terminal captures the situation and sends the image data to the server. The server generates an emergency notification and sends a warning such as "The user has fallen" to the family. At the same time, through the voice response, it can confirm with the user "Are you injured?" and provide further instructions if necessary.

[0286] An example of the prompt text input to the generation AI model is "Please explain in detail the response system when an elderly person falls". With this system, the safety and health of the user are efficiently managed.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The terminal collects biometric information such as the user's heart rate, body temperature, and activity level using a sensor device. The input is the biometric data from the sensor, which is temporarily stored and prepared to be periodically sent to the server. The output is the encrypted biometric information data for transfer to the server. As a specific operation, the wearable device tracks the movements of the user wearing it and updates the data at regular intervals.

[0290] Step 2:

[0291] The device acquires the user's visual information using a camera. The input is real-time video captured by the camera, which is converted into a format suitable for image processing algorithms. The output is processed image data, which is prepared to be sent to the server. Specifically, the camera scans the user's environment at regular intervals and captures changes in movement as needed.

[0292] Step 3:

[0293] The terminal collects user speech using a microphone to acquire audio information. The input is audio data, which is converted into an audio format and configured for speech recognition. The output is an audio file to be later sent to the server. Specifically, it performs noise reduction and necessary amplification processing on the audio to prepare clear audio data.

[0294] Step 4:

[0295] The terminal transfers collected biometric, visual, and audio information to the server using wireless communication. Input consists of various data temporarily stored within the terminal. Output is an encrypted, comprehensive data packet sent to the server via a wireless communication protocol. Specifically, the terminal calculates the optimal timing and sends the data in a batch processing format.

[0296] Step 5:

[0297] The server analyzes the user's health status and behavior using various received information. Inputs are biometric and visual information transmitted from the terminal. Data processing involves applying an anomaly detection algorithm, and if an anomaly is detected, its nature is determined. Outputs include the anomaly detection result and its details. Specifically, it uses image data to perform motion recognition and identify abnormal postures.

[0298] Step 6:

[0299] When an abnormality is recognized, the server immediately generates an emergency notification and sends it to the family members and supporters. The input is the abnormality detection result. The output is an emergency notification message that details the user's situation. As a specific operation, the server sets the priority of the notification and summarizes and distributes the necessary information in the notification text.

[0300] Step 7:

[0301] The server analyzes the voice information using natural language processing technology to understand the user's intention. The input is the voice data transmitted from the terminal. The output is the content of the analyzed intention and request. As a specific operation, the server extracts keywords from the voice and analyzes the context.

[0302] Step 8:

[0303] The server generates an appropriate voice response based on the analysis result and provides feedback to the user through the terminal. The input is the result of the voice analysis, and the output is the generated voice response data. As a specific operation, the server selects the playback voice according to the user's state and issues a playback instruction to the terminal.

[0304] (Application Example 1)

[0305] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0306] There is a need to monitor the health status and safety of the elderly in real time, detect abnormalities early, and respond promptly. However, in existing systems, there is a problem that the technology for comprehensively analyzing individual biometric information, image information, and voice information is not sufficient, making it difficult for caregivers to quickly grasp the situation. Furthermore, there is also a lack of means for caregivers to easily check the status of the elderly.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0308] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to comprehensively analyze multiple pieces of information and detect user abnormalities in real time. Furthermore, caregivers can quickly check the user's condition via a portable device.

[0309] "Biometric information" refers to data that shows the user's physical characteristics, such as heart rate, body temperature, and activity level.

[0310] "Image information" refers to visual data acquired by a camera to understand the user's movements and posture.

[0311] "Audio information" refers to data that includes the voice spoken by the user, and is used to extract meaning and intent.

[0312] "Means for detecting anomalies" refers to the process of analyzing biological and image information to identify unexpected conditions or dangerous situations.

[0313] "Means of generating notifications" refer to actions taken to quickly communicate important information based on the detection of anomalies.

[0314] "Means of language processing" refers to technologies that analyze natural language from audio information to understand the user's intent.

[0315] "Means of providing audio output" refers to the process of conveying information and instructions by voice in a format that is easy for users to understand.

[0316] A "portable device" is a device that caregivers can carry around and use to check information about the user in real time.

[0317] "Means of providing time-series information" refers to the process of managing and presenting a series of information, including historical data.

[0318] This invention is a system for monitoring the health status and daily life of elderly people in real time. The system mainly consists of a terminal, a server, and a group of sensors installed in the user's living space.

[0319] The device is equipped with sensors that continuously collect biometric information such as heart rate and body temperature. This biometric information is transmitted to the server via wireless communication. Cameras are installed in the living space to capture image information of the user's movements and posture. Similarly, the device's microphone collects voice information emitted by the user. The server processes this data in real time to detect anomalies. In particular, image processing libraries such as OpenCV and scikit-learn are used to analyze the user's movements.

[0320] If an anomaly is detected, the server generates an emergency notification and sends it to the caregiver via a portable device. This portable device is a smartphone or smart glasses, allowing the caregiver to check it immediately. Furthermore, based on collected biometric information, the system predicts the user's health status and provides voice feedback. This voice feedback is generated using natural language processing, specifically Google Dialogflow and Microsoft Azure's NLP capabilities.

[0321] For example, if a user falls, the server analyzes the camera data, immediately sends a notification to the caregiver, and plays a voice message to the user asking, "Are you injured?" This enables a quick response.

[0322] An example of a prompt is: "Generate a notification and follow-up scenario regarding fall detection in elderly individuals. Create notification messages and voice prompts, and describe in detail the steps to take after a fall is detected."

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensors. This biometric information is then transmitted to the server using wireless communication. At this stage, the data is still in raw format.

[0326] Step 2:

[0327] Cameras installed in the living space capture the user's movements and posture, acquiring image information. The captured images become input and are immediately sent to the server. At this point, the system is ready for image processing algorithms to be applied for motion recognition.

[0328] Step 3:

[0329] The device's microphone collects the user's voice and sends it to the server as audio data. The audio data becomes the input, and the server is ready to apply natural language processing.

[0330] Step 4:

[0331] The server initiates real-time analysis using biometric and image data. An algorithm for detecting anomalies in biometric data is applied using scikit-learn. Simultaneously, image data is analyzed using OpenCV to detect abnormal behavior. If an anomaly is detected, data reporting the anomaly is generated as output.

[0332] Step 5:

[0333] If an anomaly is detected, the server immediately generates an emergency notification. The notification data is used as input and output to a portable device. This notification is quickly sent to the caregiver's smartphone or glasses using the NF communication protocol, alerting the caregiver.

[0334] Step 6:

[0335] The server performs natural language processing based on the collected audio information. Using Google Dialogflow and Microsoft Azure's NLP capabilities, it analyzes the audio data to understand the user's intent. The resulting audio feedback is then sent to the device as output.

[0336] Step 7:

[0337] When a user needs feedback in an abnormal situation, the server generates voice feedback and provides it to the user via the terminal. For example, when a fall is detected, a voice message such as "Are you injured?" is output. This output allows the user to receive instructions appropriate to the situation.

[0338] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0339] The elderly care system of the present invention not only comprehensively monitors the user's daily life by combining biometric data, image data, and voice data, but also recognizes the user's emotional state in real time using an additional emotion engine and provides appropriate feedback based on that.

[0340] First, the device uses sensor devices to collect biometric data such as the user's heart rate, body temperature, and activity level. Next, cameras installed in the room record the user's movements and facial expressions, acquiring image data. This image data plays a crucial role in analyzing the user's facial expressions. In addition, the device's microphone captures audio data, accurately recording the user's speech.

[0341] All of this data is transmitted to the server in real time. The server combines and analyzes the biometric and image data to detect abnormalities in the user's health and behavior. The emotion engine uses image and audio data to analyze the user's facial expressions and tone of voice to recognize the user's emotions. For example, if a user is stressed, this is detected as a change in facial expression and tone of voice.

[0342] Based on the identified emotions, the server adjusts the voice feedback to the user to provide more appropriate communication. For example, if the user is feeling anxious, the server provides feedback in a calm and reassuring tone. Furthermore, by sharing the user's emotional state with family members and caregivers, the server supports the provision of more appropriate care.

[0343] As a concrete example, consider a case where a user is feeling unwell and anxious. Biometric data indicates an abnormality, and the emotion engine detects anxiety from the user's facial expressions and voice. Based on this information, the server generates an emergency notification and sends an alert to family members. At the same time, the device plays a reassuring message to the user through voice feedback, such as, "Are you okay? If you're worried, we'll call support right away."

[0344] As a result, the present invention not only ensures the safety and security of users, but also enables meticulous care that takes their emotions into consideration.

[0345] The following describes the processing flow.

[0346] Step 1:

[0347] The terminal collects biometric data from the user's wearable device. This data includes heart rate, body temperature, and activity level, and is transferred to a server using a secure communication protocol.

[0348] Step 2:

[0349] A camera installed on the terminal records video of the room, capturing the user's movements and facial expressions. The video data is transmitted to the server in real time. This allows for continuous monitoring of the user's posture and changes in the environment.

[0350] Step 3:

[0351] The device acquires user voice data via its microphone. It captures the user's everyday conversations and speech content and accurately transmits it to the server.

[0352] Step 4:

[0353] The server analyzes the received biometric data and determines whether it falls within the normal range. If an abnormal value is detected, an alert is immediately generated and preparations are made.

[0354] Step 5:

[0355] The server analyzes image and audio data and uses an emotion engine to determine the user's emotional state. It analyzes changes in facial expressions from the image data and the tone and volume of the voice from the audio data to identify emotions.

[0356] Step 6:

[0357] Based on the results of anomaly detection and emotion assessment, the server decides whether to send an emergency notification to family members or caregivers. Information is shared via email or push notifications as needed.

[0358] Step 7:

[0359] The server adjusts the voice feedback it provides to the user, generating feedback in an appropriate tone and content based on the user's emotions. For example, if anxiety is detected, it selects a calming message to reassure the user.

[0360] Step 8:

[0361] The device uses speech synthesis technology to provide the user with voice feedback generated from the server. It then provides necessary support through interaction with the user.

[0362] Step 9:

[0363] Users react to voice feedback and adjust their daily behaviors accordingly. This allows the entire system to receive useful feedback in the next observation cycle, improving the quality of care provided to the user.

[0364] (Example 2)

[0365] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0366] In elderly care, there is a need to monitor individual health and emotional states in real time and to respond quickly and appropriately when abnormalities are detected. However, conventional systems have difficulty providing integrated and immediate feedback of biometric and emotional data, making it challenging to adequately ensure user confidence.

[0367] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0368] In this invention, the server includes means for analyzing biometric information to detect abnormalities, means for performing emotion analysis and adjusting feedback according to the emotional state, and means for generating voice feedback based on the user's emotional state using a generative AI model. This enables a comprehensive understanding of the health and emotional state of elderly individuals, and allows for rapid and appropriate feedback tailored to individual circumstances.

[0369] "Biometric information" refers to numerical values ​​or data that indicate a person's physical state, such as heart rate, body temperature, and activity level.

[0370] "Image information" refers to visual data used to capture the user's facial expressions and movements.

[0371] "Audio information" refers to acoustic data that includes the user's voice, including the content of their speech and the tone of their voice.

[0372] "Anomaly detection" refers to the process of analyzing biological and image information to identify abnormal physical or behavioral conditions.

[0373] An "emergency notification" is a notification that is immediately sent to family members or caregivers in an emergency to ensure the user's safety.

[0374] "Speech analysis" is a process that uses natural language processing based on speech information to understand the content of speech and the tone of voice.

[0375] "Voice response" refers to voice feedback provided to the user, intended to convey information or soothe emotions.

[0376] "Emotional analysis" is a technology that identifies a user's emotional state by analyzing their facial expressions and tone of voice.

[0377] A "generative AI model" is a model that uses artificial intelligence techniques to generate output in response to a specific input.

[0378] This invention is a system for elderly care that monitors the user's daily health and emotional state in real time and provides appropriate responses. The system mainly consists of a "terminal" and a "server".

[0379] The device is hardware that is worn directly by the user or installed in their room, and is equipped with sensors for collecting biometric information (e.g., heart rate monitor and temperature sensor), a camera for recording movements and facial expressions, and a microphone for capturing audio. This device transmits the collected data to a server in real time.

[0380] The server integrates and processes received biometric, image, and audio information. Biometric information is used for detecting abnormalities in health status, image information is particularly used for analyzing the user's facial expressions, and audio information is used to analyze speech content and tone. The server also incorporates an emotion analysis engine and a generative AI model, which are used to analyze the user's emotional state and generate appropriate audio feedback based on the results. The generative AI model can receive prompts such as: "The user appears anxious; please generate reassuring words."

[0381] As a concrete example, if a user feels unwell and anxious, the device records an abnormally high heart rate and sends it to the server. Simultaneously, the camera captures the user's troubled expression, and the microphone records a tense voice. The server comprehensively analyzes this data, and if it detects both physical abnormality and anxiety, it uses a generative AI model to generate voice feedback such as, "Are you okay? We'll call support right away, so please don't worry," and conveys it to the user via the device.

[0382] This system allows users to centrally manage their health and emotional state and receive prompt and appropriate care as needed. Furthermore, family members and caregivers can monitor the user's condition in real time, enabling them to provide better support.

[0383] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0384] Step 1:

[0385] The device collects biometric information from the user. Specifically, it collects data such as heart rate, body temperature, and activity level using heart rate monitors and temperature sensors. The input is a digital signal from the biosensor, which is converted into a visualizeable format within the device. The output is formatted biometric data.

[0386] Step 2:

[0387] The device acquires image information using a camera installed in the room. This allows the user's movements and facial expressions to be captured in real time. The input is visual data received by the camera sensor, which is then processed and converted into a format that is easy to analyze. The output is the processed image data.

[0388] Step 3:

[0389] The device captures audio information using a microphone. The user's speech and tone are recorded, forming the basis for speech recognition. The input is an audio signal, which undergoes noise reduction and sampling. The output is clean audio data.

[0390] Step 4:

[0391] The device transmits all collected biometric, image, and audio information to the server in real time. Input is integrated user data transmitted using a secure protocol. Output is the data batch received by the server.

[0392] Step 5:

[0393] The server analyzes received biometric data to detect anomalies. Specifically, it analyzes sudden increases in heart rate and fluctuations in body temperature. The input is received biometric data, and data mining techniques are used to identify anomaly patterns. The output is an anomaly detection flag.

[0394] Step 6:

[0395] The server performs sentiment analysis using image and audio information. It uses facial expression recognition technology and voice sentiment analysis to determine the user's emotional state. The input consists of features extracted from image and audio data, and a machine learning model is used to identify emotion categories. The output is a label representing the emotional state.

[0396] Step 7:

[0397] The server uses an AI model to generate voice feedback that responds to the user's emotions. A prompt is input to the model, and a feedback message is output. The input is a prompt such as, "The user is feeling anxious; please generate a reassuring message." The output is the generated voice response message.

[0398] Step 8:

[0399] The device receives feedback from the server and provides voice feedback to the user. The input is a generated voice message, which is transmitted to the user through the speaker. The output is the voice feedback that the user hears.

[0400] (Application Example 2)

[0401] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0402] In elderly care, it is necessary not only to monitor health status but also to understand emotional states in real time. However, conventional technologies have made it difficult to comprehensively analyze biometric information, images, and audio data to provide appropriate feedback tailored to the emotional state of individual users. Furthermore, there has been a challenge in sharing timely and effective information with family members and caregivers, which prevents prompt and appropriate responses.

[0403] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0404] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to analyze the user's health and emotional state in real time and quickly share necessary information with family members and caregivers.

[0405] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and activity level.

[0406] "Image information" refers to visual data of a user's face and movements acquired using a camera.

[0407] "Voice information" refers to data about the user's speech content and tone collected by a microphone.

[0408] "Emotion recognition means" refers to an algorithm that analyzes image and audio information to identify the user's emotional state.

[0409] An "emergency notification" is an alert message generated when an abnormality is detected, and is sent to family members or caregivers.

[0410] "Voice feedback" refers to reactions or instructions provided to the user via voice.

[0411] "Natural language processing" is a technology that analyzes speech information to understand the intent behind a user's utterances.

[0412] "Means of sharing emotional states with family and caregivers" refers to methods for transmitting a user's emotional information to relevant parties in real time.

[0413] The system for implementing this invention comprehensively collects and analyzes the user's biometric information, image information, and voice information to monitor the user's health and emotional state in real time and provide necessary information to family members and caregivers.

[0414] The server runs a program that collects biometric information such as heart rate, body temperature, and activity level from sensor devices. This data is necessary for analyzing the user's health status. Furthermore, it uses a camera to acquire image information of the user and analyzes the user's movements and facial expressions. Image recognition algorithms are used for this analysis. Audio information is acquired via a microphone, and natural language processing technology is used to analyze the user's speech content and tone.

[0415] The server integrates this data and uses a generative AI model to recognize the user's emotions. Once the emotional state is determined, it generates voice feedback based on the result and provides appropriate feedback to the user. For example, if the user is feeling anxious, it communicates with them in a calm voice to provide reassurance. Furthermore, if emotional information is important, it is shared in real time with family members or caregivers.

[0416] For example, suppose the system detects surprise from the user's facial expression along with an increase in heart rate while the user is in the living room. The server analyzes this information and provides voice feedback such as, "Calm down, take a deep breath," while simultaneously sending a notification to the caregiver.

[0417] An example of a prompt message would be, "If an elderly person experiences anxiety, how would you provide feedback using heart rate, facial expression, and voice data?"

[0418] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0419] Step 1:

[0420] The device collects biometric information from sensor devices. This input data includes heart rate, body temperature, and activity level. The device records the digital biometric data and prepares to send it to the server.

[0421] Step 2:

[0422] Using the camera, the device acquires image information of the user. This input consists of image data of the user's face and body movements. The device applies an image recognition algorithm to extract features for analyzing facial expressions and movements.

[0423] Step 3:

[0424] The device acquires voice information using a microphone. The input data includes the user's speech content and voice tone. The device uses natural language processing technology to convert the voice data into text and perform voice tone analysis.

[0425] Step 4:

[0426] The server integrates the data collected in steps 1 through 3. The input includes biometric information, image information, and audio information. The server uses a generative AI model to infer the user's health and emotional state from this data.

[0427] Step 5:

[0428] The server generates voice feedback for the user based on the emotion recognition results. The input is data indicating the emotional state, and the output is instructions for the voice feedback to be delivered to the user. The server performs speech synthesis to prepare the feedback with the appropriate tone and content.

[0429] Step 6:

[0430] The user receives generated audio feedback. The device plays the feedback through its speaker and delivers a message to the user to provide reassurance.

[0431] Step 7:

[0432] The server shares information about the user's health and emotional state with family and caregivers in real time. Inputs include analysis results, and outputs generate alerts and notifications that are sent to family and caregiver devices.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] The elderly care system of the present invention combines biometric data, image data, and voice data to monitor the user's daily life in real time and manage their health. Specific embodiments are described below.

[0450] First, the sensor devices equipped in the terminal continuously monitor the user's heart rate, body temperature, activity level, etc., and collect biometric data. Next, cameras installed in the living room or living space capture images of the user's movements and environment, acquiring image data. This data is then transmitted to a server via wireless communication.

[0451] The server analyzes biometric and image data in real time. Specifically, it uses image processing algorithms to recognize user movements and postures and detect abnormal behavior. If an abnormality is detected, the server immediately generates an emergency notification and sends it to family members or caregivers. Furthermore, it predicts the user's health status based on the collected biometric data and provides optimal health advice.

[0452] Furthermore, the voice spoken by the user is collected through the device's microphone and analyzed on the server using natural language processing. This makes it possible to understand the user's intent and generate appropriate voice feedback.

[0453] As a concrete example, consider the case of a user falling. The device's camera captures the user's fall, and image data is sent to the server. The server detects the fall through image analysis, generates an emergency notification, and sends it to the family. At the same time, voice feedback can be used to confirm with the user, "Are you injured?", and further instructions can be given as needed.

[0454] This system not only monitors the user's status in real time but also allows for rapid response in case of abnormalities. This helps support the safe living of the elderly and reduces the burden on family members and caregivers.

[0455] The following describes the processing flow.

[0456] Step 1:

[0457] The device collects biometric data through wearable devices and smart home sensors. This data includes heart rate, body temperature, and activity level. The collected data is transmitted to a server in real time.

[0458] Step 2:

[0459] The device uses its built-in camera to capture image data of the user's movements and environment. For example, it records the user's posture and movement as video and sends the data to a server.

[0460] Step 3:

[0461] The server monitors the user's current health status based on the biometric data it receives. Predictive algorithms are executed to detect abnormal values ​​or changes.

[0462] Step 4:

[0463] The server processes image data and analyzes user behavior and environmental changes. Deep learning technology is used to detect falls and abnormal movements.

[0464] Step 5:

[0465] The device uses its microphone to collect voice data and sends it to the server. Voice input is used to recognize the user's natural speech.

[0466] Step 6:

[0467] The server performs speech recognition and natural language processing to analyze the user's utterances. This allows it to accurately understand the user's requests and instructions.

[0468] Step 7:

[0469] Based on the analysis results, the server will notify family members or caregivers of any anomalies detected in emergencies. Real-time alerts will be sent via email or app notifications.

[0470] Step 8:

[0471] The device uses speech synthesis technology to provide feedback to the user. For example, it can deliver reminders or health advice to the user via voice.

[0472] Step 9:

[0473] Users act according to the advice and instructions provided, which then provides feedback for the next data collection cycle, ensuring continued personalized care.

[0474] (Example 1)

[0475] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] There is a need to provide a system that ensures the safe living of the elderly while allowing family members and caregivers in remote locations to monitor their health and potential dangers in real time. However, existing systems lack sufficient real-time capabilities and immediate response to emergencies, making it difficult to monitor them with peace of mind.

[0477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0478] In this invention, the server includes a device for collecting biometric information, a device for acquiring visual information, and a device for processing audio information. This enables real-time monitoring of the user's health status, immediate response to abnormalities, and provision of appropriate feedback.

[0479] "Biometric information" refers to data collected from the human body, such as heart rate, body temperature, and activity level.

[0480] "Visual information" refers to video and image data acquired through cameras and sensors.

[0481] "Voice information" refers to user speech and voice data collected using a microphone.

[0482] A "device that recognizes abnormalities" is a device that uses collected data to detect conditions that are different from the normal state.

[0483] A "device for generating emergency notifications" is a device that promptly issues warnings or notifications to relevant parties when an anomaly is detected.

[0484] A "language processing device" is a device that analyzes speech information as natural language and understands its meaning and intent.

[0485] A "device that provides voice response" is a device that provides appropriate voice feedback to the user based on analyzed information.

[0486] A "device that transmits wirelessly" is a device that uses wireless technology to transmit collected information to other devices or servers without using cables.

[0487] "Supporter" refers to the user's family, caregiver, or someone responsible for their health management.

[0488] "Real-time information sharing" refers to sharing information with people in remote locations in near real-time.

[0489] This invention relates to a system that monitors a user's health status in real time and enables immediate response in the event of an abnormality. The system mainly consists of terminals and servers.

[0490] The device incorporates a sensor device for collecting biometric information, continuously acquiring data such as heart rate, body temperature, and activity level. This sensor device is integrated into typical wearable devices. In addition, the device is equipped with a camera to acquire visual information, monitoring the user's movements and environment. Audio information is collected through the device's microphone. This information is transmitted from the device to a server via wireless communication.

[0491] The server analyzes received biometric and visual information in real time. Image processing algorithms running on the server accurately recognize the user's movements and posture, and immediately generate an emergency notification if an anomaly is detected. The emergency notification is sent to family members or caregivers via email or SMS. At the same time, the system predicts the user's health status based on biometric information and provides appropriate advice. Voice information is analyzed using natural language processing technology and used to understand the user's intentions and requests. Based on the results, a voice response is generated and fed back to the user through the terminal.

[0492] For example, if a user falls, the device's camera captures the situation and sends image data to the server. The server generates an emergency notification and sends a warning to family members such as, "The user has fallen." At the same time, it can check with the user via voice response, asking, "Are you injured?" and provide further instructions as needed.

[0493] An example of a prompt to input into the generating AI model is, "Please describe in detail the response system for when an elderly person falls." This system will efficiently manage the safety and health of users.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensor devices. The input is biometric data from the sensors, which is temporarily stored and prepared for periodic transmission to the server. The output is encrypted biometric data for transfer to the server. Specifically, the wearable device tracks the movements of the user wearing it and updates the data at regular intervals.

[0497] Step 2:

[0498] The device acquires the user's visual information using a camera. The input is real-time video captured by the camera, which is converted into a format suitable for image processing algorithms. The output is processed image data, which is prepared to be sent to the server. Specifically, the camera scans the user's environment at regular intervals and captures changes in movement as needed.

[0499] Step 3:

[0500] The terminal collects user speech using a microphone to acquire audio information. The input is audio data, which is converted into an audio format and configured for speech recognition. The output is an audio file to be later sent to the server. Specifically, it performs noise reduction and necessary amplification processing on the audio to prepare clear audio data.

[0501] Step 4:

[0502] The terminal transfers collected biometric, visual, and audio information to the server using wireless communication. Input consists of various data temporarily stored within the terminal. Output is an encrypted, comprehensive data packet sent to the server via a wireless communication protocol. Specifically, the terminal calculates the optimal timing and sends the data in a batch processing format.

[0503] Step 5:

[0504] The server analyzes the user's health status and behavior using various received information. Inputs are biometric and visual information transmitted from the terminal. Data processing involves applying an anomaly detection algorithm, and if an anomaly is detected, its nature is determined. Outputs include the anomaly detection result and its details. Specifically, it uses image data to perform motion recognition and identify abnormal postures.

[0505] Step 6:

[0506] The server immediately generates an emergency notification and sends it to family members or caregivers upon detecting an anomaly. The input is the anomaly detection result. The output is an emergency notification message that provides a detailed explanation of the user's situation. Specifically, it prioritizes notifications and delivers the necessary information in a single notification message.

[0507] Step 7:

[0508] The server analyzes voice information using natural language processing techniques to understand the user's intent. The input is voice data transmitted from the terminal. The output is the analyzed intent and request content. Specifically, it extracts keywords from the voice and analyzes their context.

[0509] Step 8:

[0510] The server generates an appropriate voice response based on the analysis results and provides feedback to the user through the terminal. The input is the result of the voice analysis, and the output is the generated voice response data. Specifically, it selects the appropriate voice to play based on the user's state and issues a playback command on the terminal.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] There is a need to monitor the health and safety of the elderly in real time, detect abnormalities early, and respond quickly. However, existing systems lack sufficient technology to integrate and analyze individual biometric, image, and audio information, making it difficult for caregivers to quickly grasp the situation. Furthermore, there is a lack of easy ways for caregivers to check the condition of the elderly.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to comprehensively analyze multiple pieces of information and detect user abnormalities in real time. Furthermore, caregivers can quickly check the user's condition via a portable device.

[0516] "Biometric information" refers to data that shows the user's physical characteristics, such as heart rate, body temperature, and activity level.

[0517] "Image information" refers to visual data acquired by a camera to understand the user's movements and posture.

[0518] "Audio information" refers to data that includes the voice spoken by the user, and is used to extract meaning and intent.

[0519] "Means for detecting anomalies" refers to the process of analyzing biological and image information to identify unexpected conditions or dangerous situations.

[0520] "Means of generating notifications" refer to actions taken to quickly communicate important information based on the detection of anomalies.

[0521] "Means of language processing" refers to technologies that analyze natural language from audio information to understand the user's intent.

[0522] "Means of providing audio output" refers to the process of conveying information and instructions by voice in a format that is easy for users to understand.

[0523] A "portable device" is a device that caregivers can carry around and use to check information about the user in real time.

[0524] "Means of providing time-series information" refers to the process of managing and presenting a series of information, including historical data.

[0525] This invention is a system for monitoring the health status and daily life of elderly people in real time. The system mainly consists of a terminal, a server, and a group of sensors installed in the user's living space.

[0526] The device is equipped with sensors that continuously collect biometric information such as heart rate and body temperature. This biometric information is transmitted to the server via wireless communication. Cameras are installed in the living space to capture image information of the user's movements and posture. Similarly, the device's microphone collects voice information emitted by the user. The server processes this data in real time to detect anomalies. In particular, image processing libraries such as OpenCV and scikit-learn are used to analyze the user's movements.

[0527] If an anomaly is detected, the server generates an emergency notification and sends it to the caregiver via a portable device. This portable device is a smartphone or smart glasses, allowing the caregiver to check it immediately. Furthermore, based on collected biometric information, the system predicts the user's health status and provides voice feedback. This voice feedback is generated using natural language processing, specifically Google Dialogflow and Microsoft Azure's NLP capabilities.

[0528] For example, if a user falls, the server analyzes the camera data, immediately sends a notification to the caregiver, and plays a voice message to the user asking, "Are you injured?" This enables a quick response.

[0529] An example of a prompt is: "Generate a notification and follow-up scenario regarding fall detection in elderly individuals. Create notification messages and voice prompts, and describe in detail the steps to take after a fall is detected."

[0530] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0531] Step 1:

[0532] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensors. This biometric information is then transmitted to the server using wireless communication. At this stage, the data is still in raw format.

[0533] Step 2:

[0534] Cameras installed in the living space capture the user's movements and posture, acquiring image information. The captured images become input and are immediately sent to the server. At this point, the system is ready for image processing algorithms to be applied for motion recognition.

[0535] Step 3:

[0536] The device's microphone collects the user's voice and sends it to the server as audio data. The audio data becomes the input, and the server is ready to apply natural language processing.

[0537] Step 4:

[0538] The server initiates real-time analysis using biometric and image data. An algorithm for detecting anomalies in biometric data is applied using scikit-learn. Simultaneously, image data is analyzed using OpenCV to detect abnormal behavior. If an anomaly is detected, data reporting the anomaly is generated as output.

[0539] Step 5:

[0540] If an anomaly is detected, the server immediately generates an emergency notification. The notification data is used as input and output to a portable device. This notification is quickly sent to the caregiver's smartphone or glasses using the NF communication protocol, alerting the caregiver.

[0541] Step 6:

[0542] The server performs natural language processing based on the collected audio information. Using Google Dialogflow and Microsoft Azure's NLP capabilities, it analyzes the audio data to understand the user's intent. The resulting audio feedback is then sent to the device as output.

[0543] Step 7:

[0544] When a user needs feedback in an abnormal situation, the server generates voice feedback and provides it to the user via the terminal. For example, when a fall is detected, a voice message such as "Are you injured?" is output. This output allows the user to receive instructions appropriate to the situation.

[0545] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0546] The elderly care system of the present invention not only comprehensively monitors the user's daily life by combining biometric data, image data, and voice data, but also recognizes the user's emotional state in real time using an additional emotion engine and provides appropriate feedback based on that.

[0547] First, the device uses sensor devices to collect biometric data such as the user's heart rate, body temperature, and activity level. Next, cameras installed in the room record the user's movements and facial expressions, acquiring image data. This image data plays a crucial role in analyzing the user's facial expressions. In addition, the device's microphone captures audio data, accurately recording the user's speech.

[0548] All of this data is transmitted to the server in real time. The server combines and analyzes the biometric and image data to detect abnormalities in the user's health and behavior. The emotion engine uses image and audio data to analyze the user's facial expressions and tone of voice to recognize the user's emotions. For example, if a user is stressed, this is detected as a change in facial expression and tone of voice.

[0549] Based on the identified emotions, the server adjusts the voice feedback to the user to provide more appropriate communication. For example, if the user is feeling anxious, the server provides feedback in a calm and reassuring tone. Furthermore, by sharing the user's emotional state with family members and caregivers, the server supports the provision of more appropriate care.

[0550] As a concrete example, consider a case where a user is feeling unwell and anxious. Biometric data indicates an abnormality, and the emotion engine detects anxiety from the user's facial expressions and voice. Based on this information, the server generates an emergency notification and sends an alert to family members. At the same time, the device plays a reassuring message to the user through voice feedback, such as, "Are you okay? If you're worried, we'll call support right away."

[0551] As a result, the present invention not only ensures the safety and security of users, but also enables meticulous care that takes their emotions into consideration.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] The terminal collects biometric data from the user's wearable device. This data includes heart rate, body temperature, and activity level, and is transferred to a server using a secure communication protocol.

[0555] Step 2:

[0556] A camera installed on the terminal records video of the room, capturing the user's movements and facial expressions. The video data is transmitted to the server in real time. This allows for continuous monitoring of the user's posture and changes in the environment.

[0557] Step 3:

[0558] The device acquires user voice data via its microphone. It captures the user's everyday conversations and speech content and accurately transmits it to the server.

[0559] Step 4:

[0560] The server analyzes the received biometric data and determines whether it falls within the normal range. If an abnormal value is detected, an alert is immediately generated and preparations are made.

[0561] Step 5:

[0562] The server analyzes image and audio data and uses an emotion engine to determine the user's emotional state. It analyzes changes in facial expressions from the image data and the tone and volume of the voice from the audio data to identify emotions.

[0563] Step 6:

[0564] Based on the results of anomaly detection and emotion assessment, the server decides whether to send an emergency notification to family members or caregivers. Information is shared via email or push notifications as needed.

[0565] Step 7:

[0566] The server adjusts the voice feedback it provides to the user, generating feedback in an appropriate tone and content based on the user's emotions. For example, if anxiety is detected, it selects a calming message to reassure the user.

[0567] Step 8:

[0568] The device uses speech synthesis technology to provide the user with voice feedback generated from the server. It then provides necessary support through interaction with the user.

[0569] Step 9:

[0570] Users react to voice feedback and adjust their daily behaviors accordingly. This allows the entire system to receive useful feedback in the next observation cycle, improving the quality of care provided to the user.

[0571] (Example 2)

[0572] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] In elderly care, there is a need to monitor individual health and emotional states in real time and to respond quickly and appropriately when abnormalities are detected. However, conventional systems have difficulty providing integrated and immediate feedback of biometric and emotional data, making it challenging to adequately ensure user confidence.

[0574] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0575] In this invention, the server includes means for analyzing biometric information to detect abnormalities, means for performing emotion analysis and adjusting feedback according to the emotional state, and means for generating voice feedback based on the user's emotional state using a generative AI model. This enables a comprehensive understanding of the health and emotional state of elderly individuals, and allows for rapid and appropriate feedback tailored to individual circumstances.

[0576] "Biometric information" refers to numerical values ​​or data that indicate a person's physical state, such as heart rate, body temperature, and activity level.

[0577] "Image information" refers to visual data used to capture the user's facial expressions and movements.

[0578] "Audio information" refers to acoustic data that includes the user's voice, including the content of their speech and the tone of their voice.

[0579] "Anomaly detection" refers to the process of analyzing biological and image information to identify abnormal physical or behavioral conditions.

[0580] An "emergency notification" is a notification that is immediately sent to family members or caregivers in an emergency to ensure the user's safety.

[0581] "Speech analysis" is a process that uses natural language processing based on speech information to understand the content of speech and the tone of voice.

[0582] "Voice response" refers to voice feedback provided to the user, intended to convey information or soothe emotions.

[0583] "Emotional analysis" is a technology that identifies a user's emotional state by analyzing their facial expressions and tone of voice.

[0584] A "generative AI model" is a model that uses artificial intelligence techniques to generate output in response to a specific input.

[0585] This invention is a system for elderly care that monitors the user's daily health and emotional state in real time and provides appropriate responses. The system mainly consists of a "terminal" and a "server".

[0586] The device is hardware that is worn directly by the user or installed in their room, and is equipped with sensors for collecting biometric information (e.g., heart rate monitor and temperature sensor), a camera for recording movements and facial expressions, and a microphone for capturing audio. This device transmits the collected data to a server in real time.

[0587] The server integrates and processes received biometric, image, and audio information. Biometric information is used for detecting abnormalities in health status, image information is particularly used for analyzing the user's facial expressions, and audio information is used to analyze speech content and tone. The server also incorporates an emotion analysis engine and a generative AI model, which are used to analyze the user's emotional state and generate appropriate audio feedback based on the results. The generative AI model can receive prompts such as: "The user appears anxious; please generate reassuring words."

[0588] As a concrete example, if a user feels unwell and anxious, the device records an abnormally high heart rate and sends it to the server. Simultaneously, the camera captures the user's troubled expression, and the microphone records a tense voice. The server comprehensively analyzes this data, and if it detects both physical abnormality and anxiety, it uses a generative AI model to generate voice feedback such as, "Are you okay? We'll call support right away, so please don't worry," and conveys it to the user via the device.

[0589] This system allows users to centrally manage their health and emotional state and receive prompt and appropriate care as needed. Furthermore, family members and caregivers can monitor the user's condition in real time, enabling them to provide better support.

[0590] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0591] Step 1:

[0592] The device collects biometric information from the user. Specifically, it collects data such as heart rate, body temperature, and activity level using heart rate monitors and temperature sensors. The input is a digital signal from the biosensor, which is converted into a visualizeable format within the device. The output is formatted biometric data.

[0593] Step 2:

[0594] The device acquires image information using a camera installed in the room. This allows the user's movements and facial expressions to be captured in real time. The input is visual data received by the camera sensor, which is then processed and converted into a format that is easy to analyze. The output is the processed image data.

[0595] Step 3:

[0596] The device captures audio information using a microphone. The user's speech and tone are recorded, forming the basis for speech recognition. The input is an audio signal, which undergoes noise reduction and sampling. The output is clean audio data.

[0597] Step 4:

[0598] The device transmits all collected biometric, image, and audio information to the server in real time. Input is integrated user data transmitted using a secure protocol. Output is the data batch received by the server.

[0599] Step 5:

[0600] The server analyzes received biometric data to detect anomalies. Specifically, it analyzes sudden increases in heart rate and fluctuations in body temperature. The input is received biometric data, and data mining techniques are used to identify anomaly patterns. The output is an anomaly detection flag.

[0601] Step 6:

[0602] The server performs sentiment analysis using image and audio information. It uses facial expression recognition technology and voice sentiment analysis to determine the user's emotional state. The input consists of features extracted from image and audio data, and a machine learning model is used to identify emotion categories. The output is a label representing the emotional state.

[0603] Step 7:

[0604] The server uses an AI model to generate voice feedback that responds to the user's emotions. A prompt is input to the model, and a feedback message is output. The input is a prompt such as, "The user is feeling anxious; please generate a reassuring message." The output is the generated voice response message.

[0605] Step 8:

[0606] The device receives feedback from the server and provides voice feedback to the user. The input is a generated voice message, which is transmitted to the user through the speaker. The output is the voice feedback that the user hears.

[0607] (Application Example 2)

[0608] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0609] In elderly care, it is necessary not only to monitor health status but also to understand emotional states in real time. However, conventional technologies have made it difficult to comprehensively analyze biometric information, images, and audio data to provide appropriate feedback tailored to the emotional state of individual users. Furthermore, there has been a challenge in sharing timely and effective information with family members and caregivers, which prevents prompt and appropriate responses.

[0610] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0611] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to analyze the user's health and emotional state in real time and quickly share necessary information with family members and caregivers.

[0612] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and activity level.

[0613] "Image information" refers to visual data of a user's face and movements acquired using a camera.

[0614] "Voice information" refers to data about the user's speech content and tone collected by a microphone.

[0615] "Emotion recognition means" refers to an algorithm that analyzes image and audio information to identify the user's emotional state.

[0616] An "emergency notification" is an alert message generated when an abnormality is detected, and is sent to family members or caregivers.

[0617] "Voice feedback" refers to reactions or instructions provided to the user via voice.

[0618] "Natural language processing" is a technology that analyzes speech information to understand the intent behind a user's utterances.

[0619] "Means of sharing emotional states with family and caregivers" refers to methods for transmitting a user's emotional information to relevant parties in real time.

[0620] The system for implementing this invention comprehensively collects and analyzes the user's biometric information, image information, and voice information to monitor the user's health and emotional state in real time and provide necessary information to family members and caregivers.

[0621] The server runs a program that collects biometric information such as heart rate, body temperature, and activity level from sensor devices. This data is necessary for analyzing the user's health status. Furthermore, it uses a camera to acquire image information of the user and analyzes the user's movements and facial expressions. Image recognition algorithms are used for this analysis. Audio information is acquired via a microphone, and natural language processing technology is used to analyze the user's speech content and tone.

[0622] The server integrates this data and uses a generative AI model to recognize the user's emotions. Once the emotional state is determined, it generates voice feedback based on the result and provides appropriate feedback to the user. For example, if the user is feeling anxious, it communicates with them in a calm voice to provide reassurance. Furthermore, if emotional information is important, it is shared in real time with family members or caregivers.

[0623] For example, suppose the system detects surprise from the user's facial expression along with an increase in heart rate while the user is in the living room. The server analyzes this information and provides voice feedback such as, "Calm down, take a deep breath," while simultaneously sending a notification to the caregiver.

[0624] An example of a prompt message would be, "If an elderly person experiences anxiety, how would you provide feedback using heart rate, facial expression, and voice data?"

[0625] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0626] Step 1:

[0627] The device collects biometric information from sensor devices. This input data includes heart rate, body temperature, and activity level. The device records the digital biometric data and prepares to send it to the server.

[0628] Step 2:

[0629] Using the camera, the device acquires image information of the user. This input consists of image data of the user's face and body movements. The device applies an image recognition algorithm to extract features for analyzing facial expressions and movements.

[0630] Step 3:

[0631] The device acquires voice information using a microphone. The input data includes the user's speech content and voice tone. The device uses natural language processing technology to convert the voice data into text and perform voice tone analysis.

[0632] Step 4:

[0633] The server integrates the data collected in steps 1 through 3. The input includes biometric information, image information, and audio information. The server uses a generative AI model to infer the user's health and emotional state from this data.

[0634] Step 5:

[0635] The server generates voice feedback for the user based on the emotion recognition results. The input is data indicating the emotional state, and the output is instructions for the voice feedback to be delivered to the user. The server performs speech synthesis to prepare the feedback with the appropriate tone and content.

[0636] Step 6:

[0637] The user receives generated audio feedback. The device plays the feedback through its speaker and delivers a message to the user to provide reassurance.

[0638] Step 7:

[0639] The server shares information about the user's health and emotional state with family and caregivers in real time. Inputs include analysis results, and outputs generate alerts and notifications that are sent to family and caregiver devices.

[0640] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0641] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0643] [Fourth Embodiment]

[0644] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0645] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0646] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0648] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0650] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0651] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0652] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0653] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0654] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0655] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0656] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0657] The elderly care system of the present invention combines biometric data, image data, and voice data to monitor the user's daily life in real time and manage their health. Specific embodiments are described below.

[0658] First, the sensor devices equipped in the terminal continuously monitor the user's heart rate, body temperature, activity level, etc., and collect biometric data. Next, cameras installed in the living room or living space capture images of the user's movements and environment, acquiring image data. This data is then transmitted to a server via wireless communication.

[0659] The server analyzes biometric and image data in real time. Specifically, it uses image processing algorithms to recognize user movements and postures and detect abnormal behavior. If an abnormality is detected, the server immediately generates an emergency notification and sends it to family members or caregivers. Furthermore, it predicts the user's health status based on the collected biometric data and provides optimal health advice.

[0660] Furthermore, the voice spoken by the user is collected through the device's microphone and analyzed on the server using natural language processing. This makes it possible to understand the user's intent and generate appropriate voice feedback.

[0661] As a concrete example, consider the case of a user falling. The device's camera captures the user's fall, and image data is sent to the server. The server detects the fall through image analysis, generates an emergency notification, and sends it to the family. At the same time, voice feedback can be used to confirm with the user, "Are you injured?", and further instructions can be given as needed.

[0662] This system not only monitors the user's status in real time but also allows for rapid response in case of abnormalities. This helps support the safe living of the elderly and reduces the burden on family members and caregivers.

[0663] The following describes the processing flow.

[0664] Step 1:

[0665] The device collects biometric data through wearable devices and smart home sensors. This data includes heart rate, body temperature, and activity level. The collected data is transmitted to a server in real time.

[0666] Step 2:

[0667] The device uses its built-in camera to capture image data of the user's movements and environment. For example, it records the user's posture and movement as video and sends the data to a server.

[0668] Step 3:

[0669] The server monitors the user's current health status based on the biometric data it receives. Predictive algorithms are executed to detect abnormal values ​​or changes.

[0670] Step 4:

[0671] The server processes image data and analyzes user behavior and environmental changes. Deep learning technology is used to detect falls and abnormal movements.

[0672] Step 5:

[0673] The device uses its microphone to collect voice data and sends it to the server. Voice input is used to recognize the user's natural speech.

[0674] Step 6:

[0675] The server performs speech recognition and natural language processing to analyze the user's utterances. This allows it to accurately understand the user's requests and instructions.

[0676] Step 7:

[0677] Based on the analysis results, the server will notify family members or caregivers of any anomalies detected in emergencies. Real-time alerts will be sent via email or app notifications.

[0678] Step 8:

[0679] The device uses speech synthesis technology to provide feedback to the user. For example, it can deliver reminders or health advice to the user via voice.

[0680] Step 9:

[0681] Users act according to the advice and instructions provided, which then provides feedback for the next data collection cycle, ensuring continued personalized care.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] There is a need to provide a system that ensures the safe living of the elderly while allowing family members and caregivers in remote locations to monitor their health and potential dangers in real time. However, existing systems lack sufficient real-time capabilities and immediate response to emergencies, making it difficult to monitor them with peace of mind.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes a device for collecting biometric information, a device for acquiring visual information, and a device for processing audio information. This enables real-time monitoring of the user's health status, immediate response to abnormalities, and provision of appropriate feedback.

[0687] "Biometric information" refers to data collected from the human body, such as heart rate, body temperature, and activity level.

[0688] "Visual information" refers to video and image data acquired through cameras and sensors.

[0689] "Voice information" refers to user speech and voice data collected using a microphone.

[0690] A "device that recognizes abnormalities" is a device that uses collected data to detect conditions that are different from the normal state.

[0691] A "device for generating emergency notifications" is a device that promptly issues warnings or notifications to relevant parties when an anomaly is detected.

[0692] A "language processing device" is a device that analyzes speech information as natural language and understands its meaning and intent.

[0693] A "device that provides voice response" is a device that provides appropriate voice feedback to the user based on analyzed information.

[0694] A "device that transmits wirelessly" is a device that uses wireless technology to transmit collected information to other devices or servers without using cables.

[0695] "Supporter" refers to the user's family, caregiver, or someone responsible for their health management.

[0696] "Real-time information sharing" refers to sharing information with people in remote locations in near real-time.

[0697] This invention relates to a system that monitors a user's health status in real time and enables immediate response in the event of an abnormality. The system mainly consists of terminals and servers.

[0698] The device incorporates a sensor device for collecting biometric information, continuously acquiring data such as heart rate, body temperature, and activity level. This sensor device is integrated into typical wearable devices. In addition, the device is equipped with a camera to acquire visual information, monitoring the user's movements and environment. Audio information is collected through the device's microphone. This information is transmitted from the device to a server via wireless communication.

[0699] The server analyzes received biometric and visual information in real time. Image processing algorithms running on the server accurately recognize the user's movements and posture, and immediately generate an emergency notification if an anomaly is detected. The emergency notification is sent to family members or caregivers via email or SMS. At the same time, the system predicts the user's health status based on biometric information and provides appropriate advice. Voice information is analyzed using natural language processing technology and used to understand the user's intentions and requests. Based on the results, a voice response is generated and fed back to the user through the terminal.

[0700] For example, if a user falls, the device's camera captures the situation and sends image data to the server. The server generates an emergency notification and sends a warning to family members such as, "The user has fallen." At the same time, it can check with the user via voice response, asking, "Are you injured?" and provide further instructions as needed.

[0701] An example of a prompt to input into the generating AI model is, "Please describe in detail the response system for when an elderly person falls." This system will efficiently manage the safety and health of users.

[0702] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0703] Step 1:

[0704] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensor devices. The input is biometric data from the sensors, which is temporarily stored and prepared for periodic transmission to the server. The output is encrypted biometric data for transfer to the server. Specifically, the wearable device tracks the movements of the user wearing it and updates the data at regular intervals.

[0705] Step 2:

[0706] The device acquires the user's visual information using a camera. The input is real-time video captured by the camera, which is converted into a format suitable for image processing algorithms. The output is processed image data, which is prepared to be sent to the server. Specifically, the camera scans the user's environment at regular intervals and captures changes in movement as needed.

[0707] Step 3:

[0708] The terminal collects user speech using a microphone to acquire audio information. The input is audio data, which is converted into an audio format and configured for speech recognition. The output is an audio file to be later sent to the server. Specifically, it performs noise reduction and necessary amplification processing on the audio to prepare clear audio data.

[0709] Step 4:

[0710] The terminal transfers collected biometric, visual, and audio information to the server using wireless communication. Input consists of various data temporarily stored within the terminal. Output is an encrypted, comprehensive data packet sent to the server via a wireless communication protocol. Specifically, the terminal calculates the optimal timing and sends the data in a batch processing format.

[0711] Step 5:

[0712] The server analyzes the user's health status and behavior using various received information. Inputs are biometric and visual information transmitted from the terminal. Data processing involves applying an anomaly detection algorithm, and if an anomaly is detected, its nature is determined. Outputs include the anomaly detection result and its details. Specifically, it uses image data to perform motion recognition and identify abnormal postures.

[0713] Step 6:

[0714] The server immediately generates an emergency notification and sends it to family members or caregivers upon detecting an anomaly. The input is the anomaly detection result. The output is an emergency notification message that provides a detailed explanation of the user's situation. Specifically, it prioritizes notifications and delivers the necessary information in a single notification message.

[0715] Step 7:

[0716] The server analyzes voice information using natural language processing techniques to understand the user's intent. The input is voice data transmitted from the terminal. The output is the analyzed intent and request content. Specifically, it extracts keywords from the voice and analyzes their context.

[0717] Step 8:

[0718] The server generates an appropriate voice response based on the analysis results and provides feedback to the user through the terminal. The input is the result of the voice analysis, and the output is the generated voice response data. Specifically, it selects the appropriate voice to play based on the user's state and issues a playback command on the terminal.

[0719] (Application Example 1)

[0720] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0721] There is a need to monitor the health and safety of the elderly in real time, detect abnormalities early, and respond quickly. However, existing systems lack sufficient technology to integrate and analyze individual biometric, image, and audio information, making it difficult for caregivers to quickly grasp the situation. Furthermore, there is a lack of easy ways for caregivers to check the condition of the elderly.

[0722] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0723] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to comprehensively analyze multiple pieces of information and detect user abnormalities in real time. Furthermore, caregivers can quickly check the user's condition via a portable device.

[0724] "Biometric information" refers to data that shows the user's physical characteristics, such as heart rate, body temperature, and activity level.

[0725] "Image information" refers to visual data acquired by a camera to understand the user's movements and posture.

[0726] "Audio information" refers to data that includes the voice spoken by the user, and is used to extract meaning and intent.

[0727] "Means for detecting anomalies" refers to the process of analyzing biological and image information to identify unexpected conditions or dangerous situations.

[0728] "Means of generating notifications" refer to actions taken to quickly communicate important information based on the detection of anomalies.

[0729] "Means of language processing" refers to technologies that analyze natural language from audio information to understand the user's intent.

[0730] "Means of providing audio output" refers to the process of conveying information and instructions by voice in a format that is easy for users to understand.

[0731] A "portable device" is a device that caregivers can carry around and use to check information about the user in real time.

[0732] "Means of providing time-series information" refers to the process of managing and presenting a series of information, including historical data.

[0733] This invention is a system for monitoring the health status and daily life of elderly people in real time. The system mainly consists of a terminal, a server, and a group of sensors installed in the user's living space.

[0734] The device is equipped with sensors that continuously collect biometric information such as heart rate and body temperature. This biometric information is transmitted to the server via wireless communication. Cameras are installed in the living space to capture image information of the user's movements and posture. Similarly, the device's microphone collects voice information emitted by the user. The server processes this data in real time to detect anomalies. In particular, image processing libraries such as OpenCV and scikit-learn are used to analyze the user's movements.

[0735] If an anomaly is detected, the server generates an emergency notification and sends it to the caregiver via a portable device. This portable device is a smartphone or smart glasses, allowing the caregiver to check it immediately. Furthermore, based on collected biometric information, the system predicts the user's health status and provides voice feedback. This voice feedback is generated using natural language processing, specifically Google Dialogflow and Microsoft Azure's NLP capabilities.

[0736] For example, if a user falls, the server analyzes the camera data, immediately sends a notification to the caregiver, and plays a voice message to the user asking, "Are you injured?" This enables a quick response.

[0737] An example of a prompt is: "Generate a notification and follow-up scenario regarding fall detection in elderly individuals. Create notification messages and voice prompts, and describe in detail the steps to take after a fall is detected."

[0738] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0739] Step 1:

[0740] The device collects biometric information such as the user's heart rate, body temperature, and activity level using sensors. This biometric information is then transmitted to the server using wireless communication. At this stage, the data is still in raw format.

[0741] Step 2:

[0742] Cameras installed in the living space capture the user's movements and posture, acquiring image information. The captured images become input and are immediately sent to the server. At this point, the system is ready for image processing algorithms to be applied for motion recognition.

[0743] Step 3:

[0744] The device's microphone collects the user's voice and sends it to the server as audio data. The audio data becomes the input, and the server is ready to apply natural language processing.

[0745] Step 4:

[0746] The server initiates real-time analysis using biometric and image data. An algorithm for detecting anomalies in biometric data is applied using scikit-learn. Simultaneously, image data is analyzed using OpenCV to detect abnormal behavior. If an anomaly is detected, data reporting the anomaly is generated as output.

[0747] Step 5:

[0748] If an anomaly is detected, the server immediately generates an emergency notification. The notification data is used as input and output to a portable device. This notification is quickly sent to the caregiver's smartphone or glasses using the NF communication protocol, alerting the caregiver.

[0749] Step 6:

[0750] The server performs natural language processing based on the collected audio information. Using Google Dialogflow and Microsoft Azure's NLP capabilities, it analyzes the audio data to understand the user's intent. The resulting audio feedback is then sent to the device as output.

[0751] Step 7:

[0752] When a user needs feedback in an abnormal situation, the server generates voice feedback and provides it to the user via the terminal. For example, when a fall is detected, a voice message such as "Are you injured?" is output. This output allows the user to receive instructions appropriate to the situation.

[0753] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0754] The elderly care system of the present invention not only comprehensively monitors the user's daily life by combining biometric data, image data, and voice data, but also recognizes the user's emotional state in real time using an additional emotion engine and provides appropriate feedback based on that.

[0755] First, the device uses sensor devices to collect biometric data such as the user's heart rate, body temperature, and activity level. Next, cameras installed in the room record the user's movements and facial expressions, acquiring image data. This image data plays a crucial role in analyzing the user's facial expressions. In addition, the device's microphone captures audio data, accurately recording the user's speech.

[0756] All of this data is transmitted to the server in real time. The server combines and analyzes the biometric and image data to detect abnormalities in the user's health and behavior. The emotion engine uses image and audio data to analyze the user's facial expressions and tone of voice to recognize the user's emotions. For example, if a user is stressed, this is detected as a change in facial expression and tone of voice.

[0757] Based on the identified emotions, the server adjusts the voice feedback to the user to provide more appropriate communication. For example, if the user is feeling anxious, the server provides feedback in a calm and reassuring tone. Furthermore, by sharing the user's emotional state with family members and caregivers, the server supports the provision of more appropriate care.

[0758] As a concrete example, consider a case where a user is feeling unwell and anxious. Biometric data indicates an abnormality, and the emotion engine detects anxiety from the user's facial expressions and voice. Based on this information, the server generates an emergency notification and sends an alert to family members. At the same time, the device plays a reassuring message to the user through voice feedback, such as, "Are you okay? If you're worried, we'll call support right away."

[0759] As a result, the present invention not only ensures the safety and security of users, but also enables meticulous care that takes their emotions into consideration.

[0760] The following describes the processing flow.

[0761] Step 1:

[0762] The terminal collects biometric data from the user's wearable device. This data includes heart rate, body temperature, and activity level, and is transferred to a server using a secure communication protocol.

[0763] Step 2:

[0764] A camera installed on the terminal records video of the room, capturing the user's movements and facial expressions. The video data is transmitted to the server in real time. This allows for continuous monitoring of the user's posture and changes in the environment.

[0765] Step 3:

[0766] The device acquires user voice data via its microphone. It captures the user's everyday conversations and speech content and accurately transmits it to the server.

[0767] Step 4:

[0768] The server analyzes the received biometric data and determines whether it falls within the normal range. If an abnormal value is detected, an alert is immediately generated and preparations are made.

[0769] Step 5:

[0770] The server analyzes image and audio data and uses an emotion engine to determine the user's emotional state. It analyzes changes in facial expressions from the image data and the tone and volume of the voice from the audio data to identify emotions.

[0771] Step 6:

[0772] Based on the results of anomaly detection and emotion assessment, the server decides whether to send an emergency notification to family members or caregivers. Information is shared via email or push notifications as needed.

[0773] Step 7:

[0774] The server adjusts the voice feedback it provides to the user, generating feedback in an appropriate tone and content based on the user's emotions. For example, if anxiety is detected, it selects a calming message to reassure the user.

[0775] Step 8:

[0776] The device uses speech synthesis technology to provide the user with voice feedback generated from the server. It then provides necessary support through interaction with the user.

[0777] Step 9:

[0778] Users react to voice feedback and adjust their daily behaviors accordingly. This allows the entire system to receive useful feedback in the next observation cycle, improving the quality of care provided to the user.

[0779] (Example 2)

[0780] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0781] In elderly care, there is a need to monitor individual health and emotional states in real time and to respond quickly and appropriately when abnormalities are detected. However, conventional systems have difficulty providing integrated and immediate feedback of biometric and emotional data, making it challenging to adequately ensure user confidence.

[0782] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0783] In this invention, the server includes means for analyzing biometric information to detect abnormalities, means for performing emotion analysis and adjusting feedback according to the emotional state, and means for generating voice feedback based on the user's emotional state using a generative AI model. This enables a comprehensive understanding of the health and emotional state of elderly individuals, and allows for rapid and appropriate feedback tailored to individual circumstances.

[0784] "Biometric information" refers to numerical values ​​or data that indicate a person's physical state, such as heart rate, body temperature, and activity level.

[0785] "Image information" refers to visual data used to capture the user's facial expressions and movements.

[0786] "Audio information" refers to acoustic data that includes the user's voice, including the content of their speech and the tone of their voice.

[0787] "Anomaly detection" refers to the process of analyzing biological and image information to identify abnormal physical or behavioral conditions.

[0788] An "emergency notification" is a notification that is immediately sent to family members or caregivers in an emergency to ensure the user's safety.

[0789] "Speech analysis" is a process that uses natural language processing based on speech information to understand the content of speech and the tone of voice.

[0790] "Voice response" refers to voice feedback provided to the user, intended to convey information or soothe emotions.

[0791] "Emotional analysis" is a technology that identifies a user's emotional state by analyzing their facial expressions and tone of voice.

[0792] A "generative AI model" is a model that uses artificial intelligence techniques to generate output in response to a specific input.

[0793] This invention is a system for elderly care that monitors the user's daily health and emotional state in real time and provides appropriate responses. The system mainly consists of a "terminal" and a "server".

[0794] The device is hardware that is worn directly by the user or installed in their room, and is equipped with sensors for collecting biometric information (e.g., heart rate monitor and temperature sensor), a camera for recording movements and facial expressions, and a microphone for capturing audio. This device transmits the collected data to a server in real time.

[0795] The server integrates and processes received biometric, image, and audio information. Biometric information is used for detecting abnormalities in health status, image information is particularly used for analyzing the user's facial expressions, and audio information is used to analyze speech content and tone. The server also incorporates an emotion analysis engine and a generative AI model, which are used to analyze the user's emotional state and generate appropriate audio feedback based on the results. The generative AI model can receive prompts such as: "The user appears anxious; please generate reassuring words."

[0796] As a concrete example, if a user feels unwell and anxious, the device records an abnormally high heart rate and sends it to the server. Simultaneously, the camera captures the user's troubled expression, and the microphone records a tense voice. The server comprehensively analyzes this data, and if it detects both physical abnormality and anxiety, it uses a generative AI model to generate voice feedback such as, "Are you okay? We'll call support right away, so please don't worry," and conveys it to the user via the device.

[0797] This system allows users to centrally manage their health and emotional state and receive prompt and appropriate care as needed. Furthermore, family members and caregivers can monitor the user's condition in real time, enabling them to provide better support.

[0798] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0799] Step 1:

[0800] The device collects biometric information from the user. Specifically, it collects data such as heart rate, body temperature, and activity level using heart rate monitors and temperature sensors. The input is a digital signal from the biosensor, which is converted into a visualizeable format within the device. The output is formatted biometric data.

[0801] Step 2:

[0802] The device acquires image information using a camera installed in the room. This allows the user's movements and facial expressions to be captured in real time. The input is visual data received by the camera sensor, which is then processed and converted into a format that is easy to analyze. The output is the processed image data.

[0803] Step 3:

[0804] The device captures audio information using a microphone. The user's speech and tone are recorded, forming the basis for speech recognition. The input is an audio signal, which undergoes noise reduction and sampling. The output is clean audio data.

[0805] Step 4:

[0806] The device transmits all collected biometric, image, and audio information to the server in real time. Input is integrated user data transmitted using a secure protocol. Output is the data batch received by the server.

[0807] Step 5:

[0808] The server analyzes received biometric data to detect anomalies. Specifically, it analyzes sudden increases in heart rate and fluctuations in body temperature. The input is received biometric data, and data mining techniques are used to identify anomaly patterns. The output is an anomaly detection flag.

[0809] Step 6:

[0810] The server performs sentiment analysis using image and audio information. It uses facial expression recognition technology and voice sentiment analysis to determine the user's emotional state. The input consists of features extracted from image and audio data, and a machine learning model is used to identify emotion categories. The output is a label representing the emotional state.

[0811] Step 7:

[0812] The server uses an AI model to generate voice feedback that responds to the user's emotions. A prompt is input to the model, and a feedback message is output. The input is a prompt such as, "The user is feeling anxious; please generate a reassuring message." The output is the generated voice response message.

[0813] Step 8:

[0814] The device receives feedback from the server and provides voice feedback to the user. The input is a generated voice message, which is transmitted to the user through the speaker. The output is the voice feedback that the user hears.

[0815] (Application Example 2)

[0816] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0817] In elderly care, it is necessary not only to monitor health status but also to understand emotional states in real time. However, conventional technologies have made it difficult to comprehensively analyze biometric information, images, and audio data to provide appropriate feedback tailored to the emotional state of individual users. Furthermore, there has been a challenge in sharing timely and effective information with family members and caregivers, which prevents prompt and appropriate responses.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0819] In this invention, the server includes means for collecting biometric information, means for acquiring image information, and means for processing audio information. This makes it possible to analyze the user's health and emotional state in real time and quickly share necessary information with family members and caregivers.

[0820] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and activity level.

[0821] "Image information" refers to visual data of a user's face and movements acquired using a camera.

[0822] "Voice information" refers to data about the user's speech content and tone collected by a microphone.

[0823] "Emotion recognition means" refers to an algorithm that analyzes image and audio information to identify the user's emotional state.

[0824] An "emergency notification" is an alert message generated when an abnormality is detected, and is sent to family members or caregivers.

[0825] "Voice feedback" refers to reactions or instructions provided to the user via voice.

[0826] "Natural language processing" is a technology that analyzes speech information to understand the intent behind a user's utterances.

[0827] "Means of sharing emotional states with family and caregivers" refers to methods for transmitting a user's emotional information to relevant parties in real time.

[0828] The system for implementing this invention comprehensively collects and analyzes the user's biometric information, image information, and voice information to monitor the user's health and emotional state in real time and provide necessary information to family members and caregivers.

[0829] The server runs a program that collects biometric information such as heart rate, body temperature, and activity level from sensor devices. This data is necessary for analyzing the user's health status. Furthermore, it uses a camera to acquire image information of the user and analyzes the user's movements and facial expressions. Image recognition algorithms are used for this analysis. Audio information is acquired via a microphone, and natural language processing technology is used to analyze the user's speech content and tone.

[0830] The server integrates this data and uses a generative AI model to recognize the user's emotions. Once the emotional state is determined, it generates voice feedback based on the result and provides appropriate feedback to the user. For example, if the user is feeling anxious, it communicates with them in a calm voice to provide reassurance. Furthermore, if emotional information is important, it is shared in real time with family members or caregivers.

[0831] For example, suppose the system detects surprise from the user's facial expression along with an increase in heart rate while the user is in the living room. The server analyzes this information and provides voice feedback such as, "Calm down, take a deep breath," while simultaneously sending a notification to the caregiver.

[0832] An example of a prompt message would be, "If an elderly person experiences anxiety, how would you provide feedback using heart rate, facial expression, and voice data?"

[0833] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0834] Step 1:

[0835] The device collects biometric information from sensor devices. This input data includes heart rate, body temperature, and activity level. The device records the digital biometric data and prepares to send it to the server.

[0836] Step 2:

[0837] Using the camera, the device acquires image information of the user. This input consists of image data of the user's face and body movements. The device applies an image recognition algorithm to extract features for analyzing facial expressions and movements.

[0838] Step 3:

[0839] The device acquires voice information using a microphone. The input data includes the user's speech content and voice tone. The device uses natural language processing technology to convert the voice data into text and perform voice tone analysis.

[0840] Step 4:

[0841] The server integrates the data collected in steps 1 through 3. The input includes biometric information, image information, and audio information. The server uses a generative AI model to infer the user's health and emotional state from this data.

[0842] Step 5:

[0843] The server generates voice feedback for the user based on the emotion recognition results. The input is data indicating the emotional state, and the output is instructions for the voice feedback to be delivered to the user. The server performs speech synthesis to prepare the feedback with the appropriate tone and content.

[0844] Step 6:

[0845] The user receives generated audio feedback. The device plays the feedback through its speaker and delivers a message to the user to provide reassurance.

[0846] Step 7:

[0847] The server shares information about the user's health and emotional state with family and caregivers in real time. Inputs include analysis results, and outputs generate alerts and notifications that are sent to family and caregiver devices.

[0848] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0849] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0851] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0852] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0853] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0854] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0855] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0856] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0857] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0858] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0859] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0860] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0861] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0862] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0863] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0864] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0865] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0866] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0867] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0868] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0869] The following is further disclosed regarding the embodiments described above.

[0870] (Claim 1)

[0871] Means of collecting biometric data,

[0872] Means for acquiring image data,

[0873] Means for processing audio data,

[0874] A means for analyzing the aforementioned biological data and image data to detect abnormalities,

[0875] A means for generating emergency notifications based on anomaly detection,

[0876] Means for performing natural language processing based on the aforementioned audio data,

[0877] A means of providing voice feedback to the user,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, further comprising means for predicting health status based on the aforementioned biometric data.

[0881] (Claim 3)

[0882] The system according to claim 1, further comprising means for sharing information in real time with family members or caregivers.

[0883] "Example 1"

[0884] (Claim 1)

[0885] A device for collecting biological information,

[0886] A device for acquiring visual information,

[0887] A device for processing audio information,

[0888] A device that analyzes the aforementioned biological and visual information to recognize abnormalities,

[0889] A device that generates emergency notifications based on the recognition of an anomaly,

[0890] A device that performs language processing based on the aforementioned audio information,

[0891] A device that provides voice responses to the user,

[0892] A device that transmits data wirelessly,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, further comprising a device that predicts health status based on the aforementioned biological information.

[0896] (Claim 3)

[0897] The system according to claim 1, further comprising a device for immediately sharing information with family members or caregivers.

[0898] "Application Example 1"

[0899] (Claim 1)

[0900] Means for collecting biometric information,

[0901] Means for acquiring image information,

[0902] Means for processing audio information,

[0903] A means for analyzing the aforementioned biological information and image information to detect abnormalities,

[0904] A means for generating notifications based on anomaly detection,

[0905] Means for performing language processing based on the aforementioned audio information,

[0906] A means of providing audio output to the user,

[0907] A means including a portable device that allows caregivers to check user information in real time,

[0908] A system that includes this.

[0909] (Claim 2)

[0910] The system according to claim 1, further comprising means for predicting health status based on the aforementioned biological information.

[0911] (Claim 3)

[0912] The system according to claim 1, further comprising means for providing time-series information to a family member or caregiver.

[0913] "Example 2 of combining an emotion engine"

[0914] (Claim 1)

[0915] Means for collecting biometric information,

[0916] Means for acquiring image information,

[0917] Means for processing audio information,

[0918] A means for analyzing the aforementioned biological information and image information to detect abnormalities,

[0919] A means for generating emergency alerts based on anomaly detection,

[0920] Means for performing speech analysis based on the aforementioned speech information,

[0921] A means of providing voice responses to the user,

[0922] A means of performing emotion analysis and adjusting feedback according to the emotional state,

[0923] Means of communicating emotional and health status to family members or caregivers,

[0924] A system that includes this.

[0925] (Claim 2)

[0926] The system according to claim 1, further comprising means for estimating health status based on the aforementioned biological information.

[0927] (Claim 3)

[0928] The system according to claim 1, further comprising means for generating voice feedback based on the user's emotional state using a generative AI model.

[0929] "Application example 2 when combining with an emotional engine"

[0930] (Claim 1)

[0931] Means for collecting biometric information,

[0932] Means for acquiring image information,

[0933] Means for processing audio information,

[0934] A means for analyzing the aforementioned biological information and image information to detect abnormalities,

[0935] A means for generating emergency notifications based on anomaly detection,

[0936] Means for performing natural language processing based on the aforementioned audio information,

[0937] A means of providing voice feedback to the user,

[0938] An emotion recognition method for analyzing the user's emotional state,

[0939] Means of sharing emotional states with family and caregivers,

[0940] A system that includes this.

[0941] (Claim 2)

[0942] The system according to claim 1, further comprising means for performing health prediction based on the aforementioned biological information.

[0943] (Claim 3)

[0944] The system according to claim 1, further comprising means for adjusting voice guidance according to the user's emotional state. [Explanation of Symbols]

[0945] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of collecting biometric data, Means for acquiring image data, Means for processing audio data, A means for analyzing the aforementioned biological data and image data to detect abnormalities, A means for generating emergency notifications based on anomaly detection, Means for performing natural language processing based on the aforementioned audio data, A means of providing voice feedback to the user, A system that includes this.

2. The system according to claim 1, further comprising means for predicting health status based on the aforementioned biometric data.

3. The system according to claim 1, further comprising means for sharing information in real time with family members or caregivers.

Images