system

Abstract

We provide the system. [Solution] A means of monitoring the user's health status and detecting abnormalities by collecting and analyzing biometric information obtained from the user, A means of notifying via a pre-configured communication channel when an anomaly is detected, A means of analyzing collected information based on machine learning algorithms to identify trends in health status, A means of tracking a user's movements using augmented reality technology and providing feedback to support their exercise, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an aging society, an effective in-home care support system is necessary for the elderly to live a secure and independent life. However, currently, monitoring of health conditions and prompt response in case of abnormalities are insufficient, increasing the care burden and the anxiety of the elderly. Also, the problems of shortage of care personnel and loneliness of the elderly have not been solved. It is required to comprehensively solve these problems and improve the quality of life of the elderly.

Means for Solving the Problems

[0005] This invention provides a system that monitors health status and detects abnormalities by collecting and analyzing health data obtained from users. When an abnormality is detected, it quickly notifies the user through a pre-configured communication channel, allowing for necessary action to be taken. Furthermore, it analyzes the collected data based on a machine learning model to predict future health status. In addition, it effectively supports rehabilitation and daily health maintenance by tracking the user's movements in real time and providing appropriate feedback. Moreover, it facilitates communication with users and reduces feelings of isolation through a dialogue function using voice and video data. Including a function that automatically requests assistance in emergencies, these means make it possible to comprehensively support the independent living of the elderly.

[0006] "Health data" refers to a collection of information that includes indicators of a user's health status, such as heart rate, steps taken, body temperature, and activity level.

[0007] "Monitoring" refers to the act of continuously observing and recording health data in order to understand the user's health status.

[0008] An "abnormal" value or pattern in a user's health data that exceeds the normal range indicates a condition that requires attention.

[0009] A "communication channel" refers to a means of transmitting information, such as email, SMS, or voice messages, used for sending notifications.

[0010] A "machine learning model" is an algorithmic framework that learns from past data and identifies and predicts users' health trends based on new data.

[0011] "Real-time" refers to the characteristic that data processing and analysis are performed immediately on the spot, and feedback is provided without delay.

[0012] "Feedback" refers to information, including advice and evaluations, that the system provides to users based on the results of its analysis of the data it has obtained.

[0013] The "dialogue function" is a feature that allows the system and the user to communicate using voice and video, enabling smooth interaction.

[0014] A "request for assistance" is the act of seeking help from others or specific organizations in an emergency to protect the safety and health of the user. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The care support system of the present invention includes a program for comprehensively supporting the user's health condition. The system functions through the cooperation of a terminal, a server, and the user, and comprehensively manages the user's health.

[0037] The terminal collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. This collected data is temporarily stored on the terminal and then sent to the server at regular intervals.

[0038] The server stores the received data and analyzes it using a machine learning model to determine if there are any anomalies by comparing it with past data. If an anomaly is detected, the server sends a notification to the user and the necessary medical institutions through a pre-configured communication channel. This notification includes an overview of the health status and details of the anomaly, prompting the user to take prompt action.

[0039] Furthermore, the server generates feedback based on data analysis, indicating the user's health trends. This feedback is sent to the device as advice for maintaining or improving the user's health, and the device presents it to the user in either voice or text format.

[0040] Furthermore, the device tracks the user's movements through an AR device to support rehabilitation and exercise. The server analyzes the movement data and provides feedback on appropriate exercise methods and posture to support effective rehabilitation.

[0041] For the dialogue function, the terminal collects voice and video data from the user, and the server analyzes it using natural language processing technology. Based on the analysis results, the server generates an appropriate response and interacts with the user through the terminal, thereby reducing the user's feelings of isolation.

[0042] For example, if a user experiences an emergency such as a fall, the device uses an accelerometer and acoustic analysis to detect the anomaly and immediately reports it to the server. The server then automatically notifies registered emergency contacts, enabling a rapid response.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. The collected data is stored on the device for a short period of time.

[0046] Step 2:

[0047] The device sends the accumulated data to the server as packets at regular intervals (e.g., every 30 minutes). Data that has been successfully sent is deleted from the device.

[0048] Step 3:

[0049] The server stores health data received from the terminal in a database. The data is categorized by user and stored along with past history data.

[0050] Step 4:

[0051] The server inputs the stored data into a machine learning model to analyze health trends. If an anomaly is detected, the server evaluates the type and nature of the anomaly.

[0052] Step 5:

[0053] The server immediately determines the action to take when an anomaly is detected and sends a notification to the user or healthcare provider through the registered communication channel. The notification includes an overview of the health condition and how to proceed.

[0054] Step 6:

[0055] The server generates health maintenance feedback based on the analysis results and sends it to the device. The device then presents this feedback to the user in audio or text format.

[0056] Step 7:

[0057] When users perform rehabilitation or exercise, the device uses an AR device to track their movements. The server analyzes the movement data in real time and provides appropriate advice.

[0058] Step 8:

[0059] When a user uses the conversational function, the device collects audio and video data and sends it to the server. The server analyzes this data using natural language processing technology, generates the most appropriate response for the user, and communicates it through the device.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] Efficiently monitoring the health of the elderly and those with chronic illnesses in real time, and responding quickly to abnormalities, is crucial in modern welfare and healthcare. However, existing systems require the combination of multiple different technologies to efficiently analyze biometric information and provide integrated motion tracking and conversational functions, resulting in complexity and high costs. Therefore, effective methods to solve these problems are needed.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for monitoring the user's physical condition and detecting abnormalities by collecting and analyzing biometric information obtained from the user; means for notifying the user through a pre-configured communication path when an abnormality is detected; means for analyzing the collected information based on a learning model and identifying trends in the physical condition; means for generating and presenting advice useful for maintaining the user's health based on the analysis results; and means for tracking the user's movements using augmented reality technology and providing rehabilitation and exercise support. This makes it possible to integrate health monitoring, rapid response to abnormalities, and the provision of advice for improving health conditions.

[0065] "Users" refer to individuals who use the system to receive services such as health monitoring and rehabilitation.

[0066] "Biometric information" refers to data such as heart rate, steps taken, and body temperature, which are collected to understand the user's health status.

[0067] A "server" refers to a computing device that stores and analyzes collected data and generates necessary notifications and feedback.

[0068] A "learning model" refers to an algorithm used to analyze data using machine learning techniques and identify trends and anomalies.

[0069] "Communication path" refers to the means of communication used to transmit abnormality notifications and feedback to users and medical institutions, and includes email, SMS, etc.

[0070] Augmented reality technology is a technology that overlays virtual information onto the real world environment and is used to track user movements and provide health support.

[0071] "Health monitoring" refers to the process of observing the user's health status in real time and detecting any abnormalities.

[0072] "Rehabilitation" refers to the process of providing exercise therapy and feedback aimed at improving the physical function of the user.

[0073] This invention provides a care support system that collects and analyzes user health information from multiple perspectives, enabling real-time health management and rapid response to abnormalities. The system operates with terminals (including wearable devices), a server, and the user working together as a single unit.

[0074] The terminal collects biometric information (e.g., heart rate, steps, body temperature) from wearable devices worn by the user. These devices could be commonly referred to as portable vital signs sensors. The terminal acquires data in real time via Bluetooth or Wi-Fi and temporarily stores it on its own.

[0075] The server receives and stores data transmitted from the terminal and analyzes it using a learning model (e.g., an algorithm using TENSORFLOW®). Based on the analysis results, the server immediately notifies the user and, if necessary, the medical institution if an anomaly is detected. This is done using a configured communication channel (e.g., email or SMS).

[0076] The server also uses accumulated data to understand long-term health trends and generates advice that can help improve the user's health. This advice is provided through the terminal and can also be suggested in voice using speech synthesis software (e.g., a general term for a speech generation system).

[0077] Furthermore, the device utilizes augmented reality technology to track and analyze the user's movements, providing support for rehabilitation and exercise. This data is used for analysis on a server, and feedback is provided on appropriate exercise methods.

[0078] If a user encounters an emergency, the device uses its built-in sensors to detect the situation and report it to the server. This enables a quick and appropriate response. For example, there is a function that issues an alarm based on acceleration data when a user falls.

[0079] Examples of prompts for a generative AI model:

[0080] "Design an algorithm to analyze heart rate data collected from wearable sensors and detect anomalies."

[0081] "Please develop a system that suggests appropriate rehabilitation methods based on user activity data."

[0082] Thus, this invention is a system that utilizes advanced technology to comprehensively support the user's daily health.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] The terminal collects biometric information such as heart rate, steps taken, and body temperature in real time from wearable devices. The input is sensor data obtained from the wearable device. Based on this, the terminal temporarily stores the data and organizes it for transmission in the next step. Specifically, the device measures the heart rate every minute, and the terminal receives the data via Bluetooth.

[0086] Step 2:

[0087] The device sends the collected data to the server at regular time intervals (for example, every hour). The input is biometric information stored on the device, and the output is the transmission of data to the server. As part of the data processing, the device converts the data format and sends it using the HTTPS protocol. Specifically, the device establishes a secure connection and uploads the compiled data to the server.

[0088] Step 3:

[0089] The server opens the data received from the terminal and simultaneously stores it in the database. The input is biometric information sent from the terminal, and the output is a record in the database. In terms of data processing, the server manages the time series of the data and prepares to compare it with anomaly detection borderlines. Specifically, the server creates a new entry in the database and adds the data to the history of the corresponding user.

[0090] Step 4:

[0091] The server analyzes accumulated data using a machine learning model to detect anomalies. The input is historical biometric information stored in the database, and the output is a determination of whether or not an anomaly exists. As part of the data analysis, the server filters the data through the model and makes predictive judgments. Specifically, the server executes an anomaly detection algorithm to identify data points that fall outside the normal range.

[0092] Step 5:

[0093] If an anomaly is detected, the server notifies the user through a pre-configured communication channel. The input is the anomaly detection result, and the output is an alarm message to the user. As part of the data processing, the server generates a notification message and prepares it for sending via email or SMS. Specifically, the server creates a notification message and sends it based on the registered contact information.

[0094] Step 6:

[0095] The server analyzes health trends using collected data and performs long-term health status assessments. The input is the entire historical data in the database, and the output is a health trend report. As a data calculation, the server applies trend analysis algorithms and generates indicators to visualize health status. Specifically, the server generates a health trend report and visualizes the results based on the algorithm used.

[0096] Step 7:

[0097] The server sends the generated health trend report and advice to the terminal, suggesting it to the user. The input is the health trend report, and the output is the feedback the user receives. As part of data processing, the server converts the content into audio or text and presents it to the user. Specifically, the terminal uses AV technology to read the report aloud and provides the user with specific health advice.

[0098] Step 8:

[0099] The device utilizes augmented reality technology to monitor user movements and provide rehabilitation and exercise support. Input is movement data from the AR system installed on the device, and output is feedback on how to improve. For data analysis, the device analyzes movement parameters and guides the user toward correct form and exercise methods. Specifically, the device uses AR feedback to guide the user toward correct posture during exercise.

[0100] (Application Example 1)

[0101] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0102] In an aging society, there is a growing need for individual health management and prompt response to emergencies. Traditional manual health monitoring has limitations due to labor shortages and lack of knowledge, and further challenges include the inability to alleviate feelings of isolation and provide appropriate exercise guidance. In particular, the insufficient ability to respond quickly in emergencies and to continuously monitor health trends are problematic.

[0103] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0104] In this invention, the server includes means for collecting and analyzing biometric information in real time, means for tracking actions and providing feedback using augmented reality technology, and means for analyzing conversational data and generating responses using named entity recognition technology. This enables comprehensive monitoring of the user's health status, rapid detection and response to abnormalities, and furthermore, reduces feelings of loneliness and provides effective exercise guidance.

[0105] "Biometric information" refers to data such as heart rate, body temperature, and step count that indicates the user's health status.

[0106] "Analysis" refers to the data processing activity used to identify anomalies and trends in collected data.

[0107] "Augmented reality technology" is a technology that captures a user's physical actions as digital data and provides appropriate feedback.

[0108] "Motion tracking" is the process of observing a user's body movements in real time and analyzing them as digital information.

[0109] "Feedback" refers to guidance and advice provided to users based on the results of analysis.

[0110] Named entity recognition technology is a technology that automatically analyzes audio and video data and extracts meaningful information.

[0111] "Generating a response" refers to the act of creating an appropriate reply to the user based on the analyzed data.

[0112] An "emergency situation" refers to a situation where a user's normal health condition is abnormal and requires a swift response.

[0113] To implement this invention, a system is constructed in which a wearable device worn by the user, a smartphone, and a server work together. The wearable device is equipped with sensors that measure heart rate, body temperature, steps taken, etc. This data is transmitted to the smartphone via Bluetooth communication, and the smartphone temporarily stores this data.

[0114] The server stores biometric data received from smartphones at specific time intervals. Using machine learning algorithms based on TensorFlow, the server analyzes this data to identify abnormalities and health trends. The analysis results are fed back to the user in real time, and in the event of an anomaly, notifications are sent to the user and medical institutions via pre-configured communication channels. These notifications are typically sent as text messages.

[0115] The smart glasses worn by the user utilize augmented reality technology to track the user's movements in real time. A server analyzes the movement data and provides effective exercise guidance by suggesting a suitable rehabilitation plan. This feedback is displayed through the smart glasses.

[0116] Furthermore, the smartphone collects the user's voice and camera footage, and the server employs named entity recognition technology to analyze this data. The server generates appropriate responses from the analyzed data, enabling conversation with the user. This process can utilize natural language generation models such as those from OpenAI®.

[0117] As a concrete example, if the system detects a sudden decrease in a user's step count during the daytime, it recognizes this as an anomaly. To investigate the reason, the server generates questions such as, "How did you spend your day?" and confirms with the user via voice. An example of a prompt to the generating AI model could be, "Please provide questions to identify the reason for the decrease in user activity."

[0118] In this way, the present invention enables comprehensive health management and supports early detection of abnormalities and rapid response.

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The terminal receives biometric information from wearable devices, collecting data such as heart rate, body temperature, and steps taken. The input is sensor data from the wearable device, and the output is the temporary storage of the collected data within the terminal. Data processing includes data format conversion and initial filtering.

[0122] Step 2:

[0123] The server receives data from terminals at regular intervals. The input is biometric information transmitted from the terminals, and the output is stored in the server's database. As a data processing step, the data is organized chronologically and saved to storage.

[0124] Step 3:

[0125] The server analyzes received biometric data using a machine learning algorithm based on TensorFlow to perform anomaly detection. The input is accumulated biometric data, and the output is the result of the anomaly detection. As part of data processing, it compares the data with past data to identify outliers and anomaly patterns.

[0126] Step 4:

[0127] If an anomaly is detected, the server sends a notification to the user or designated contact via the communication channel. The input is the anomaly detection result, and the output is a notification in text message format. The operation involves formatting and sending the message according to the communication protocol.

[0128] Step 5:

[0129] The device receives health trend feedback from the server and presents it to the user in audio or text. The input is an analysis report from the server, and the output is the feedback information displayed to the user. The system performs UI rendering to present the information in an easily understandable format for the user.

[0130] Step 6:

[0131] The server receives motion data from smart glasses and analyzes the motion using augmented reality technology. The input is motion data from the smart glasses, and the output is appropriate rehabilitation feedback. The data processing involves identifying the motion and evaluating its quality.

[0132] Step 7:

[0133] The user inputs audio and video data into the terminal, and the server analyzes the data using named entity recognition technology to generate a dialogue. The input is audio and video data, and the output is an appropriate response. The operation involves response generation using a natural language generation model. An example of a prompt sentence for the generating AI model is, "Generate a response to continue a natural conversation with the user."

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] The care support system incorporating the emotion recognition function of the present invention not only monitors the user's health condition but also recognizes the user's emotional state in detail and provides support based on that data.

[0136] The device is equipped with a camera and microphone that captures the user's voice and video in real time. This allows for the acquisition of data in a natural way as the user interacts with the system.

[0137] The server utilizes an emotion engine to process the collected audio and video data. This emotion engine analyzes the tone of voice and facial expression patterns to determine the user's emotional state. The determined emotional state is stored on the platform and integrated with health data to be used for an overall assessment of the user's health and psychological state.

[0138] For example, while a user is receiving exercise guidance using the conversational function, the device captures changes in the user's facial expressions and tone of voice. If the server determines that the user is expressing feelings of fatigue, it can provide feedback suggesting adjustments to the exercise intensity.

[0139] Furthermore, if the system detects that a user is emotionally unstable, it provides support to reduce anxiety by offering relaxation advice and engaging in reassuring conversations. This function helps alleviate the user's psychological burden and supports them in living their daily lives with peace of mind.

[0140] Through these implementations, the system integrates users' physical health data and emotional data to provide personalized and optimal care.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The device collects health data from the wearable device worn by the user and simultaneously captures the user's voice and video data. This data is temporarily stored within the device.

[0144] Step 2:

[0145] The device sends health data, audio data, and video data to the server at regular intervals. The data is organized for each user and stored in the server's database.

[0146] Step 3:

[0147] The server inputs audio and video data into an emotion engine, which analyzes the intonation of the voice and changes in facial expressions. Based on this analysis, it identifies the user's emotional state.

[0148] Step 4:

[0149] The server integrates identified emotional states with accumulated health data to assess the user's overall psychological and health status. Machine learning models are used to predict abnormal trends and emotional trends.

[0150] Step 5:

[0151] If abnormalities or emotional instability are detected, the server generates optimal feedback and dialogue. This includes health management advice and relaxation suggestions.

[0152] Step 6:

[0153] The terminal presents the user with feedback and conversation content provided by the server. This is done through voice messages and screen displays.

[0154] Step 7:

[0155] Users act based on feedback from their devices and continuously manage their health and emotional state. User responses and additional data are collected again from the device and used in the system as a feedback loop.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0158] For elderly users and those with health concerns, it is challenging to simultaneously assess not only their physical health but also their psychological health and provide appropriate support. This project aims to improve the quality of life for users by addressing this challenge.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes means for collecting physical and psychological data obtained from the user and determining the emotional state by analyzing it using a generative model; means for generating personalized feedback for the user based on the determined emotional state; and means for notifying the user of the feedback via a terminal and supporting appropriate care. This makes it possible to comprehensively understand the user's physical and psychological health status and provide appropriate support.

[0161] "Physical and psychological data" refers to physiological information indicating the user's health status and psychological indicators representing their emotional state.

[0162] A "generative model" refers to an artificial intelligence technology that uses algorithms learned through machine learning to generate responses to new data.

[0163] "Emotional state" refers to the user's current psychological feelings and mood, and is analyzed from voice tone and facial expression data.

[0164] "Feedback" refers to the information and suggestions provided to users based on analysis results, with the aim of improving the users' health and well-being.

[0165] "Personalized feedback" refers to information and suggestions that are customized to take into account the user's specific health and emotional state.

[0166] A "terminal" refers to a digital device used to collect, transmit, and notify data between the user and the system.

[0167] This invention is an integrated system for monitoring the physical and psychological health status of users and providing care support. The system mainly consists of terminals and a server.

[0168] The device is equipped with a camera and microphone, and collects the user's voice and video data in real time. This allows for the acquisition of physical and psychological data, including changes in the user's facial expressions and voice tone.

[0169] The server analyzes the received data in detail using a dedicated emotion analysis engine. This emotion analysis engine uses a generative AI model to determine the user's emotional state based on information obtained from audio and video. This analysis identifies the user's emotional state.

[0170] Based on the identified emotional state, the server generates personalized feedback. This feedback is tailored to support the user's health and psychological well-being. For example, if fatigue is detected, a suggestion such as "Take a short break and take some deep breaths" might be generated. Additionally, relaxation music or simple exercises may be suggested as needed.

[0171] The generated feedback is communicated to the user through the device. The user is informed either by a message appearing on the device's display or by a voice assistant reading the feedback aloud.

[0172] For example, if a user is practicing the piano on their device, but the neural network detects that their concentration is declining based on facial recognition, the server will send feedback to the device saying, "We recommend you take a break and refresh yourself." This allows the user to take appropriate action immediately.

[0173] An example of a prompt message might be, "Please help the user relax. Advice that takes their current emotional state into consideration is needed." In this way, the system provides valuable support to the user.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The device acquires the user's voice and video data. This is done using the high-performance camera and microphone built into the device. Inputs include the user's facial expressions and voice. Outputs include high-resolution image data and audio data, which are sent to the server in real time.

[0177] Step 2:

[0178] The server analyzes the received data. This analysis uses an emotion analysis engine based on a generative AI model. The input consists of audio and video data obtained in step 1. The server first extracts the tone of the audio and the facial expression patterns from the video. Based on this, it determines the user's emotional state (e.g., joy, sadness, fatigue). The output is a description of the determined emotional state.

[0179] Step 3:

[0180] The server generates feedback based on the identified emotional state. The input is the emotional state output in step 2. The server utilizes a generative AI model to generate a prompt corresponding to this emotional state. Specifically, the feedback includes optimal action suggestions, such as "We recommend taking a short break." The output is the generated feedback message.

[0181] Step 4:

[0182] The terminal notifies the user of feedback sent from the server. The input is the feedback message generated in step 3. The terminal either displays the message on its screen or reads the feedback aloud via voice output. The output consists of specific suggestions and advice conveyed to the user.

[0183] Step 5:

[0184] The user modifies their behavior based on the feedback provided. The input is the feedback received in step 4. For example, the user might take a break or do a simple exercise to change their mood. The expected output is an improvement in the user's state.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] In households with elderly individuals or those living alone, there is a need to manage not only the physical health but also the psychological health of users simultaneously. However, conventional health monitoring systems have difficulty providing individualized feedback that takes emotional states into account, and have been insufficient to support the overall health of users. Furthermore, there is a need to identify changes in emotions in real time and provide appropriate support accordingly.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] This invention includes a server that collects physiological and emotional data obtained from users, analyzes this data to monitor the user's health and emotional state, and detects abnormalities; means for notifying users through a pre-configured communication channel when an abnormality is detected; and means for analyzing the collected data based on a machine learning model to identify trends in health and emotional state. This makes it possible to comprehensively manage the user's physical and emotional health and provide individually optimized feedback.

[0190] "Physiological data" refers to information that indicates the user's physical health status, including vital signs such as heart rate, blood pressure, and body temperature.

[0191] "Emotional data" refers to information that indicates the user's psychological state, and includes emotional indicators derived from facial expressions, tone of voice, and linguistic expressions.

[0192] A "machine learning model" is an algorithm that learns from large amounts of data and identifies patterns, and is a method for making predictions and classifications based on new data.

[0193] A "communication channel" is a path for sending and receiving data, and is a means of transmitting information using technologies such as the internet, telephone lines, and wireless communication.

[0194] "Feedback" refers to the information and instructions that a system provides to a user, which are adjusted based on the user's actions and circumstances.

[0195] A "server" is a computer system that provides data processing and storage functions, and is a device that provides services while communicating with other devices over a network.

[0196] The system that realizes this invention is superior in that it monitors the user's emotional state and provides dynamically adaptable responses. The terminal is equipped with hardware such as a camera and microphone, which are used to acquire the user's voice and video data in real time. This data is transmitted to a server equipped with an emotion recognition engine.

[0197] After receiving the data, the server uses a machine learning model to analyze the tone of voice and facial expression patterns. This model utilizes software such as TensorFlow and PyTorch. The trend information on emotional states obtained through the analysis is integrated with accumulated health data to evaluate the user's overall health and psychological state.

[0198] Based on the evaluation results, the server generates appropriate feedback and provides it to the user through the terminal. The feedback is delivered via audio, video, or a combination of both, and may include relaxing music or encouraging words depending on the user's state.

[0199] For example, if a user indicates fatigue, the server generates feedback such as, "You seem a little tired. I'll play some music of your choice to help you relax." Furthermore, by inputting instructions such as, "If the user is feeling stressed, suggest actions to alleviate it, such as playing music or suggesting stretching," into the AI ​​model, personalized responses can be provided. This system enables users to live a richer and more comfortable life.

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The device acquires the user's voice and video data. It captures the user's facial expressions through the camera and records the user's voice in real time using the microphone. Since this data is processed digitally, signal processing is performed to reduce noise. The input is the user's real-world facial expressions and voice, and the output is digital data.

[0203] Step 2:

[0204] The terminal transmits the acquired audio and video data to the server. A secure communication channel is used for transmission, and the data is encrypted to maintain confidentiality. The input is the digital data obtained in the previous step, and the output is the encrypted data that has reached the server.

[0205] Step 3:

[0206] The server decodes the received data and analyzes the user's emotional state using an emotion recognition engine. The data is then fed into a machine learning model, which uses a specific algorithm (e.g., a neural network) to analyze the tone of voice and facial features. The input is the decoded data, and the output is a metric indicating the emotional state.

[0207] Step 4:

[0208] The server performs a comprehensive assessment based on emotional states, integrating them with health data. It identifies emotional trends and compares them to past health database data to evaluate the user's psychological and physical state. Inputs are emotional metrics and existing health data, and output is the integrated assessment result.

[0209] Step 5:

[0210] The server generates appropriate feedback based on the evaluation results. It utilizes a generative AI model and constructs personalized responses using prompts. For example, it might generate feedback such as, "If the user appears stressed, recommend relaxing music to alleviate it." The input is the evaluation result, and the output is the specific feedback content.

[0211] Step 6:

[0212] The server sends the generated feedback to the terminal, and the terminal provides the feedback to the user. The feedback is visualized to the user in the form of audio or video, improving the quality of the interaction. The input is the feedback content, and the output is the audio or visual information received by the user.

[0213] Step 7:

[0214] The user receives feedback from the device and provides further interaction or instructions as needed. This allows the system to continuously adapt to the user's situation and maintain real-time support. The input is feedback input, and the output is the user's response.

[0215] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0216] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0217] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0218] [Second Embodiment]

[0219] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0220] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0221] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0222] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0223] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0225] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0226] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0227] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0228] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0229] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0230] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0231] The care support system of the present invention includes a program for comprehensively supporting the user's health condition. The system functions through the cooperation of a terminal, a server, and the user, and comprehensively manages the user's health.

[0232] The terminal collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. This collected data is temporarily stored on the terminal and then sent to the server at regular intervals.

[0233] The server stores the received data and analyzes it using a machine learning model to determine if there are any anomalies by comparing it with past data. If an anomaly is detected, the server sends a notification to the user and the necessary medical institutions through a pre-configured communication channel. This notification includes an overview of the health status and details of the anomaly, prompting the user to take prompt action.

[0234] Furthermore, the server generates feedback based on data analysis, indicating the user's health trends. This feedback is sent to the device as advice for maintaining or improving the user's health, and the device presents it to the user in either voice or text format.

[0235] Furthermore, the device tracks the user's movements through an AR device to support rehabilitation and exercise. The server analyzes the movement data and provides feedback on appropriate exercise methods and posture to support effective rehabilitation.

[0236] For the dialogue function, the terminal collects voice and video data from the user, and the server analyzes it using natural language processing technology. Based on the analysis results, the server generates an appropriate response and interacts with the user through the terminal, thereby reducing the user's feelings of isolation.

[0237] For example, if a user experiences an emergency such as a fall, the device uses an accelerometer and acoustic analysis to detect the anomaly and immediately reports it to the server. The server then automatically notifies registered emergency contacts, enabling a rapid response.

[0238] The following describes the processing flow.

[0239] Step 1:

[0240] The device collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. The collected data is stored on the device for a short period of time.

[0241] Step 2:

[0242] The device sends the accumulated data to the server as packets at regular intervals (e.g., every 30 minutes). Data that has been successfully sent is deleted from the device.

[0243] Step 3:

[0244] The server stores health data received from the terminal in a database. The data is categorized by user and stored along with past history data.

[0245] Step 4:

[0246] The server inputs the stored data into a machine learning model to analyze health trends. If an anomaly is detected, the server evaluates the type and nature of the anomaly.

[0247] Step 5:

[0248] The server immediately determines the action to take when an anomaly is detected and sends a notification to the user or healthcare provider through the registered communication channel. The notification includes an overview of the health condition and how to proceed.

[0249] Step 6:

[0250] The server generates health maintenance feedback based on the analysis results and sends it to the device. The device then presents this feedback to the user in audio or text format.

[0251] Step 7:

[0252] When users perform rehabilitation or exercise, the device uses an AR device to track their movements. The server analyzes the movement data in real time and provides appropriate advice.

[0253] Step 8:

[0254] When a user uses the conversational function, the device collects audio and video data and sends it to the server. The server analyzes this data using natural language processing technology, generates the most appropriate response for the user, and communicates it through the device.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] Efficiently monitoring the health of the elderly and those with chronic illnesses in real time, and responding quickly to abnormalities, is crucial in modern welfare and healthcare. However, existing systems require the combination of multiple different technologies to efficiently analyze biometric information and provide integrated motion tracking and conversational functions, resulting in complexity and high costs. Therefore, effective methods to solve these problems are needed.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes means for monitoring the user's physical condition and detecting abnormalities by collecting and analyzing biometric information obtained from the user; means for notifying the user through a pre-configured communication path when an abnormality is detected; means for analyzing the collected information based on a learning model and identifying trends in the physical condition; means for generating and presenting advice useful for maintaining the user's health based on the analysis results; and means for tracking the user's movements using augmented reality technology and providing rehabilitation and exercise support. This makes it possible to integrate health monitoring, rapid response to abnormalities, and the provision of advice for improving health conditions.

[0260] "Users" refer to individuals who use the system to receive services such as health monitoring and rehabilitation.

[0261] "Biometric information" refers to data such as heart rate, steps taken, and body temperature, which are collected to understand the user's health status.

[0262] A "server" refers to a computing device that stores and analyzes collected data and generates necessary notifications and feedback.

[0263] A "learning model" refers to an algorithm used to analyze data using machine learning techniques and identify trends and anomalies.

[0264] "Communication path" refers to the means of communication used to transmit abnormality notifications and feedback to users and medical institutions, and includes email, SMS, etc.

[0265] Augmented reality technology is a technology that overlays virtual information onto the real world environment and is used to track user movements and provide health support.

[0266] "Health monitoring" refers to the process of observing the user's health status in real time and detecting any abnormalities.

[0267] "Rehabilitation" refers to the process of providing exercise therapy and feedback aimed at improving the physical function of the user.

[0268] This invention provides a care support system that collects and analyzes user health information from multiple perspectives, enabling real-time health management and rapid response to abnormalities. The system operates with terminals (including wearable devices), a server, and the user working together as a single unit.

[0269] The terminal collects biometric information (e.g., heart rate, steps, body temperature) from wearable devices worn by the user. These devices could be commonly referred to as portable vital signs sensors. The terminal acquires data in real time via Bluetooth or Wi-Fi and temporarily stores it on its own.

[0270] The server receives and stores data transmitted from the terminal and analyzes it using a learning model (e.g., an algorithm using TensorFlow). Based on the analysis results, the server immediately notifies the user and, if necessary, the medical institution if an anomaly is detected. This is done using a configured communication channel (e.g., email or SMS).

[0271] The server also uses accumulated data to understand long-term health trends and generates advice that can help improve the user's health. This advice is provided through the terminal and can also be suggested in voice using speech synthesis software (e.g., a general term for a speech generation system).

[0272] Furthermore, the device utilizes augmented reality technology to track and analyze the user's movements, providing support for rehabilitation and exercise. This data is used for analysis on a server, and feedback is provided on appropriate exercise methods.

[0273] If a user encounters an emergency, the device uses its built-in sensors to detect the situation and report it to the server. This enables a quick and appropriate response. For example, there is a function that issues an alarm based on acceleration data when a user falls.

[0274] Examples of prompts for a generative AI model:

[0275] "Design an algorithm to analyze heart rate data collected from wearable sensors and detect anomalies."

[0276] "Please develop a system that suggests appropriate rehabilitation methods based on user activity data."

[0277] Thus, this invention is a system that utilizes advanced technology to comprehensively support the user's daily health.

[0278] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0279] Step 1:

[0280] The terminal collects biometric information such as heart rate, step count, body temperature, etc. from the wearable device in real time. The input is the sensor data obtained from the wearable device. Based on this, the terminal temporarily stores the data and organizes it for transmission in the next step. As a specific operation, the device measures the heart rate per minute, and the terminal receives the data via Bluetooth.

[0281] Step 2:

[0282] The terminal transmits the collected data to the server at regular time intervals (e.g., every hour). The input is the biometric information stored in the terminal, and the output is the data transmission to the server. As data processing, the terminal converts the data format and transmits it via the HTTPS protocol. As a specific operation, the terminal establishes a secure connection and uploads the aggregated data to the server.

[0283] Step 3:

[0284] The server opens the data received from the terminal and accumulates it in the database at the same time. The input is the biometric information transmitted from the terminal, and the output is the record to the database. As data calculation, the server manages the time series of the data and prepares to compare the anomaly detection border lines. As a specific operation, the server creates a new entry in the database and adds the data to the corresponding user's history.

[0285] Step 4:

[0286] The server analyzes the accumulated data using a machine learning model to detect anomalies. The input is the past biometric information stored in the database, and the output is the result of determining whether there is an anomaly. As data analysis, the server filters the data through the model and makes a prediction determination. As a specific operation, the server executes an anomaly detection algorithm to identify data points that exceed the normal range.

[0287] Step 5:

[0288] If an anomaly is detected, the server notifies the user through a pre-set communication path. The input is the detection result of the anomaly, and the output is an alarm message to the user. As data processing, the server generates a notification message and prepares to send it via email or SMS. As a specific operation, the server creates a notification message and sends it based on the registered contact information.

[0289] Step 6:

[0290] The server analyzes the health trends using the collected data and conducts a long-term health status assessment. The input is all the historical data in the database, and the output is a report on the health trends. As data calculation, the server applies a trend analysis algorithm to generate indicators for visualizing the health status. As a specific operation, the server generates a health trend report and visualizes the results based on the algorithm used.

[0291] Step 7:

[0292] The server sends the generated health trend report and advice to the terminal and proposes them to the user. The input is the health trend report, and the output is the feedback received by the user. As data processing, the server converts the content into voice or text and presents it to the user. As a specific operation, the terminal uses AV technology to read out the report and provide specific health advice to the user.

[0293] Step 8:

[0294] The device utilizes augmented reality technology to monitor user movements and provide rehabilitation and exercise support. Input is movement data from the AR system installed on the device, and output is feedback on how to improve. For data analysis, the device analyzes movement parameters and guides the user toward correct form and exercise methods. Specifically, the device uses AR feedback to guide the user toward correct posture during exercise.

[0295] (Application Example 1)

[0296] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0297] In an aging society, there is a growing need for individual health management and prompt response to emergencies. Traditional manual health monitoring has limitations due to labor shortages and lack of knowledge, and further challenges include the inability to alleviate feelings of isolation and provide appropriate exercise guidance. In particular, the insufficient ability to respond quickly in emergencies and to continuously monitor health trends are problematic.

[0298] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0299] In this invention, the server includes means for collecting and analyzing biometric information in real time, means for tracking actions and providing feedback using augmented reality technology, and means for analyzing conversational data and generating responses using named entity recognition technology. This enables comprehensive monitoring of the user's health status, rapid detection and response to abnormalities, and furthermore, reduces feelings of loneliness and provides effective exercise guidance.

[0300] "Biometric information" refers to data such as heart rate, body temperature, and step count that indicates the user's health status.

[0301] "Analysis" refers to the data processing activity used to identify anomalies and trends in collected data.

[0302] "Augmented Reality Technology" is a technology that captures the physical actions of users as digital data and provides appropriate feedback.

[0303] "Tracking actions" is the process of observing the movements of the user's body in real time and analyzing them as digital information.

[0304] "Feedback" refers to the guidance or advice provided to the user based on the analysis results.

[0305] "Proprietary Expression Processing Technology" is a technology that automatically analyzes audio and video data and extracts meaningful information.

[0306] "Generating a response" is the act of creating an appropriate response to the user based on the analyzed data.

[0307] "Emergency" refers to a situation where an abnormality occurs in the user's normal health condition and a prompt response is required.

[0308] To implement this invention, a system is constructed in which a wearable device worn by the user, a smartphone, and a server cooperate. The wearable device is equipped with sensors for measuring heart rate, body temperature, number of steps, etc. These data are transmitted to the smartphone via Bluetooth communication, and the smartphone temporarily stores the data.

[0309] The server accumulates the biometric information received from the smartphone at specific time intervals. The server analyzes these data using a machine learning algorithm using TensorFlow to identify the presence or absence of abnormalities and health trends. The analysis results are fed back to the user in real time, and when an abnormality is detected, the user and medical institutions are notified through a pre-set communication channel. The notification is generally sent as a text message.

[0310] The smart glasses worn by the user utilize augmented reality technology to track the user's movements in real time. A server analyzes the movement data and provides effective exercise guidance by suggesting a suitable rehabilitation plan. This feedback is displayed through the smart glasses.

[0311] Furthermore, the smartphone collects the user's voice and camera footage, and the server employs named entity recognition technology to analyze this data. The server generates appropriate responses from the analyzed data, enabling conversation with the user. This process can utilize natural language generation models such as OpenAI.

[0312] As a concrete example, if the system detects a sudden decrease in a user's step count during the daytime, it recognizes this as an anomaly. To investigate the reason, the server generates questions such as, "How did you spend your day?" and confirms with the user via voice. An example of a prompt to the generating AI model could be, "Please provide questions to identify the reason for the decrease in user activity."

[0313] In this way, the present invention enables comprehensive health management and supports early detection of abnormalities and rapid response.

[0314] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0315] Step 1:

[0316] The terminal receives biometric information from wearable devices, collecting data such as heart rate, body temperature, and steps taken. The input is sensor data from the wearable device, and the output is the temporary storage of the collected data within the terminal. Data processing includes data format conversion and initial filtering.

[0317] Step 2:

[0318] The server receives data from terminals at regular intervals. The input is biometric information transmitted from the terminals, and the output is stored in the server's database. As a data processing step, the data is organized chronologically and saved to storage.

[0319] Step 3:

[0320] The server analyzes received biometric data using a machine learning algorithm based on TensorFlow to perform anomaly detection. The input is accumulated biometric data, and the output is the result of the anomaly detection. As part of data processing, it compares the data with past data to identify outliers and anomaly patterns.

[0321] Step 4:

[0322] If an anomaly is detected, the server sends a notification to the user or designated contact via the communication channel. The input is the anomaly detection result, and the output is a notification in text message format. The operation involves formatting and sending the message according to the communication protocol.

[0323] Step 5:

[0324] The device receives health trend feedback from the server and presents it to the user in audio or text. The input is an analysis report from the server, and the output is the feedback information displayed to the user. The system performs UI rendering to present the information in an easily understandable format for the user.

[0325] Step 6:

[0326] The server receives motion data from smart glasses and analyzes the motion using augmented reality technology. The input is motion data from the smart glasses, and the output is appropriate rehabilitation feedback. The data processing involves identifying the motion and evaluating its quality.

[0327] Step 7:

[0328] The user inputs audio and video data into the terminal, and the server analyzes the data using named entity recognition technology to generate a dialogue. The input is audio and video data, and the output is an appropriate response. The operation involves response generation using a natural language generation model. An example of a prompt sentence for the generating AI model is, "Generate a response to continue a natural conversation with the user."

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] The care support system incorporating the emotion recognition function of the present invention not only monitors the user's health condition but also recognizes the user's emotional state in detail and provides support based on that data.

[0331] The device is equipped with a camera and microphone that captures the user's voice and video in real time. This allows for the acquisition of data in a natural way as the user interacts with the system.

[0332] The server utilizes an emotion engine to process the collected audio and video data. This emotion engine analyzes the tone of voice and facial expression patterns to determine the user's emotional state. The determined emotional state is stored on the platform and integrated with health data to be used for an overall assessment of the user's health and psychological state.

[0333] For example, while a user is receiving exercise guidance using the conversational function, the device captures changes in the user's facial expressions and tone of voice. If the server determines that the user is expressing feelings of fatigue, it can provide feedback suggesting adjustments to the exercise intensity.

[0334] Furthermore, if the system detects that a user is emotionally unstable, it provides support to reduce anxiety by offering relaxation advice and engaging in reassuring conversations. This function helps alleviate the user's psychological burden and supports them in living their daily lives with peace of mind.

[0335] Through these implementations, the system integrates users' physical health data and emotional data to provide personalized and optimal care.

[0336] The following describes the processing flow.

[0337] Step 1:

[0338] The device collects health data from the wearable device worn by the user and simultaneously captures the user's voice and video data. This data is temporarily stored within the device.

[0339] Step 2:

[0340] The device sends health data, audio data, and video data to the server at regular intervals. The data is organized for each user and stored in the server's database.

[0341] Step 3:

[0342] The server inputs audio and video data into an emotion engine, which analyzes the intonation of the voice and changes in facial expressions. Based on this analysis, it identifies the user's emotional state.

[0343] Step 4:

[0344] The server integrates identified emotional states with accumulated health data to assess the user's overall psychological and health status. Machine learning models are used to predict abnormal trends and emotional trends.

[0345] Step 5:

[0346] If abnormalities or emotional instability are detected, the server generates optimal feedback and dialogue. This includes health management advice and relaxation suggestions.

[0347] Step 6:

[0348] The terminal presents the user with feedback and conversation content provided by the server. This is done through voice messages and screen displays.

[0349] Step 7:

[0350] Users act based on feedback from their devices and continuously manage their health and emotional state. User responses and additional data are collected again from the device and used as a feedback loop in the system.

[0351] (Example 2)

[0352] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0353] For elderly users and those with health concerns, it is challenging to simultaneously assess not only their physical health but also their psychological health and provide appropriate support. This project aims to improve the quality of life for users by addressing this challenge.

[0354] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0355] In this invention, the server includes means for collecting physical and psychological data obtained from the user and determining the emotional state by analyzing it using a generative model; means for generating personalized feedback for the user based on the determined emotional state; and means for notifying the user of the feedback via a terminal and supporting appropriate care. This makes it possible to comprehensively understand the user's physical and psychological health status and provide appropriate support.

[0356] "Physical and psychological data" refers to physiological information indicating the user's health status and psychological indicators representing their emotional state.

[0357] A "generative model" refers to an artificial intelligence technology that uses algorithms learned through machine learning to generate responses to new data.

[0358] "Emotional state" refers to the user's current psychological feelings and mood, and is analyzed from voice tone and facial expression data.

[0359] "Feedback" refers to the information and suggestions provided to users based on analysis results, with the aim of improving the users' health and well-being.

[0360] "Personalized feedback" refers to information and suggestions that are customized to take into account the user's specific health and emotional state.

[0361] A "terminal" refers to a digital device used to collect, transmit, and notify data between the user and the system.

[0362] This invention is an integrated system for monitoring the physical and psychological health status of users and providing care support. The system mainly consists of terminals and a server.

[0363] The device is equipped with a camera and microphone, and collects the user's voice and video data in real time. This allows for the acquisition of physical and psychological data, including changes in the user's facial expressions and voice tone.

[0364] The server analyzes the received data in detail using a dedicated emotion analysis engine. This emotion analysis engine uses a generative AI model to determine the user's emotional state based on information obtained from audio and video. This analysis identifies the user's emotional state.

[0365] Based on the identified emotional state, the server generates personalized feedback. This feedback is tailored to support the user's health and psychological well-being. For example, if fatigue is detected, a suggestion such as "Take a short break and take some deep breaths" might be generated. Additionally, relaxation music or simple exercises may be suggested as needed.

[0366] The generated feedback is communicated to the user through the device. The user is informed either by a message appearing on the device's display or by a voice assistant reading the feedback aloud.

[0367] For example, if a user is practicing the piano on their device, but the neural network detects that their concentration is declining based on facial recognition, the server will send feedback to the device saying, "We recommend you take a break and refresh yourself." This allows the user to take appropriate action immediately.

[0368] An example of a prompt message might be, "Please help the user relax. Advice that takes their current emotional state into consideration is needed." In this way, the system provides valuable support to the user.

[0369] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0370] Step 1:

[0371] The device acquires the user's voice and video data. This is done using the high-performance camera and microphone built into the device. Inputs include the user's facial expressions and voice. Outputs include high-resolution image data and audio data, which are sent to the server in real time.

[0372] Step 2:

[0373] The server analyzes the received data. This analysis uses an emotion analysis engine based on a generative AI model. The input consists of audio and video data obtained in step 1. The server first extracts the tone of the audio and the facial expression patterns from the video. Based on this, it determines the user's emotional state (e.g., joy, sadness, fatigue). The output is a description of the determined emotional state.

[0374] Step 3:

[0375] The server generates feedback based on the identified emotional state. The input is the emotional state output in step 2. The server utilizes a generative AI model to generate a prompt corresponding to this emotional state. Specifically, the feedback includes optimal action suggestions, such as "We recommend taking a short break." The output is the generated feedback message.

[0376] Step 4:

[0377] The terminal notifies the user of feedback sent from the server. The input is the feedback message generated in step 3. The terminal either displays the message on its screen or reads the feedback aloud via voice output. The output consists of specific suggestions and advice conveyed to the user.

[0378] Step 5:

[0379] The user modifies their behavior based on the feedback provided. The input is the feedback received in step 4. For example, the user might take a break or do a simple exercise to change their mood. The expected output is an improvement in the user's state.

[0380] (Application Example 2)

[0381] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0382] In households with elderly individuals or those living alone, there is a need to manage not only the physical health but also the psychological health of users simultaneously. However, conventional health monitoring systems have difficulty providing individualized feedback that takes emotional states into account, and have been insufficient to support the overall health of users. Furthermore, there is a need to identify changes in emotions in real time and provide appropriate support accordingly.

[0383] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0384] This invention includes a server that collects physiological and emotional data obtained from users, analyzes this data to monitor the user's health and emotional state, and detects abnormalities; means for notifying users through a pre-configured communication channel when an abnormality is detected; and means for analyzing the collected data based on a machine learning model to identify trends in health and emotional state. This makes it possible to comprehensively manage the user's physical and emotional health and provide individually optimized feedback.

[0385] "Physiological data" refers to information that indicates the user's physical health status, including vital signs such as heart rate, blood pressure, and body temperature.

[0386] "Emotional data" refers to information that indicates the user's psychological state, and includes emotional indicators derived from facial expressions, tone of voice, and linguistic expressions.

[0387] A "machine learning model" is an algorithm that learns from large amounts of data and identifies patterns, and is a method for making predictions and classifications based on new data.

[0388] A "communication channel" is a path for sending and receiving data, and is a means of transmitting information using technologies such as the internet, telephone lines, and wireless communication.

[0389] "Feedback" refers to the information and instructions that a system provides to a user, which are adjusted based on the user's actions and circumstances.

[0390] A "server" is a computer system that provides data processing and storage functions, and is a device that provides services while communicating with other devices over a network.

[0391] The system that realizes this invention is superior in that it monitors the user's emotional state and provides dynamically adaptable responses. The terminal is equipped with hardware such as a camera and microphone, which are used to acquire the user's voice and video data in real time. This data is transmitted to a server equipped with an emotion recognition engine.

[0392] After receiving the data, the server uses a machine learning model to analyze the tone of voice and facial expression patterns. This model utilizes software such as TensorFlow and PyTorch. The trend information on emotional states obtained through the analysis is integrated with accumulated health data to evaluate the user's overall health and psychological state.

[0393] Based on the evaluation results, the server generates appropriate feedback and provides it to the user through the terminal. The feedback is delivered via audio, video, or a combination of both, and may include relaxing music or encouraging words depending on the user's state.

[0394] For example, if a user indicates fatigue, the server generates feedback such as, "You seem a little tired. I'll play some music of your choice to help you relax." Furthermore, by inputting instructions such as, "If the user is feeling stressed, suggest actions to alleviate it, such as playing music or suggesting stretching," into the AI ​​model, personalized responses can be provided. This system enables users to live a richer and more comfortable life.

[0395] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0396] Step 1:

[0397] The device acquires the user's voice and video data. It captures the user's facial expressions through the camera and records the user's voice in real time using the microphone. Since this data is processed digitally, signal processing is performed to reduce noise. The input is the user's real-world facial expressions and voice, and the output is digital data.

[0398] Step 2:

[0399] The terminal transmits the acquired audio and video data to the server. A secure communication channel is used for transmission, and the data is encrypted to maintain confidentiality. The input is the digital data obtained in the previous step, and the output is the encrypted data that has reached the server.

[0400] Step 3:

[0401] The server decodes the received data and analyzes the user's emotional state using an emotion recognition engine. The data is then fed into a machine learning model, which uses a specific algorithm (e.g., a neural network) to analyze the tone of voice and facial features. The input is the decoded data, and the output is a metric indicating the emotional state.

[0402] Step 4:

[0403] The server performs a comprehensive assessment based on emotional states, integrating them with health data. It identifies emotional trends and compares them to past health database data to evaluate the user's psychological and physical state. Inputs are emotional metrics and existing health data, and output is the integrated assessment result.

[0404] Step 5:

[0405] The server generates appropriate feedback based on the evaluation results. It utilizes a generative AI model and constructs personalized responses using prompts. For example, it might generate feedback such as, "If the user appears stressed, recommend relaxing music to alleviate it." The input is the evaluation result, and the output is the specific feedback content.

[0406] Step 6:

[0407] The server sends the generated feedback to the terminal, and the terminal provides the feedback to the user. The feedback is visualized to the user in the form of audio or video, improving the quality of the interaction. The input is the feedback content, and the output is the audio or visual information received by the user.

[0408] Step 7:

[0409] The user receives feedback from the device and provides further interaction or instructions as needed. This allows the system to continuously adapt to the user's situation and maintain real-time support. The input is feedback input, and the output is the user's response.

[0410] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0411] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0412] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0413] [Third Embodiment]

[0414] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0415] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0416] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0417] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0418] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0420] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0421] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0422] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0423] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0424] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0425] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0426] The care support system of the present invention includes a program for comprehensively supporting the user's health condition. The system functions through the cooperation of a terminal, a server, and the user, and comprehensively manages the user's health.

[0427] The terminal collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. This collected data is temporarily stored on the terminal and then sent to the server at regular intervals.

[0428] The server stores the received data and analyzes it using a machine learning model to determine if there are any anomalies by comparing it with past data. If an anomaly is detected, the server sends a notification to the user and the necessary medical institutions through a pre-configured communication channel. This notification includes an overview of the health status and details of the anomaly, prompting the user to take prompt action.

[0429] Furthermore, the server generates feedback based on data analysis, indicating the user's health trends. This feedback is sent to the device as advice for maintaining or improving the user's health, and the device presents it to the user in either voice or text format.

[0430] Furthermore, the device tracks the user's movements through an AR device to support rehabilitation and exercise. The server analyzes the movement data and provides feedback on appropriate exercise methods and posture to support effective rehabilitation.

[0431] For the dialogue function, the terminal collects voice and video data from the user, and the server analyzes it using natural language processing technology. Based on the analysis results, the server generates an appropriate response and interacts with the user through the terminal, thereby reducing the user's feelings of isolation.

[0432] For example, if a user experiences an emergency such as a fall, the device uses an accelerometer and acoustic analysis to detect the anomaly and immediately reports it to the server. The server then automatically notifies registered emergency contacts, enabling a rapid response.

[0433] The following describes the processing flow.

[0434] Step 1:

[0435] The device collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. The collected data is stored on the device for a short period of time.

[0436] Step 2:

[0437] The device sends the accumulated data to the server as packets at regular intervals (e.g., every 30 minutes). Data that has been successfully sent is deleted from the device.

[0438] Step 3:

[0439] The server stores health data received from the terminal in a database. The data is categorized by user and stored along with past history data.

[0440] Step 4:

[0441] The server inputs the stored data into a machine learning model to analyze health trends. If an anomaly is detected, the server evaluates the type and nature of the anomaly.

[0442] Step 5:

[0443] The server immediately determines the action to take when an anomaly is detected and sends a notification to the user or healthcare provider through the registered communication channel. The notification includes an overview of the health condition and how to proceed.

[0444] Step 6:

[0445] The server generates health maintenance feedback based on the analysis results and sends it to the device. The device then presents this feedback to the user in audio or text format.

[0446] Step 7:

[0447] When users perform rehabilitation or exercise, the device uses an AR device to track their movements. The server analyzes the movement data in real time and provides appropriate advice.

[0448] Step 8:

[0449] When a user uses the conversational function, the device collects audio and video data and sends it to the server. The server analyzes this data using natural language processing technology, generates the most appropriate response for the user, and communicates it through the device.

[0450] (Example 1)

[0451] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0452] Efficiently monitoring the health of the elderly and those with chronic illnesses in real time, and responding quickly to abnormalities, is crucial in modern welfare and healthcare. However, existing systems require the combination of multiple different technologies to efficiently analyze biometric information and provide integrated motion tracking and conversational functions, resulting in complexity and high costs. Therefore, effective methods to solve these problems are needed.

[0453] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0454] In this invention, the server includes means for monitoring the user's physical condition and detecting abnormalities by collecting and analyzing biometric information obtained from the user; means for notifying the user through a pre-configured communication path when an abnormality is detected; means for analyzing the collected information based on a learning model and identifying trends in the physical condition; means for generating and presenting advice useful for maintaining the user's health based on the analysis results; and means for tracking the user's movements using augmented reality technology and providing rehabilitation and exercise support. This makes it possible to integrate health monitoring, rapid response to abnormalities, and the provision of advice for improving health conditions.

[0455] "Users" refer to individuals who use the system to receive services such as health monitoring and rehabilitation.

[0456] "Biometric information" refers to data such as heart rate, steps taken, and body temperature, which are collected to understand the user's health status.

[0457] A "server" refers to a computing device that stores and analyzes collected data and generates necessary notifications and feedback.

[0458] A "learning model" refers to an algorithm used to analyze data using machine learning techniques and identify trends and anomalies.

[0459] "Communication path" refers to the means of communication used to transmit abnormality notifications and feedback to users and medical institutions, and includes email, SMS, etc.

[0460] Augmented reality technology is a technology that overlays virtual information onto the real world environment and is used to track user movements and provide health support.

[0461] "Health monitoring" refers to the process of observing the user's health status in real time and detecting any abnormalities.

[0462] "Rehabilitation" refers to the process of providing exercise therapy and feedback aimed at improving the physical function of the user.

[0463] This invention provides a care support system that collects and analyzes user health information from multiple perspectives, enabling real-time health management and rapid response to abnormalities. The system operates with terminals (including wearable devices), a server, and the user working together as a single unit.

[0464] The terminal collects biometric information (e.g., heart rate, steps, body temperature) from wearable devices worn by the user. These devices could be commonly referred to as portable vital signs sensors. The terminal acquires data in real time via Bluetooth or Wi-Fi and temporarily stores it on its own.

[0465] The server receives and stores data transmitted from the terminal and analyzes it using a learning model (e.g., an algorithm using TensorFlow). Based on the analysis results, the server immediately notifies the user and, if necessary, the medical institution if an anomaly is detected. This is done using a configured communication channel (e.g., email or SMS).

[0466] The server also uses accumulated data to understand long-term health trends and generates advice that can help improve the user's health. This advice is provided through the terminal and can also be suggested in voice using speech synthesis software (e.g., a general term for a speech generation system).

[0467] Furthermore, the device utilizes augmented reality technology to track and analyze the user's movements, providing support for rehabilitation and exercise. This data is used for analysis on a server, and feedback is provided on appropriate exercise methods.

[0468] If a user encounters an emergency, the device uses its built-in sensors to detect the situation and report it to the server. This enables a quick and appropriate response. For example, there is a function that issues an alarm based on acceleration data when a user falls.

[0469] Examples of prompts for a generative AI model:

[0470] "Design an algorithm to analyze heart rate data collected from wearable sensors and detect anomalies."

[0471] "Please develop a system that suggests appropriate rehabilitation methods based on user activity data."

[0472] Thus, this invention is a system that utilizes advanced technology to comprehensively support the user's daily health.

[0473] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0474] Step 1:

[0475] The terminal collects biometric information such as heart rate, steps taken, and body temperature in real time from wearable devices. The input is sensor data obtained from the wearable device. Based on this, the terminal temporarily stores the data and organizes it for transmission in the next step. Specifically, the device measures the heart rate every minute, and the terminal receives the data via Bluetooth.

[0476] Step 2:

[0477] The device sends the collected data to the server at regular time intervals (for example, every hour). The input is biometric information stored on the device, and the output is the transmission of data to the server. As part of the data processing, the device converts the data format and sends it using the HTTPS protocol. Specifically, the device establishes a secure connection and uploads the compiled data to the server.

[0478] Step 3:

[0479] The server opens the data received from the terminal and simultaneously stores it in the database. The input is biometric information sent from the terminal, and the output is a record in the database. In terms of data processing, the server manages the time series of the data and prepares to compare it with anomaly detection borderlines. Specifically, the server creates a new entry in the database and adds the data to the history of the corresponding user.

[0480] Step 4:

[0481] The server analyzes accumulated data using a machine learning model to detect anomalies. The input is historical biometric information stored in the database, and the output is a determination of whether or not an anomaly exists. As part of the data analysis, the server filters the data through the model and makes predictive judgments. Specifically, the server executes an anomaly detection algorithm to identify data points that fall outside the normal range.

[0482] Step 5:

[0483] If an anomaly is detected, the server notifies the user through a pre-configured communication channel. The input is the anomaly detection result, and the output is an alarm message to the user. As part of the data processing, the server generates a notification message and prepares it for sending via email or SMS. Specifically, the server creates a notification message and sends it based on the registered contact information.

[0484] Step 6:

[0485] The server analyzes health trends using collected data and performs long-term health status assessments. The input is the entire historical data in the database, and the output is a health trend report. As a data calculation, the server applies trend analysis algorithms and generates indicators to visualize health status. Specifically, the server generates a health trend report and visualizes the results based on the algorithm used.

[0486] Step 7:

[0487] The server sends the generated health trend report and advice to the terminal, suggesting it to the user. The input is the health trend report, and the output is the feedback the user receives. As part of data processing, the server converts the content into audio or text and presents it to the user. Specifically, the terminal uses AV technology to read the report aloud and provides the user with specific health advice.

[0488] Step 8:

[0489] The device utilizes augmented reality technology to monitor user movements and provide rehabilitation and exercise support. Input is movement data from the AR system installed on the device, and output is feedback on how to improve. For data analysis, the device analyzes movement parameters and guides the user toward correct form and exercise methods. Specifically, the device uses AR feedback to guide the user toward correct posture during exercise.

[0490] (Application Example 1)

[0491] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0492] In an aging society, there is a growing need for individual health management and prompt response to emergencies. Traditional manual health monitoring has limitations due to labor shortages and lack of knowledge, and further challenges include the inability to alleviate feelings of isolation and provide appropriate exercise guidance. In particular, the insufficient ability to respond quickly in emergencies and to continuously monitor health trends are problematic.

[0493] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0494] In this invention, the server includes means for collecting and analyzing biometric information in real time, means for tracking actions and providing feedback using augmented reality technology, and means for analyzing conversational data and generating responses using named entity recognition technology. This enables comprehensive monitoring of the user's health status, rapid detection and response to abnormalities, and furthermore, reduces feelings of loneliness and provides effective exercise guidance.

[0495] "Biometric information" refers to data such as heart rate, body temperature, and step count that indicates the user's health status.

[0496] "Analysis" refers to the data processing activity used to identify anomalies and trends in collected data.

[0497] "Augmented reality technology" is a technology that captures a user's physical actions as digital data and provides appropriate feedback.

[0498] "Motion tracking" is the process of observing a user's body movements in real time and analyzing them as digital information.

[0499] "Feedback" refers to guidance and advice provided to users based on the results of analysis.

[0500] Named entity recognition technology is a technology that automatically analyzes audio and video data and extracts meaningful information.

[0501] "Generating a response" refers to the act of creating an appropriate reply to the user based on the analyzed data.

[0502] An "emergency situation" refers to a situation where a user's normal health condition is abnormal and requires a swift response.

[0503] To implement this invention, a system is constructed in which a wearable device worn by the user, a smartphone, and a server work together. The wearable device is equipped with sensors that measure heart rate, body temperature, steps taken, etc. This data is transmitted to the smartphone via Bluetooth communication, and the smartphone temporarily stores this data.

[0504] The server stores biometric data received from smartphones at specific time intervals. Using machine learning algorithms based on TensorFlow, the server analyzes this data to identify abnormalities and health trends. The analysis results are fed back to the user in real time, and in the event of an anomaly, notifications are sent to the user and medical institutions via pre-configured communication channels. These notifications are typically sent as text messages.

[0505] The smart glasses worn by the user utilize augmented reality technology to track the user's movements in real time. A server analyzes the movement data and provides effective exercise guidance by suggesting a suitable rehabilitation plan. This feedback is displayed through the smart glasses.

[0506] Furthermore, the smartphone collects the user's voice and camera footage, and the server employs named entity recognition technology to analyze this data. The server generates appropriate responses from the analyzed data, enabling conversation with the user. This process can utilize natural language generation models such as OpenAI.

[0507] As a concrete example, if the system detects a sudden decrease in a user's step count during the daytime, it recognizes this as an anomaly. To investigate the reason, the server generates questions such as, "How did you spend your day?" and confirms with the user via voice. An example of a prompt to the generating AI model could be, "Please provide questions to identify the reason for the decrease in user activity."

[0508] In this way, the present invention enables comprehensive health management and supports early detection of abnormalities and rapid response.

[0509] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0510] Step 1:

[0511] The terminal receives biometric information from wearable devices, collecting data such as heart rate, body temperature, and steps taken. The input is sensor data from the wearable device, and the output is the temporary storage of the collected data within the terminal. Data processing includes data format conversion and initial filtering.

[0512] Step 2:

[0513] The server receives data from terminals at regular intervals. The input is biometric information transmitted from the terminals, and the output is stored in the server's database. As a data processing step, the data is organized chronologically and saved to storage.

[0514] Step 3:

[0515] The server analyzes received biometric data using a machine learning algorithm based on TensorFlow to perform anomaly detection. The input is accumulated biometric data, and the output is the result of the anomaly detection. As part of data processing, it compares the data with past data to identify outliers and anomaly patterns.

[0516] Step 4:

[0517] If an anomaly is detected, the server sends a notification to the user or designated contact via the communication channel. The input is the anomaly detection result, and the output is a notification in text message format. The operation involves formatting and sending the message according to the communication protocol.

[0518] Step 5:

[0519] The device receives health trend feedback from the server and presents it to the user in audio or text. The input is an analysis report from the server, and the output is the feedback information displayed to the user. The system performs UI rendering to present the information in an easily understandable format for the user.

[0520] Step 6:

[0521] The server receives motion data from smart glasses and analyzes the motion using augmented reality technology. The input is motion data from the smart glasses, and the output is appropriate rehabilitation feedback. The data processing involves identifying the motion and evaluating its quality.

[0522] Step 7:

[0523] The user inputs audio and video data into the terminal, and the server analyzes the data using named entity recognition technology to generate a dialogue. The input is audio and video data, and the output is an appropriate response. The operation involves response generation using a natural language generation model. An example of a prompt sentence for the generating AI model is, "Generate a response to continue a natural conversation with the user."

[0524] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0525] The care support system incorporating the emotion recognition function of the present invention not only monitors the user's health condition but also recognizes the user's emotional state in detail and provides support based on that data.

[0526] The device is equipped with a camera and microphone that captures the user's voice and video in real time. This allows for the acquisition of data in a natural way as the user interacts with the system.

[0527] The server utilizes an emotion engine to process the collected audio and video data. This emotion engine analyzes the tone of voice and facial expression patterns to determine the user's emotional state. The determined emotional state is stored on the platform and integrated with health data to be used for an overall assessment of the user's health and psychological state.

[0528] For example, while a user is receiving exercise guidance using the conversational function, the device captures changes in the user's facial expressions and tone of voice. If the server determines that the user is expressing feelings of fatigue, it can provide feedback suggesting adjustments to the exercise intensity.

[0529] Furthermore, if the system detects that a user is emotionally unstable, it provides support to reduce anxiety by offering relaxation advice and engaging in reassuring conversations. This function helps alleviate the user's psychological burden and supports them in living their daily lives with peace of mind.

[0530] Through these implementations, the system integrates users' physical health data and emotional data to provide personalized and optimal care.

[0531] The following describes the processing flow.

[0532] Step 1:

[0533] The device collects health data from the wearable device worn by the user and simultaneously captures the user's voice and video data. This data is temporarily stored within the device.

[0534] Step 2:

[0535] The device sends health data, audio data, and video data to the server at regular intervals. The data is organized for each user and stored in the server's database.

[0536] Step 3:

[0537] The server inputs audio and video data into an emotion engine, which analyzes the intonation of the voice and changes in facial expressions. Based on this analysis, it identifies the user's emotional state.

[0538] Step 4:

[0539] The server integrates identified emotional states with accumulated health data to assess the user's overall psychological and health status. Machine learning models are used to predict abnormal trends and emotional trends.

[0540] Step 5:

[0541] If abnormalities or emotional instability are detected, the server generates optimal feedback and dialogue. This includes health management advice and relaxation suggestions.

[0542] Step 6:

[0543] The terminal presents the user with feedback and conversation content provided by the server. This is done through voice messages and screen displays.

[0544] Step 7:

[0545] Users act based on feedback from their devices and continuously manage their health and emotional state. User responses and additional data are collected again from the device and used as a feedback loop in the system.

[0546] (Example 2)

[0547] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0548] For elderly users and those with health concerns, it is challenging to simultaneously assess not only their physical health but also their psychological health and provide appropriate support. This project aims to improve the quality of life for users by addressing this challenge.

[0549] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0550] In this invention, the server includes means for collecting physical and psychological data obtained from the user and determining the emotional state by analyzing it using a generative model; means for generating personalized feedback for the user based on the determined emotional state; and means for notifying the user of the feedback via a terminal and supporting appropriate care. This makes it possible to comprehensively understand the user's physical and psychological health status and provide appropriate support.

[0551] "Physical and psychological data" refers to physiological information indicating the user's health status and psychological indicators representing their emotional state.

[0552] A "generative model" refers to an artificial intelligence technology that uses algorithms learned through machine learning to generate responses to new data.

[0553] "Emotional state" refers to the user's current psychological feelings and mood, and is analyzed from voice tone and facial expression data.

[0554] "Feedback" refers to the information and suggestions provided to users based on analysis results, with the aim of improving the users' health and well-being.

[0555] "Personalized feedback" refers to information and suggestions that are customized to take into account the user's specific health and emotional state.

[0556] A "terminal" refers to a digital device used to collect, transmit, and notify data between the user and the system.

[0557] This invention is an integrated system for monitoring the physical and psychological health status of users and providing care support. The system mainly consists of terminals and a server.

[0558] The device is equipped with a camera and microphone, and collects the user's voice and video data in real time. This allows for the acquisition of physical and psychological data, including changes in the user's facial expressions and voice tone.

[0559] The server analyzes the received data in detail using a dedicated emotion analysis engine. This emotion analysis engine uses a generative AI model to determine the user's emotional state based on information obtained from audio and video. This analysis identifies the user's emotional state.

[0560] Based on the identified emotional state, the server generates personalized feedback. This feedback is tailored to support the user's health and psychological well-being. For example, if fatigue is detected, a suggestion such as "Take a short break and take some deep breaths" might be generated. Additionally, relaxation music or simple exercises may be suggested as needed.

[0561] The generated feedback is communicated to the user through the device. The user is informed either by a message appearing on the device's display or by a voice assistant reading the feedback aloud.

[0562] For example, if a user is practicing the piano on their device, but the neural network detects that their concentration is declining based on facial recognition, the server will send feedback to the device saying, "We recommend you take a break and refresh yourself." This allows the user to take appropriate action immediately.

[0563] An example of a prompt message might be, "Please help the user relax. Advice that takes their current emotional state into consideration is needed." In this way, the system provides valuable support to the user.

[0564] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0565] Step 1:

[0566] The device acquires the user's voice and video data. This is done using the high-performance camera and microphone built into the device. Inputs include the user's facial expressions and voice. Outputs include high-resolution image data and audio data, which are sent to the server in real time.

[0567] Step 2:

[0568] The server analyzes the received data. This analysis uses an emotion analysis engine based on a generative AI model. The input consists of audio and video data obtained in step 1. The server first extracts the tone of the audio and the facial expression patterns from the video. Based on this, it determines the user's emotional state (e.g., joy, sadness, fatigue). The output is a description of the determined emotional state.

[0569] Step 3:

[0570] The server generates feedback based on the identified emotional state. The input is the emotional state output in step 2. The server utilizes a generative AI model to generate a prompt corresponding to this emotional state. Specifically, the feedback includes optimal action suggestions, such as "We recommend taking a short break." The output is the generated feedback message.

[0571] Step 4:

[0572] The terminal notifies the user of feedback sent from the server. The input is the feedback message generated in step 3. The terminal either displays the message on its screen or reads the feedback aloud via voice output. The output consists of specific suggestions and advice conveyed to the user.

[0573] Step 5:

[0574] The user modifies their behavior based on the feedback provided. The input is the feedback received in step 4. For example, the user might take a break or do a simple exercise to change their mood. The expected output is an improvement in the user's state.

[0575] (Application Example 2)

[0576] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0577] In households with elderly individuals or those living alone, there is a need to manage not only the physical health but also the psychological health of users simultaneously. However, conventional health monitoring systems have difficulty providing individualized feedback that takes emotional states into account, and have been insufficient to support the overall health of users. Furthermore, there is a need to identify changes in emotions in real time and provide appropriate support accordingly.

[0578] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0579] This invention includes a server that collects physiological and emotional data obtained from users, analyzes this data to monitor the user's health and emotional state, and detects abnormalities; means for notifying users through a pre-configured communication channel when an abnormality is detected; and means for analyzing the collected data based on a machine learning model to identify trends in health and emotional state. This makes it possible to comprehensively manage the user's physical and emotional health and provide individually optimized feedback.

[0580] "Physiological data" refers to information that indicates the user's physical health status, including vital signs such as heart rate, blood pressure, and body temperature.

[0581] "Emotional data" refers to information that indicates the user's psychological state, and includes emotional indicators derived from facial expressions, tone of voice, and linguistic expressions.

[0582] A "machine learning model" is an algorithm that learns from large amounts of data and identifies patterns, and is a method for making predictions and classifications based on new data.

[0583] A "communication channel" is a path for sending and receiving data, and is a means of transmitting information using technologies such as the internet, telephone lines, and wireless communication.

[0584] "Feedback" refers to the information and instructions that a system provides to a user, which are adjusted based on the user's actions and circumstances.

[0585] A "server" is a computer system that provides data processing and storage functions, and is a device that provides services while communicating with other devices over a network.

[0586] The system that realizes this invention is superior in that it monitors the user's emotional state and provides dynamically adaptable responses. The terminal is equipped with hardware such as a camera and microphone, which are used to acquire the user's voice and video data in real time. This data is transmitted to a server equipped with an emotion recognition engine.

[0587] After receiving the data, the server uses a machine learning model to analyze the tone of voice and facial expression patterns. This model utilizes software such as TensorFlow and PyTorch. The trend information on emotional states obtained through the analysis is integrated with accumulated health data to evaluate the user's overall health and psychological state.

[0588] Based on the evaluation results, the server generates appropriate feedback and provides it to the user through the terminal. The feedback is delivered via audio, video, or a combination of both, and may include relaxing music or encouraging words depending on the user's state.

[0589] For example, if a user indicates fatigue, the server generates feedback such as, "You seem a little tired. I'll play some music of your choice to help you relax." Furthermore, by inputting instructions such as, "If the user is feeling stressed, suggest actions to alleviate it, such as playing music or suggesting stretching," into the AI ​​model, personalized responses can be provided. This system enables users to live a richer and more comfortable life.

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] The device acquires the user's voice and video data. It captures the user's facial expressions through the camera and records the user's voice in real time using the microphone. Since this data is processed digitally, signal processing is performed to reduce noise. The input is the user's real-world facial expressions and voice, and the output is digital data.

[0593] Step 2:

[0594] The terminal transmits the acquired audio and video data to the server. A secure communication channel is used for transmission, and the data is encrypted to maintain confidentiality. The input is the digital data obtained in the previous step, and the output is the encrypted data that has reached the server.

[0595] Step 3:

[0596] The server decodes the received data and analyzes the user's emotional state using an emotion recognition engine. The data is then fed into a machine learning model, which uses a specific algorithm (e.g., a neural network) to analyze the tone of voice and facial features. The input is the decoded data, and the output is a metric indicating the emotional state.

[0597] Step 4:

[0598] The server performs a comprehensive assessment based on emotional states, integrating them with health data. It identifies emotional trends and compares them to past health database data to evaluate the user's psychological and physical state. Inputs are emotional metrics and existing health data, and output is the integrated assessment result.

[0599] Step 5:

[0600] The server generates appropriate feedback based on the evaluation results. It utilizes a generative AI model and constructs personalized responses using prompts. For example, it might generate feedback such as, "If the user appears stressed, recommend relaxing music to alleviate it." The input is the evaluation result, and the output is the specific feedback content.

[0601] Step 6:

[0602] The server sends the generated feedback to the terminal, and the terminal provides the feedback to the user. The feedback is visualized to the user in the form of audio or video, improving the quality of the interaction. The input is the feedback content, and the output is the audio or visual information received by the user.

[0603] Step 7:

[0604] The user receives feedback from the device and provides further interaction or instructions as needed. This allows the system to continuously adapt to the user's situation and maintain real-time support. The input is feedback input, and the output is the user's response.

[0605] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0606] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0607] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0608] [Fourth Embodiment]

[0609] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0610] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0611] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0612] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0613] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0614] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0615] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0616] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0617] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0618] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0619] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0620] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0621] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] The care support system of the present invention includes a program for comprehensively supporting the user's health condition. The system functions through the cooperation of a terminal, a server, and the user, and comprehensively manages the user's health.

[0623] The terminal collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. This collected data is temporarily stored on the terminal and then sent to the server at regular intervals.

[0624] The server stores the received data and analyzes it using a machine learning model to determine if there are any anomalies by comparing it with past data. If an anomaly is detected, the server sends a notification to the user and the necessary medical institutions through a pre-configured communication channel. This notification includes an overview of the health status and details of the anomaly, prompting the user to take prompt action.

[0625] Furthermore, the server generates feedback based on data analysis, indicating the user's health trends. This feedback is sent to the device as advice for maintaining or improving the user's health, and the device presents it to the user in either voice or text format.

[0626] Furthermore, the device tracks the user's movements through an AR device to support rehabilitation and exercise. The server analyzes the movement data and provides feedback on appropriate exercise methods and posture to support effective rehabilitation.

[0627] For the dialogue function, the terminal collects voice and video data from the user, and the server analyzes it using natural language processing technology. Based on the analysis results, the server generates an appropriate response and interacts with the user through the terminal, thereby reducing the user's feelings of isolation.

[0628] For example, if a user experiences an emergency such as a fall, the device uses an accelerometer and acoustic analysis to detect the anomaly and immediately reports it to the server. The server then automatically notifies registered emergency contacts, enabling a rapid response.

[0629] The following describes the processing flow.

[0630] Step 1:

[0631] The device collects health data such as heart rate, steps taken, and body temperature in real time from the wearable device worn by the user. The collected data is stored on the device for a short period of time.

[0632] Step 2:

[0633] The device sends the accumulated data to the server as packets at regular intervals (e.g., every 30 minutes). Data that has been successfully sent is deleted from the device.

[0634] Step 3:

[0635] The server stores health data received from the terminal in a database. The data is categorized by user and stored along with past history data.

[0636] Step 4:

[0637] The server inputs the stored data into a machine learning model to analyze health trends. If an anomaly is detected, the server evaluates the type and nature of the anomaly.

[0638] Step 5:

[0639] The server immediately determines the action to take when an anomaly is detected and sends a notification to the user or healthcare provider through the registered communication channel. The notification includes an overview of the health condition and how to proceed.

[0640] Step 6:

[0641] The server generates health maintenance feedback based on the analysis results and sends it to the device. The device then presents this feedback to the user in audio or text format.

[0642] Step 7:

[0643] When users perform rehabilitation or exercise, the device uses an AR device to track their movements. The server analyzes the movement data in real time and provides appropriate advice.

[0644] Step 8:

[0645] When a user uses the conversational function, the device collects audio and video data and sends it to the server. The server analyzes this data using natural language processing technology, generates the most appropriate response for the user, and communicates it through the device.

[0646] (Example 1)

[0647] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0648] Efficiently monitoring the health of the elderly and those with chronic illnesses in real time, and responding quickly to abnormalities, is crucial in modern welfare and healthcare. However, existing systems require the combination of multiple different technologies to efficiently analyze biometric information and provide integrated motion tracking and conversational functions, resulting in complexity and high costs. Therefore, effective methods to solve these problems are needed.

[0649] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0650] In this invention, the server includes means for monitoring the user's physical condition and detecting abnormalities by collecting and analyzing biometric information obtained from the user; means for notifying the user through a pre-configured communication path when an abnormality is detected; means for analyzing the collected information based on a learning model and identifying trends in the physical condition; means for generating and presenting advice useful for maintaining the user's health based on the analysis results; and means for tracking the user's movements using augmented reality technology and providing rehabilitation and exercise support. This makes it possible to integrate health monitoring, rapid response to abnormalities, and the provision of advice for improving health conditions.

[0651] "Users" refer to individuals who use the system to receive services such as health monitoring and rehabilitation.

[0652] "Biometric information" refers to data such as heart rate, steps taken, and body temperature, which are collected to understand the user's health status.

[0653] A "server" refers to a computing device that stores and analyzes collected data and generates necessary notifications and feedback.

[0654] A "learning model" refers to an algorithm used to analyze data using machine learning techniques and identify trends and anomalies.

[0655] "Communication path" refers to the means of communication used to transmit abnormality notifications and feedback to users and medical institutions, and includes email, SMS, etc.

[0656] Augmented reality technology is a technology that overlays virtual information onto the real world environment and is used to track user movements and provide health support.

[0657] "Health monitoring" refers to the process of observing the user's health status in real time and detecting any abnormalities.

[0658] "Rehabilitation" refers to the process of providing exercise therapy and feedback aimed at improving the physical function of the user.

[0659] This invention provides a care support system that collects and analyzes user health information from multiple perspectives, enabling real-time health management and rapid response to abnormalities. The system operates with terminals (including wearable devices), a server, and the user working together as a single unit.

[0660] The terminal collects biometric information (e.g., heart rate, steps, body temperature) from wearable devices worn by the user. These devices could be commonly referred to as portable vital signs sensors. The terminal acquires data in real time via Bluetooth or Wi-Fi and temporarily stores it on its own.

[0661] The server receives and stores data transmitted from the terminal and analyzes it using a learning model (e.g., an algorithm using TensorFlow). Based on the analysis results, the server immediately notifies the user and, if necessary, the medical institution if an anomaly is detected. This is done using a configured communication channel (e.g., email or SMS).

[0662] The server also uses accumulated data to understand long-term health trends and generates advice that can help improve the user's health. This advice is provided through the terminal and can also be suggested in voice using speech synthesis software (e.g., a general term for a speech generation system).

[0663] Furthermore, the device utilizes augmented reality technology to track and analyze the user's movements, providing support for rehabilitation and exercise. This data is used for analysis on a server, and feedback is provided on appropriate exercise methods.

[0664] If a user encounters an emergency, the device uses its built-in sensors to detect the situation and report it to the server. This enables a quick and appropriate response. For example, there is a function that issues an alarm based on acceleration data when a user falls.

[0665] Examples of prompts for a generative AI model:

[0666] "Design an algorithm to analyze heart rate data collected from wearable sensors and detect anomalies."

[0667] "Please develop a system that suggests appropriate rehabilitation methods based on user activity data."

[0668] Thus, this invention is a system that utilizes advanced technology to comprehensively support the user's daily health.

[0669] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0670] Step 1:

[0671] The terminal collects biometric information such as heart rate, steps taken, and body temperature in real time from wearable devices. The input is sensor data obtained from the wearable device. Based on this, the terminal temporarily stores the data and organizes it for transmission in the next step. Specifically, the device measures the heart rate every minute, and the terminal receives the data via Bluetooth.

[0672] Step 2:

[0673] The device sends the collected data to the server at regular time intervals (for example, every hour). The input is biometric information stored on the device, and the output is the transmission of data to the server. As part of the data processing, the device converts the data format and sends it using the HTTPS protocol. Specifically, the device establishes a secure connection and uploads the compiled data to the server.

[0674] Step 3:

[0675] The server opens the data received from the terminal and simultaneously stores it in the database. The input is biometric information sent from the terminal, and the output is a record in the database. In terms of data processing, the server manages the time series of the data and prepares to compare it with anomaly detection borderlines. Specifically, the server creates a new entry in the database and adds the data to the history of the corresponding user.

[0676] Step 4:

[0677] The server analyzes accumulated data using a machine learning model to detect anomalies. The input is historical biometric information stored in the database, and the output is a determination of whether or not an anomaly exists. As part of the data analysis, the server filters the data through the model and makes predictive judgments. Specifically, the server executes an anomaly detection algorithm to identify data points that fall outside the normal range.

[0678] Step 5:

[0679] If an anomaly is detected, the server notifies the user through a pre-configured communication channel. The input is the anomaly detection result, and the output is an alarm message to the user. As part of the data processing, the server generates a notification message and prepares it for sending via email or SMS. Specifically, the server creates a notification message and sends it based on the registered contact information.

[0680] Step 6:

[0681] The server analyzes health trends using collected data and performs long-term health status assessments. The input is the entire historical data in the database, and the output is a health trend report. As a data calculation, the server applies trend analysis algorithms and generates indicators to visualize health status. Specifically, the server generates a health trend report and visualizes the results based on the algorithm used.

[0682] Step 7:

[0683] The server sends the generated health trend report and advice to the terminal, suggesting it to the user. The input is the health trend report, and the output is the feedback the user receives. As part of data processing, the server converts the content into audio or text and presents it to the user. Specifically, the terminal uses AV technology to read the report aloud and provides the user with specific health advice.

[0684] Step 8:

[0685] The device utilizes augmented reality technology to monitor user movements and provide rehabilitation and exercise support. Input is movement data from the AR system installed on the device, and output is feedback on how to improve. For data analysis, the device analyzes movement parameters and guides the user toward correct form and exercise methods. Specifically, the device uses AR feedback to guide the user toward correct posture during exercise.

[0686] (Application Example 1)

[0687] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0688] In an aging society, there is a growing need for individual health management and prompt response to emergencies. Traditional manual health monitoring has limitations due to labor shortages and lack of knowledge, and further challenges include the inability to alleviate feelings of isolation and provide appropriate exercise guidance. In particular, the insufficient ability to respond quickly in emergencies and to continuously monitor health trends are problematic.

[0689] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0690] In this invention, the server includes means for collecting and analyzing biometric information in real time, means for tracking actions and providing feedback using augmented reality technology, and means for analyzing conversational data and generating responses using named entity recognition technology. This enables comprehensive monitoring of the user's health status, rapid detection and response to abnormalities, and furthermore, reduces feelings of loneliness and provides effective exercise guidance.

[0691] "Biometric information" refers to data such as heart rate, body temperature, and step count that indicates the user's health status.

[0692] "Analysis" refers to the data processing activity used to identify anomalies and trends in collected data.

[0693] "Augmented reality technology" is a technology that captures a user's physical actions as digital data and provides appropriate feedback.

[0694] "Motion tracking" is the process of observing a user's body movements in real time and analyzing them as digital information.

[0695] "Feedback" refers to guidance and advice provided to users based on the results of analysis.

[0696] Named entity recognition technology is a technology that automatically analyzes audio and video data and extracts meaningful information.

[0697] "Generating a response" refers to the act of creating an appropriate reply to the user based on the analyzed data.

[0698] An "emergency situation" refers to a situation where a user's normal health condition is abnormal and requires a swift response.

[0699] To implement this invention, a system is constructed in which a wearable device worn by the user, a smartphone, and a server work together. The wearable device is equipped with sensors that measure heart rate, body temperature, steps taken, etc. This data is transmitted to the smartphone via Bluetooth communication, and the smartphone temporarily stores this data.

[0700] The server stores biometric data received from smartphones at specific time intervals. Using machine learning algorithms based on TensorFlow, the server analyzes this data to identify abnormalities and health trends. The analysis results are fed back to the user in real time, and in the event of an anomaly, notifications are sent to the user and medical institutions via pre-configured communication channels. These notifications are typically sent as text messages.

[0701] The smart glasses worn by the user utilize augmented reality technology to track the user's movements in real time. A server analyzes the movement data and provides effective exercise guidance by suggesting a suitable rehabilitation plan. This feedback is displayed through the smart glasses.

[0702] Furthermore, the smartphone collects the user's voice and camera footage, and the server employs named entity recognition technology to analyze this data. The server generates appropriate responses from the analyzed data, enabling conversation with the user. This process can utilize natural language generation models such as OpenAI.

[0703] As a concrete example, if the system detects a sudden decrease in a user's step count during the daytime, it recognizes this as an anomaly. To investigate the reason, the server generates questions such as, "How did you spend your day?" and confirms with the user via voice. An example of a prompt to the generating AI model could be, "Please provide questions to identify the reason for the decrease in user activity."

[0704] In this way, the present invention enables comprehensive health management and supports early detection of abnormalities and rapid response.

[0705] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0706] Step 1:

[0707] The terminal receives biometric information from wearable devices, collecting data such as heart rate, body temperature, and steps taken. The input is sensor data from the wearable device, and the output is the temporary storage of the collected data within the terminal. Data processing includes data format conversion and initial filtering.

[0708] Step 2:

[0709] The server receives data from terminals at regular intervals. The input is biometric information transmitted from the terminals, and the output is stored in the server's database. As a data processing step, the data is organized chronologically and saved to storage.

[0710] Step 3:

[0711] The server analyzes received biometric data using a machine learning algorithm based on TensorFlow to perform anomaly detection. The input is accumulated biometric data, and the output is the result of the anomaly detection. As part of data processing, it compares the data with past data to identify outliers and anomaly patterns.

[0712] Step 4:

[0713] If an anomaly is detected, the server sends a notification to the user or designated contact via the communication channel. The input is the anomaly detection result, and the output is a notification in text message format. The operation involves formatting and sending the message according to the communication protocol.

[0714] Step 5:

[0715] The device receives health trend feedback from the server and presents it to the user in audio or text. The input is an analysis report from the server, and the output is the feedback information displayed to the user. The system performs UI rendering to present the information in an easily understandable format for the user.

[0716] Step 6:

[0717] The server receives motion data from smart glasses and analyzes the motion using augmented reality technology. The input is motion data from the smart glasses, and the output is appropriate rehabilitation feedback. The data processing involves identifying the motion and evaluating its quality.

[0718] Step 7:

[0719] The user inputs audio and video data into the terminal, and the server analyzes the data using named entity recognition technology to generate a dialogue. The input is audio and video data, and the output is an appropriate response. The operation involves response generation using a natural language generation model. An example of a prompt sentence for the generating AI model is, "Generate a response to continue a natural conversation with the user."

[0720] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0721] The care support system incorporating the emotion recognition function of the present invention not only monitors the user's health condition but also recognizes the user's emotional state in detail and provides support based on that data.

[0722] The device is equipped with a camera and microphone that captures the user's voice and video in real time. This allows for the acquisition of data in a natural way as the user interacts with the system.

[0723] The server utilizes an emotion engine to process the collected audio and video data. This emotion engine analyzes the tone of voice and facial expression patterns to determine the user's emotional state. The determined emotional state is stored on the platform and integrated with health data to be used for an overall assessment of the user's health and psychological state.

[0724] For example, while a user is receiving exercise guidance using the conversational function, the device captures changes in the user's facial expressions and tone of voice. If the server determines that the user is expressing feelings of fatigue, it can provide feedback suggesting adjustments to the exercise intensity.

[0725] Furthermore, if the system detects that a user is emotionally unstable, it provides support to reduce anxiety by offering relaxation advice and engaging in reassuring conversations. This function helps alleviate the user's psychological burden and supports them in living their daily lives with peace of mind.

[0726] Through these implementations, the system integrates users' physical health data and emotional data to provide personalized and optimal care.

[0727] The following describes the processing flow.

[0728] Step 1:

[0729] The device collects health data from the wearable device worn by the user and simultaneously captures the user's voice and video data. This data is temporarily stored within the device.

[0730] Step 2:

[0731] The device sends health data, audio data, and video data to the server at regular intervals. The data is organized for each user and stored in the server's database.

[0732] Step 3:

[0733] The server inputs audio and video data into an emotion engine, which analyzes the intonation of the voice and changes in facial expressions. Based on this analysis, it identifies the user's emotional state.

[0734] Step 4:

[0735] The server integrates identified emotional states with accumulated health data to assess the user's overall psychological and health status. Machine learning models are used to predict abnormal trends and emotional trends.

[0736] Step 5:

[0737] If abnormalities or emotional instability are detected, the server generates optimal feedback and dialogue. This includes health management advice and relaxation suggestions.

[0738] Step 6:

[0739] The terminal presents the user with feedback and conversation content provided by the server. This is done through voice messages and screen displays.

[0740] Step 7:

[0741] Users act based on feedback from their devices and continuously manage their health and emotional state. User responses and additional data are collected again from the device and used as a feedback loop in the system.

[0742] (Example 2)

[0743] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0744] For elderly users and those with health concerns, it is challenging to simultaneously assess not only their physical health but also their psychological health and provide appropriate support. This project aims to improve the quality of life for users by addressing this challenge.

[0745] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0746] In this invention, the server includes means for collecting physical and psychological data obtained from the user and determining the emotional state by analyzing it using a generative model; means for generating personalized feedback for the user based on the determined emotional state; and means for notifying the user of the feedback via a terminal and supporting appropriate care. This makes it possible to comprehensively understand the user's physical and psychological health status and provide appropriate support.

[0747] "Physical and psychological data" refers to physiological information indicating the user's health status and psychological indicators representing their emotional state.

[0748] A "generative model" refers to an artificial intelligence technology that uses algorithms learned through machine learning to generate responses to new data.

[0749] "Emotional state" refers to the user's current psychological feelings and mood, and is analyzed from voice tone and facial expression data.

[0750] "Feedback" refers to the information and suggestions provided to users based on analysis results, with the aim of improving the users' health and well-being.

[0751] "Personalized feedback" refers to information and suggestions that are customized to take into account the user's specific health and emotional state.

[0752] A "terminal" refers to a digital device used to collect, transmit, and notify data between the user and the system.

[0753] This invention is an integrated system for monitoring the physical and psychological health status of users and providing care support. The system mainly consists of terminals and a server.

[0754] The device is equipped with a camera and microphone, and collects the user's voice and video data in real time. This allows for the acquisition of physical and psychological data, including changes in the user's facial expressions and voice tone.

[0755] The server analyzes the received data in detail using a dedicated emotion analysis engine. This emotion analysis engine uses a generative AI model to determine the user's emotional state based on information obtained from audio and video. This analysis identifies the user's emotional state.

[0756] Based on the identified emotional state, the server generates personalized feedback. This feedback is tailored to support the user's health and psychological well-being. For example, if fatigue is detected, a suggestion such as "Take a short break and take some deep breaths" might be generated. Additionally, relaxation music or simple exercises may be suggested as needed.

[0757] The generated feedback is communicated to the user through the device. The user is informed either by a message appearing on the device's display or by a voice assistant reading the feedback aloud.

[0758] For example, if a user is practicing the piano on their device, but the neural network detects that their concentration is declining based on facial recognition, the server will send feedback to the device saying, "We recommend you take a break and refresh yourself." This allows the user to take appropriate action immediately.

[0759] An example of a prompt message might be, "Please help the user relax. Advice that takes their current emotional state into consideration is needed." In this way, the system provides valuable support to the user.

[0760] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0761] Step 1:

[0762] The device acquires the user's voice and video data. This is done using the high-performance camera and microphone built into the device. Inputs include the user's facial expressions and voice. Outputs include high-resolution image data and audio data, which are sent to the server in real time.

[0763] Step 2:

[0764] The server analyzes the received data. This analysis uses an emotion analysis engine based on a generative AI model. The input consists of audio and video data obtained in step 1. The server first extracts the tone of the audio and the facial expression patterns from the video. Based on this, it determines the user's emotional state (e.g., joy, sadness, fatigue). The output is a description of the determined emotional state.

[0765] Step 3:

[0766] The server generates feedback based on the identified emotional state. The input is the emotional state output in step 2. The server utilizes a generative AI model to generate a prompt corresponding to this emotional state. Specifically, the feedback includes optimal action suggestions, such as "We recommend taking a short break." The output is the generated feedback message.

[0767] Step 4:

[0768] The terminal notifies the user of feedback sent from the server. The input is the feedback message generated in step 3. The terminal either displays the message on its screen or reads the feedback aloud via voice output. The output consists of specific suggestions and advice conveyed to the user.

[0769] Step 5:

[0770] The user modifies their behavior based on the feedback provided. The input is the feedback received in step 4. For example, the user might take a break or do a simple exercise to change their mood. The expected output is an improvement in the user's state.

[0771] (Application Example 2)

[0772] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] In households with elderly individuals or those living alone, there is a need to manage not only the physical health but also the psychological health of users simultaneously. However, conventional health monitoring systems have difficulty providing individualized feedback that takes emotional states into account, and have been insufficient to support the overall health of users. Furthermore, there is a need to identify changes in emotions in real time and provide appropriate support accordingly.

[0774] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0775] This invention includes a server that collects physiological and emotional data obtained from users, analyzes this data to monitor the user's health and emotional state, and detects abnormalities; means for notifying users through a pre-configured communication channel when an abnormality is detected; and means for analyzing the collected data based on a machine learning model to identify trends in health and emotional state. This makes it possible to comprehensively manage the user's physical and emotional health and provide individually optimized feedback.

[0776] "Physiological data" refers to information that indicates the user's physical health status, including vital signs such as heart rate, blood pressure, and body temperature.

[0777] "Emotional data" refers to information that indicates the user's psychological state, and includes emotional indicators derived from facial expressions, tone of voice, and linguistic expressions.

[0778] A "machine learning model" is an algorithm that learns from large amounts of data and identifies patterns, and is a method for making predictions and classifications based on new data.

[0779] A "communication channel" is a path for sending and receiving data, and is a means of transmitting information using technologies such as the internet, telephone lines, and wireless communication.

[0780] "Feedback" refers to the information and instructions that a system provides to a user, which are adjusted based on the user's actions and circumstances.

[0781] A "server" is a computer system that provides data processing and storage functions, and is a device that provides services while communicating with other devices over a network.

[0782] The system that realizes this invention is superior in that it monitors the user's emotional state and provides dynamically adaptable responses. The terminal is equipped with hardware such as a camera and microphone, which are used to acquire the user's voice and video data in real time. This data is transmitted to a server equipped with an emotion recognition engine.

[0783] After receiving the data, the server uses a machine learning model to analyze the tone of voice and facial expression patterns. This model utilizes software such as TensorFlow and PyTorch. The trend information on emotional states obtained through the analysis is integrated with accumulated health data to evaluate the user's overall health and psychological state.

[0784] Based on the evaluation results, the server generates appropriate feedback and provides it to the user through the terminal. The feedback is delivered via audio, video, or a combination of both, and may include relaxing music or encouraging words depending on the user's state.

[0785] For example, if a user indicates fatigue, the server generates feedback such as, "You seem a little tired. I'll play some music of your choice to help you relax." Furthermore, by inputting instructions such as, "If the user is feeling stressed, suggest actions to alleviate it, such as playing music or suggesting stretching," into the AI ​​model, personalized responses can be provided. This system enables users to live a richer and more comfortable life.

[0786] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0787] Step 1:

[0788] The device acquires the user's voice and video data. It captures the user's facial expressions through the camera and records the user's voice in real time using the microphone. Since this data is processed digitally, signal processing is performed to reduce noise. The input is the user's real-world facial expressions and voice, and the output is digital data.

[0789] Step 2:

[0790] The terminal transmits the acquired audio and video data to the server. A secure communication channel is used for transmission, and the data is encrypted to maintain confidentiality. The input is the digital data obtained in the previous step, and the output is the encrypted data that has reached the server.

[0791] Step 3:

[0792] The server decodes the received data and analyzes the user's emotional state using an emotion recognition engine. The data is then fed into a machine learning model, which uses a specific algorithm (e.g., a neural network) to analyze the tone of voice and facial features. The input is the decoded data, and the output is a metric indicating the emotional state.

[0793] Step 4:

[0794] The server performs a comprehensive assessment based on emotional states, integrating them with health data. It identifies emotional trends and compares them to past health database data to evaluate the user's psychological and physical state. Inputs are emotional metrics and existing health data, and output is the integrated assessment result.

[0795] Step 5:

[0796] The server generates appropriate feedback based on the evaluation results. It utilizes a generative AI model and constructs personalized responses using prompts. For example, it might generate feedback such as, "If the user appears stressed, recommend relaxing music to alleviate it." The input is the evaluation result, and the output is the specific feedback content.

[0797] Step 6:

[0798] The server sends the generated feedback to the terminal, and the terminal provides the feedback to the user. The feedback is visualized to the user in the form of audio or video, improving the quality of the interaction. The input is the feedback content, and the output is the audio or visual information received by the user.

[0799] Step 7:

[0800] The user receives feedback from the device and provides further interaction or instructions as needed. This allows the system to continuously adapt to the user's situation and maintain real-time support. The input is feedback input, and the output is the user's response.

[0801] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0802] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0803] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0804] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0805] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0806] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0807] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0808] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0809] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0810] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0811] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0812] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0813] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0814] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0815] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0816] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0817] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0818] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0819] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0820] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0821] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0822] The following is further disclosed regarding the embodiments described above.

[0823] (Claim 1)

[0824] A means of monitoring users' health status and detecting abnormalities by collecting and analyzing health data obtained from users,

[0825] A means of notifying via a pre-configured communication channel when an anomaly is detected,

[0826] A means of analyzing collected data based on machine learning models to identify trends in health status,

[0827] A system that includes this.

[0828] (Claim 2)

[0829] The system according to claim 1, which provides appropriate feedback by tracking and analyzing the user's actions in real time.

[0830] (Claim 3)

[0831] The system according to claim 1, which collects audio and video data, analyzes it, and generates an appropriate response in order to enable interaction between the user and the system.

[0832] "Example 1"

[0833] (Claim 1)

[0834] A means of monitoring the user's physical condition and detecting abnormalities by collecting and analyzing biometric information obtained from the user,

[0835] A means of notifying via a pre-configured communication path when an anomaly is detected,

[0836] A means of analyzing collected information based on a learning model to identify trends in physical condition,

[0837] A means of generating and presenting advice to users that is useful for maintaining their health, based on the analysis results.

[0838] A means of tracking users' movements using augmented reality technology and providing rehabilitation and exercise support,

[0839] A system that includes this.

[0840] (Claim 2)

[0841] The system according to claim 1, which collects user voice and video information, analyzes it, and generates an appropriate response to enable interaction with the user.

[0842] (Claim 3)

[0843] The system according to claim 1, which analyzes acceleration information obtained from a portable device worn by the user and automatically sends an alert to a registered emergency contact when an emergency is detected.

[0844] "Application Example 1"

[0845] (Claim 1)

[0846] A means of monitoring the user's health status and detecting abnormalities by collecting and analyzing biometric information obtained from the user,

[0847] A means of notifying via a pre-configured communication channel when an anomaly is detected,

[0848] A means of analyzing collected information based on machine learning algorithms to identify trends in health status,

[0849] A means of tracking a user's movements using augmented reality technology and providing feedback to support their exercise,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which analyzes conversation data using named entity recognition technology and generates a response tailored to the user.

[0853] (Claim 3)

[0854] The system according to claim 1, comprising a mechanism for quickly detecting an emergency and automatically notifying registered contacts.

[0855] "Example 2 of combining an emotion engine"

[0856] (Claim 1)

[0857] A means of determining emotional states by collecting physical and psychological data obtained from users and analyzing it using a generative model,

[0858] A means for generating personalized feedback for the user based on the identified emotional state,

[0859] A means of notifying users of feedback via their devices and supporting appropriate care,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, which tracks the user's actions and emotional state in real time and provides optimal feedback based on the analyzed data.

[0863] (Claim 3)

[0864] The system according to claim 1, which collects audio and video data in real time to enable interaction with users, analyzes this data, and generates an appropriate response using an emotion engine.

[0865] "Application example 2 when combining with an emotional engine"

[0866] (Claim 1)

[0867] A means of monitoring the user's health and emotional state and detecting abnormalities by collecting and analyzing physiological and emotional data obtained from the user,

[0868] A means of notifying via a pre-configured communication channel when an anomaly is detected,

[0869] A means for analyzing collected data based on machine learning models to identify trends in health and emotional states,

[0870] A means of generating and providing appropriate feedback in audio and video format based on the user's emotional state,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, which tracks and analyzes the user's emotional state in real time to provide appropriate emotional feedback.

[0874] (Claim 3)

[0875] The system according to claim 1, which collects voice and video data, analyzes it, generates a response according to the user's emotional state, and provides it in order to enable interaction between the user and the system. [Explanation of Symbols]

[0876] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of monitoring the user's health status and detecting abnormalities by collecting and analyzing biometric information obtained from the user, A means of notifying via a pre-configured communication channel when an anomaly is detected, A means of analyzing collected information based on machine learning algorithms to identify trends in health status, A means of tracking a user's movements using augmented reality technology and providing feedback to support their exercise, A system that includes this.

2. The system according to claim 1, which analyzes conversation data using named entity recognition technology and generates a response tailored to the user.

3. The system according to claim 1, comprising a mechanism for quickly detecting an emergency and automatically notifying registered contacts.

Images