system

Abstract

We provide the system. [Solution] A natural language processing means for receiving voice input, analyzing it, and generating an appropriate response, A health management system for collecting and analyzing the health data of individual users, An augmented reality display means for individually customizing and presenting rehabilitation programs based on collected data, Emergency response measures to detect and notify of emergencies, A system that includes this.

Description

Technical Field

[0001] The technology disclosed herein relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In caregiving support in the homes of the elderly, shortages of personnel in home care services and feelings of isolation among the elderly have become problems. Also, lack of prompt response in emergencies, individually optimized health management, and communication deficiencies are degrading the quality of life of the elderly. To solve these problems, a flexible and comprehensive caregiving support system is needed.

Means for Solving the Problems

[0005] This invention solves the aforementioned problems by providing a system that includes a natural language processing means for analyzing the user's voice and enabling natural language dialogue, a means for collecting and analyzing the user's health data and supporting health management, an augmented reality display means for customizing and presenting rehabilitation programs, and an emergency response means for detecting emergencies and providing rapid notification. This system enables comprehensive care support at home and creates an environment in which the elderly can live with greater peace of mind.

[0006] "Natural language processing means" refers to a technology or device for analyzing natural language in speech or text format and generating an appropriate response.

[0007] "Health management means" refers to technologies or devices that monitor a user's health status, collect and analyze data, and optimize their health.

[0008] "Augmented reality display means" refers to a technology or device that overlays information onto a real environment and presents special programs, such as rehabilitation programs, to users.

[0009] "Emergency response measures" refer to technologies or devices that monitor the user's environment and circumstances and promptly notify or respond when an emergency is detected. [Brief explanation of the drawing]

[0010] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0011] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0012] First, let's explain the terminology used in the following explanation.

[0013] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0014] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0015] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0016] In the following embodiments, the labeled communication I / F (Interface) is an interface that includes a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0018] [First Embodiment]

[0019] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0020] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0021] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0022] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0023] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0025] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0026] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0027] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0028] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0029] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0030] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0031] This invention is a system that comprehensively supports home care for the elderly by combining natural language processing means, health management means, augmented reality display means, and emergency response means. The system realizes its functions based on the interaction of a server, terminals, and users.

[0032] The server receives user voice input and performs natural language processing to control the operation of necessary electronic devices. If the user commands "Turn on the TV," the server converts the command into text and sends a signal to the device to turn on the TV. In this way, it facilitates accurate communication between the user and the device.

[0033] Health management is performed by collecting the user's vital data via a device and sending it to a server. The server analyzes this data and provides nutritional management and medical advice based on the user's health indicators. For example, if it detects a regular heart rate abnormality, the server will issue a warning and suggest necessary actions.

[0034] Augmented reality display provides users with rehabilitation content in real time via an augmented reality device. The server customizes the rehabilitation plan based on the user's progress data and provides appropriate guidance through the device. For example, it provides feedback to confirm whether the user is performing the exercises correctly and adjusts the content and difficulty level as needed.

[0035] Emergency response is achieved by the device monitoring the user's surroundings. If the user falls, the device immediately sends an alert signal to the server. The server automatically notifies emergency contacts, informing them that a rapid response is required.

[0036] As described above, by coordinating these various methods, it becomes possible to provide the safe and independent living environment that elderly people desire.

[0037] The following describes the processing flow.

[0038] Step 1:

[0039] The device captures the user's voice using a microphone. The voice data is immediately converted to a digital format and undergoes initial pre-processing.

[0040] Step 2:

[0041] The terminal sends voice data to the server. The server receives the voice data and converts it into text data using automatic speech recognition technology.

[0042] Step 3:

[0043] The server analyzes the converted text data and uses natural language processing to understand the user's intent. This analysis includes understanding the context and interpreting commands.

[0044] Step 4:

[0045] Based on the analysis results, the server determines the appropriate action. For example, if the user requests to operate a home appliance, the server sends that command to the terminal.

[0046] Step 5:

[0047] The terminal receives instructions from the server and performs the necessary device operations. It also provides audio or visual feedback to the user.

[0048] Step 6:

[0049] The device continuously collects the user's vital data and periodically sends this data to the server.

[0050] Step 7:

[0051] The server analyzes the received vital data and assesses the patient's health status. If an abnormality is detected, it issues an alert and notifies medical professionals and family members.

[0052] Step 8:

[0053] The device displays a customized rehabilitation program through an augmented reality device worn by the user and monitors the user's progress.

[0054] Step 9:

[0055] The device monitors the surrounding environment and immediately sends an alert to the server if it detects an emergency such as a fall.

[0056] Step 10:

[0057] The server receives emergency alerts and automatically sends notifications to pre-registered emergency contacts, prompting them to take the necessary action quickly.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] For elderly people to live safely and independently at home, a comprehensive support system is needed that includes assistance with home appliances, health management, rehabilitation, and prompt emergency response. However, current technology only addresses these issues individually, and a system that provides comprehensive support is lacking. Therefore, there is a need to improve the quality of life for the elderly by coordinating these various functions.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes processing means for receiving and analyzing audio signals and converting them into commands, processing means for evaluating the user's health status by collecting and analyzing biometric information, and augmented reality provision means for customizing and displaying rehabilitation based on the user's progress data. This improves safety in the living environment of the elderly and enables life support tailored to individual needs.

[0063] A "processing means for receiving, analyzing, and converting audio signals into commands" is a device that receives audio input from a user as a digital signal and converts that audio data into text commands using natural language processing technology.

[0064] "Processing means for evaluating a user's health status by collecting and analyzing biometric information" refers to a device that collects health-related data from users, analyzes that data to evaluate their health status, and provides appropriate advice.

[0065] "An augmented reality provisioning method for customizing and displaying rehabilitation based on user progress data" refers to a device that analyzes the user's rehabilitation progress, creates an individually optimized rehabilitation plan based on that data, and displays it using augmented reality technology.

[0066] An "emergency response system for detecting and promptly notifying of environmental changes and abnormalities" is a device that continuously monitors the environment around the user, immediately detects falls or other abnormal situations, and promptly notifies the appropriate emergency contacts.

[0067] This invention is a comprehensive system designed to support safe and independent living for the elderly in their own homes. This system integrates four functions—voice recognition, health management, augmented reality (AR), and emergency response—through the mutual cooperation of a server, terminal, and user.

[0068] The server uses natural language processing technology to receive and analyze audio signals. Specifically, it converts audio data into text using software such as "SpeechRecognition" or "Google Cloud Speech-to-Text API," and then uses this text to instruct users to operate home appliances. For example, a user can say "Turn off the lights" and the server will then turn off the lights.

[0069] The device transmits biometric information collected from the wearable device to a server. The server analyzes this data using machine learning algorithms such as "TENSORFLOW®" or "scikit-learn" to assess the user's health status. If an abnormality is detected, the user is immediately notified and given specific advice, such as "Drink some water."

[0070] Furthermore, in augmented reality-based rehabilitation, the device utilizes platforms such as "ARKit" or "Vuforia" to display a rehabilitation program customized for the user. The server adjusts the plan based on the user's progress data and provides real-time feedback. For example, if the user's exercise is insufficient, it will provide guidance such as, "Let's lift your knee a little higher."

[0071] Furthermore, the device uses an accelerometer and other sensors to monitor the user's surroundings and detect anomalies. In the event of an emergency such as a fall, the device immediately sends an alert to the server. The server then uses the Twilio API to automatically notify emergency contacts, supporting a rapid response.

[0072] This system allows users to enjoy a safe and comfortable life at home, and improve their quality of life. An example of a prompt for the generated AI model would be the instruction, "Simulate a situation where the user changes the lighting settings by voice."

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The user issues a voice command. The voice signal is received via the terminal's microphone. The terminal converts this voice signal into digital data and sends it to the server. The input is the voice instruction from the user, and the output is digital voice data. This data conversion transforms the voice command into a format that can be parsed within the system.

[0076] Step 2:

[0077] The server converts received digital audio data into text using natural language processing techniques. Specifically, it analyzes the audio data using software such as the "Google Cloud Speech-to-Text API" to obtain text commands. The input is digital audio data, and the output is text-formatted commands. Through this process, the server understands the user's intent.

[0078] Step 3:

[0079] The server analyzes the converted text command and generates specific commands to operate electronic devices. For example, if a user says "Turn on the TV," the server generates a TV-on command and sends it to the terminal. The input is the text command resulting from the speech analysis, and the output is an executable device control command. This process enables operation based on voice commands.

[0080] Step 4:

[0081] The terminal receives control commands from the server and controls the target electronic device. For example, the terminal turns on the television via a smart home device. The input is the control command from the server, and the output is the actual physical operation of the device. This process enables physical operation based on the user's intent.

[0082] Step 5:

[0083] The terminal collects biometric information from wearable devices and transmits it to a server. This includes data such as heart rate and body temperature. The input is biometric data from the wearable device, and the output is digital health data transmitted to the server. This data collection forms the foundation for health management.

[0084] Step 6:

[0085] The server analyzes received health data and evaluates the user's health status. It uses machine learning algorithms to check for deviations from the normal range. The input is biometric data, and the output is the health status evaluation result. This analysis enables the provision of necessary health advice.

[0086] Step 7:

[0087] The server generates necessary health advice and warnings based on the evaluation results and notifies the user via the terminal. For example, a message such as "We recommend you drink plenty of water" is sent as needed. The input is the health status evaluation result, and the output is a specific notification to the user. This notification allows the user to deepen their awareness of their own health status.

[0088] Step 8:

[0089] The terminal uses augmented reality devices to provide rehabilitation information to the user. The server customizes the rehabilitation plan based on the user's data and sends it to the terminal. The input is the user's health status and progress data, and the output is the customized rehabilitation plan and its implementation instructions. This support enables more effective rehabilitation.

[0090] Step 9:

[0091] The terminal monitors environmental changes and anomalies, and sends an alert to the server when an anomaly is detected. Specifically, it may use an accelerometer to detect falls. The input is data from environmental sensors, and the output is an alert signal for an anomaly. This monitoring function enables a rapid response.

[0092] Step 10:

[0093] When the server receives an alert, it sends a notification to the emergency contact requesting prompt assistance. For example, it can use the Twilio API to send a message to the emergency contact. The input is an anomaly alert from the device, and the output is a notification to the emergency contact. This notification ensures that appropriate action is taken immediately if necessary.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In modern society, the elderly face many difficulties in maintaining independent daily living. In particular, health management, emergency response, and operating household devices pose significant burdens. Furthermore, effective rehabilitation and individualized support for health management are essential for achieving a better quality of life. However, systems to comprehensively support these aspects are currently inadequate.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes language processing means for receiving and analyzing voice input and generating an appropriate response; health management means for collecting and analyzing biometric information of individual users; augmented reality display means for individually customizing and presenting a functional recovery program based on the collected information; alarm response means for detecting and notifying of emergencies; and wide-area control means for controlling the operation of electronic devices. This enables elderly people to live independently, safely, and healthily.

[0099] A "language processing system" is a system equipped with the function of receiving voice input, analyzing it, and generating an appropriate response.

[0100] A "health management system" is a system that collects biometric information from individual users and analyzes that information to manage their health status.

[0101] An "augmented reality display means" is a system that provides technology to individually customize and visually present a functional recovery program to the user based on collected information.

[0102] An "alarm response system" is a function that enables a rapid response by promptly notifying users when an emergency is detected.

[0103] "Wide-area control means" refers to technology that controls the operation of electronic devices in a home or other space based on voice commands or other inputs.

[0104] The system implementing this invention aims to provide comprehensive support for elderly people to live their daily lives safely and independently. The entire system consists of a server, terminals, and users, each playing a specific role.

[0105] The server first receives the user's voice input via a language processing device. This voice is converted into text using a natural language processing library (e.g., Google Cloud Speech-to-Text), and then into commands for operating electronic devices within the home and living space. This enables voice-controlled automation within the home.

[0106] The device collects the user's biometric information in real time through health management tools. This involves using wearable sensors to collect data such as heart rate and blood pressure. This data is analyzed by a cloud-based data management system, and alerts are issued via the server if any abnormalities are detected.

[0107] Furthermore, as a means of displaying augmented reality, the terminal uses augmented reality (AR) technology (e.g., ARCore) to visually provide rehabilitation programs to the user. The server transfers personalized programs to the terminal based on the collected data and provides appropriate guidance.

[0108] Furthermore, the device is equipped with sensors that monitor the user's surroundings as a means of responding to alarms. This allows for the detection of emergencies such as falls, prompt notification to emergency contacts, and the taking of appropriate measures as needed.

[0109] As a concrete example, let's consider a scenario involving an elderly person named Mr. Tanaka. If Mr. Tanaka says "Make me some tea" using voice command, the server will perform the appropriate action. Also, if an abnormality in heart rate is detected during exercise, an alert will be immediately issued to notify medical professionals or family members.

[0110] Examples of prompts for a generative AI model include the following:

[0111] "Design the optimal voice commands for when users want to control home appliances using their voice."

[0112] "Could you please explain methods for analyzing health data to customize exercise plans for older adults?"

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The server receives voice input from the user. The input voice data is converted into text data by the server using a natural language processing library. This prepares the server to analyze the specific instructions.

[0116] Step 2:

[0117] Based on the analyzed text data, the server generates appropriate electronic device control commands in response to user instructions. For example, if the instruction "Turn on the TV" is analyzed, the server generates a control signal for the corresponding device and sends it to the terminal to turn on the TV.

[0118] Step 3:

[0119] The device collects biometric information from the user in real time as a means of health management. The collected data (e.g., heart rate, blood pressure) is sent from the device to a server and analyzed by a cloud-based data management system. Based on the results of this analysis, health status is monitored and continuously managed.

[0120] Step 4:

[0121] The server, as an augmented reality display method, individually customizes a rehabilitation program tailored to each user based on collected health data. The server then delivers this program to the terminal, which uses augmented reality technology to provide visual guidance to the user.

[0122] Step 5:

[0123] The device monitors the user's surroundings using environmental monitoring sensors. If an emergency (e.g., a fall) is detected, the device sends an alert to the server. The server automatically notifies emergency contacts as needed to encourage a quick response.

[0124] Step 6:

[0125] The server uses a generative AI model to respond to detailed information processing requests from users. The generative AI model receives and analyzes prompt messages, providing appropriate information and suggesting service improvements. In this way, it expands the overall system knowledge and improves the user experience.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention is a system that incorporates an emotion engine in addition to natural language processing means, health management means, augmented reality display means, and emergency response means, in order to comprehensively support the lives of the elderly. This enables personalized responses that take into account the user's emotions.

[0128] The emotion engine is implemented by having the device acquire the user's voice and facial expressions using its camera and microphone, and by having the server analyze that data in real time. If the user says, "I'm not feeling well today," the emotion engine will determine the user's emotional state from their tone of voice and facial expressions. The server will feed the emotional data back into a natural language processing system to generate the most appropriate response based on the emotion. This response will be communicated to the user through the device as needed, enabling more empathetic and appropriate communication.

[0129] Furthermore, emotional data is used in conjunction with health management tools. The server analyzes the user's emotional change patterns and evaluates them as part of their overall health status. For example, if a user frequently experiences stress, the health management tools will suggest nutritional plans and relaxation menus based on that data. In addition, rehabilitation programs can be customized to maintain user motivation by adjusting the difficulty level based on the emotional engine data.

[0130] Emergency response measures include a rapid response to sudden changes in the user's emotional state. When the server receives an alert from the emotion engine, it sends a notification to family members or caregivers via the emergency contact network. In this way, the system, with the addition of the emotion engine, constantly monitors the user's emotional state, providing greater flexibility and individualized support in communication and health management.

[0131] The following describes the processing flow.

[0132] Step 1:

[0133] The device captures the user's voice and facial expressions in real time. This is done using a camera and microphone.

[0134] Step 2:

[0135] The terminal sends the acquired audio and video data to the server. The data is converted into a format for processing.

[0136] Step 3:

[0137] The server converts the audio data into text and performs natural language processing to analyze the user's intent.

[0138] Step 4:

[0139] The server uses video data to perform facial recognition and estimate the user's emotional state from their facial expressions. An emotion engine is used for the analysis.

[0140] Step 5:

[0141] The server integrates the analyzed intent and emotional state to determine the optimal response for the user. The response will have a tone that matches the emotional state.

[0142] Step 6:

[0143] The device communicates optimized responses from the server to the user via voice or text. This allows the user to receive feedback that takes emotions into account.

[0144] Step 7:

[0145] The device continuously collects the user's health data and sends it to the server along with emotional data.

[0146] Step 8:

[0147] The server comprehensively analyzes the received health and emotional data to assess the user's health status. Health management tools provide appropriate advice as needed.

[0148] Step 9:

[0149] The device displays information to help users improve their rehabilitation and lifestyle based on insights from the server. Augmented reality may be used for this purpose.

[0150] Step 10:

[0151] In the event of an emergency, the device immediately sends an alert to the server via its sensors. Sudden changes in emotions are also monitored.

[0152] Step 11:

[0153] The server receives alerts and quickly notifies relevant parties through the emergency contact network. Sentiment data is also used as a basis for making decisions regarding emergency response.

[0154] (Example 2)

[0155] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0156] In the daily lives of users, including the elderly, it was difficult with conventional systems to appropriately understand emotional changes and to provide appropriate communication and health management. Furthermore, the provision of individualized rehabilitation programs tailored to the emotional state of users and prompt responses in emergencies were not adequately implemented.

[0157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0158] In this invention, the server includes analysis means for collecting and analyzing voice data and facial expression data to determine the emotional state, natural language processing means for generating appropriate responses according to the emotional state, and health management means for collecting and analyzing individual users' health information and using it in conjunction with emotional data. This makes it possible to grasp changes in the user's emotions in real time and to quickly implement individualized and emergency responses accordingly.

[0159] "Voice data" refers to information recorded in a digital format of the user's voice for use in analysis.

[0160] "Facial expression data" refers to information recorded in image or video format, capturing the movements and expressions of a user's face, and used for emotion analysis.

[0161] "Analysis means" refers to technologies and devices for processing collected voice data and facial expression data to estimate the user's emotional state.

[0162] "Natural language processing means" refers to technologies and devices that generate appropriate responses based on analyzed emotion data and facilitate dialogue.

[0163] "Health management tools" refer to technologies and devices that collect and analyze users' health information, integrate it with the resulting emotional data, and then evaluate and manage their health status.

[0164] "Information presentation means" refers to technologies and devices used to convey individually tailored rehabilitation programs and health information to users.

[0165] A "warning mechanism" refers to a technology or device that monitors changes in a user's emotions and issues a notification if an anomaly is detected.

[0166] This invention is a system that supports the daily lives of users, including the elderly, and includes features such as emotional state analysis, response generation using natural language processing, integration with health management, customization of rehabilitation programs, and emergency response.

[0167] Specific implementations of the system

[0168] The device collects the user's voice and facial expressions in real time. Specific hardware examples include communication devices with built-in cameras and microphones. This allows for the acquisition of the user's visual and auditory information.

[0169] The server receives audio and facial expression data transmitted from the terminal. Using speech recognition technology, the audio data is converted into text, and then facial expression analysis technology is used to estimate emotions from the facial expression data. This process quantifies emotions from the tone of voice and facial movements, and generates a response using natural language processing technology.

[0170] The generated response is transmitted to the user via the terminal. This enables natural and empathetic communication for the user. Possible software used for this purpose includes speech synthesis software and a GUI for display.

[0171] The server integrates the analyzed emotional data with a health management system. For example, if stress levels are high, it suggests relaxation techniques. Rehabilitation programs are tailored based on emotional data and designed to help motivate the user.

[0172] Furthermore, the server has a monitoring function to detect sudden changes in emotional state. If an anomaly is detected, it will alert the user's family or health support team and prompt them to take emergency action.

[0173] Specific example

[0174] For example, if a user says, "I'm not feeling well today," the device collects that voice and sends it to the server. The server uses speech recognition technology to analyze the phrase "I'm not feeling well" and determines the emotional state to be "anxious." As a result, natural language processing generates a response such as, "Please rest well. Shall I play some relaxing music?" and conveys it to the user through the device.

[0175] Example of a prompt

[0176] "Analyze the user's voice and facial expression data, determine their emotional state, and generate an appropriate response."

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The device collects the user's voice and facial expressions. Specifically, it records the user's face with a camera and records their voice with a microphone. The input is real-time visual and audio data, which is then converted into packets and prepared for transmission to the server. The output is structured data packets.

[0180] Step 2:

[0181] The server receives audio and facial expression data transmitted from the terminal. The input here is the data packets sent from the terminal. The server converts this data into an internal data structure for analysis. The output is in a parseable data format.

[0182] Step 3:

[0183] The server converts audio data into text data using speech recognition software. The input is audio data. The server processes the data, converting the audio into text. The output is text data.

[0184] Step 4:

[0185] The server analyzes facial expression data to estimate the user's emotional state. The input is image data representing facial expressions. The server uses a facial expression recognition algorithm to quantify emotions. The output is numerical data representing the emotional state.

[0186] Step 5:

[0187] The server generates a response using natural language processing based on the emotional state. The input consists of text data and emotional state data. A generative AI model is used to perform calculations that produce an appropriate response. The output is the response text to be conveyed to the user.

[0188] Step 6:

[0189] The server sends the generated response text to the terminal. The input is the generated response text. The server uses a communication protocol to send this to the terminal in order to convey it to the user. The output is the response data delivered to the terminal.

[0190] Step 7:

[0191] The terminal communicates the response received from the server to the user. Specifically, it displays text on the screen and plays audio through the speaker. The input is the response data from the server, and the output is the user's perception.

[0192] (Application Example 2)

[0193] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0194] In the lives of the elderly, not only physical health management but also emotional support is essential. Traditional systems have been unable to adequately address emotional changes, making it difficult to provide appropriate support. In particular, the lack of a comprehensive support system that considers the impact of emotional changes on health is a significant challenge.

[0195] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0196] In this invention, the server includes information processing means for receiving and analyzing acoustic input and generating an appropriate response, health management means for collecting and analyzing the biometric data of individual users, and emotion analysis means for analyzing the emotional state of users and providing empathetic support based on that analysis. This enables comprehensive support that responds immediately to changes in the user's physical and emotional health.

[0197] "Acoustic input" refers to signals that are mechanically received from the user's voice and surrounding environmental sounds.

[0198] "Information processing means" refers to technologies for analyzing received data and generating appropriate responses or operations based on that analysis.

[0199] "Biometric data" refers to various types of physiological information that indicate the user's health status.

[0200] A "health management system" is a system that analyzes a user's health status based on biometric data and provides appropriate management and recommendations.

[0201] "Visual augmentation display means" refers to a technology that displays virtual information overlaid on real-world visual information.

[0202] An "immediate response mechanism" is a system for quickly providing necessary notifications and taking appropriate action in response to detected emergencies.

[0203] "Emotional analysis methods" refer to technologies that analyze a user's emotional state from their voice and facial expressions, and then use the results to respond accordingly.

[0204] The system for carrying out this invention combines the functions of acoustic input, information processing means, biometric data collection, health management means, visual augmentation display, immediate response, and emotion analysis means. A specific embodiment thereof is shown below.

[0205] The server acquires user voice and ambient sounds through acoustic input. The audio data is analyzed using a natural language processing model with Python and deep learning frameworks such as TensorFlow and PyTorch. An appropriate response is then generated from the analysis results.

[0206] User biometric data is collected by smartphones or dedicated wearable devices. These devices transmit data such as body temperature and heart rate to a server in real time. The server uses this data to analyze the user's health status and provides feedback based on the results as a health management tool.

[0207] Emotion analysis is performed by capturing the user's facial expressions and tone of voice using the device's camera and microphone. This data is instantly analyzed using image processing libraries such as OpenCV and deep learning models to evaluate the user's emotional state. Based on the evaluation results, empathetic responses and health suggestions are generated.

[0208] For example, if a user says, "I'm tired today," the system will determine their stress and fatigue level from their tone of voice and facial expression, and then play relaxing music or suggest stretching exercises. A concrete example of a prompt would be to instruct the AI ​​to "generate health advice to improve the user's mood."

[0209] This system allows users to receive not only physical health management but also mental support, enabling them to live a more comfortable daily life.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The device's microphone and camera capture the user's voice and facial expression data in real time. This input data is primary information indicating the user's current state.

[0213] Step 2:

[0214] The device sends the acquired audio data to the server. The server analyzes the audio data using a natural language processing model based on TensorFlow or PyTorch. The analyzed data is output as text data to understand the user's emotional state and health status.

[0215] Step 3:

[0216] The server analyzes facial expression data sent from the camera using OpenCV and a deep learning model. The analyzed emotion data is output as numerical information to evaluate the user's emotional state.

[0217] Step 4:

[0218] The server acquires data from wearable devices containing biometric information and performs data analysis using health management tools. This output information is used as foundational data to evaluate the user's health status and generate health management advice as needed.

[0219] Step 5:

[0220] The server integrates the analyzed emotional data and health information, and uses a generative AI model to generate empathetic responses and health suggestions optimized for the user. This output response is based on the prompt "Please generate health advice to improve the user's mood" and is sent to the terminal as a response expressed in natural language.

[0221] Step 6:

[0222] The user's device displays the generated response on the screen or plays it back as audio. This allows the user to receive health management and emotional support suggested by the server.

[0223] This processing flow allows users to receive appropriate feedback on their physical and mental health status in real time.

[0224] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0225] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0226] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0227] [Second Embodiment]

[0228] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0229] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0230] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0231] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0232] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0234] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0235] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0236] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0237] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0238] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0239] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0240] This invention is a system that comprehensively supports home care for the elderly by combining natural language processing means, health management means, augmented reality display means, and emergency response means. The system realizes its functions based on the interaction of a server, terminals, and users.

[0241] The server receives user voice input and performs natural language processing to control the operation of necessary electronic devices. If the user commands "Turn on the TV," the server converts the command into text and sends a signal to the device to turn on the TV. In this way, it facilitates accurate communication between the user and the device.

[0242] Health management is performed by collecting the user's vital data via a device and sending it to a server. The server analyzes this data and provides nutritional management and medical advice based on the user's health indicators. For example, if it detects a regular heart rate abnormality, the server will issue a warning and suggest necessary actions.

[0243] Augmented reality display provides users with rehabilitation content in real time via an augmented reality device. The server customizes the rehabilitation plan based on the user's progress data and provides appropriate guidance through the device. For example, it provides feedback to confirm whether the user is performing the exercises correctly and adjusts the content and difficulty level as needed.

[0244] Emergency response is achieved by the device monitoring the user's surroundings. If the user falls, the device immediately sends an alert signal to the server. The server automatically notifies emergency contacts, informing them that a rapid response is required.

[0245] As described above, by coordinating these various methods, it becomes possible to provide the safe and independent living environment that elderly people desire.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The device captures the user's voice using a microphone. The voice data is immediately converted to a digital format and undergoes initial pre-processing.

[0249] Step 2:

[0250] The terminal sends voice data to the server. The server receives the voice data and converts it into text data using automatic speech recognition technology.

[0251] Step 3:

[0252] The server analyzes the converted text data and uses natural language processing to understand the user's intent. This analysis includes understanding the context and interpreting commands.

[0253] Step 4:

[0254] Based on the analysis results, the server determines the appropriate action. For example, if the user requests to operate a home appliance, the server sends that command to the terminal.

[0255] Step 5:

[0256] The terminal receives instructions from the server and performs the necessary device operations. It also provides audio or visual feedback to the user.

[0257] Step 6:

[0258] The device continuously collects the user's vital data and periodically sends this data to the server.

[0259] Step 7:

[0260] The server analyzes the received vital data and assesses the patient's health status. If an abnormality is detected, it issues an alert and notifies medical professionals and family members.

[0261] Step 8:

[0262] The device displays a customized rehabilitation program through an augmented reality device worn by the user and monitors the user's progress.

[0263] Step 9:

[0264] The device monitors the surrounding environment and immediately sends an alert to the server if it detects an emergency such as a fall.

[0265] Step 10:

[0266] The server receives emergency alerts and automatically sends notifications to pre-registered emergency contacts, prompting them to take the necessary action quickly.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] For elderly people to live safely and independently at home, a comprehensive support system is needed that includes assistance with home appliances, health management, rehabilitation, and prompt emergency response. However, current technology only addresses these issues individually, and a system that provides comprehensive support is lacking. Therefore, there is a need to improve the quality of life for the elderly by coordinating these various functions.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes processing means for receiving and analyzing audio signals and converting them into commands, processing means for evaluating the user's health status by collecting and analyzing biometric information, and augmented reality provision means for customizing and displaying rehabilitation based on the user's progress data. This improves safety in the living environment of the elderly and enables life support tailored to individual needs.

[0272] A "processing means for receiving, analyzing, and converting audio signals into commands" is a device that receives audio input from a user as a digital signal and converts that audio data into text commands using natural language processing technology.

[0273] "Processing means for evaluating a user's health status by collecting and analyzing biometric information" refers to a device that collects health-related data from users, analyzes that data to evaluate their health status, and provides appropriate advice.

[0274] "An augmented reality provisioning method for customizing and displaying rehabilitation based on user progress data" refers to a device that analyzes the user's rehabilitation progress, creates an individually optimized rehabilitation plan based on that data, and displays it using augmented reality technology.

[0275] An "emergency response system for detecting and promptly notifying of environmental changes and abnormalities" is a device that continuously monitors the environment around the user, immediately detects falls or other abnormal situations, and promptly notifies the appropriate emergency contacts.

[0276] This invention is a comprehensive system designed to support safe and independent living for the elderly in their own homes. This system integrates four functions—voice recognition, health management, augmented reality (AR), and emergency response—through the mutual cooperation of a server, terminal, and user.

[0277] The server uses natural language processing technology to receive and analyze audio signals. Specifically, it converts audio data into text using software such as "SpeechRecognition" or "Google Cloud Speech-to-Text API," and then uses this text to instruct users to operate home appliances. For example, a user can say "Turn off the lights" and the server will then turn off the lights.

[0278] The device sends biometric information collected from the wearable device to a server. The server analyzes this data using machine learning algorithms such as "TensorFlow" or "scikit-learn" to assess the user's health status. If an abnormality is detected, the user is immediately notified and given specific advice, such as "Drink some water."

[0279] In rehabilitation using augmented reality, the terminal utilizes platforms such as "ARKit" or "Vuforia" to display a customized rehabilitation program for the user. The server adjusts the plan based on the user's progress data and provides real-time feedback. For example, if the user's movement is insufficient, guidance such as "Raise your knee a little more" is given.

[0280] Furthermore, the terminal uses an acceleration sensor etc. to monitor the user's surrounding environment and detect abnormalities. When an emergency situation such as a fall occurs, the terminal immediately sends an alert to the server. The server uses the "Twilio API" to automatically notify the emergency contacts and assist with a prompt response.

[0281] With this system, the user can enjoy a safe and comfortable life at home and can improve the quality of that life. As an example of a prompt sentence for the generative AI model, an instruction such as "Please simulate the situation where the user changes the lighting settings by voice." can be considered.

[0282] The flow of the specific process in Example 1 will be described using FIG. 11.

[0283] Step 1:

[0284] The user issues a voice command. The voice signal is received via the microphone of the terminal. The terminal converts this voice signal into digital data and sends it to the server. The input is the voice instruction from the user, and the output is digital voice data. By this data conversion, the voice command is converted into a form that can be analyzed within the system.

[0285] Step 2:

[0286] The server converts the received digital audio data into text using natural language processing technology. Specifically, the audio data is analyzed by software such as the "Google Cloud Speech-to-Text API" to obtain text instructions. The input is digital audio data, and the output is instructions in text format. Through this process, the server understands the user's intention.

[0287] Step 3:

[0288] The server analyzes the converted text instructions and generates specific operation instructions for electronic devices. For example, when the user says "Turn on the TV", the server generates a TV on instruction and sends it to the terminal. The input is the text instruction of the voice analysis result, and the output is an executable device control instruction. Through this process, operations based on voice commands become possible.

[0289] Step 4:

[0290] The terminal receives the operation instruction from the server and controls the target electronic device. For example, the terminal turns on the TV via a smart home device. The input is the control instruction from the server, and the output is the actual physical operation of the device. Through this process, physical operations based on the user's intention are realized.

[0291] Step 5:

[0292] The terminal collects biometric information from the wearable device and sends it to the server. For example, data such as heart rate and body temperature is included. The input is biometric data from the wearable device, and the output is digital health data sent to the server. Through this data collection, the basis for health management is formed.

[0293] Step 6:

[0294] The server analyzes received health data and evaluates the user's health status. It uses machine learning algorithms to check for deviations from the normal range. The input is biometric data, and the output is the health status evaluation result. This analysis enables the provision of necessary health advice.

[0295] Step 7:

[0296] The server generates necessary health advice and warnings based on the evaluation results and notifies the user via the terminal. For example, a message such as "We recommend you drink plenty of water" is sent as needed. The input is the health status evaluation result, and the output is a specific notification to the user. This notification allows the user to deepen their awareness of their own health status.

[0297] Step 8:

[0298] The terminal uses augmented reality devices to provide rehabilitation information to the user. The server customizes the rehabilitation plan based on the user's data and sends it to the terminal. The input is the user's health status and progress data, and the output is the customized rehabilitation plan and its implementation instructions. This support enables more effective rehabilitation.

[0299] Step 9:

[0300] The terminal monitors environmental changes and anomalies, and sends an alert to the server when an anomaly is detected. Specifically, it may use an accelerometer to detect falls. The input is data from environmental sensors, and the output is an alert signal for an anomaly. This monitoring function enables a rapid response.

[0301] Step 10:

[0302] When the server receives an alert, it sends a notification to the emergency contact and requests prompt assistance. For example, it uses the "Twilio API" to send a message to the emergency contact. The input is an abnormal alert from the terminal, and the output is a notification to the emergency contact. This notification enables appropriate action to be taken immediately if necessary.

[0303] (Application Example 1)

[0304] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0305] In modern society, the elderly face many difficulties in leading an independent daily life. In particular, health management, response to emergencies, and operation of household devices are a heavy burden for them. Also, individual support for effective rehabilitation and health management is essential for achieving a better quality of life. However, the current situation is that systems for comprehensively supporting these are not fully developed.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0307] In this invention, the server includes language processing means for receiving voice input, analyzing it, and generating an appropriate response, health management means for collecting biometric information of individual users and analyzing that information, extended reality display means for individually customizing a function recovery program based on the collected information and presenting that program, alert response means for detecting an emergency and giving a notification, and wide-area control means for controlling the operation of electronic devices. Thereby, it becomes possible for the elderly to live an independent, safe, and healthy life.

[0308] The "language processing means" is a system having the function of receiving voice input, analyzing it, and generating an appropriate response.

[0309] A "health management system" is a system that collects biometric information from individual users and analyzes that information to manage their health status.

[0310] An "augmented reality display means" is a system that provides technology to individually customize and visually present a functional recovery program to the user based on collected information.

[0311] An "alarm response system" is a function that enables a rapid response by promptly notifying users when an emergency is detected.

[0312] "Wide-area control means" refers to technology that controls the operation of electronic devices in a home or other space based on voice commands or other inputs.

[0313] The system implementing this invention aims to provide comprehensive support for elderly people to live their daily lives safely and independently. The entire system consists of a server, terminals, and users, each playing a specific role.

[0314] The server first receives the user's voice input via a language processing device. This voice is converted into text using a natural language processing library (e.g., Google Cloud Speech-to-Text), and then into commands for operating electronic devices within the home and living space. This enables voice-controlled automation within the home.

[0315] The device collects the user's biometric information in real time through health management tools. This involves using wearable sensors to collect data such as heart rate and blood pressure. This data is analyzed by a cloud-based data management system, and alerts are issued via the server if any abnormalities are detected.

[0316] Furthermore, as a means of displaying augmented reality, the terminal uses augmented reality (AR) technology (e.g., ARCore) to visually provide rehabilitation programs to the user. The server transfers personalized programs to the terminal based on the collected data and provides appropriate guidance.

[0317] Furthermore, the device is equipped with sensors that monitor the user's surroundings as a means of responding to alarms. This allows for the detection of emergencies such as falls, prompt notification to emergency contacts, and the taking of appropriate measures as needed.

[0318] As a concrete example, let's consider a scenario involving an elderly person named Mr. Tanaka. If Mr. Tanaka says "Make me some tea" using voice command, the server will perform the appropriate action. Also, if an abnormality in heart rate is detected during exercise, an alert will be immediately issued to notify medical professionals or family members.

[0319] Examples of prompts for a generative AI model include the following:

[0320] "Design the optimal voice commands for when users want to control home appliances using their voice."

[0321] "Could you please explain methods for analyzing health data to customize exercise plans for older adults?"

[0322] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0323] Step 1:

[0324] The server receives voice input from the user. The input voice data is converted into text data by the server using a natural language processing library. This prepares the server to analyze the specific instructions.

[0325] Step 2:

[0326] Based on the analyzed text data, the server generates appropriate electronic device control commands in response to user instructions. For example, if the instruction "Turn on the TV" is analyzed, the server generates a control signal for the corresponding device and sends it to the terminal to turn on the TV.

[0327] Step 3:

[0328] The device collects biometric information from the user in real time as a means of health management. The collected data (e.g., heart rate, blood pressure) is sent from the device to a server and analyzed by a cloud-based data management system. Based on the results of this analysis, health status is monitored and continuously managed.

[0329] Step 4:

[0330] The server, as an augmented reality display method, individually customizes a rehabilitation program tailored to each user based on collected health data. The server then delivers this program to the terminal, which uses augmented reality technology to provide visual guidance to the user.

[0331] Step 5:

[0332] The device monitors the user's surroundings using environmental monitoring sensors. If an emergency (e.g., a fall) is detected, the device sends an alert to the server. The server automatically notifies emergency contacts as needed to encourage a quick response.

[0333] Step 6:

[0334] The server uses a generative AI model to respond to detailed information processing requests from users. The generative AI model receives and analyzes prompt messages, providing appropriate information and suggesting service improvements. In this way, it expands the overall system knowledge and improves the user experience.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] This invention is a system that incorporates an emotion engine in addition to natural language processing means, health management means, augmented reality display means, and emergency response means, in order to comprehensively support the lives of the elderly. This enables personalized responses that take into account the user's emotions.

[0337] The emotion engine is implemented by having the device acquire the user's voice and facial expressions using its camera and microphone, and by having the server analyze that data in real time. If the user says, "I'm not feeling well today," the emotion engine will determine the user's emotional state from their tone of voice and facial expressions. The server will feed the emotional data back into a natural language processing system to generate the most appropriate response based on the emotion. This response will be communicated to the user through the device as needed, enabling more empathetic and appropriate communication.

[0338] Furthermore, emotional data is used in conjunction with health management tools. The server analyzes the user's emotional change patterns and evaluates them as part of their overall health status. For example, if a user frequently experiences stress, the health management tools will suggest nutritional plans and relaxation menus based on that data. In addition, rehabilitation programs can be customized to maintain user motivation by adjusting the difficulty level based on the emotional engine data.

[0339] Emergency response measures include a rapid response to sudden changes in the user's emotional state. When the server receives an alert from the emotion engine, it sends a notification to family members or caregivers via the emergency contact network. In this way, the system, with the addition of the emotion engine, constantly monitors the user's emotional state, providing greater flexibility and individualized support in communication and health management.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The device captures the user's voice and facial expressions in real time. This is done using a camera and microphone.

[0343] Step 2:

[0344] The terminal sends the acquired audio and video data to the server. The data is converted into a format for processing.

[0345] Step 3:

[0346] The server converts the audio data into text and performs natural language processing to analyze the user's intent.

[0347] Step 4:

[0348] The server uses video data to perform facial recognition and estimate the user's emotional state from their facial expressions. An emotion engine is used for the analysis.

[0349] Step 5:

[0350] The server integrates the analyzed intent and emotional state to determine the optimal response for the user. The response will have a tone that matches the emotional state.

[0351] Step 6:

[0352] The device communicates optimized responses from the server to the user via voice or text. This allows the user to receive feedback that takes emotions into account.

[0353] Step 7:

[0354] The device continuously collects the user's health data and sends it to the server along with emotional data.

[0355] Step 8:

[0356] The server comprehensively analyzes the received health and emotional data to assess the user's health status. Health management tools provide appropriate advice as needed.

[0357] Step 9:

[0358] The device displays information to help users improve their rehabilitation and lifestyle based on insights from the server. Augmented reality may be used for this purpose.

[0359] Step 10:

[0360] In the event of an emergency, the device immediately sends an alert to the server via its sensors. Sudden changes in emotions are also monitored.

[0361] Step 11:

[0362] The server receives alerts and quickly notifies relevant parties through the emergency contact network. Sentiment data is also used as a basis for making decisions regarding emergency response.

[0363] (Example 2)

[0364] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0365] In the daily lives of users, including the elderly, it was difficult with conventional systems to appropriately understand emotional changes and to provide appropriate communication and health management. Furthermore, the provision of individualized rehabilitation programs tailored to the emotional state of users and prompt responses in emergencies were not adequately implemented.

[0366] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0367] In this invention, the server includes analysis means for collecting and analyzing voice data and facial expression data to determine the emotional state, natural language processing means for generating appropriate responses according to the emotional state, and health management means for collecting and analyzing individual users' health information and using it in conjunction with emotional data. This makes it possible to grasp changes in the user's emotions in real time and to quickly implement individualized and emergency responses accordingly.

[0368] "Voice data" refers to information recorded in a digital format of the user's voice for use in analysis.

[0369] "Facial expression data" refers to information recorded in image or video format, capturing the movements and expressions of a user's face, and used for emotion analysis.

[0370] "Analysis means" refers to technologies and devices for processing collected voice data and facial expression data to estimate the user's emotional state.

[0371] "Natural language processing means" refers to technologies and devices that generate appropriate responses based on analyzed emotion data and facilitate dialogue.

[0372] "Health management tools" refer to technologies and devices that collect and analyze users' health information, integrate it with the resulting emotional data, and then evaluate and manage their health status.

[0373] "Information presentation means" refers to technologies and devices used to convey individually tailored rehabilitation programs and health information to users.

[0374] A "warning mechanism" refers to a technology or device that monitors changes in a user's emotions and issues a notification if an anomaly is detected.

[0375] This invention is a system that supports the daily lives of users, including the elderly, and includes features such as emotional state analysis, response generation using natural language processing, integration with health management, customization of rehabilitation programs, and emergency response.

[0376] Specific implementations of the system

[0377] The device collects the user's voice and facial expressions in real time. Specific hardware examples include communication devices with built-in cameras and microphones. This allows for the acquisition of the user's visual and auditory information.

[0378] The server receives audio and facial expression data transmitted from the terminal. Using speech recognition technology, the audio data is converted into text, and then facial expression analysis technology is used to estimate emotions from the facial expression data. This process quantifies emotions from the tone of voice and facial movements, and generates a response using natural language processing technology.

[0379] The generated response is transmitted to the user via the terminal. This enables natural and empathetic communication for the user. Possible software used for this purpose includes speech synthesis software and a GUI for display.

[0380] The server integrates the analyzed emotional data with a health management system. For example, if stress levels are high, it suggests relaxation techniques. Rehabilitation programs are tailored based on emotional data and designed to help motivate the user.

[0381] Furthermore, the server has a monitoring function to detect sudden changes in emotional state. If an anomaly is detected, it will alert the user's family or health support team and prompt them to take emergency action.

[0382] Specific example

[0383] For example, if a user says, "I'm not feeling well today," the device collects that voice and sends it to the server. The server uses speech recognition technology to analyze the phrase "I'm not feeling well" and determines the emotional state to be "anxious." As a result, natural language processing generates a response such as, "Please rest well. Shall I play some relaxing music?" and conveys it to the user through the device.

[0384] Example of a prompt

[0385] "Analyze the user's voice and facial expression data, determine their emotional state, and generate an appropriate response."

[0386] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0387] Step 1:

[0388] The device collects the user's voice and facial expressions. Specifically, it records the user's face with a camera and records their voice with a microphone. The input is real-time visual and audio data, which is then converted into packets and prepared for transmission to the server. The output is structured data packets.

[0389] Step 2:

[0390] The server receives audio and facial expression data transmitted from the terminal. The input here is the data packets sent from the terminal. The server converts this data into an internal data structure for analysis. The output is in a parseable data format.

[0391] Step 3:

[0392] The server converts audio data into text data using speech recognition software. The input is audio data. The server processes the data, converting the audio into text. The output is text data.

[0393] Step 4:

[0394] The server analyzes facial expression data to estimate the user's emotional state. The input is image data representing facial expressions. The server uses a facial expression recognition algorithm to quantify emotions. The output is numerical data representing the emotional state.

[0395] Step 5:

[0396] The server generates a response using natural language processing based on the emotional state. The input consists of text data and emotional state data. A generative AI model is used to perform calculations that produce an appropriate response. The output is the response text to be conveyed to the user.

[0397] Step 6:

[0398] The server sends the generated response text to the terminal. The input is the generated response text. The server uses a communication protocol to send this to the terminal in order to convey it to the user. The output is the response data delivered to the terminal.

[0399] Step 7:

[0400] The terminal communicates the response received from the server to the user. Specifically, it displays text on the screen and plays audio through the speaker. The input is the response data from the server, and the output is the user's perception.

[0401] (Application Example 2)

[0402] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0403] In the lives of the elderly, not only physical health management but also emotional support is essential. Traditional systems have been unable to adequately address emotional changes, making it difficult to provide appropriate support. In particular, the lack of a comprehensive support system that considers the impact of emotional changes on health is a significant challenge.

[0404] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0405] In this invention, the server includes information processing means for receiving and analyzing acoustic input and generating an appropriate response, health management means for collecting and analyzing the biometric data of individual users, and emotion analysis means for analyzing the emotional state of users and providing empathetic support based on that analysis. This enables comprehensive support that responds immediately to changes in the user's physical and emotional health.

[0406] "Acoustic input" refers to signals that are mechanically received from the user's voice and surrounding environmental sounds.

[0407] "Information processing means" refers to technologies for analyzing received data and generating appropriate responses or operations based on that analysis.

[0408] "Biometric data" refers to various types of physiological information that indicate the user's health status.

[0409] A "health management system" is a system that analyzes a user's health status based on biometric data and provides appropriate management and recommendations.

[0410] "Visual augmentation display means" refers to a technology that displays virtual information overlaid on real-world visual information.

[0411] An "immediate response mechanism" is a system for quickly providing necessary notifications and taking appropriate action in response to detected emergencies.

[0412] "Emotional analysis methods" refer to technologies that analyze a user's emotional state from their voice and facial expressions, and then use the results to respond accordingly.

[0413] The system for carrying out this invention combines the functions of acoustic input, information processing means, biometric data collection, health management means, visual augmentation display, immediate response, and emotion analysis means. A specific embodiment thereof is shown below.

[0414] The server acquires user voice and ambient sounds through acoustic input. The audio data is analyzed using a natural language processing model with Python and deep learning frameworks such as TensorFlow and PyTorch. An appropriate response is then generated from the analysis results.

[0415] User biometric data is collected by smartphones or dedicated wearable devices. These devices transmit data such as body temperature and heart rate to a server in real time. The server uses this data to analyze the user's health status and provides feedback based on the results as a health management tool.

[0416] Emotion analysis is performed by capturing the user's facial expressions and tone of voice using the device's camera and microphone. This data is instantly analyzed using image processing libraries such as OpenCV and deep learning models to evaluate the user's emotional state. Based on the evaluation results, empathetic responses and health suggestions are generated.

[0417] For example, if a user says, "I'm tired today," the system will determine their stress and fatigue level from their tone of voice and facial expression, and then play relaxing music or suggest stretching exercises. A concrete example of a prompt would be to instruct the AI ​​to "generate health advice to improve the user's mood."

[0418] This system allows users to receive not only physical health management but also mental support, enabling them to live a more comfortable daily life.

[0419] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0420] Step 1:

[0421] The device's microphone and camera capture the user's voice and facial expression data in real time. This input data is primary information indicating the user's current state.

[0422] Step 2:

[0423] The device sends the acquired audio data to the server. The server analyzes the audio data using a natural language processing model based on TensorFlow or PyTorch. The analyzed data is output as text data to understand the user's emotional state and health status.

[0424] Step 3:

[0425] The server analyzes facial expression data sent from the camera using OpenCV and a deep learning model. The analyzed emotion data is output as numerical information to evaluate the user's emotional state.

[0426] Step 4:

[0427] The server acquires data from wearable devices containing biometric information and performs data analysis using health management tools. This output information is used as foundational data to evaluate the user's health status and generate health management advice as needed.

[0428] Step 5:

[0429] The server integrates the analyzed emotional data and health information, and uses a generative AI model to generate empathetic responses and health suggestions optimized for the user. This output response is based on the prompt "Please generate health advice to improve the user's mood" and is sent to the terminal as a response expressed in natural language.

[0430] Step 6:

[0431] The user's device displays the generated response on the screen or plays it back as audio. This allows the user to receive health management and emotional support suggested by the server.

[0432] This processing flow allows users to receive appropriate feedback on their physical and mental health status in real time.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] This invention is a system that comprehensively supports home care for the elderly by combining natural language processing means, health management means, augmented reality display means, and emergency response means. The system realizes its functions based on the interaction of a server, terminals, and users.

[0450] The server receives user voice input and performs natural language processing to control the operation of necessary electronic devices. If the user commands "Turn on the TV," the server converts the command into text and sends a signal to the device to turn on the TV. In this way, it facilitates accurate communication between the user and the device.

[0451] Health management is performed by collecting the user's vital data via a device and sending it to a server. The server analyzes this data and provides nutritional management and medical advice based on the user's health indicators. For example, if it detects a regular heart rate abnormality, the server will issue a warning and suggest necessary actions.

[0452] Augmented reality display provides users with rehabilitation content in real time via an augmented reality device. The server customizes the rehabilitation plan based on the user's progress data and provides appropriate guidance through the device. For example, it provides feedback to confirm whether the user is performing the exercises correctly and adjusts the content and difficulty level as needed.

[0453] Emergency response is achieved by the device monitoring the user's surroundings. If the user falls, the device immediately sends an alert signal to the server. The server automatically notifies emergency contacts, informing them that a rapid response is required.

[0454] As described above, by coordinating these various methods, it becomes possible to provide the safe and independent living environment that elderly people desire.

[0455] The following describes the processing flow.

[0456] Step 1:

[0457] The device captures the user's voice using a microphone. The voice data is immediately converted to a digital format and undergoes initial pre-processing.

[0458] Step 2:

[0459] The terminal sends voice data to the server. The server receives the voice data and converts it into text data using automatic speech recognition technology.

[0460] Step 3:

[0461] The server analyzes the converted text data and uses natural language processing to understand the user's intent. This analysis includes understanding the context and interpreting commands.

[0462] Step 4:

[0463] Based on the analysis results, the server determines the appropriate action. For example, if the user requests to operate a home appliance, the server sends that command to the terminal.

[0464] Step 5:

[0465] The terminal receives instructions from the server and performs the necessary device operations. It also provides audio or visual feedback to the user.

[0466] Step 6:

[0467] The device continuously collects the user's vital data and periodically sends this data to the server.

[0468] Step 7:

[0469] The server analyzes the received vital data and assesses the patient's health status. If an abnormality is detected, it issues an alert and notifies medical professionals and family members.

[0470] Step 8:

[0471] The device displays a customized rehabilitation program through an augmented reality device worn by the user and monitors the user's progress.

[0472] Step 9:

[0473] The device monitors the surrounding environment and immediately sends an alert to the server if it detects an emergency such as a fall.

[0474] Step 10:

[0475] The server receives emergency alerts and automatically sends notifications to pre-registered emergency contacts, prompting them to take the necessary action quickly.

[0476] (Example 1)

[0477] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] For elderly people to live safely and independently at home, a comprehensive support system is needed that includes assistance with home appliances, health management, rehabilitation, and prompt emergency response. However, current technology only addresses these issues individually, and a system that provides comprehensive support is lacking. Therefore, there is a need to improve the quality of life for the elderly by coordinating these various functions.

[0479] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0480] In this invention, the server includes processing means for receiving and analyzing audio signals and converting them into commands, processing means for evaluating the user's health status by collecting and analyzing biometric information, and augmented reality provision means for customizing and displaying rehabilitation based on the user's progress data. This improves safety in the living environment of the elderly and enables life support tailored to individual needs.

[0481] A "processing means for receiving, analyzing, and converting audio signals into commands" is a device that receives audio input from a user as a digital signal and converts that audio data into text commands using natural language processing technology.

[0482] "Processing means for evaluating a user's health status by collecting and analyzing biometric information" refers to a device that collects health-related data from users, analyzes that data to evaluate their health status, and provides appropriate advice.

[0483] "An augmented reality provisioning method for customizing and displaying rehabilitation based on user progress data" refers to a device that analyzes the user's rehabilitation progress, creates an individually optimized rehabilitation plan based on that data, and displays it using augmented reality technology.

[0484] An "emergency response system for detecting and promptly notifying of environmental changes and abnormalities" is a device that continuously monitors the environment around the user, immediately detects falls or other abnormal situations, and promptly notifies the appropriate emergency contacts.

[0485] This invention is a comprehensive system designed to support safe and independent living for the elderly in their own homes. This system integrates four functions—voice recognition, health management, augmented reality (AR), and emergency response—through the mutual cooperation of a server, terminal, and user.

[0486] The server uses natural language processing technology to receive and analyze audio signals. Specifically, it converts audio data into text using software such as "SpeechRecognition" or "Google Cloud Speech-to-Text API," and then uses this text to instruct users to operate home appliances. For example, a user can say "Turn off the lights" and the server will then turn off the lights.

[0487] The device sends biometric information collected from the wearable device to a server. The server analyzes this data using machine learning algorithms such as "TensorFlow" or "scikit-learn" to assess the user's health status. If an abnormality is detected, the user is immediately notified and given specific advice, such as "Drink some water."

[0488] Furthermore, in augmented reality-based rehabilitation, the device utilizes platforms such as "ARKit" or "Vuforia" to display a rehabilitation program customized for the user. The server adjusts the plan based on the user's progress data and provides real-time feedback. For example, if the user's exercise is insufficient, it will provide guidance such as, "Let's lift your knee a little higher."

[0489] Furthermore, the device uses an accelerometer and other sensors to monitor the user's surroundings and detect anomalies. In the event of an emergency such as a fall, the device immediately sends an alert to the server. The server then uses the Twilio API to automatically notify emergency contacts, supporting a rapid response.

[0490] This system allows users to enjoy a safe and comfortable life at home, and improve their quality of life. An example of a prompt for the generated AI model would be the instruction, "Simulate a situation where the user changes the lighting settings by voice."

[0491] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0492] Step 1:

[0493] The user issues a voice command. The voice signal is received via the terminal's microphone. The terminal converts this voice signal into digital data and sends it to the server. The input is the voice instruction from the user, and the output is digital voice data. This data conversion transforms the voice command into a format that can be parsed within the system.

[0494] Step 2:

[0495] The server converts received digital audio data into text using natural language processing techniques. Specifically, it analyzes the audio data using software such as the "Google Cloud Speech-to-Text API" to obtain text commands. The input is digital audio data, and the output is text-formatted commands. Through this process, the server understands the user's intent.

[0496] Step 3:

[0497] The server analyzes the converted text command and generates specific commands to operate electronic devices. For example, if a user says "Turn on the TV," the server generates a TV-on command and sends it to the terminal. The input is the text command resulting from the speech analysis, and the output is an executable device control command. This process enables operation based on voice commands.

[0498] Step 4:

[0499] The terminal receives control commands from the server and controls the target electronic device. For example, the terminal turns on the television via a smart home device. The input is the control command from the server, and the output is the actual physical operation of the device. This process enables physical operation based on the user's intent.

[0500] Step 5:

[0501] The terminal collects biometric information from wearable devices and transmits it to a server. This includes data such as heart rate and body temperature. The input is biometric data from the wearable device, and the output is digital health data transmitted to the server. This data collection forms the foundation for health management.

[0502] Step 6:

[0503] The server analyzes received health data and evaluates the user's health status. It uses machine learning algorithms to check for deviations from the normal range. The input is biometric data, and the output is the health status evaluation result. This analysis enables the provision of necessary health advice.

[0504] Step 7:

[0505] The server generates necessary health advice and warnings based on the evaluation results and notifies the user via the terminal. For example, a message such as "We recommend you drink plenty of water" is sent as needed. The input is the health status evaluation result, and the output is a specific notification to the user. This notification allows the user to deepen their awareness of their own health status.

[0506] Step 8:

[0507] The terminal uses augmented reality devices to provide rehabilitation information to the user. The server customizes the rehabilitation plan based on the user's data and sends it to the terminal. The input is the user's health status and progress data, and the output is the customized rehabilitation plan and its implementation instructions. This support enables more effective rehabilitation.

[0508] Step 9:

[0509] The terminal monitors environmental changes and anomalies, and sends an alert to the server when an anomaly is detected. Specifically, it may use an accelerometer to detect falls. The input is data from environmental sensors, and the output is an alert signal for an anomaly. This monitoring function enables a rapid response.

[0510] Step 10:

[0511] When the server receives an alert, it sends a notification to the emergency contact requesting prompt assistance. For example, it can use the Twilio API to send a message to the emergency contact. The input is an anomaly alert from the device, and the output is a notification to the emergency contact. This notification ensures that appropriate action is taken immediately if necessary.

[0512] (Application Example 1)

[0513] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0514] In modern society, the elderly face many difficulties in maintaining independent daily living. In particular, health management, emergency response, and operating household devices pose significant burdens. Furthermore, effective rehabilitation and individualized support for health management are essential for achieving a better quality of life. However, systems to comprehensively support these aspects are currently inadequate.

[0515] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0516] In this invention, the server includes language processing means for receiving and analyzing voice input and generating an appropriate response; health management means for collecting and analyzing biometric information of individual users; augmented reality display means for individually customizing and presenting a functional recovery program based on the collected information; alarm response means for detecting and notifying of emergencies; and wide-area control means for controlling the operation of electronic devices. This enables elderly people to live independently, safely, and healthily.

[0517] A "language processing system" is a system equipped with the function of receiving voice input, analyzing it, and generating an appropriate response.

[0518] A "health management system" is a system that collects biometric information from individual users and analyzes that information to manage their health status.

[0519] An "augmented reality display means" is a system that provides technology to individually customize and visually present a functional recovery program to the user based on collected information.

[0520] An "alarm response system" is a function that enables a rapid response by promptly notifying users when an emergency is detected.

[0521] "Wide-area control means" refers to technology that controls the operation of electronic devices in a home or other space based on voice commands or other inputs.

[0522] The system implementing this invention aims to provide comprehensive support for elderly people to live their daily lives safely and independently. The entire system consists of a server, terminals, and users, each playing a specific role.

[0523] The server first receives the user's voice input via a language processing device. This voice is converted into text using a natural language processing library (e.g., Google Cloud Speech-to-Text), and then into commands for operating electronic devices within the home and living space. This enables voice-controlled automation within the home.

[0524] The device collects the user's biometric information in real time through health management tools. This involves using wearable sensors to collect data such as heart rate and blood pressure. This data is analyzed by a cloud-based data management system, and alerts are issued via the server if any abnormalities are detected.

[0525] Furthermore, as a means of displaying augmented reality, the terminal uses augmented reality (AR) technology (e.g., ARCore) to visually provide rehabilitation programs to the user. The server transfers personalized programs to the terminal based on the collected data and provides appropriate guidance.

[0526] Furthermore, the device is equipped with sensors that monitor the user's surroundings as a means of responding to alarms. This allows for the detection of emergencies such as falls, prompt notification to emergency contacts, and the taking of appropriate measures as needed.

[0527] As a concrete example, let's consider a scenario involving an elderly person named Mr. Tanaka. If Mr. Tanaka says "Make me some tea" using voice command, the server will perform the appropriate action. Also, if an abnormality in heart rate is detected during exercise, an alert will be immediately issued to notify medical professionals or family members.

[0528] Examples of prompts for a generative AI model include the following:

[0529] "Design the optimal voice commands for when users want to control home appliances using their voice."

[0530] "Could you please explain methods for analyzing health data to customize exercise plans for older adults?"

[0531] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0532] Step 1:

[0533] The server receives voice input from the user. The input voice data is converted into text data by the server using a natural language processing library. This prepares the server to analyze the specific instructions.

[0534] Step 2:

[0535] Based on the analyzed text data, the server generates appropriate electronic device control commands in response to user instructions. For example, if the instruction "Turn on the TV" is analyzed, the server generates a control signal for the corresponding device and sends it to the terminal to turn on the TV.

[0536] Step 3:

[0537] The device collects biometric information from the user in real time as a means of health management. The collected data (e.g., heart rate, blood pressure) is sent from the device to a server and analyzed by a cloud-based data management system. Based on the results of this analysis, health status is monitored and continuously managed.

[0538] Step 4:

[0539] The server, as an augmented reality display method, individually customizes a rehabilitation program tailored to each user based on collected health data. The server then delivers this program to the terminal, which uses augmented reality technology to provide visual guidance to the user.

[0540] Step 5:

[0541] The device monitors the user's surroundings using environmental monitoring sensors. If an emergency (e.g., a fall) is detected, the device sends an alert to the server. The server automatically notifies emergency contacts as needed to encourage a quick response.

[0542] Step 6:

[0543] The server uses a generative AI model to respond to detailed information processing requests from users. The generative AI model receives and analyzes prompt messages, providing appropriate information and suggesting service improvements. In this way, it expands the overall system knowledge and improves the user experience.

[0544] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0545] This invention is a system that incorporates an emotion engine in addition to natural language processing means, health management means, augmented reality display means, and emergency response means, in order to comprehensively support the lives of the elderly. This enables personalized responses that take into account the user's emotions.

[0546] The emotion engine is implemented by having the device acquire the user's voice and facial expressions using its camera and microphone, and by having the server analyze that data in real time. If the user says, "I'm not feeling well today," the emotion engine will determine the user's emotional state from their tone of voice and facial expressions. The server will feed the emotional data back into a natural language processing system to generate the most appropriate response based on the emotion. This response will be communicated to the user through the device as needed, enabling more empathetic and appropriate communication.

[0547] Furthermore, emotional data is used in conjunction with health management tools. The server analyzes the user's emotional change patterns and evaluates them as part of their overall health status. For example, if a user frequently experiences stress, the health management tools will suggest nutritional plans and relaxation menus based on that data. In addition, rehabilitation programs can be customized to maintain user motivation by adjusting the difficulty level based on the emotional engine data.

[0548] Emergency response measures include a rapid response to sudden changes in the user's emotional state. When the server receives an alert from the emotion engine, it sends a notification to family members or caregivers via the emergency contact network. In this way, the system, with the addition of the emotion engine, constantly monitors the user's emotional state, providing greater flexibility and individualized support in communication and health management.

[0549] The following describes the processing flow.

[0550] Step 1:

[0551] The device captures the user's voice and facial expressions in real time. This is done using a camera and microphone.

[0552] Step 2:

[0553] The terminal sends the acquired audio and video data to the server. The data is converted into a format for processing.

[0554] Step 3:

[0555] The server converts the audio data into text and performs natural language processing to analyze the user's intent.

[0556] Step 4:

[0557] The server uses video data to perform facial recognition and estimate the user's emotional state from their facial expressions. An emotion engine is used for the analysis.

[0558] Step 5:

[0559] The server integrates the analyzed intent and emotional state to determine the optimal response for the user. The response will have a tone that matches the emotional state.

[0560] Step 6:

[0561] The device communicates optimized responses from the server to the user via voice or text. This allows the user to receive feedback that takes emotions into account.

[0562] Step 7:

[0563] The device continuously collects the user's health data and sends it to the server along with emotional data.

[0564] Step 8:

[0565] The server comprehensively analyzes the received health and emotional data to assess the user's health status. Health management tools provide appropriate advice as needed.

[0566] Step 9:

[0567] The device displays information to help users improve their rehabilitation and lifestyle based on insights from the server. Augmented reality may be used for this purpose.

[0568] Step 10:

[0569] In the event of an emergency, the device immediately sends an alert to the server via its sensors. Sudden changes in emotions are also monitored.

[0570] Step 11:

[0571] The server receives alerts and quickly notifies relevant parties through the emergency contact network. Sentiment data is also used as a basis for making decisions regarding emergency response.

[0572] (Example 2)

[0573] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0574] In the daily lives of users, including the elderly, it was difficult with conventional systems to appropriately understand emotional changes and to provide appropriate communication and health management. Furthermore, the provision of individualized rehabilitation programs tailored to the emotional state of users and prompt responses in emergencies were not adequately implemented.

[0575] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0576] In this invention, the server includes analysis means for collecting and analyzing voice data and facial expression data to determine the emotional state, natural language processing means for generating appropriate responses according to the emotional state, and health management means for collecting and analyzing individual users' health information and using it in conjunction with emotional data. This makes it possible to grasp changes in the user's emotions in real time and to quickly implement individualized and emergency responses accordingly.

[0577] "Voice data" refers to information recorded in a digital format of the user's voice for use in analysis.

[0578] "Facial expression data" refers to information recorded in image or video format, capturing the movements and expressions of a user's face, and used for emotion analysis.

[0579] "Analysis means" refers to technologies and devices for processing collected voice data and facial expression data to estimate the user's emotional state.

[0580] "Natural language processing means" refers to technologies and devices that generate appropriate responses based on analyzed emotion data and facilitate dialogue.

[0581] "Health management tools" refer to technologies and devices that collect and analyze users' health information, integrate it with the resulting emotional data, and then evaluate and manage their health status.

[0582] "Information presentation means" refers to technologies and devices used to convey individually tailored rehabilitation programs and health information to users.

[0583] A "warning mechanism" refers to a technology or device that monitors changes in a user's emotions and issues a notification if an anomaly is detected.

[0584] This invention is a system that supports the daily lives of users, including the elderly, and includes features such as emotional state analysis, response generation using natural language processing, integration with health management, customization of rehabilitation programs, and emergency response.

[0585] Specific implementations of the system

[0586] The device collects the user's voice and facial expressions in real time. Specific hardware examples include communication devices with built-in cameras and microphones. This allows for the acquisition of the user's visual and auditory information.

[0587] The server receives audio and facial expression data transmitted from the terminal. Using speech recognition technology, the audio data is converted into text, and then facial expression analysis technology is used to estimate emotions from the facial expression data. This process quantifies emotions from the tone of voice and facial movements, and generates a response using natural language processing technology.

[0588] The generated response is transmitted to the user via the terminal. This enables natural and empathetic communication for the user. Possible software used for this purpose includes speech synthesis software and a GUI for display.

[0589] The server integrates the analyzed emotional data with a health management system. For example, if stress levels are high, it suggests relaxation techniques. Rehabilitation programs are tailored based on emotional data and designed to help motivate the user.

[0590] Furthermore, the server has a monitoring function to detect sudden changes in emotional state. If an anomaly is detected, it will alert the user's family or health support team and prompt them to take emergency action.

[0591] Specific example

[0592] For example, if a user says, "I'm not feeling well today," the device collects that voice and sends it to the server. The server uses speech recognition technology to analyze the phrase "I'm not feeling well" and determines the emotional state to be "anxious." As a result, natural language processing generates a response such as, "Please rest well. Shall I play some relaxing music?" and conveys it to the user through the device.

[0593] Example of a prompt

[0594] "Analyze the user's voice and facial expression data, determine their emotional state, and generate an appropriate response."

[0595] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0596] Step 1:

[0597] The device collects the user's voice and facial expressions. Specifically, it records the user's face with a camera and records their voice with a microphone. The input is real-time visual and audio data, which is then converted into packets and prepared for transmission to the server. The output is structured data packets.

[0598] Step 2:

[0599] The server receives audio and facial expression data transmitted from the terminal. The input here is the data packets sent from the terminal. The server converts this data into an internal data structure for analysis. The output is in a parseable data format.

[0600] Step 3:

[0601] The server converts audio data into text data using speech recognition software. The input is audio data. The server processes the data, converting the audio into text. The output is text data.

[0602] Step 4:

[0603] The server analyzes facial expression data to estimate the user's emotional state. The input is image data representing facial expressions. The server uses a facial expression recognition algorithm to quantify emotions. The output is numerical data representing the emotional state.

[0604] Step 5:

[0605] The server generates a response using natural language processing based on the emotional state. The input consists of text data and emotional state data. A generative AI model is used to perform calculations that produce an appropriate response. The output is the response text to be conveyed to the user.

[0606] Step 6:

[0607] The server sends the generated response text to the terminal. The input is the generated response text. The server uses a communication protocol to send this to the terminal in order to convey it to the user. The output is the response data delivered to the terminal.

[0608] Step 7:

[0609] The terminal communicates the response received from the server to the user. Specifically, it displays text on the screen and plays audio through the speaker. The input is the response data from the server, and the output is the user's perception.

[0610] (Application Example 2)

[0611] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0612] In the lives of the elderly, not only physical health management but also emotional support is essential. Traditional systems have been unable to adequately address emotional changes, making it difficult to provide appropriate support. In particular, the lack of a comprehensive support system that considers the impact of emotional changes on health is a significant challenge.

[0613] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0614] In this invention, the server includes information processing means for receiving and analyzing acoustic input and generating an appropriate response, health management means for collecting and analyzing the biometric data of individual users, and emotion analysis means for analyzing the emotional state of users and providing empathetic support based on that analysis. This enables comprehensive support that responds immediately to changes in the user's physical and emotional health.

[0615] "Acoustic input" refers to signals that are mechanically received from the user's voice and surrounding environmental sounds.

[0616] "Information processing means" refers to technologies for analyzing received data and generating appropriate responses or operations based on that analysis.

[0617] "Biometric data" refers to various types of physiological information that indicate the user's health status.

[0618] A "health management system" is a system that analyzes a user's health status based on biometric data and provides appropriate management and recommendations.

[0619] "Visual augmentation display means" refers to a technology that displays virtual information overlaid on real-world visual information.

[0620] An "immediate response mechanism" is a system for quickly providing necessary notifications and taking appropriate action in response to detected emergencies.

[0621] "Emotional analysis methods" refer to technologies that analyze a user's emotional state from their voice and facial expressions, and then use the results to respond accordingly.

[0622] The system for carrying out this invention combines the functions of acoustic input, information processing means, biometric data collection, health management means, visual augmentation display, immediate response, and emotion analysis means. A specific embodiment thereof is shown below.

[0623] The server acquires user voice and ambient sounds through acoustic input. The audio data is analyzed using a natural language processing model with Python and deep learning frameworks such as TensorFlow and PyTorch. An appropriate response is then generated from the analysis results.

[0624] User biometric data is collected by smartphones or dedicated wearable devices. These devices transmit data such as body temperature and heart rate to a server in real time. The server uses this data to analyze the user's health status and provides feedback based on the results as a health management tool.

[0625] Emotion analysis is performed by capturing the user's facial expressions and tone of voice using the device's camera and microphone. This data is instantly analyzed using image processing libraries such as OpenCV and deep learning models to evaluate the user's emotional state. Based on the evaluation results, empathetic responses and health suggestions are generated.

[0626] For example, if a user says, "I'm tired today," the system will determine their stress and fatigue level from their tone of voice and facial expression, and then play relaxing music or suggest stretching exercises. A concrete example of a prompt would be to instruct the AI ​​to "generate health advice to improve the user's mood."

[0627] This system allows users to receive not only physical health management but also mental support, enabling them to live a more comfortable daily life.

[0628] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0629] Step 1:

[0630] The device's microphone and camera capture the user's voice and facial expression data in real time. This input data is primary information indicating the user's current state.

[0631] Step 2:

[0632] The device sends the acquired audio data to the server. The server analyzes the audio data using a natural language processing model based on TensorFlow or PyTorch. The analyzed data is output as text data to understand the user's emotional state and health status.

[0633] Step 3:

[0634] The server analyzes facial expression data sent from the camera using OpenCV and a deep learning model. The analyzed emotion data is output as numerical information to evaluate the user's emotional state.

[0635] Step 4:

[0636] The server acquires data from wearable devices containing biometric information and performs data analysis using health management tools. This output information is used as foundational data to evaluate the user's health status and generate health management advice as needed.

[0637] Step 5:

[0638] The server integrates the analyzed emotional data and health information, and uses a generative AI model to generate empathetic responses and health suggestions optimized for the user. This output response is based on the prompt "Please generate health advice to improve the user's mood" and is sent to the terminal as a response expressed in natural language.

[0639] Step 6:

[0640] The user's device displays the generated response on the screen or plays it back as audio. This allows the user to receive health management and emotional support suggested by the server.

[0641] This processing flow allows users to receive appropriate feedback on their physical and mental health status in real time.

[0642] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0643] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0644] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0645] [Fourth Embodiment]

[0646] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0647] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0648] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0649] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0650] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0652] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0653] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0654] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0655] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0658] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0659] This invention is a system that comprehensively supports home care for the elderly by combining natural language processing means, health management means, augmented reality display means, and emergency response means. The system realizes its functions based on the interaction of a server, terminals, and users.

[0660] The server receives user voice input and performs natural language processing to control the operation of necessary electronic devices. If the user commands "Turn on the TV," the server converts the command into text and sends a signal to the device to turn on the TV. In this way, it facilitates accurate communication between the user and the device.

[0661] Health management is performed by collecting the user's vital data via a device and sending it to a server. The server analyzes this data and provides nutritional management and medical advice based on the user's health indicators. For example, if it detects a regular heart rate abnormality, the server will issue a warning and suggest necessary actions.

[0662] Augmented reality display provides users with rehabilitation content in real time via an augmented reality device. The server customizes the rehabilitation plan based on the user's progress data and provides appropriate guidance through the device. For example, it provides feedback to confirm whether the user is performing the exercises correctly and adjusts the content and difficulty level as needed.

[0663] Emergency response is achieved by the device monitoring the user's surroundings. If the user falls, the device immediately sends an alert signal to the server. The server automatically notifies emergency contacts, informing them that a rapid response is required.

[0664] As described above, by coordinating these various methods, it becomes possible to provide the safe and independent living environment that elderly people desire.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] The device captures the user's voice using a microphone. The voice data is immediately converted to a digital format and undergoes initial pre-processing.

[0668] Step 2:

[0669] The terminal sends voice data to the server. The server receives the voice data and converts it into text data using automatic speech recognition technology.

[0670] Step 3:

[0671] The server analyzes the converted text data and uses natural language processing to understand the user's intent. This analysis includes understanding the context and interpreting commands.

[0672] Step 4:

[0673] Based on the analysis results, the server determines the appropriate action. For example, if the user requests to operate a home appliance, the server sends that command to the terminal.

[0674] Step 5:

[0675] The terminal receives instructions from the server and performs the necessary device operations. It also provides audio or visual feedback to the user.

[0676] Step 6:

[0677] The device continuously collects the user's vital data and periodically sends this data to the server.

[0678] Step 7:

[0679] The server analyzes the received vital data and assesses the patient's health status. If an abnormality is detected, it issues an alert and notifies medical professionals and family members.

[0680] Step 8:

[0681] The device displays a customized rehabilitation program through an augmented reality device worn by the user and monitors the user's progress.

[0682] Step 9:

[0683] The device monitors the surrounding environment and immediately sends an alert to the server if it detects an emergency such as a fall.

[0684] Step 10:

[0685] The server receives emergency alerts and automatically sends notifications to pre-registered emergency contacts, prompting them to take the necessary action quickly.

[0686] (Example 1)

[0687] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0688] For elderly people to live safely and independently at home, a comprehensive support system is needed that includes assistance with home appliances, health management, rehabilitation, and prompt emergency response. However, current technology only addresses these issues individually, and a system that provides comprehensive support is lacking. Therefore, there is a need to improve the quality of life for the elderly by coordinating these various functions.

[0689] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0690] In this invention, the server includes processing means for receiving and analyzing audio signals and converting them into commands, processing means for evaluating the user's health status by collecting and analyzing biometric information, and augmented reality provision means for customizing and displaying rehabilitation based on the user's progress data. This improves safety in the living environment of the elderly and enables life support tailored to individual needs.

[0691] A "processing means for receiving, analyzing, and converting audio signals into commands" is a device that receives audio input from a user as a digital signal and converts that audio data into text commands using natural language processing technology.

[0692] "Processing means for evaluating a user's health status by collecting and analyzing biometric information" refers to a device that collects health-related data from users, analyzes that data to evaluate their health status, and provides appropriate advice.

[0693] "An augmented reality provisioning method for customizing and displaying rehabilitation based on user progress data" refers to a device that analyzes the user's rehabilitation progress, creates an individually optimized rehabilitation plan based on that data, and displays it using augmented reality technology.

[0694] An "emergency response system for detecting and promptly notifying of environmental changes and abnormalities" is a device that continuously monitors the environment around the user, immediately detects falls or other abnormal situations, and promptly notifies the appropriate emergency contacts.

[0695] This invention is a comprehensive system designed to support safe and independent living for the elderly in their own homes. This system integrates four functions—voice recognition, health management, augmented reality (AR), and emergency response—through the mutual cooperation of a server, terminal, and user.

[0696] The server uses natural language processing technology to receive and analyze audio signals. Specifically, it converts audio data into text using software such as "SpeechRecognition" or "Google Cloud Speech-to-Text API," and then uses this text to instruct users to operate home appliances. For example, a user can say "Turn off the lights" and the server will then turn off the lights.

[0697] The device sends biometric information collected from the wearable device to a server. The server analyzes this data using machine learning algorithms such as "TensorFlow" or "scikit-learn" to assess the user's health status. If an abnormality is detected, the user is immediately notified and given specific advice, such as "Drink some water."

[0698] Furthermore, in augmented reality-based rehabilitation, the device utilizes platforms such as "ARKit" or "Vuforia" to display a rehabilitation program customized for the user. The server adjusts the plan based on the user's progress data and provides real-time feedback. For example, if the user's exercise is insufficient, it will provide guidance such as, "Let's lift your knee a little higher."

[0699] Furthermore, the device uses an accelerometer and other sensors to monitor the user's surroundings and detect anomalies. In the event of an emergency such as a fall, the device immediately sends an alert to the server. The server then uses the Twilio API to automatically notify emergency contacts, supporting a rapid response.

[0700] This system allows users to enjoy a safe and comfortable life at home, and improve their quality of life. An example of a prompt for the generated AI model would be the instruction, "Simulate a situation where the user changes the lighting settings by voice."

[0701] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0702] Step 1:

[0703] The user issues a voice command. The voice signal is received via the terminal's microphone. The terminal converts this voice signal into digital data and sends it to the server. The input is the voice instruction from the user, and the output is digital voice data. This data conversion transforms the voice command into a format that can be parsed within the system.

[0704] Step 2:

[0705] The server converts received digital audio data into text using natural language processing techniques. Specifically, it analyzes the audio data using software such as the "Google Cloud Speech-to-Text API" to obtain text commands. The input is digital audio data, and the output is text-formatted commands. Through this process, the server understands the user's intent.

[0706] Step 3:

[0707] The server analyzes the converted text command and generates specific commands to operate electronic devices. For example, if a user says "Turn on the TV," the server generates a TV-on command and sends it to the terminal. The input is the text command resulting from the speech analysis, and the output is an executable device control command. This process enables operation based on voice commands.

[0708] Step 4:

[0709] The terminal receives control commands from the server and controls the target electronic device. For example, the terminal turns on the television via a smart home device. The input is the control command from the server, and the output is the actual physical operation of the device. This process enables physical operation based on the user's intent.

[0710] Step 5:

[0711] The terminal collects biometric information from wearable devices and transmits it to a server. This includes data such as heart rate and body temperature. The input is biometric data from the wearable device, and the output is digital health data transmitted to the server. This data collection forms the foundation for health management.

[0712] Step 6:

[0713] The server analyzes received health data and evaluates the user's health status. It uses machine learning algorithms to check for deviations from the normal range. The input is biometric data, and the output is the health status evaluation result. This analysis enables the provision of necessary health advice.

[0714] Step 7:

[0715] The server generates necessary health advice and warnings based on the evaluation results and notifies the user via the terminal. For example, a message such as "We recommend you drink plenty of water" is sent as needed. The input is the health status evaluation result, and the output is a specific notification to the user. This notification allows the user to deepen their awareness of their own health status.

[0716] Step 8:

[0717] The terminal uses augmented reality devices to provide rehabilitation information to the user. The server customizes the rehabilitation plan based on the user's data and sends it to the terminal. The input is the user's health status and progress data, and the output is the customized rehabilitation plan and its implementation instructions. This support enables more effective rehabilitation.

[0718] Step 9:

[0719] The terminal monitors environmental changes and anomalies, and sends an alert to the server when an anomaly is detected. Specifically, it may use an accelerometer to detect falls. The input is data from environmental sensors, and the output is an alert signal for an anomaly. This monitoring function enables a rapid response.

[0720] Step 10:

[0721] When the server receives an alert, it sends a notification to the emergency contact requesting prompt assistance. For example, it can use the Twilio API to send a message to the emergency contact. The input is an anomaly alert from the device, and the output is a notification to the emergency contact. This notification ensures that appropriate action is taken immediately if necessary.

[0722] (Application Example 1)

[0723] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] In modern society, the elderly face many difficulties in maintaining independent daily living. In particular, health management, emergency response, and operating household devices pose significant burdens. Furthermore, effective rehabilitation and individualized support for health management are essential for achieving a better quality of life. However, systems to comprehensively support these aspects are currently inadequate.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0726] In this invention, the server includes language processing means for receiving and analyzing voice input and generating an appropriate response; health management means for collecting and analyzing biometric information of individual users; augmented reality display means for individually customizing and presenting a functional recovery program based on the collected information; alarm response means for detecting and notifying of emergencies; and wide-area control means for controlling the operation of electronic devices. This enables elderly people to live independently, safely, and healthily.

[0727] A "language processing system" is a system equipped with the function of receiving voice input, analyzing it, and generating an appropriate response.

[0728] A "health management system" is a system that collects biometric information from individual users and analyzes that information to manage their health status.

[0729] An "augmented reality display means" is a system that provides technology to individually customize and visually present a functional recovery program to the user based on collected information.

[0730] An "alarm response system" is a function that enables a rapid response by promptly notifying users when an emergency is detected.

[0731] "Wide-area control means" refers to technology that controls the operation of electronic devices in a home or other space based on voice commands or other inputs.

[0732] The system implementing this invention aims to provide comprehensive support for elderly people to live their daily lives safely and independently. The entire system consists of a server, terminals, and users, each playing a specific role.

[0733] The server first receives the user's voice input via a language processing device. This voice is converted into text using a natural language processing library (e.g., Google Cloud Speech-to-Text), and then into commands for operating electronic devices within the home and living space. This enables voice-controlled automation within the home.

[0734] The device collects the user's biometric information in real time through health management tools. This involves using wearable sensors to collect data such as heart rate and blood pressure. This data is analyzed by a cloud-based data management system, and alerts are issued via the server if any abnormalities are detected.

[0735] Furthermore, as a means of displaying augmented reality, the terminal uses augmented reality (AR) technology (e.g., ARCore) to visually provide rehabilitation programs to the user. The server transfers personalized programs to the terminal based on the collected data and provides appropriate guidance.

[0736] Furthermore, the device is equipped with sensors that monitor the user's surroundings as a means of responding to alarms. This allows for the detection of emergencies such as falls, prompt notification to emergency contacts, and the taking of appropriate measures as needed.

[0737] As a concrete example, let's consider a scenario involving an elderly person named Mr. Tanaka. If Mr. Tanaka says "Make me some tea" using voice command, the server will perform the appropriate action. Also, if an abnormality in heart rate is detected during exercise, an alert will be immediately issued to notify medical professionals or family members.

[0738] Examples of prompts for a generative AI model include the following:

[0739] "Design the optimal voice commands for when users want to control home appliances using their voice."

[0740] "Could you please explain methods for analyzing health data to customize exercise plans for older adults?"

[0741] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0742] Step 1:

[0743] The server receives voice input from the user. The input voice data is converted into text data by the server using a natural language processing library. This prepares the server to analyze the specific instructions.

[0744] Step 2:

[0745] Based on the analyzed text data, the server generates appropriate electronic device control commands in response to user instructions. For example, if the instruction "Turn on the TV" is analyzed, the server generates a control signal for the corresponding device and sends it to the terminal to turn on the TV.

[0746] Step 3:

[0747] The device collects biometric information from the user in real time as a means of health management. The collected data (e.g., heart rate, blood pressure) is sent from the device to a server and analyzed by a cloud-based data management system. Based on the results of this analysis, health status is monitored and continuously managed.

[0748] Step 4:

[0749] The server, as an augmented reality display method, individually customizes a rehabilitation program tailored to each user based on collected health data. The server then delivers this program to the terminal, which uses augmented reality technology to provide visual guidance to the user.

[0750] Step 5:

[0751] The device monitors the user's surroundings using environmental monitoring sensors. If an emergency (e.g., a fall) is detected, the device sends an alert to the server. The server automatically notifies emergency contacts as needed to encourage a quick response.

[0752] Step 6:

[0753] The server uses a generative AI model to respond to detailed information processing requests from users. The generative AI model receives and analyzes prompt messages, providing appropriate information and suggesting service improvements. In this way, it expands the overall system knowledge and improves the user experience.

[0754] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0755] This invention is a system that incorporates an emotion engine in addition to natural language processing means, health management means, augmented reality display means, and emergency response means, in order to comprehensively support the lives of the elderly. This enables personalized responses that take into account the user's emotions.

[0756] The emotion engine is implemented by having the device acquire the user's voice and facial expressions using its camera and microphone, and by having the server analyze that data in real time. If the user says, "I'm not feeling well today," the emotion engine will determine the user's emotional state from their tone of voice and facial expressions. The server will feed the emotional data back into a natural language processing system to generate the most appropriate response based on the emotion. This response will be communicated to the user through the device as needed, enabling more empathetic and appropriate communication.

[0757] Furthermore, emotional data is used in conjunction with health management tools. The server analyzes the user's emotional change patterns and evaluates them as part of their overall health status. For example, if a user frequently experiences stress, the health management tools will suggest nutritional plans and relaxation menus based on that data. In addition, rehabilitation programs can be customized to maintain user motivation by adjusting the difficulty level based on the emotional engine data.

[0758] Emergency response measures include a rapid response to sudden changes in the user's emotional state. When the server receives an alert from the emotion engine, it sends a notification to family members or caregivers via the emergency contact network. In this way, the system, with the addition of the emotion engine, constantly monitors the user's emotional state, providing greater flexibility and individualized support in communication and health management.

[0759] The following describes the processing flow.

[0760] Step 1:

[0761] The device captures the user's voice and facial expressions in real time. This is done using a camera and microphone.

[0762] Step 2:

[0763] The terminal sends the acquired audio and video data to the server. The data is converted into a format for processing.

[0764] Step 3:

[0765] The server converts the audio data into text and performs natural language processing to analyze the user's intent.

[0766] Step 4:

[0767] The server uses video data to perform facial recognition and estimate the user's emotional state from their facial expressions. An emotion engine is used for the analysis.

[0768] Step 5:

[0769] The server integrates the analyzed intent and emotional state to determine the optimal response for the user. The response will have a tone that matches the emotional state.

[0770] Step 6:

[0771] The device communicates optimized responses from the server to the user via voice or text. This allows the user to receive feedback that takes emotions into account.

[0772] Step 7:

[0773] The device continuously collects the user's health data and sends it to the server along with emotional data.

[0774] Step 8:

[0775] The server comprehensively analyzes the received health and emotional data to assess the user's health status. Health management tools provide appropriate advice as needed.

[0776] Step 9:

[0777] The device displays information to help users improve their rehabilitation and lifestyle based on insights from the server. Augmented reality may be used for this purpose.

[0778] Step 10:

[0779] In the event of an emergency, the device immediately sends an alert to the server via its sensors. Sudden changes in emotions are also monitored.

[0780] Step 11:

[0781] The server receives alerts and quickly notifies relevant parties through the emergency contact network. Sentiment data is also used as a basis for making decisions regarding emergency response.

[0782] (Example 2)

[0783] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0784] In the daily lives of users, including the elderly, it was difficult with conventional systems to appropriately understand emotional changes and to provide appropriate communication and health management. Furthermore, the provision of individualized rehabilitation programs tailored to the emotional state of users and prompt responses in emergencies were not adequately implemented.

[0785] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0786] In this invention, the server includes analysis means for collecting and analyzing voice data and facial expression data to determine the emotional state, natural language processing means for generating appropriate responses according to the emotional state, and health management means for collecting and analyzing individual users' health information and using it in conjunction with emotional data. This makes it possible to grasp changes in the user's emotions in real time and to quickly implement individualized and emergency responses accordingly.

[0787] "Voice data" refers to information recorded in a digital format of the user's voice for use in analysis.

[0788] "Facial expression data" refers to information recorded in image or video format, capturing the movements and expressions of a user's face, and used for emotion analysis.

[0789] "Analysis means" refers to technologies and devices for processing collected voice data and facial expression data to estimate the user's emotional state.

[0790] "Natural language processing means" refers to technologies and devices that generate appropriate responses based on analyzed emotion data and facilitate dialogue.

[0791] "Health management tools" refer to technologies and devices that collect and analyze users' health information, integrate it with the resulting emotional data, and then evaluate and manage their health status.

[0792] "Information presentation means" refers to technologies and devices used to convey individually tailored rehabilitation programs and health information to users.

[0793] A "warning mechanism" refers to a technology or device that monitors changes in a user's emotions and issues a notification if an anomaly is detected.

[0794] This invention is a system that supports the daily lives of users, including the elderly, and includes features such as emotional state analysis, response generation using natural language processing, integration with health management, customization of rehabilitation programs, and emergency response.

[0795] Specific implementations of the system

[0796] The device collects the user's voice and facial expressions in real time. Specific hardware examples include communication devices with built-in cameras and microphones. This allows for the acquisition of the user's visual and auditory information.

[0797] The server receives audio and facial expression data transmitted from the terminal. Using speech recognition technology, the audio data is converted into text, and then facial expression analysis technology is used to estimate emotions from the facial expression data. This process quantifies emotions from the tone of voice and facial movements, and generates a response using natural language processing technology.

[0798] The generated response is transmitted to the user via the terminal. This enables natural and empathetic communication for the user. Possible software used for this purpose includes speech synthesis software and a GUI for display.

[0799] The server integrates the analyzed emotional data with a health management system. For example, if stress levels are high, it suggests relaxation techniques. Rehabilitation programs are tailored based on emotional data and designed to help motivate the user.

[0800] Furthermore, the server has a monitoring function to detect sudden changes in emotional state. If an anomaly is detected, it will alert the user's family or health support team and prompt them to take emergency action.

[0801] Specific example

[0802] For example, if a user says, "I'm not feeling well today," the device collects that voice and sends it to the server. The server uses speech recognition technology to analyze the phrase "I'm not feeling well" and determines the emotional state to be "anxious." As a result, natural language processing generates a response such as, "Please rest well. Shall I play some relaxing music?" and conveys it to the user through the device.

[0803] Example of a prompt

[0804] "Analyze the user's voice and facial expression data, determine their emotional state, and generate an appropriate response."

[0805] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0806] Step 1:

[0807] The device collects the user's voice and facial expressions. Specifically, it records the user's face with a camera and records their voice with a microphone. The input is real-time visual and audio data, which is then converted into packets and prepared for transmission to the server. The output is structured data packets.

[0808] Step 2:

[0809] The server receives audio and facial expression data transmitted from the terminal. The input here is the data packets sent from the terminal. The server converts this data into an internal data structure for analysis. The output is in a parseable data format.

[0810] Step 3:

[0811] The server converts audio data into text data using speech recognition software. The input is audio data. The server processes the data, converting the audio into text. The output is text data.

[0812] Step 4:

[0813] The server analyzes facial expression data to estimate the user's emotional state. The input is image data representing facial expressions. The server uses a facial expression recognition algorithm to quantify emotions. The output is numerical data representing the emotional state.

[0814] Step 5:

[0815] The server generates a response using natural language processing based on the emotional state. The input consists of text data and emotional state data. A generative AI model is used to perform calculations that produce an appropriate response. The output is the response text to be conveyed to the user.

[0816] Step 6:

[0817] The server sends the generated response text to the terminal. The input is the generated response text. The server uses a communication protocol to send this to the terminal in order to convey it to the user. The output is the response data delivered to the terminal.

[0818] Step 7:

[0819] The terminal communicates the response received from the server to the user. Specifically, it displays text on the screen and plays audio through the speaker. The input is the response data from the server, and the output is the user's perception.

[0820] (Application Example 2)

[0821] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] In the lives of the elderly, not only physical health management but also emotional support is essential. Traditional systems have been unable to adequately address emotional changes, making it difficult to provide appropriate support. In particular, the lack of a comprehensive support system that considers the impact of emotional changes on health is a significant challenge.

[0823] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0824] In this invention, the server includes information processing means for receiving and analyzing acoustic input and generating an appropriate response, health management means for collecting and analyzing the biometric data of individual users, and emotion analysis means for analyzing the emotional state of users and providing empathetic support based on that analysis. This enables comprehensive support that responds immediately to changes in the user's physical and emotional health.

[0825] "Acoustic input" refers to signals that are mechanically received from the user's voice and surrounding environmental sounds.

[0826] "Information processing means" refers to technologies for analyzing received data and generating appropriate responses or operations based on that analysis.

[0827] "Biometric data" refers to various types of physiological information that indicate the user's health status.

[0828] A "health management system" is a system that analyzes a user's health status based on biometric data and provides appropriate management and recommendations.

[0829] "Visual augmentation display means" refers to a technology that displays virtual information overlaid on real-world visual information.

[0830] An "immediate response mechanism" is a system for quickly providing necessary notifications and taking appropriate action in response to detected emergencies.

[0831] "Emotional analysis methods" refer to technologies that analyze a user's emotional state from their voice and facial expressions, and then use the results to respond accordingly.

[0832] The system for carrying out this invention combines the functions of acoustic input, information processing means, biometric data collection, health management means, visual augmentation display, immediate response, and emotion analysis means. A specific embodiment thereof is shown below.

[0833] The server acquires user voice and ambient sounds through acoustic input. The audio data is analyzed using a natural language processing model with Python and deep learning frameworks such as TensorFlow and PyTorch. An appropriate response is then generated from the analysis results.

[0834] User biometric data is collected by smartphones or dedicated wearable devices. These devices transmit data such as body temperature and heart rate to a server in real time. The server uses this data to analyze the user's health status and provides feedback based on the results as a health management tool.

[0835] Emotion analysis is performed by capturing the user's facial expressions and tone of voice using the device's camera and microphone. This data is instantly analyzed using image processing libraries such as OpenCV and deep learning models to evaluate the user's emotional state. Based on the evaluation results, empathetic responses and health suggestions are generated.

[0836] For example, if a user says, "I'm tired today," the system will determine their stress and fatigue level from their tone of voice and facial expression, and then play relaxing music or suggest stretching exercises. A concrete example of a prompt would be to instruct the AI ​​to "generate health advice to improve the user's mood."

[0837] This system allows users to receive not only physical health management but also mental support, enabling them to live a more comfortable daily life.

[0838] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0839] Step 1:

[0840] The device's microphone and camera capture the user's voice and facial expression data in real time. This input data is primary information indicating the user's current state.

[0841] Step 2:

[0842] The device sends the acquired audio data to the server. The server analyzes the audio data using a natural language processing model based on TensorFlow or PyTorch. The analyzed data is output as text data to understand the user's emotional state and health status.

[0843] Step 3:

[0844] The server analyzes facial expression data sent from the camera using OpenCV and a deep learning model. The analyzed emotion data is output as numerical information to evaluate the user's emotional state.

[0845] Step 4:

[0846] The server acquires data from wearable devices containing biometric information and performs data analysis using health management tools. This output information is used as foundational data to evaluate the user's health status and generate health management advice as needed.

[0847] Step 5:

[0848] The server integrates the analyzed emotional data and health information, and uses a generative AI model to generate empathetic responses and health suggestions optimized for the user. This output response is based on the prompt "Please generate health advice to improve the user's mood" and is sent to the terminal as a response expressed in natural language.

[0849] Step 6:

[0850] The user's device displays the generated response on the screen or plays it back as audio. This allows the user to receive health management and emotional support suggested by the server.

[0851] This processing flow allows users to receive appropriate feedback on their physical and mental health status in real time.

[0852] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0853] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0854] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0855] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0856] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0857] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0858] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0859] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0860] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0861] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0862] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0863] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0864] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0865] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0866] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0867] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0868] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0869] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0870] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0871] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0872] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0873] The following is further disclosed regarding the embodiments described above.

[0874] (Claim 1)

[0875] A natural language processing means for receiving voice input, analyzing it, and generating an appropriate response,

[0876] A health management system for collecting and analyzing the health data of individual users,

[0877] An augmented reality display means for individually customizing and presenting rehabilitation programs based on collected data,

[0878] Emergency response measures to detect and notify of emergencies,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, wherein the natural language processing means performs operations on an electronic device in response to voice instructions from a user.

[0882] (Claim 3)

[0883] The system according to claim 1, wherein the health management means generates a meal plan for nutritional management based on the user's health condition.

[0884] "Example 1"

[0885] (Claim 1)

[0886] A processing means that receives and analyzes an audio signal and converts it into a command,

[0887] A processing means for evaluating the user's health status by collecting and analyzing biometric information,

[0888] An augmented reality provisioning method for customizing and displaying rehabilitation based on user progress data,

[0889] Emergency response measures to detect and quickly notify of environmental changes and anomalies,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, wherein the processing means controls the operation of a physical device based on the user's voice instructions.

[0893] (Claim 3)

[0894] The system according to claim 1, wherein the processing means creates a meal management plan based on health data analysis.

[0895] "Application Example 1"

[0896] (Claim 1)

[0897] A language processing means for receiving voice input, analyzing it, and generating an appropriate response,

[0898] A health management system for collecting and analyzing biometric information of individual users,

[0899] An augmented reality display means for individually customizing a function recovery program based on collected information and presenting that program,

[0900] An alarm response system for detecting and notifying of emergencies,

[0901] Wide-area control means for controlling the operation of electronic devices,

[0902] A system that includes this.

[0903] (Claim 2)

[0904] The system according to claim 1, wherein the language processing means performs operations on a home device in response to voice instructions from a user.

[0905] (Claim 3)

[0906] The system according to claim 1, wherein the health management means generates a meal plan for nutritional management based on the user's health condition.

[0907] "Example 2 of combining an emotion engine"

[0908] (Claim 1)

[0909] An analytical means for collecting voice data and facial expression data, and analyzing them to determine emotional state,

[0910] A natural language processing means that generates an appropriate response according to the emotional state,

[0911] A health management system that collects and analyzes individual users' health information and uses it in conjunction with emotional data,

[0912] Based on the collected data, a rehabilitation program is individually tailored, and a means of presenting information to display that program is provided.

[0913] A warning system for detecting and notifying of abnormal emotional changes,

[0914] A system that includes this.

[0915] (Claim 2)

[0916] The system according to claim 1, wherein the analysis means collects and processes voice and facial expressions from the user.

[0917] (Claim 3)

[0918] The system according to claim 1, wherein the health management means generates a health plan based on the user's emotional data.

[0919] "Application example 2 when combining with an emotional engine"

[0920] (Claim 1)

[0921] Information processing means for receiving acoustic input, analyzing it, and generating an appropriate response,

[0922] A health management system for collecting and analyzing the biometric data of individual users,

[0923] A visual augmentation display means for individually customizing rehabilitation measures based on collected information and presenting those measures,

[0924] An immediate response mechanism for detecting and notifying of emergency situations,

[0925] An emotion analysis tool for analyzing the emotional state of users and providing empathetic responses based on that analysis,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] The system according to claim 1, wherein the information processing means performs operations on an electronic device in response to an audible instruction from a user.

[0929] (Claim 3)

[0930] The system according to claim 1, wherein the health management means generates a meal plan for nutritional management based on the user's health status, and further includes suggestions based on emotional data. [Explanation of symbols]

[0931] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A natural language processing means for receiving voice input, analyzing it, and generating an appropriate response, A health management system for collecting and analyzing the health data of individual users, An augmented reality display means for individually customizing and presenting rehabilitation programs based on collected data, Emergency response measures to detect and notify of emergencies, A system that includes this.

2. The system according to claim 1, wherein the natural language processing means performs operations on an electronic device in response to voice instructions from a user.

3. The system according to claim 1, wherein the health management means generates a meal plan for nutritional management based on the user's health condition.

Images